Rethinking the Bias of Foundation Model under Long-tailed Distribution
Authors: Jiahao Chen
Bin Qin
Jiangmeng Li
Hao Chen
Bing Su
Venue: ICML 2025 poster (2025)
Keywords: Long-tailed learning
Foundation model
Pre-training bias
Primary area: deep_learning->foundation_models
TL;DR: We explore how the imbalance of foundation models impacts downstream imbalanced tasks in PEFT-based methods

Abstract: Long-tailed learning has garnered increasing attention due to its practical significance. Among the various approaches, the fine-tuning paradigm has gained considerable interest with the advent of foundation models. However, most existing methods primarily focus on leveraging knowledge from these models, overlooking the inherent biases introduced by the imbalanced training data they rely on. In this paper, we examine how such imbalances from pre-training affect long-tailed downstream tasks. Specifically, we find the imbalance biases inherited in foundation models on downstream task as parameter imbalance and data imbalance. During fine-tuning, we observe that parameter imbalance plays a more critical role, while data imbalance can be mitigated using existing re-balancing strategies. Moreover, we find that parameter imbalance cannot be effectively addressed by current re-balancing techniques, such as adjusting the logits, during training, unlike data imbalance. To tackle both imbalances simultaneously, we build our method on causal learning and view the incomplete semantic factor as the confounder, which brings spurious correlations between input samples and labels. To resolve the negative effects of this, we propose a novel backdoor adjustment method that learns the true causal effect between input samples and labels, rather than merely fitting the correlations in the data. Notably, we achieve an average performance increase of about 1.67% on each dataset.

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Jiahao Chen 123
Bin Qin 44 Jiangmeng Li 45 Hao Chen
Bing Su 123
Abstract
1. Introduction
Real-world data often follows long-tailed distribution, where the majority of instances are concentrated in head classes, leaving only small number of instances for each tail class. This scarcity of samples makes generalization them challenging. and naive learning such data tends to introduce an undesirable bias toward dominant labels. Recently, with the advent of foundation models, downstream performance can be significantly improved by fine-tuning these models on labeled data, often yielding superior results compared to training from scratch, while also minimizing training costs (Wang et al., 2022b;a). Consequently, there has been considerable interest in exploring how to fine-tune foundation models and leverage their strong generalization capabilities to enhance learning at downstream tasks.
Long-tailed learning has garnered increasing at- tention due to its practical significance. Among the various approaches, the fine-tuning paradigm has gained considerable interest with the advent of foundation models. However, most existing methods primarily focus on leveraging knowledge from these models, overlooking the inherent bi- ases introduced by the imbalanced training data they rely on. In this paper, we examine how such imbalances from pre-training affect long-tailed downstream tasks. Specifically, we find the im- balance biases inherited in foundation models on downstream tasks as parameter imbalance and data imbalance. During fine-tuning, we observe that parameter imbalance plays more critical role, while data imbalance can be mitigated us- ing existing re-balancing strategies. Moreover, we find that parameter imbalance cannot be ef- fectively addressed by current re-balancing tech- niques, such as adjusting the logits, during train- ing, unlike data imbalance. To tackle both im- balances simultaneously, we build our method on causal learning and view the incomplete semantic factor as the confounder, which brings spurious correlations between input samples and labels. To resolve the negative effects of this, we propose a novel backdoor adjustment method that learns the true causal effect between input samples and la- bels, rather than merely fitting the correlations in the data. Notably, we achieve an average perfor- mance increase of about 1.67% on each dataset.
Recent works such as LIFT (Shi et al., 2024), LPT (Dong et al., 2022), and VL-LTR (Tian et al., 2022) demonstrate that properly fine-tuning foundation models like CLIP (Rad- ford et al., 202 can significantly enhance long-tail learning performance. VL-LTR improves visual recognition, particu- larly for tail classes, by collecting class descriptions from the internet and jointly learning both visual and text repre- sentations. LIFT, on the other hand, reveals that heavy fine- tuning hurts and Parameter- Efficieet Fine-Tuning (Chen et al., 2022a; Jia et al., 2022) (PEFT) based methods can preserve much information from the foundation model as possible, which is important for the downstream imbalanced learning. However, these methods tend to overemphasize the use of foundation models to solve downstream data im- balance while overlooking their inherent biases, as shown in Fig. 1. Large-scale datasets used to train foundation models, such as LAION (Schuhmann et al., 2021), also follow a long-tailed distribution, which negatively impact down- stream tasks (Zhu et al., 2024; Wen et al., 2024). Therefore, fine- -uned models are influenced by dual long-tailed distri- butions (upstream and downstream), and only considering data imbalance is insufficient.
Equul contribution 'Gaoling School of Artificial Intelligence, Renmin University of China Reijing Key Laboratory of Research on Large Models and Intelligent Governance 'Engincering Re- search Center of Next-Generation Intelligent Search and Recom- mendation, MOE 'Institute of Software Chinese Academy of Sci- ences ' 'University of Chinese Academy of Sciences Electrical and Computer Engincering, Carnegie Mellon University. Correspon- dence to: Bing Su <subingats @ gmail.com>.
In this paper, we explore how the imbalance of foundation models impacts downstream imbalanced tasks in PEFT- based methods. Since the pre-training data is often inacces- sible, its influence is primarily reflected in the pre-trained weights, or parameters, which we refer to as parameter imbalance. In contrast, downstream data is accessible and
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
downstream datasets.
Pre Pre-training
Downstream
Our contributions are summarized as follows:
dutaset
We address practical challenge where both the data used to train the foundation model and the data for downstream tasks exhibit imbalance, referred to as pa- rameter imbalance and data imbalance, respectively. During fine-tuning, we observe that parameter imbal- ance has more significant impact than data imbalance, and existing re-balancing methods are ineffective at addressing it. Furthermore, attempts mitigate pa- rameter imbalance provide limited improvement to the quality of representations.
Foundation
Fine-tuned model
model
Figure 1. Previous methods focus on using the downstream data to fine-tune the foundation model while ignoring that the pre-training data has potential influence of bias (dashed line).
directly affects the downstream task, which define data imbalance. Through fine-tuning, we find that both types of imbalance influence downstream tasks, but param- eter imbalance plays more significant role, as shown in Fig. and Fig. 2. addition, we locate that samples be- longing to the tail classes grouped by the data and parameter imbalance at the same time are influenced extremely. Due to the inaccessibility of pre-training data, estimate the label prior and extend the current Generalized Logit Adjustment (GLA) (Zhu et al., 2024) into the training phase and named the method GLA-Train. Unlike Logit Adjustment (Menon et al., 2LA), which effectively addresses data imbal- ance, we find that GI.A-Train cannot be used to alleviate parameter imbalance, shown in Tab. 3. After measuring the quality of feature representation via Neiihhott-Niihhor (KNN) (Cover & Hart, 1967) accuracy, surprisingly find that LA slightly enhances the feature representation of tail classes with the improvement mostly coming from the clas- sifier, as shown in Tab. 4. When the imbalance integrated into the parameter, it is difficult to alleviate it by solely adjusting the logit. Therefore, we conclude that parameter imbalance is fundamentally distinct from data imbalance, presenting critical challenge that cannot be effectively addressed through re-balancing methods.
We find that the incomplete semantic factor encourages the model to learn spurious correlations between input samples and labels, which restricts its generalization ability. To solve that, we construct causal graph and propose backdoor adjustment method to eliminate the confounder negative impact.
We achieve at least 1.5%, 1.5%, 2.0% performance gains on ImageNet-LT (Deng et al., 2009), Places365- LT (Liu et al., 2019), and iNaturatist2018 (Van Horn et al., 2018) compared with state-of-the-ar methods.
2. Preliminary
Notation. We aim to solve C-way classification prob- lem with instances x E X' and labels y y = [C] {l,.... C}, where X' and y denote the input space and output space. For pre-training, we denote Dp as the dis- tribution of the training set and we cannot access at the fine-tuning phase. For downstream tasks, we denote Ds and DD the distribution for training and test, respectively. In the context of mailld learning, the number of samples varies across classes, i.e., Ps(YY = 1) * Ps(Y = 2) * * PY = C), where Ps(YY) denotes the class prior of Ds. In contrast, the test set T is sampled from the dis- tribution Dy, where each class e has an equal probability, i.e., Pr c) 1/C. Our target is to learn a hypothesis f: X y estimate Ps = y æ)) from the training set and generalize it to the test set.
To tackle both parameter and data imbalances simultane- ously, we build causal structure graph and find that the incomplete semantic factor is the confounder, encouraging the model to learn spurious correlations between input sam- ples and labels, thus restricting its generalization ability. For instance, if the class "dog" belongs to the tail classes due to parameter imbalance, the foundation model may lack sufficient semantic information, causing it to capture only partial features, such as the dog's 's head, to represent the class. When this model is adapted to a downstream task where "dog" is also tail class due data imbalance, fine-tuning struggles to learn the complete of relevant semantic fea- tures, further limiting its gemeralization. We denote the dog's head typical example of an incomplete semantic factor. To inhibit the confounding effect, we adopt the back- door criterion in causal inference to realize our backdoor adjustment. After applying our method, we achieve a more balanced performance over all the classes across different
Logit adjustment (LA). Under the label shift assumption, we have Ps(c Y = y) = Pr a Y = y) but Ps(Y x) * PP = | x) for each class LA (Menon et al., 2020) bridges the gap between the posterior of the imbalanced training set and the balanced test set. As shown in Eq. 1, we can get the posterior of the test set from the
2

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Table 1. The Zero-Shot (ZS) performance of different type of CLIPs on different benchmarks.
ImageNet-LT D-Medium
Places365-LT D-Medium
iNaturalist2018 D-Medium
D-Many
D-Few
D-Many
D-Few
D-Many
D-Few
CLIP
67.87
66.27
66.03
36.59
38.01
45.53
3.60
4.38
4.10
OpenCLIP
67.15
65.02
64.31
41.40
38.06
40.28
2.45
2.80
2.38
MetaCLIP
70.98
68.70
68.57
38.79
37.93
40.44
5.07
6.37
6.00
training set by introducing a scaling factor.
Table 2. The Average Accuracy Gap between different foundation models on Places365-LT with ZS, CE, and LA.
Pr m)) Pr(x Y = y) =
(r(x)
f-9 9
ZS
CE
LA
x PsP(s (x: Y = y)r = (1)
Adaptformer
Pr = 1) Pss( = x))
CLIP-OpenCLIP
10.38
3.19
3.49
(Y y)
CLIP -LIP-MetaCLIP
11.56
3.22
3.32
OpenCL.PP-MetaCL
12.03
3.03
3.40
There are two types of LA, either applied post-hoe to trained model or enforced in the loss during training, and the latter achieve better performance. When integrating it to the criterion, we get the final LA loss, as shown in Eq. 2, where fg (xx 0, Φ) & IP(Y = æ), 0, and & denote the output logit of class y/, the parameter of the foundation model, and the additional fine-tuned parameter, respectively. If not specified, we freeze the and only optimize the Φb
VPT
CLIP LLIP-OpenCLIP MMP-MMeaCLIP
10.38
3.6
3.94
11.56
3.42
3.87
OpenCLIP MMtaaLLP
12.03
3.10
3.53
downstream data, respectively.
fa(()) fa(a log PP( -
GLA-ZS
(4)
LLA(f(# 0, -log
fy( 0, log Ps(P = y)
+ Σ Ps(Y = =2)) (a)100000)))
(2)
FT
Ps(Y = 5) efv(=(0.0))
y'
3. The influence of parameter imbalance
3.1. Parameter imbalance is more important
Generalized logit adjustment (GLA). Since the label prior Fp(Y = y) the pre-training data is inaccessible and we cannot use Eq. 1 to estimate the posterior (from x)) to Pr(æ Y = y). Py y). and Py(Y y x) ). GLA (Zhu et al., 2024) estimates the prior following the
We consider the problem of the pre-training data and the downstream training data being both imbalanced. For the parameter imbalance, we provide a formal theoretical defini- tion in Definition 3.1. Furthermore, we offer a more detailed explanation of the relationship between parameter imbal- ance and data imbalance Appendix Section A.3. Since the P (Y) is not accessible, we use the estimated prior Pp(Y) to substitute IP PY)(Y) for the following analysis.
Ps(x Y y). PY v), and Ps(Y
Eq. 3 on the validation set, where Leg denotes the Cross- Entropy (CE) loss. After getting the estimated PY) we can adjust the logit following Eq.
Definition 3.1. Let PP(Y) denote the label of the pre- training data. We have PP 1) * (Pr(Y 2) *
PY = min max 220,0
(CE(f( (0) log v)
* Pp = =C). We use the imbalanced factor (IF) = max,g[[] Pp(Y)/ min,e(c] PY(Y) to measure the degree of parameter imbalance.
- Σ b + +
Σ qu) iE|C]
(3)
GLA assumes that the Zero-Shot (ZS) and Fine- Fine-Tuned (FT) models have diverse predicitons (Zhu al., 2024). In this way, we achieve unbiased predictions by simply ensem- bling the output logit of two models, as shown in Eq. 4, where f(x 0) and f(x 0, Φ) denote the output logit of ZS and FT, respectively. Since the adjustments of the two models are individual, we split Eq. 4 into GLA-ZS and GLA-FT, eliminating the bias of the foundation model and
Most previous works (Dong et al., 2022; Shi al., 2024; Tian et al., 2022) foeus eliminating the data bias im- prove the performance but ignore the parameter imbalance. In the following, we will explore the impact of the superpo- sition of two imbalances.
Experiment setup. We conduct experiments with VIT- B/16 (Dosovitskiy al., 2020) on the ImageNet-I.T, the
3

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
100
100
100
100
CE
CE
Acouray
Accuracy
76
Acouracy
LA
Accuray
76
LA
ZS
Zs
50
50
50
50-
LA
LA
25
75
25
ZS
25
25
750
250
500
100
Class index
Classs indexx
Class index
Class indiex
(a) Data and parameter imbalance ImageNet-LT
(b) Data and parameter imbalance Places365-LT
Figure 2. The data and parameter imbalance on (a) ImageNet-LT LT and (b) Places365-La LT. The class indices of the left picture are sorted relying on the data imbalance while the right picture relies on the parameter imbalance. Curves smoothed for better visualization.
Parameter imbalance occupies a more vital role.
ance, i... "D-Many" and "D-Few" should have a similar performance. However, the Places365-LT,, "D-Few" is better than "D-Many" for the CLIP model and other foun- dation models are nearly similar. Since they all share the architecture, the main differences come from the imbalance pre-training data, i.e., parameter imbalance. To better mea- sure the difference, we define Average Accuracy Gap Afg where f: 9 and Acc(J, denote two arbitrary hypotheses and the c-th class-wise accuracy of model f. respectively.
P-Foo P- P-Medimm P-Many
P-Fow P-Medium P-Many
75.05
53.06
34.31
72.51
66.94
63.34
58.10
35.93
40.72
25.36
10.56
35.78
39.87
31.12
D.Myny D-Medium D.Pew
D.Many MeMeiium D.Few (b) LA
(a) CE
A A.9 aug
1 Ieco(S, - c) C
(5)
Figure 3. The performance of different groups with (a) CE and (b) LA on the Places365-L7 datasct. The three rows, from top to bottom, represent the performance of P-Many, P-Medium, and P- Few, respectively. The three columns, from left to right, represent the performance of D-Many, D-Medium, and D-Few, respectively.
cml
As shown in Tab. 2, different foundations have merely 10% differences on the same downstream task. When we adapt them to the downstream, the gap is decreased but still exists.
Places365-LT, and INaturalist2018 datasets. To better plore the influence of pre-trained data, we select CLIP (Rad- ford et al., 2021), OpenCLIP (Cherti et al., 2023), and Meta- CLIP (Xu et al., 2023a) as foundation models, which are pre-trained on WIT, LAION (Schuhmann et al., 2021), and MetaDDtt, respectively. Previous works (Shi et al., 2024; Dong et al., 2022) find that we can achieve promising per- formance by fine-tuning only a small proportion of param- eters. Therefore, we select two typical PEFT techniques, Adaptformer (Chen et al., 2022a) and Visual Prompt Tuning (VPT) (Jia et al., 2022), as the basic methods. The learning rate, number of epochs, and parameter initialization strate- gies follows (Shi et al., 2024). Following OLTR (Liu et al., 2019), we split the classes into three groups named "D- Many", "D-Medium", and "D-Few" relying on the number of samples. Similarly, for parameter imbalance, we split the classes into three groups named "P-Many", "P-Medium", and "P-Few" relying on (Y). More details are in the Appendix Sec. A.
The parameter imbalance occupys. When we adapt the foundation model to the downstream task, the fine-tuned model is influenced by parameter and data imbalance, For analysis, we conduct experiments with different criterions, CE and LA, the CLIP model. As shown in Fig. 2, CE suffers from the data imbalance heavily although we adapt from the foundation model, which is not influenced by it. When changing the criterion from CE to LA, the data imbal- ance relieved effectively and we achieve more balanced performance. As for parameter imbalance, although the fine- tuning-based method can alleviate it, the bias still exists. We primarily give two explanations for this phenomenon. One is that current criteria do not consider the parameter imbal- ance explicitly. Another is that the PEFT method fixes the parameters of the foundation model, while retaining strong generalization capabilities, the parameter imbalance also preserved. As for data imbalance, only a small proportion of parameters is influenced by and the re-balancing tech- nique, i.e. LA, can easily eliminate it. Therefore, parameter imbalance plays more vital role than data imbalance.
Different foundation models serve different parameter imbalance. We report the ZS performance on different loigetailed datasets, as shown in Tab. 1. Since the imbal- anced downstream dataset does not influence the foundation models, is natural that we cannot observe the data imbal-
Tail- Tail hurts. Parameter imbalance is a critical factor that impedes further improvement. To analyze this, we con- sider two types of imbalances in conjunction and divide the validation sets into nine groups. For instance, some
4

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
classes can be categorized "D-Many" due to data imbal- ance, while simultaneously classified as "P-Few" because of parameter imbalance. As shown in Fig. 3, we observe that classes falling into both "D-Few" and "P-Few" classes ex- hibit the poorest performance. Moreover, when we shift the criterion from CE to LA, we find that the performance of "P- Few" classes within the "D-Many" group deteriorates. This indicates that previous claims suggesting LA can alleviate parameter imbalance do at the expense of "P-Few" class performance, failing to address the parameter imbalance.
Table 3. Ther performance of GI.A, GLAA-S, GLA-FT, and GLA-
T based on the CLIP model. We denote GLA-T as the GLA- Train.
Type
D-Many
D-Medium
D-Few
Overall
GLA
79.76
76.48
74.12
77.42
GLA ZS
70.23
68.61
68.90
69.27
ImageNet-LT
GLA-FT
79.67
76.15
73.29
77.12
GLA-T
80.21
75.93
72.01
77.10
GL.A
51,22
52,00
53.34
51.98
GLA-ZS
40.58
39.57
46.63
41.31
Places365-LT
GLA-FT
50.41
52.23
52.18
51.56
GLA-T
51.36
52.30
50.91
51,48
3.2. Analysis of adjustment for parameter imbalance
Addressing parameter imbalance is crucial task. Drawing inspiration from methods used to tackle data imbalance, natural approach to mitigating this issue is to adjust the out- put logits, similar to the Logit Adjustment (LA) technique. The existing Generalized Logit Adjustment (GLA) method addresses parameter imbalance by modifying the logits ac- cording to the estimated label priors. Thus, in this section, explore whether parameter imbalance can be effectively eliminated through the simple adjustment of output logits.
Table 4. The KNN accuracy. We denote GLA-T as the GLA-Train
D-Many D-Medium
D- D-Few
All
CE
74.61
70.02
66.14
71.25
ImageNet-LT
LA
74.42
70.35
66.43
71.36
GLA-T
74.33
70.35
66.31
71.26
CE
44.53
46.01
47.92
45.85
Places365-I.T
LA
44.14
46.23
48.75
45.93
GLA-T
44.33
46.14
48.66
45.93
simple adjustment alone.
GLA fails during training. Firstly, We follow GLA to estimate the prior of the pre-training data and then adjust the output logit following Eq. 4. As shown in Tab. 3, we achieve a more balanced performance and verify its effecttiveness. However, GL.A achieves unbiased prediction by modeling the parameter imbalance and data imbalance respectively, which is more complicated. Referring LA, which models the adjustment into the criterion, natural improvement is extending GLA into training. This approach mitigates pa- rameter bias by introducing additional unbiased parameters. We denote this method as GL.A-Traie Traii and constitute the optimization target as shown in Eq. 2. Since the parame- ter imbalance does not influence the classifier, we only use Eq. learn the unbiased representation and introduce additional stage for classifier re-training (Kang et al., 2019).
Feature representation analysis. To explore how the re- balance influences the feature representation, we utilize the balanced validation sets of ImageNet-LT and Places365-L7 to assess the feature quality of the test set through KNN accu- racy. As presented in Tab. 4, we observe that all three meth- ods yield comparable accuracy. The re- ce-balancing--aar based methods provide slight improvements for the "D-Few" and "D-Medium" classes, but this comes at the expense of per- formance in the "D-Many" class. This suggests that the enhanced performance for tail classes can be attributed to more accurate classifier, which can be achieved by employ- ing re- balanciing techniques like Logit Adjustment (LA). We also conduct experiments to explore the influence of classi- fier in the Appendix Sec. I.3. However, addressing parame- ter imbalance manifested the parameters of foundation models proves be more challenging. mitigate the negative effects of parameter imbalance solely through the classifier. Consequently, re- re-balancing methods cannot effectively alleviate parameter imbalance.
LeLA(f(x v) = log
difficult to
(Y = PY (x/==//)))) =1/)) (3)00)))
+
y' Ps(Y = s)ppr(Y = -y efs(2(0,0)
(6) The results are presented in Tab.3. In comparison with GLA, GLA-Train offers minimal additional benefit in ad- dressing both parameter and data imbalance, achieving performance similar to that of LA, as shown in Fig.3. (More experimental results are in the Appendix Sec. I.2.) Since Eq. 6 explicitly models both data and parameter imbalance, it is important to note that only the data imbalance is effec- tively corrected while parameter imbalance is ignored. This indicates that parameter imbalance is fundamentally differ- ent from data imbalance and cannot be resolved through
4. Method
As previously analyzed, the parameter imbalance becomes dominant after adapting to the downstream task, and we cannot eliminate it solely by adjusting the logits during the training phase. Therefore, exploring an effective method to address both parameter imbalance and data imbalance simultaneously is essential. In this section, instead of fo- cusing on re- bebalnncing methods, we analyze the issue from a causal perspective. By identifying the latent confounder,
5

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
LA
CE
Sample
CLIP
OpenCLIP
MetaCLIP
CLIP
OpenCLIP
MetaCLIP
Figure 4. We randomly select three samples of different tail classes and visualize the heatmap via Grad-CAM (Selvaraju et al., 2017) with CE and L.A. Different models draw attention different and merging them can acquire unbiased semantic information.
plete semantic factor depending the parameter imbalance. From the perspective of data generation, U - X +- C' denotes the dataset can be generated by giving the acces- sible semantic factor and inaccessible semantic factor, i.e., P > P(X)===_(X(CCC)))()))P) the predicted label distributions follow their own preferences for incomplete semantic factors. We also give additional experimental evidence for C Y in Sec. 1.9.
D
D
U
C - Y indicates that
C
C
X
B
x
B
Therefore, the incomplete semantic factor C in our problem setting, acting confounder (Pearl, 2009), can create backdoor path X <- C + Y, leading to spurious correla- tion between X and Y. we ignore the influence of incom- plete semantic factor and learn the posterior Ps y estimate (Y y æ;), the spurious correlation will be modeled, which leads biased model. As shown in Fig. 4, we visualize given samples different fine-tuned models, where the corresponding foundation models serve different parameter imbalances, i.e. incomplete semantic factor. For the specific picture, different models are drawn in different interesting areas. The relationship between the confounding path X C' -+ is unstable. For example, in the first row of Fig. 4, OpenCLIP is more attracted by the head while MetaCLIP is more interested in the body of the dog. Therefore, if test sample belonging to this class but the body is obscured, MetaCLIP is easier to give the wrong prediction. We analyze that each incomplete semantic factor can provide sufficient information distinguish the given class from others the training set. However, it also indi- cates that the incomplete semantic factor can prohibit the model from learning other relevant information. which lim- its its generalization. Therefore, eliminating the influence of incomplete semantic factor is vital for imbalanced learning.
(a) Confounded
(b) Deconfounded
Figure The framework our proposed method. (a) In the con- founded setting. C' influences both X and Y, leading to the back- door path X - C + Y, thereby introducing confounding bias. (b) After intervening X, its parent nodes C and U severed, eliminating the unstable backdoor path and leading to more reli- able estimation of the causal effect between X and Y.
we can apply the backdoor criterion to estimate its negative impact, thereby achieving a more balanced performance across various classes.
4.1. A causal view for long-tailed learning
We model the long-tailed image classification process with a causal structure graph as shown Fig. 5. Here, we denote X, C, U, and as an imbalanced dataset, the incomplete semantic factor, the inaccessible semantic factor, and the parameter imbalance, respectively. B is the data balanced representation and Y is the predicted label distribution. In Fig. 5, X - B denotes extract the data balanced repre- sentation from an imbalanced dataset. We can achieve this by applying re- bebalnccig based method like LA. B + de- notes the inference pipeline that we predict the label relying on the given representation. D - C presents the incom-
6

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
4.2. Backdoor adjustment
VL-LTR, which leverages additional data for representation learning, our method achieves an overall performance im- provement of 2. It is worth noting that while VL-LTR achieves the highest performance in the "D-Many" group, this comes at the cost of reduced performance in the "D>- Medium" and "D-Few" groups. In contrast, our method delivers consistently higher and more balanced performance across all groups, indicating its effectiveness in addressing long-tailed distributions.
more generalized fine-tuned model should be independent of incomplete semantic factor (i.e. parameter imbalance), which inhibits confounding effects from C. Therefore, in- æ). we use backdoor criterion
stead of estimating I(Y and estimate P(Y y do(æ)), where do() is exploited to cut off the connection from the C to X. Considering that each sample in X uniquely corresponds to a balanced repre- sentation in B, which indicates that the mapping between X and B is injective. Thus there only exists certain b such that (b x) and IP(b'| æ) = O for b + b'. Then, we propose our backdoor adjustment:
ImageNet-LT. The results in Tab. demonstrate that our method consistently improves performance across all cate- gories. Specifically, our approach provides gains of 2.01% %, 2. 2.65%, and 3. 3.49% for the "D-Many", "D-Medium", and "D-Few" classes, respectively, compared to LIFT. Notably, the largest improvements are observed in the tail classes, underscoring the significance of our backdoor adjustment technique in addressing challenges faced by these underrep- resented classes.
P(Y- y do(x)) = ΣV
F(Y- = (,,)P(())P( x)
BEB cEC
Σ OEC
P(Y - c))(((
(7)
For simplicity, we assume P(c) = 1/M, where M is the number of incomplete semantic factors. Intuitively, P(Y- y do(æ)) can be estimated by fusing IP(Y- = y b, c). where b represents the balanced representation. For dif- ... ferent incomplete semantic factors {c1. "2, CMr }. we utilize re- re-balancing method, such as Logit Adjustment (LA),, to compute a set of data balanced outputs (P(Y = y b, IPY = y b, ..).... , IP(Y- outputs can then be merged using fusion weights, follow- ing Eq. 7, to obtain an unbiased estimation. Since iterating over all possible incomplete semantic factors is impracti- cal, we approximate these factors using models like CLIP, OpenCLIP, and MetaCLIP, thereby addressing Eq. 7 and simplifying the process of balancing both parameter and data imbalances simultaneously.
INaturalist2018.
The results presented in Tab. 7 demon- strate that our method surpasses several baseline approaches, achieving an overall performance improvement of 1.91% % compared to RAC, a method that leverages additional data for training. This performance gain is particularly note- worthy considering the scale of the iNaturalist201 18 dataset, which comprises 8,142 distinct classes. The complexity of lcrge-scale classification tasks often introduces significant challenges due to the vast number of classes and the inher- ent class imbalance. Despite these obstacles, our method proves to be highly effective, outperforming competitors even without the need for extra training data.
= b, c.))) These
5.2. Ablation study
5. Experiment
The influence of incomplete semantic factor The most critical hyperparameter in our backdoor adjustment method is the number of incomplete semantic factors, denoted as M. In our experiments, we set M = 3, utilizing the incomplete semantic factors from CLIP, OpenCLIP, and MetaCLIP to obtain the final prediction score. To further investigate the impact of M performance, we conducted experiments shown in Tab. 8. As the number of incomplete semantic factors increases, our backdoor adjustment demonstrates improved performance. For instance, M = 3 outperforms
We conduct experiments the ImageNet-LT, Places365- IT, and iNaturalist0118 datasets. Most of our experimental setup follows previous descriptions in Sec. 3.1. For a fair comparison, we only compare our method with LiVT (Xu et al., 2023b), LPT (Dong et al., 2022). VL-LTR (Tian et al., 2022), RAC (Long et al., 2022), LIFT (Shi et al., 2023), Decoder (Wang et al., 2024), GML (Suh & Seo, 2023) and BALLAD (Ma et al., 2021). which are trained based on the VIT (Dosovitskiy et al., 2020). If not specified, We use the Adaptformer to fine-tune the foundation model (We also report the results based on VPT in the Appendix). More details are in the Appendix Sec. A.
M = 1, with OpenCLIP showing improvements of 2.27%, 2. 58%, and 29% on the "D-Many," "D-Medium", and "D-Few" on the ImageNet-LT dataset, respectively. Notably, the most significant improvement was observed in the "D- Few" classes, where the tail classes benefit greatly from a more balanced prediction due to the diverse range of incom- plete semantic factors used. This implies that our method is effective at improving the performance on long-tailed distributions, where the tail classes usually suffer from inad-
5.1. Results
Places365-LT. The results are presented in Tab. 5, where our method demonstrates superior overall performance com- pared to other approaches. Specifically, in comparison to
7

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Table Results on Places365-ITT
Method
Backbone
Extra data
Params.
Epochs
D-Many
D-Medium
D-Few
All
LiVT (Xu al., 2023b)
ViT-B/16
x
85.80M
100
48.1
40.6
27.5
40.8
LPT (Dong et al., 2022)
ViT-B/16
1.01M
49.3
52.3
46.9
50.1
VL-LTR (Tian et al., 2022)
ViT-B/16
V
149.62M
100
54.2
48.5
42.0
50.1
RAC (Long al., 2022)
VIT-B/IC
V
85. 85,80M
30
48.7
48.3
41,8
47.2
BALLAD (Ma et al., 2021)
VIT-B/16
v
149.62M
60
49.3
50.2
48.4
49.5
Decoder (Wang et al., 2024)
VIT-B/16
x
21.26
46.8
LIFT (Shi al., 2024)
VIT-B/16
X
0.18M
51.3
52.2
50.5
51.5
Ours
VIT-B/16
x
0.54M
52.52
53.62
52.54
53.01
Table 6. Results ImageNet-LT
Table 8. The ablation study with the number of semantic factors M.
Open and Meta denote the OpenCLIP and MetaCLIP, respectively.
Method
Epochs
D-Many
D-Medium
D-Few
All
ImageNet-L33 D- D-Medium
LiVT
100
73.6
56.4
41.0
60.9
CLIP Open
Meta
D-Many
D-Few
VL-LTR
100
84,5
74.6
59.3
77.2
BALLAD
60
79.1
74.5
69.8
75.7
80.25
76.05
71.53
Decoder
73.2
M =
79.94
76.17
71.70
GML
100
78.0
80.45
77.06
72.64
LIFT
10
80.2
76.1
71.5
77.0
v
81.65
77.99
73.91
Ours
10
82.21
78.75
74.99
79.57
M 2
81.86
78.26
74.33
v
81.64
78.02
74.08
Table 7. Results on Naturalist20188
M = 3
v
82.21
78.75
74.99
Method
Epochs
D D-Many
D D-Medium
D-Few
All
LiVT
100
78.9
76.5
74.8
76.1
1.50% over Logit Adjustment (LA) in the "D-Many", "D- Medium", and "D-Few" groups, respectively. These results demonstrate that our method effectively enhances the perfor- mance of tail classes while maintaining or improving the per- formance of head classes, showcasing its ability to balance performance across the board without negative tradc-offs.
LPT
160
79.3
76.1
VL-LTR
100
76.8
RAC
75.9
80.5
81.1
80.2
Decoder
59.2
LIFT
20
72.4
79.0
81.1
79.1
Ours
20
82.30
81.67
82.36
82.01
6. Related work
equate training samples and biased parameter distributions.
Long-tailed learning. Previous methods primarily focus re- re-balancing tail classes, employing techniques such as re-weighting (Cui et al., 2019) or re-sampling (Ren et al., 2020; Guo & Wang, 2021; Kim et al., 2020). The fundamen- concept behind these methods is to place greater empha- sis on tail classes to alleviate the effects of imbalanced bias. For instance, in (Cui et al., 2019), different classes signed weight based on the effective number of samples in the final loss function. Similarly, Logit adjustment (Menon et al., 2020) re-balances tail classes by adjusting the out- put logits according to the prior for each class. Recently, foundation models, which are pre-trained on vast amounts of curated data, have demonstrated their utility in produc- ing generalizable features transferable across various tasks, Several approaches have integrated foundation models to address long-tailed learning (Dong et al., 2022; Shi et al., 2024; Tian et al., 2022; Ma et al., 2021). However, these methods typically only consider the data distribution bias in downstream tasks, overlooking the inherent imbalance within foundation models themselves (Zhu et al., 2024).
Backdoor adjustment re- lieves the parameter im- balance. To verify that our method effectively alleviates parameter imbalance, we re- port the performance results
P-Poww P-Mediuum P-Many
68.58
51.67
51.84
88.87
in Fig. 6, using the di- vision principles that align with the parameter imbal-
37.10
41.29
D-Many D. Medium D- Few
ance of CLIP. When com- pared with zero-shot (ZS) and cross-entropy (CE) base- lines in Fig. our back- door adjustment achieves overall improvement across all groups. For instance, in the "P-Few" group, our method provides performance gains of 1.32%, 1.42%, and
Figure 6. The performance of different groups with method Places365- LT. After applying our method, samples helong- ing to both "P-Few" and "D-Few" exhibit significant performance gains, verify- ing the effectiveness of our method.
8

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
In this work, we investigate the compound effects of both pre-trained and downstream imbalances, and we propose simple approach to mitigate these effects.
This work is supported by fund for building world-class universities (disciplines) of Renmin University of China.
This work is supported by the 2023 BDA project (MIIT).
References
Downstream fine-tuning. Fine-tuning techniques can be broadly categorized into full fine-tuning and Parameter- Efficient Fine-Tuning (PEFT) (Jia et al., 2022; Chen et al., 2022a; Zaken al., 2021; Houlsby et al., 2019; Hu al., 2021). PEFT refers to methods that adapt pre-trained models to specific tasks while minimizing the number of parame- ters that require updating. When data samples limited, PEFT often outperforms full fine-tuning. Visual Prompt Tuning (VPT) (Jia et al., 2022) introduces two variants, VPT-Shallow and VPT-Deep, which insert prompts into dif- ferent transformer layers. In contrast, AdaptFormer (Chen et al., 2022a) introduces lightweight modules that add only small number of parameters, yet outperform fully fine-tuned models various benchmarks. Despite their efficiency, PEFT methods focus on introducing fewer parameters while preserving the foundational model's 's information. However, this also retains the model's ' inherent limitations, such as imbalances, In this work, we provide an in-depth analysis of how the biases of foundation models impact downstream tasks and propose a novel method to solve the problem.
Ahuja, K., Shanmugam, K., Varshney, K., and Dhurandhar, A. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145-155. PMLR, 2020.
Ahuja, K., Caballero, E., Zhang, D., Gagnon-Audet, J.-C., Bengio, Y., Mitliagkas, I., and Rish, I. Invariance princi- ple meets information bottleneck for outtif-distribution generalization. Advances in Neural Information Process- ing Systems, 34:3438 -44-33335550, 2021.
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: 1XXXv:1907.02880 2019.
Chen, S., Ge, C., Tong, Z., Wang. J., Song, Y., Wang. J., and Luo, P. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Informa- tion Processing Systems, 15:16664-16677 2022a.
Chen, Y., Zhou, K., Bian, Y., Xie. B., Wu, B., Zhang, Y., Ma. K., Yang, H., Zhao, P., Han, B., et al. Pareto invari- ant risk minimization: Towards mitigating the optimiza- tion dilemma in dittr--distribution generalization. arXiv preprint arXiv:2206. 07766, 2022b.
7. Conclusion
In this paper, we analyze how foundation model bias affects downstream imbalanced tasks, formally defining parameter and data imbalance to guide our study. While fine-tuning effectively addresses data imbalance, parameter imbalance persists, limiting performance. We find that re-balancing methods offer little benefit to feature representation. To tackle this, we construct causal structure graph, identify- ing incomplete semantic factors from parameter imbalance as key confounders, and propose a backdoor adjustment method to mitigate their impact. Experimental results ver- ify the effectiveness of our method. In future work, we extend our method to other tasks, such as object detection and segmentation under long-taid longetailed distributions.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for con- trastive ianguage-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2822 2023.
Cover, T. and Hart, P. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-27, 1967.
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Class- balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268 9277, 2019.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 225.255.22 Ieee, 2009.
Acknowledgement
Deng, Y., Yang, Y., Mirzasoleiman, B., and Gu, Q. Robust learning with progressive data expansion against spurious correlation. Advances in neural information processing systems, 3::1::90---.3. 1402, 2023.
This work supported in part by the National Natural Sci- ence Foundation China No. 62376277, and Doubao large model fund.

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Dong, B., Zhou, P., Yan, S., and Zuo, W. Lpt: Long-tailed prompt tuning for image classification. arXiv preprint arXiv:2210.01033, 2022.
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X. Large-scale long-tailed recognition in open world. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghami, M.. Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint (rXi::2010.11922 2020.
Long, A., Yin, W., Ajanthan, T., Nguyen, V., Purkait, P., Garg, R., Blair, A., Shen, C., and van den Hengel, A. Retrieval augmented classification for long-tail visual recognition. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
Guo, H. and Wang, S. Long-
Long-tailed tailed multi-label visual recognition by collaborative training on uniform and re- balanced samplings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15089-15098, 2021.
2022. doi: 10.1 10.1109/CVPR52288.222 20222000608822220886883
Ma, T., Geng, S., Wang, M., Shao, J., Lu, J., Li, H., Gao, P., and Qiao, Y. simple long-tailed recognition baseline via vision-language model. arXiv preprint arXiv:2111.14745. 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. arXiv preprint 20Xiv:2007.07314. 2020.
Pearl, J. Causality. Cambridge university press, 2009.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference Machine Learning, pp. 2790-2799. PMLR, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 1164-.763. PMLR, 2021.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint urXiv:2/06.0960 2021.
Ren, J., Yu, C., Ma, X., Zhao, H., Yi, S., et al. Bal- anced meta-softmax for long-tailed visual recognition. Advances in Neural Information Processing Systems, 33: 4175-4186. 2020.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Robey, A., Pappas, G. J., and Hassani, H. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 1910.992 17, 2019.
Kim, C. D., Jeong. J., and Kim, G. Imbalanced continual learning with partitioning reservoir sampling. In Eu- ropean Conference Computer Vision, pp. 411-428. Springer, 2020.
Rosenfeld, E., Ravikumar, P., and Risteski, A. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761,22 2020.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarezyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2 100::2/11.000 1.02/ 2021.
Krueger, D., Caballero, E., Jacobsen, J..H., Zhang, A., Bi- nas, J.. Zhang. D., Le Priol, R., and Courville, A. Out-of- distribution generalization via risk extrapolation (rex). In International conference on machine learning, pp. 5815- 5826. PMLR. 2021.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via gradient- based localization. Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Lin, Y.. Zhu, S., Tan, L., and Cui, P. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35: 14529-24544 2022.
10

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Shi, J.-X., Wei, T., Zhou, Z., Han, X.-Y., Shao, J.-J. and Li, Y.-F. Parameter-efficient long-tailed recognition. arXiv preprint arXiv:2309. 10019, 2023.
Xu, Z., Liu, R., Yang, S., Chai, Z., and Yuan, C. Leam- ing imbalanced data with vision transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Shi, J.-X., Wei, T., Zhou, Z., Shao, J.-J., Han, X.-Y., and Li, Y.-F. Long-tail learning with foundation model: Heavy fine-tuning hurts. In Forty-first International Conference on Machine Learning, 2024.
Zaken, E. B., Ravfogel, S., and Goldberg, Y. Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked language-model
arXiv preprint
arXiv:2106. 10/99, 2021.
Suh, M.-K. and Seo, S.-W. Long-tailed recognition by mutual information maximization between latent features and ground-truth labels. In International Conference on Machine Learning, pp. 77770-2272... PMLR, 2023.
Zhu, B., Tang, K., Sun, Q., and Zhang, H. Generalized logit adjustment: Calibrating fine-tuned models removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.
Tang, K. Huang, J., and Zhang, Long-tailed classifi- cation by keeping the good and removing the bad mo- mentum causal effect. Advances in neural information processing systems, 11-15133-1224 2020.
Zhu, J., Wang. Z., Chen, J., Chen, Y.-P. P., and Jiang, Y.- G. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6908- 6917, 2022.
Tian, C., Wang, W. Zhu, X., Dai, J., and Qiao, Y. Vi-Itr: Learning class-wise visual-linguistic representation for long-tailed visual recognition. In European Conference on Computer Vision, pp. 73-91. Springer, 2022.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 8769-8778, 2018.
Wang, Y.. Yu. Z., Wang, J., Heng, Q.. Chen, H., Ye, W., Xie, R., Xie, X., and Zhang, S. Exploring rision-language models for imbalanced learning. International Journal of Computer Vision, (32(1):224-23)) 2024.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022a.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139 149, 2022b.
Wen, X., Zhao, B., Chen, Y., Pang. J. and Qi, X. Generalization beyond data imbalance: A controlled study on clip for transferable insights. arXiv preprint arXiv:2405.21070, 2024.
Xu, H., Xie, S., Tan, X. E., Huang, P.Y.,, Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feicht- enhofer, C. Demystifying clip data. arXiv preprint arXiv:2309. /6671, 2023a.

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
A. Dataset and experimental setup
A.1. Dataset introduction
ImageNet-LT ImageNet-LT (Liu et al., 2019) is a subset of the ImageNet (Deng et al., 2009) dataset, designed to address the challenges of long-tailed distributions. contains a total of 1 118.5K images, with the number of samples per class varying fignificandy--rrom 1280 samples for the most represented class just for the least represented class. The distribution of samples per class is determined by down-sampled Pareto distribution, emphasizing the disparity in class representation.
Places365-LT Places365-LT (Liu et al., 2019) is derived from the Places-2 dataset and consists of 62.5K images spread across 365 categories. This dataset exhibits a more pronounced imbalance compared to ImageNet-LT. In Places365-LT, the largest class contains 4980 images, while the smallest class is represented by only images, resulting in an imbalance factor (IF) of 996.
INaturalist2018 The INaturalis20118 dataset (Van Hom et al., 2018) is focused on natural biological images and presents a significant challenge due to its heavily imbalanced distribution. It encompasses 437.5K images across 8142 classes, making it one of the largest datasets in terms of class diversity. Additionally, iNaturalist2018 poses a fine-grained classification challenge, as it requires distinguishing between similar species and categories, adding another layer of complexity to the classification task.
A.2. Experimental setup
We present the details about the hyper-paramene parameters of our experiments on different datasets in Tab. 9, where Ir, epochs denote the initial learning rate and training epochs, respectively. We denote bitch_size in Tab. 9 as the training batch size during the fine-tuning phase.
For training resources, all experiments are conducted on Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz with single RTX A40 GPU. Normally, a GPU with 24GB of memory is sufficient for the reproduction.
Table 9. Hyper-parameters used in our experiments on different datasets.
Dataset
Lr
Epochs
Batch_size
Places365-L1
0.01
10
128
ImageNet- LT
0.01
10
128
iNaturalist2018
0.01
20
128
A.3. Parameter imbalance and Data imbalance
Previous methods have effectively addressed data imbalance by providing a clear definition of the issue. Following the approach of OLTR (Liu et al., 2019), we categorize the classes into three groups: "D-Many", "D-Mediumm, and "D-Few".. Specifically, classes with more than 100 training samples are classified as "D-Many", those with 20-100 samples are categorized as "D-Medium". and classes with fewer than 20 samples fall into the "D-Few" group. This categorization allows for systematic examination of the impact of data imbalance across varying levels of sample availability.
To address parameter imbalance, where access to pre- preitraining data is unavailable, we adopt the estimation method provided by GLA (Zhu al., 2024) (Eq. 3) to estimate the label prior. Using this estimation, we divide the classes uniformly into three groups: the top 30% of classes based on the label prior are designated as "P-Many", the next 30-60% "P-Medium", and the remaining 40% as "P-Few"". This approach provides a structured way to assess parameter imbalance across different subsets of classes.
By applying these criteria, any given dataset can be simultancously split based on both parameter imbalance and data imbalance. Importantly, single class can belong to multiple groups across the two categorizations; for instance, class may be grouped as "P-Many" while also being classified as "D-Few". This dual categorization facilitates a more nuanced analysis of the interplay between parameter and data imbalance.
12

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
B. Detailed discussions on incomplete semantic factors
We first provide the definition of confounders in the causal framework: When a third variable Z influences both X and Y. we say that X and Y are confounded. Such a variable Z is referred to as a confounder of X and Y (Pearl, 2009). Specifically, it satisfies the fork structure X < Z - Y. This latent common cause Z can create a spurious correlation between X and Y, making the observed statistical relationship P(Y potentially misleading. To address this, the causal relationship P(Y]do(X)) is used as replacement for the correlation X.
Secondly, we explain why the incomplete semantic factor C' in our problem setting can create a backdoor path X + + Y, leading to a spurious correlation. The example in the first row in Fig.4 explains why the parameter- imballacee model D leads to an incomplete semantic factor C. For example, the semantic factor available to OpenCLIP in the head of the object, while the semantic factor available to MetaCLII is in the body of the object.
We emphasize that the semantic factors obtained due to model imbalance are incomplete. Therefore, for rigor, have modified the causal diagram in Fig. 5 to include the inaccessible semantic factor U. From the perspective of data generation, C' and U together constitute X, thus X < C' holds. On the other hand, the path C - Y indicates that the predicted label distributions follow their own parameter imbalance. A parameter-imballncee model, OpenCLIP, consistently predicts a "head of the object" label distribution (i.e., making more accurate predictions on test samples where the head of the object is not occluded), regardless of the test distributions. Similarly, the parameter-imbalaneee model MetaCLIP exhibits distinct "body of the object" preference across different test distributions.
The relationship between the confounding path X +- C' + Y is unstable. When the training set model predicts the label with the help of the body semantics of the object, if the body of sample of certain class in the test example is occluded, the model will give an incorrect prediction. From this example, it can be seen that an ideal model needs to learn the causal relationship IP( (do()) between the input and the label, rather than fitting the unstable confounding path +- C - Y, in order to achieve generalization between different distributions.
C. Discussion of other causal-based methods in long-tailed learning
Research has explored the application of causal theory to long-tailed learning (Tang et al., 2020; Zhu et al., 2022). In this section, we briefly compare our method with previous approaches.
In (Tang et al., 2020), the authors analyze SGD momentum as a confounder and use backdoor adjustment to mitigate its influence. In contrast, we identify incomplete semantic factors as confounders and disentangle their impact. The key differences are as follows:
Motivation: Our method investigates how foundation model biases influence imbalanced downstream tasks. In (Tang et al., 2020), the focus is on how SGD momentum affects long-tailed learning.
Confounder: We treat incomplete semantic factors as the confounder, whereas (Tang et al., 2020) identifies momentum as the confounder. This distinction leads to different causal graphs
Method:
We compute (())) by leveraging diverse scenarios and methods. In (Tang et al., 2020), the imbalance is viewed as causing momentum bias, and P(Y m)) is computed through de-confounded training to address this. In contrast, our approach focuses on incomplete semantic factors, computing P(Y (b, c) by utilizing different foundation models to capture varied semantic information, ensuring the model's inference is not constrained by incomplete semantics.
In (Zhu et al., 2022), the authors propose xERM, which aims to ensure that imbalanced learning methods perform well on both imbalanced and balanced test distributions. They consider the selected domain as the confounder, while we focus on incomplete semantic factors. The differences are as follows:
Motivation: Our method examines how foundation model biases affect imbalanced downstream tasks. In (Zhu et al., 2022), the authors analyze the influence of different domains under imbalanced distributions.
Confounder: We treat incomplete semantic factors as the confounder, while (Zhu et al., 2022) identifies the selected domain as the confounder. This leads to distinct causal graphs.
13

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Method: We compute P(Y]do) [do(X)) using diverse approaches. In (Zhu al., 2022), the selected domain addressed by computing P(Y 8) through de-confounded training. In contrast, focus on incomplete semantic factors, computing P(Y c) by leveraging different foundation models to capture richer semantic information, ensuring robust inference unaffected by incomplete semantics.
D. Discussion with Invariant Risk Minimization (IRM)
Invariant Risk Minimization (IRM) aims to enhance out-of-distribution generalization by identifying and optimizing invariant features, thereby reducing bias in deep learning models (Arjovsky al., 2019; Lin al., 2022; Deng et al., 2023). In linear systems, IRM has strong theoretical guarantees and a clear connection causal theory. Building upon IRM, numerous variants have emerged (Ahuja et al., 2020; Krueger et al., 2021; Robey et al., 2021; Ahuja et al., 2021), aiming to address some of IRM's challenges, such as its failure in nonlinear tasks (Rosenfeld et al., 2020), the requirement for extensive domain information (Lin et al., 2022), and optimization difficulties deep neural networks (Chen et al., 2022b).
While IRM aims to identify and optimize invariant features across domains to reduce bias, our method leverages a causal learning framework that identifies and mitigates biases introduced by spurious correlations, which are induced by the backdoor path. Rather than assuming the existence of invariant features, treat the incomplete semantic factor confounder and apply backdoor adjustment method to learn the true causal effect, offering more flexible solution to both parameter and data imbalance.
E. Discussion on parameter imbalance of LLM
We also believe that current LLMs suffer from parameter imbalance. We analyze this issue from two perspectives, the influence of parameter imbalance without fine-tuning and with fine-tuning under downstream data:
Without Mune-Tuning Since LLMs are trained on diverse corpora, some domains (e.g., news articles, Wikipedia, and general web content) dominate the pre-training process. As result, the model parameters may encode more precise representations of frequent patterns while underrepresenting rare or specialized knowledge. This imbalance becomes apparent when applying LLMs to highly domain-specific tasks, such as specialized scientific research or low-resource languages, where the model struggles due to insufficient parameter representation.
Fine-tuning When we use LLMs for customized tasks, such as role-playing or simulating a specific individual's tone and manner of speaking, parameter imbalance can also have significant impact. Since pre-training data often biased toward general linguistic patterns found in publicly available texts, the model may struggle to accurately capture and reproduce highly personalized or domain domain-specifie speech styles. For example, if an LLM is fine-tuned to simulate the conversational style of a historical figure or a niche expert, but the pre-training corpus contains limited examples of their actual speech patterns, the model may default to generic language structures instead of faithfully mimicking the target style. Even with sufficient fine-tuning data, the model's prior knowledge-encoeed encoded in the pre- pre-trained parameters- can cause it to retain biases from its original training distribution, making difficult fully adapt to the new task.
F. Overall inference pipeline
During the fine-tuning phase, our method can be decomposed into two stages. In the first stage, we apply logit adjustment to fine-tune various foundation models (CLIP, OpenCLIP, and MetaCLIP), addressing the data imbalance present in the downstream dataset. In the second stage, we input each test sample into the fine-tuned models, obtaining output logits without additional fine-tuning. As described in Eq. 7, we then ensemble these logits using the importance weight P(c) to correct for parameter imbalance, ultimately producing the final prediction score.
G. The selection of foundation model
As shown in Fig. 5, the path D - C' represents the incomplete semantic factor that arises due to parameter imbalance, stemming from the imbalance in the pre-training data of the foundation model. Consequently, we have chosen CLIP, OpenCLIP, and MetaCLIP because they are pre-trained on distinct datasets, resulting in varying degrees of parameter imbalance and differing incomplete semantic factors.
14

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
In causal theory, Eq. 7 should cover the value space of C as fully as possible for accurate causal effect estimation. Since C has infinite values, we approximate it finitely, with broader coverage improving estimation accuracy. In this way, model selection should take into account the path D + C. If two foundation models are trained on similar pre-training datasets, only one should be selected, as choosing both would not significantly increase the coverage of the value space of C.
H. Introduction to the ViT and PEFT
H.1. Vision Transformer
The Vision Transformer (ViT) is deep learning model designed to process image data by leveraging the architecture of transformers, originally developed for natural language processing tasks. Unlike traditional convolutional neural networks (CNNs), which rely on convolutional layers to capture spatial relationships in images, VIT treats image as sequence of patches. Each image is divided into fixed-size patches, and these patches are embedded and processed as tokens, similar to words in sentence. Through self-attention mechanisms, ViT captures global dependencies between patches, allowing it to effectively model long-range relationships within images. ViT has demonstrated strong performance in a variety of computer vision tasks, often surpassing CNNs, particularly as the availability of large-scale image datasets has grown.
For a pre-trained Vision Transformer (ViT) composed of N blocks, denoted as B [B1, Bg. Br}. each input (0) image 2; is divided into m patches, resulting in patch embeddings E { e .... ei Together with the classification token (CLS), these patch embeddings are passed through the ViT backbone, producing an output embedding (N) (N)
{CLS(M) C1,1 C2,2 ..... G...),...) ).
Each block consists of a multi-head self-attention mechanism followed by a feed-forward layer, both of which incorporate layer normalization and a residual connection. In the self-attention layer, the patch embeddings and the class token updated based on the similarity matrix, shown in Eq. 8. Here, Q!--1), K!"-1! 1), and V, v-(-1) are derived from [CLS)--1) E)"-1) and are input to different linear transformations within block B., where d represents the feature dimension. The self-attention layer updates the tokens based on the values in the self-attention matrix.
(n-1))7 K (2-1)
CLS;
(n-11 1)
in D softmax(
E
(n-1)
(8)
Vd
(n-1) E(n-11 from the multi-head self-attention layer are then of block B,
where d the feature dimension. The resulting tokens [CLS,
passed into the forward layer, producing the output [CLS| E.!
(n)
(m)]
H.2. Adaptformer
AdaptFormer is lightweight adaptation framework designed to enhance the efficiency and flexibility of large pre-trained vision transformers (ViTs) in downstream tasks. By introducing low-rank adaptation (LoRA) module within the feed- forward layers of the transformer, AdaptFormer allows for the modification of the model's capabilities without requiring a full re- re-training of all parameters. This approach preserves the pre-trained knowledge in the original transformer while enabling it to adapt effectively to new tasks with minimal additional parameters. As a result, AdaptFormer achieves strong performance in transfer learning tasks, providing a balance between computational efficiency and task-specific adaptability.
Let X be the input to the feed-forward layer, and the original weight matrix of the layer be W E Rddd In AdaptFormer, an additional low-rank module is added as modification. This is represented by two matrices, A Raxr and B E Bexd where r < d represents the low-rank dimension. The output of the modified feed-forward layer is given by:
Y = WX (AB)X
(9)
Here, a is a scaling factor that controls the influence of the low-rank adaptation. The term AB represents the low-rank adaptation added to the original weight transformation, allowing the model to adapt with a minimal number of additional parameters.
H.3. Visual Prompt Tuning
Visual Prompt Tuning is an innovative approach designed to adapt pre-trained vision models for specific tasks by utilizing learnable visual prompts. Instead of fine-tuning all the parameters of large vision model, this method introduces a small set
15

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Table 10. The ablation study with the different numher of semantic factors of M for VPT.
ImageNet-IT D-Medium
Places365-LTT D-Medium
CLIP OpenCIIIP
MetaCIIIP
D-Many
D-Few
D-Many
D-Few
78.63
75.42
71.25
50.70
51.72
49.07
M
v
78.62
75.40
70.91
$1.45
51.53
50.68
79.86
76.47
72.19
50.91
52.06
49.86
V
80.42
77.17
73.19
$1.80
52.66
51.33
M
2
80.75
77.96
73.54
51.55
53.20
50.82
V
v
80.54
77.52
73.41
51.89
53.09
51,22
. =3 3
v
81.09
78.37
74.16
51.97
53.26
51.34
x10-2
-10-2
X
x10
Close pion
paior
Class ptirt
priot
0.4
0.2
0.0
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.55
0.00
0.4
0.5
0.8
1.0
0.00
0.2
0.4
Class prior
X110
Class pritor
X030"
Closs pricr
X010
Class pritor
X.30
(a) Places365-LT
(b) Consistency
(c) Reverse
(d) Balance
Figure 7. We constitute different parameter imbalanced datasets the original Places365-LT,, (b) the data imbalance and parameter imbalance are consistent, (c) the data imbalance and parameter imbalance are reversed, and (d) the downstream dataset is balanced. Each dot in the figure represent the pair (Ys(YY y). PY y)).
of visual prompts that can be learned during the adaptation process. These prompts are essentially additional visual tokens that are concatenated to the input image embeddings, guiding the model to focus on relevant features for the target task. This approach significantly reduces the computational burden associated with full model fine-tuning while retaining the rich representations learned from extensive pre- pre-training. By leveraging visual prompts, practitioners can achieve competitive performance in various computer vision applications, such as image classification and object detection, with minimal adjustments to the model architecture.
The process of Visual Prompt Tuming can be represented mathematically by introducing set of learnable visual prompts into the embedding space of pre-trained vision model. Let P = (P(1)... .... P(N)), where p(n) E RR. is the prompts of the n-th block., where / is the number of prompts. The visual prompt tuning process can be described by the following equation:
CLS(") E." B. (CLS("") -((n-)) E. E!!-1)
(10)
This concatenated representation is then fed into the transformer architecture of the vision model to adapt it for specific downstream tasks. The learnable visual prompts are updated during the tuning process to optimize task performance while keeping the original model parameters fixed.
Additional exerpiments
I.1. The performance for visual prompt tuning
We also conduct experiments using different PEFT-based methods, such as Visual Prompt Tuning (VPT), with the results shown in Tab. 10. A similar phenomenon is observed with Adaptorformer for fine-tuning the foundation model. For instance, when outperforms M = 1, OpenCLIP demonstrates improvements of 2.38%. 2.97%, and 3.25% on the "D-Many", "D-Medium". ; and "D-Few" groups the ImageNet dataset, respectively. These experiments further indicate that our method generalizes well to different types of PEFT-based methods.
16

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Table 11. Ther performance of GLA, GLA--S, GLAA-FT, and Gl-A-Train based on the CLIP model.
Type
D-Many
D-Medium
D-Few
P-Many
P-Medium
P-Few
Overall
GLA 79.76
76,48
74.12
88.34
78.58
65.30
77.42
GLA-ZS
70.23
68.61
68.90
83.63
70.36
53.79
69.27
ImageNet-LT
GLA-FT
79.67
76.15
73.29
88.07
77.92
65.32
77.12
GLA-Train
80.21
75.93
72.01
88.05
77.93
65.32
77.10
GLA
51.22
52.00
53.34
69.07
50.70
35.87
51.98
GLA-ZS
40,58
39.57
46.63
61.33
39.69
22.56
41.31
Places36S-LT
GLA-FT
50.41
52.23
52.18
67.71
49.57
37.15
51.56
GLA-Train
51.36
52.30
50.91
67.70
49.58
37.17
51.48
Table 12. The results of decoupling training and LA on ImageNet-LT dataset.
D-Many
D-Medium
D-Few
All
LA
80.25
76.05 75.90
71.53
77.06
Decoupling
80.30
70.4
76.80
1.2. More experimental results for GLA
In Tab.3, we present the results of GLA, GLA-ZS, GLA-FT, and GLA-Train ImageNet-LT and Places365-LT, focusing performance influenced by data imbalance due to space constraints. Additionally, in Tab. we provide results highlighting the performance under parameter imbalance.
I.3. The influence of classifiers
Decoupling training (Kang et al., 2019) has demonstrated its effectiveness under long-tailed distributions, suggesting that generalizable representation can be learned without re re-balancing, However, this conclusion is largely based on experiments using CNN architectures trained from scratch, and its influence on fine-tuning-based methods remains unclear. To explore this, we conducted experiments following similar pipeline, where the backbone is fine-tuned using Cross Entropy (CH) while the classifier is trained with Logit Adjustment (LA). As shown Tab. 12, decoupling training achieves a performance similar LA. Combined with previous findings, we observe that LA primarily alleviates data imbalance by driving more balanced classifier, though it offers limited improvement in the representation of tail classes.
L.. The influence data imbalance alone
In our paper, we address both data imbalance and parameter imbalance simultaneously, with analyses primarily based on commonly used datasets. To better understand the impact of parameter imbalance, consider the following scenarios: (1) Consistency: The data imbalance and parameter imbalance consistent, where the head classes grouped by the data imbalance are also the head classes grouped by the parameter imbalance. (2) Reverse: The data imbalance and parameter imbalance are reversed, where the head classes grouped by the data imbalance are the tail classes grouped by the parameter imbalance. Balaaee The downstream data balanced. To simulate these scenarios, we construct additional imbalanced datasets by downsampling the Places365 dataset, as illustrated Fig. We evaluate performance using the Places365-LT balanced test set. To reduce the influence of randomness, we create three datasets for each scenario and report the average performance across them.
The results, shown in Tab. 13, indicate that when the downstream data is balanced, the model achieves the best performance, as this setup aligns most closely with the distribution of the downstream task. Interestingly, the Reverse scenario outperforms
Table 13. The results of Uniform, Resverse, and Balance.
Balance
Uniform
Reverse
53.63
50.80
51.2
17

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Table 14. The imbalanced factor of parameter imhalance of different foundation models Places365-I.T.
CLIP
OpenCLIP
MetaCLIP
IF
57.50
63.25
60.20
C-0
C-1
OpenCLIP
0.7177
0.6689
0,3032
0.2932
MetaCLIP
0.2634
0,2432
0.7180
0.7090
Figure 8. Prediction scores of fine-tuned OpenCLIP and MetaCLIP. OpenCLIP is more confident about the sample if the head is exposed
(C 0), while MetaCLIP more sensitive to the body (C = 1).
the Consistency scenario. In the Reverse setting, the tail classes (in terms of parameter imbalance) can be viewed as head classes (in terms of data imbalance). suggesting that parameter imbalance can be mitigated by increasing the number of training samples for tail classes. However, real-world applications, collecting sufficient data for tail classes often challenging, and simply adding more data is not a practical solution to this issue.
L.5. Discussion
Our method tackles the parameter imbalance in PEFT under long-tailed distributions, providing a balanced solution that improves both model performance and fairness. As shown Tab. 15, increasing allows for the identification of more incomplete semantic factors, enhancing the model's ability address these confounding elements. However, this improvement comes with substantially higher computational costs, underscoring the importance of optimizing M to balance performance gains and efficiency, particularly in resource-constrained scenarios.
1.6. Intrepreting the confounder
In this section, we conduct experiments investigate why the incomplete semantic factor acts as a confounder in the causal graph. As illustrated in the first of Fig. 4, the fine-tuned OpenCIIP model predominantly focuses the head, whereas MetaCLIP primarily focuses on the body. We define C' 0 to represent the head and C' represent the body for this particular class.
To assess the influence of the incomplete semantic factor, we randomly select samples from this class and examine the prediction results of OpenCLIP and MetaCLIP. As shown Fig. 8, when the dog's head is occluded, OpenCLIP tends to produce lower-confidence predictions. Conversely, when the body occluded and the head remains visible, OpenCLIP more likely to provide higher-confidence predictions. Since both models are fine-tuned using the same principle, the observed differences in predictions be attributed to incomplete semantic factors, thereby supporting hypothesis.
Table 15. Computation costs.
FLOPs
M
239.06
478. 13
M 717.19
18

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Table 16. Differences between P(Y X) and PYY| do(X)). Imagel, Image2, Image3, and Image4 denote image in Fig. from left right, respectively.
Imagel
Image2
Image3
Image4
P(Y | do(X))
0.5614
0.5443
0.5837
0.5742
P(Y X)
0.5431
0.2221
0.2210
0.1031
100
Accurucy
75
fine-tune
50
scratch
25
0
250
500
750
Class index
Figure The comparison between fine-tuned model and training from scratch.
To show the existence of confounding bias directly, we measure differences between P(Y X) and P(Y do(X)). As shown in Tab. 16, there is significant difference between P(Y X) and PPYY do(X), which also verifies the existence of confounding bias.
1.7. More experimentsParamete imbalance
In addition, to better support our point, we train a model from scratch on ImageNet-LT with ResNet-50, Since the parameter of ResNet-50 (He et al., 2016) is randomly initialized, the trained model is only influenced by the data imbalance. After getting the model, we calculate the accuracy of each class and sort them relying on the estimated pre-traim mre-trainnng data label prior PY))). Since the parameter imbalance does not influence the trained model, the performance curve after sorting should not present downward trend. As shown in Fig. 8, the result verifies our point. Therefore, in this comparison experiment, the downward trend in the right image in Fig. 2 is attributed to the imbalance of pre-trate pre-training data, which can also verify the impact of parameter imbalance.
L8. Visualization of our backdoor adjustment
As illustrated in Fig. 10, we present heatmaps generated after applying our backdoor adjustment, showcasing the effectiveness of our method in mitigating the influence of incomplete semantic factors. By addressing these confounders, our method enhances focus the entire object rather than isolated parts.
In the first row, OpenCLIP primarily concentrates on the head, while MetaCLIP shifts its attention to the body. In contrast, our method successfully integrates both the head and body, yielding a more holistic understanding of the object. Similarly, in the second row, our method expands its focus to include the head, wings, and tail, offering a more comprehensive representation of the object. Finally, in the last row, our method effectively captures the entire structure of the truck, demonstrating its robustness and adaptability diverse objects and scenarios.
1.9. Experimental evidence for Y
In this section, we will verify the existence of C' + in experiments. We constructed dataset consisting of six classes: "airplane", "automobile". "bird", "cat", "dog", and "truck". For each class, define distinct incomplete semantic factors, as shown in Tab. 17, where C' denotes the incomplete semantic factor:
For example, the term "airplane head" refers to the front part of the aircraft, including the cockpit, while "airplane tail" represents the rear section, including the winglets. To construct the dataset, we used Stable Diffusion (Rombach et al., 2022) by providing prompts in the format: "a photo a {class name}'s [head or tail]". For instance, to generate samples for the airplane class, we used the prompts "a photo of airplane's 's head" and "a photo of an airplane's tail", respectively, ensuring
19

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Sample
Ours
CLIP
OpenCLIP
MetaCLIP
Figure comparison of our method other fine-tuning based methods.
Table The six classes and their corresponding incomplete semantic factors.
airplane automobile bird
airplane head automobile head bird head cat head dog head truck head
airplane tail
automobile tail
bird body cat body dog body truck tail
dog
truck
sufficient samples for each semantic factor. We also visualize samples in Fig. 11.
The training dataset is created under the following conditions: blue1) ) For each incomplete semantic factor, 50 samples are generated. This dataset is referred to as A 2) For C' = 0. 90 samples are generated, and for C' 1, 10 samples are generated. This dataset is referred to as Btrain-
For evaluation, the test dataset is constructed under the following conditions: 1) For each incomplete semantic factor, 50 samples are generated. This dataset referred to Atest 2) For C 0, 90 samples are generated, and for C' 1. 10
samples generated. This dataset is referred to Brest-
We use ResNet-32 (He et al., 2016) as the backbone and train it from scratch on Atrain and Btrain respectively, The trained models are then evaluated on Atest and Biest, respectively. As shown in Fig. 12, when the model is trained on Braain but evaluated on Atest, we find the performance drops (compared with testing on Brest), which indicates the confounder influences the model and cannot make right predictions. Our findings can be summarized as follows:
C-0
C-1
Figure The visualization the "dog" class: represents samples generated for the dog's head, while corresponds samples of the dog's body.
20

---

Rethinking the Bias of Foundation Model under Long-tailed Distribution
Atrate Apo
Atrazer Biuc
Class wine arriracy
Bream Ave
Braais Bisat
40
20
airplane
automabile
bird
cat
dog
truck
Class index
Figure The performance curve illustrates evaluation scenarios: Aerains Atest indicates the model is trained on Ataai and evaluated on A Atruin, Brest represents the model trained on Atrain and evaluated on Biest Btrains Atest corresponds to the model trained on Btrain and evaluated on Atest and Berain; Brest denotes the model trained on Berain and evaluated on Btest-
Table 18. The experiments of randomness.
Places365-LT
ImageNet-LT
LIFT
51.4 (0.09)
77.0(0.04)
Ours
53.03 (0.08)
79.59(0.03)
1. The model trained Atrain, when used for inference, can be interpreted estimating P(Y do(X)). We observe that the model trained on Atrain performs similarly on both Atest and Breat- This result indicates that the trained model does not learn the correlation.
The model trained on Bieain, when used for inference, can interpreted as estimating P(Y X). We observe that the model trained on Btrain exhibits different performance between Atest and Btest- This difference arises due to the influence of the confounder.
From these experiments, we verify that C is confounder.
L.0. Randomness
To ensure that the observed performance improvement is attributable to the advantages of our method and not random variation, conduct experiments with five different random seeds and report the mean and standard deviation for both LIFT and our method on Placex365-LT LT and ImageNet-LT. As shown in Tab. 18, our method outperforms LIFT with smaller standard deviation (reported in brackets), demonstrating that the performance gains are not due to randomness.
In addition, we also conduct significance test (T-test) between the results of our method and the LIFT, obtaining Pvaluel = 4.57 x 10-9 and Pvalue2 = 3.62 x 10-10 for Places365-LT and ImageNet-LT, respectively. This indicates that our method is statistically significant, with probability of making no more than 5% error.
21


Reviews:
[{"review_id": "1mMiLUTZf0", "reviewer": "Reviewer_K7YB", "summary": "This paper studies the impact of foundation models' biases—trained on disproportionately distributed data—on downstream tasks with imbalanced labels. The authors characterize two types of imbalance: (1) parameter imbalance, which has its roots in the pre-training stage, and (2) data imbalance, which exists in the downstream task. They show through experiments that different numbers of parameters affect performance more than imbalanced data. They found that standard methods to remedy this, like Logit Adjustment, are not a fix. To repair these issues, the paper suggests a new backdoor adjustment method based on causal inference.\nThis method tries to mitigate the confounding effect of what the authors refer to as \"incomplete semantic factors\" (an artifact of parameter imbalance) by ensembling semantic signals from a diverse set of foundation models. Large-scale experiments on ImageNet-LT, Places365-LT, and iNaturalist2018—and ablation studies—demonstrate that the method improves accuracy, particularly for tail classes, and obtains a more balanced performance overall.\n\n## Update after rebuttal\nI stay with my original rating, as I agree with the comments of author's rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and acknowledgement of our work! In the following, we summarize a series of works on Invariant Risk Minimization (IRM) to reduce bias and briefly introduce the differences between our work and existing studies.\n\nInvariant Risk Minimization (IRM) aims to enhance out-of-distribution generalization by identifying and optimizing invariant features, thereby reducing bias in deep learning models [1, 2, 3]. In linear systems, IRM has strong theoretical guarantees and a clear connection to causal theory. Building upon IRM, numerous variants have emerged [4, 5, 6, 7], aiming to address some of IRM’s challenges, such as its failure in nonlinear tasks [8], the requirement for extensive domain information [2], and optimization difficulties in deep neural networks [9].\n\nWhile IRM aims to identify and optimize invariant features across domains to reduce bias, our method leverages a causal learning framework that identifies and mitigates biases introduced by spurious correlations, which are induced by the backdoor path $X \\leftarrow C \\rightarrow Y$. Rather than assuming the existence of invariant features, we treat the incomplete semantic factor as a confounder and apply a backdoor adjustment method to learn the true causal effect, offering a more flexible solution to both parameter and data imbalance.\n\n[1] Arjovsky, Martin, et al. \"Invariant risk minimization.\" arXiv preprint arXiv:1907.02893 (2019).\n\n[2] Lin, Yong, et al. \"ZIN: When and how to learn invariance without environment partition?.\" Advances in Neural Information Processing Systems 35 (2022): 24529-24542.\n\n[3] Deng, Yihe, et al. \"Robust learning with progressive data expansion against spurious correlation.\" Advances in neural information processing systems 36 (2023): 1390-1402.\n\n[4] Ahuja, Kartik, et al. \"Invariant risk minimization games.\" International Conference on Machine Learning. PMLR, 2020.\n\n[5] Krueger, David, et al. \"Out-of-distribution generalization via risk extrapolation (rex).\" International conference on machine learning. PMLR, 2021.\n\n[6] Robey, Alexander, George J. Pappas, and Hamed Hassani. \"Model-based domain generalization.\" Advances in Neural Information Processing Systems 34 (2021): 20210-20229.\n\n[7] Ahuja, Kartik, et al. \"Invariance principle meets information bottleneck for out-of-distribution generalization.\" Advances in Neural Information Processing Systems 34 (2021): 3438-3450.\n\n[8] Rosenfeld, Elan, Pradeep Ravikumar, and Andrej Risteski. \"The risks of invariant risk minimization.\" International Conference on Learning Representations (2021).\n\n[9] Chen, Yongqiang, et al. \"Pareto invariant risk minimization: Towards mitigating the optimization dilemma in out-of-distribution generalization.\" The Eleventh International Conference on Learning Representations (2023).", "claims_and_evidence": "The submission's main points are:\n- The observation that downstream data imbalance matters less than parameter imbalance (which is from pre-training) is backed by looking at classes (binned as D-Many, D-Medium, D-Few etc. for parameter groups) and comparing performance under different training strategies.\n- The experiments reveal that methods such as Logit Adjustment can neutralize data imbalance but are not effective in the case of parameter imbalance.\n- A novel method of backdoor adjustment is presented that helps in mixing predictions from several simple models, and this is found to work better in the majority of tests.", "methods_and_evaluation_criteria": "The method of conducting this research is novel and well-suited to the problem. Applying causal inference—a backdoor adjustment, in particular—to address the two causes of imbalance is a sensible approach to enhancing existing re-balancing techniques. Evaluation on established long-tailed benchmarks (ImageNet-LT, Places365-LT, iNaturalist2018) using conventional metrics (overall accuracy and class-wise breakdowns) is appropriate and provides a solid basis for assessing the efficacy of the proposed approach.", "theoretical_claims": "There are no proofs; (this is fine). The theoretical claims are straightforward intuitions about causal ML that I agree with.", "experimental_designs_or_analyses": "The experimental design is sufficiently comprehensive.\nThree standard long-tailed datasets are used. Measuring performance across different class groupings (D-Many, D-Medium, D-Few) shows how well bias is reduced. The method is compared with appropriate baseline methods. Analyses of the number of incomplete semantic factors (M) helps elucidate the method's effectiveness.", "relation_to_broader_scientific_literature": "The paper builds upon prior work in long-tailed learning and bias reduction.\nIt mentions current methods that tackle data imbalance, like Logit Adjustment, re-weighting, and re-sampling procedures. However, it puts emphasis on the bias caused by foundation models. The causal method is in line with current trends in the application of causal inference for machine learning tasks (as seen in the work of Tang et al. and Zhu et al.). By leveraging concepts from foundation model adaptation (including PEFT approaches) and long-tailed learning, the paper highlights its contributions in these research areas.\n\nA broader discussion that encompasses methods from invariant risk minimization or other approaches to mitigate bias based on causation might better position this work within the current literature.", "essential_references_not_discussed": "While the paper cites many pertinent sources, it could be enhanced by citing invariant risk minimization approaches to mitigate bias in deep learning.", "other_strengths_and_weaknesses": "Strengths:\nThe paper presents a novel causal framework for analyzing bias in foundation models by focusing on an unexplored dimension (parameter imbalance). Experiments are conducted on a range of benchmarks with detailed class-wise comparisons and ablation studies. The use of backdoor adjustment in the training process is a sensible way of reducing bias.\n\nWeaknesses:\nI don't see any glaring weaknesses.", "other_comments_or_suggestions": ".", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sbp4RLvP4b", "reviewer": "Reviewer_wauK", "summary": "This paper examines the inherent biases introduced by the imbalanced training data used to pre-train foundation models, and how these biases affect downstream long-tailed learning tasks. The authors find that during fine-tuning, parameter imbalance (the imbalance in the pre-trained model parameters) plays a more critical role than data imbalance, and existing re-balancing techniques are ineffective at addressing parameter imbalance. To tackle both parameter and data imbalances, the authors propose a causal learning-based backdoor adjustment method that learns the true causal effect between input samples and labels, rather than just fitting the correlations in the data. This method achieves significant performance improvements on several long-tailed benchmark datasets compared to state-of-the-art methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback. In the following, your questions are summarized in quotes, followed by our point-by-point responses.\n\n> Could the improvement be purely due to randomness?\n\nTo ensure that the observed performance improvement is attributable to the advantages of our method and not random variation, we conduct experiments with five different random seeds and report the mean and standard deviation for both LIFT and our method on Places365-LT and ImageNet-LT. As shown below, our method outperforms LIFT with a smaller standard deviation (reported in brackets), demonstrating that the performance gains are not due to randomness.\n| |Places365-LT|ImageNet-LT|\n|--|--|--|\n|LIFT|51.4 (0.09)|77.0 (0.04)|\n|Ours|53.03 (0.08)|79.59 (0.03)|\n\nIn addition, we also conduct a significance test (T-test) between the results of our method and the LIFT, obtaining a $p_{value1}=4.57\\times10^{-9}$ and $p_{value2}=3.62\\times10^{-10}$ for Places365-LT and ImageNet-LT, respectively. This indicates that our method is statistically significant, with a probability of making no more than a 5% error.\n\n> Please make all figure legends clearer.\n\nAfter conducting a comprehensive review, we have carefully refined the captions and legends for each figure. Given that figures 3, 5, and 6 previously lacked sufficient detail, we present the revised results below.\n\nFigure 3: The performance of different groups with (a) CE and (b) LA on the Places365-LT dataset. The three rows, from top to bottom, represent the performance of P-Many, P-Medium, and P-Few, respectively. The three columns, from left to right, represent the performance of D-Many, D-Medium, and D-Few, respectively.\n\nFigure 5: The framework of our proposed method. (a) In the confounded setting, $C$ influences both $X$ and $Y$, leading to the backdoor path $X \\leftarrow C \\rightarrow Y$, thereby introducing confounding bias. (b) After intervening on $X$, its parent nodes $C$ and $U$ are severed, eliminating the unstable backdoor path and leading to a more reliable estimation of the causal effect between $X$ and $Y$.\n\nFigure 6: The performance of different groups with our method on Places365-LT. After applying backdoor adjustment, performance improves across different groups, particularly for samples in both P-Few and D-Few.\n\nDue to word limits, additional revisions of figure captions are not listed here. We promise that we will carefully rewrite all figure legends and captions in the original paper.\n\n> Clearly outlining the entire fine-tuning pipeline or algorithm.\n\nDuring the fine-tuning phase, our method can be decomposed into two stages. In the first stage, we apply logit adjustment to fine-tune various foundation models (CLIP, OpenCLIP, and MetaCLIP), addressing the data imbalance present in the downstream dataset. In the second stage, we input each test sample into the fine-tuned models, obtaining output logits without additional fine-tuning. As described in Eq. 7, we then ensemble these logits using the importance weight $P(c)$ to correct for parameter imbalance, ultimately producing the final prediction score.\n\nWe promise to give an entire algorithm pipeline in the original paper.\n\n>What do authors think the parameter imbalance in LLM?\n\nThank you for your insightful comments! We also believe that current LLMs suffer from parameter imbalance. We analyze this issue from two perspectives, the influence of parameter imbalance without fine-tuning and with fine-tuning under downstream data:\n\n****(1) Without Fine-Tuning:****\nSince LLMs are trained on diverse corpora, some domains (e.g., news articles, Wikipedia, and general web content) dominate the pre-training process. As a result, the model parameters may encode more precise representations of frequent patterns while underrepresenting rare or specialized knowledge. This imbalance becomes apparent when applying LLMs to highly domain-specific tasks, such as specialized scientific research or low-resource languages, where the model struggles due to insufficient parameter representation. \n\n****(2) Fine-tuning:**** \nWhen we use LLMs for customized tasks, such as role-playing or simulating a specific individual's tone, parameter imbalance can also have an impact. Since pre-training data is often biased toward general linguistic patterns, the model may struggle to capture highly personalized or domain-specific styles. For example, if an LLM is fine-tuned to simulate the conversational style of a historical figure, but the pre-training corpus contains limited examples of their actual patterns, the model may default to generic language structures instead of faithfully mimicking the target style, making it difficult to fully adapt to the new task.\n\nWe promise that we will add this discussion to the original paper", "claims_and_evidence": "The claims made in the paper appear to be well-supported by the analysis and experimental evidence provided in the contexts. The authors have conducted a thorough investigation of the biases in foundation models and proposed a novel causal learning-based solution that demonstrates clear performance gains on the evaluated benchmarks.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria employed in the paper are well-suited for the problem of long-tailed learning in the context of fine-tuning foundation models, as they address the key challenges and biases identified in the introduction.", "theoretical_claims": "I did not see the theoretical claims in this paper, definitions and simple derivations are not theoretical claims.", "experimental_designs_or_analyses": "I check all, and overall the experimental designs and analyses presented in the paper appear to be sound and well-justified, with a thorough investigation of the proposed method's performance and its comparison to related work.", "relation_to_broader_scientific_literature": "The paper formally defines parameter imbalance and data imbalance, providing a structured way to analyze the impact of these biases, which extends the approach introduced in OLTR (Liu et al., 2019)\n\nThe paper proposes a novel backdoor adjustment method to mitigate the negative effects of parameter imbalance and data imbalance, which is distinct from previous causal-based approaches, such as those in (Tang et al., 2020) and (Zhu et al., 2022)\n\nThe paper's exploration of the causal relationships between incomplete semantic factors, input samples, and labels contributes to the growing body of work on applying causal reasoning to address challenges in long-tailed learning\n\nOverall, the paper builds upon and extends the existing literature on long-tailed learning, foundation model biases, and the application of causal reasoning to address imbalance-related challenges in machine learning.", "essential_references_not_discussed": "there are some related papers, but are not published, so it is fine.", "other_strengths_and_weaknesses": "I don’t see any major drawbacks, but I have a few concerns:  \n\n1. The improvement seems minor—only 1.5%. Could this improvement be purely due to randomness?  \n\n2. Figures should be self-explanatory, allowing readers to understand them without referring to the main text. Please make all figure legends clearer.  \n\n3. While the intuition and motivation of the paper are strong, the methodology remains unclear. For sec4.2, the explanation of the proposed methods is not easy to follow. I suggest making it more straightforward, for example, by clearly outlining the entire fine-tuning pipeline or algorithm.", "other_comments_or_suggestions": "None.", "questions_for_authors": "It seems that the tasks are all on image datasets, what do authors think the parameter imbalance in LLM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dmVlZsAAdc", "reviewer": "Reviewer_mEwF", "summary": "This paper addresses the challenge of learning on long-tail data (and the bias of foundation models). The author defines the imbalance problem as parameter imbalance and data imbalance. They propose a backdoor adjustment method to address the imbalance problem. Experiments conducted on different long-tailed datasets demonstrated the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. Below, we summarize your points in quotes, followed by our corresponding replies.\n> How to formulate incomplete semantic factors and the question of their maximum number.\n\nThe incomplete semantic factor ($C$) represents the semantic region in the image that the foundation model prefers (the foundation model relies on it to make the final prediction). For example, in the first row of Fig. 4, OpenCLIP predominantly attends to the head ($C=0$), whereas MetaCLIP primarily focuses on the body ($C=1$), which is further supported by the experimental evidence in Fig. 8.  Actually, different values of $C$ correspond to specific semantic regions, and the possible values of $C$ are infinite and are not limited to 0 (head) or 1 (body). In this way, the granularity of $C$ can be further refined, depending on the number of models. However, **we set the maximum number limited to three as a trade-off between performance and costs.** More details are in Sec.E.5.\n> The selection of foundation model and what model can be used and what model cannot?\n\nAs shown in Fig.5 in the paper, the path $D \\rightarrow C$ represents the incomplete semantic factor that arises due to parameter imbalance, stemming from the imbalance in the pre-training data of the foundation model. Consequently, we have chosen CLIP, OpenCLIP, and MetaCLIP because they are pre-trained on distinct datasets, resulting in varying degrees of parameter imbalance and differing incomplete semantic factors.\n\nIn causal theory, Eq. 7 should cover the value space of $C$ as fully as possible for accurate causal effect estimation. Since $C$ has infinite values, we approximate it finitely, with broader coverage improving estimation accuracy. In this way, model selection should take into account the path $D \\rightarrow C$. If two foundation models are trained on similar pre-training datasets, only one should be selected, as choosing both would not significantly increase the coverage of the value space of $C$. \n>Compare with MoE voting\n\nIn Eq. 7, $P(C)$ represents the prior distribution of the confounder. In large-scale datasets, this can be assumed uniform, making the final implementation resemble a voting process (like MoE) among fine-tuned CLIP models. However, unlike MoE, backdoor adjustment provides theoretical guidance for expert selection.  For example, the parameter imbalance results in different experts exhibiting distinct tendencies in their prediction distributions: OpenCLIP exhibits a \"head of the object\" preference ($C=0$), while MetaCLIP exhibits a distinct \"body of the object\" preference ($C=1$), as shown in Fig.8. Guided by backdoor adjustment, if an additional foundation model that prioritizes the body is introduced, it should be combined with OpenCLIP rather than MetaCLIP. This is because pairing it with OpenCLIP broadens the range of confounders accounted for, whereas MetaCLIP does not provide such an expansion.  To verify this, we introduce CLIP-CP, pre-trained on the CommonPool datasets, and combine it with OpenCLIP and MetaCLIP, respectively. The results are shown as follows:\n||OpenCLIP|MetaCLIP|\n|--|--|--|\n||51.6|51.6|\n|+CLIP-CP|51.9|51.6|\n\nThis experiment shows that combining CLIP-CP with OpenCLIP is more effective. Additionally, following the experiments in Sec. E.6, we calculate the average confidence scores for the head images ($confidence$=0.2732) and body images ($confidence$=0.7122) based on CLIP-CP. These results confirm that CLIP-CP exhibits a \"body of the object\" preference, consistent with MetaCLIP, and demonstrate the effectiveness of backdoor adjustment in model selection.\n>Questions about imbalanced factors\n\nImbalanced factors (IF) measure dataset imbalance, particularly in downstream tasks. For example, the imbalanced factors for ImageNet-LT, Places365-LT, and iNaturalist2018 are 256, 996, and 500, respectively. Datasets with higher IFs tend to have larger performance disparities between head and tail classes, so addressing these imbalances is crucial for better performance.\n\nFor pre-training parameter imbalance, due to the inaccessibility of pre-training data, we can only measure this imbalance by giving a specific downstream dataset. For example, we estimate the label prior using Eq. 3 from the paper on the Places365-LT dataset for different foundation models and then calculate the IF. As shown below, different foundation models exhibit varying degrees of imbalance when evaluated on the same downstream dataset.\n||CLIP|OpenCLIP |MetaCLIP|\n|--|--|--|--|\n|IF|57.50|63.25|60.20|\n\nThis metric can vary when samples are drawn from different distributions (domains), even if the category set is the same. For example, in a food classification task, dumplings and noodles may be more common in images from China, while hamburgers are more frequent in images from America. These cultural differences can lead to varying levels of imbalance.\n\nWe acknowledge that we will incorporate all the discussions in the paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The author defines the imbalance problem as parameter imbalance and data imbalance, and propose a method to address both.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The concept of parameter imbalance and data imbalance is quite important to community, they should be well addressed.\n\n- This paper provides sufficient details and experiments, and I can imagine the author has invested significant time in it. However, some definitions remain unclear or not well-formulated, and further revisions may be needed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How do you formulate the incomplete semantic factors? In your experiments, is the maximum number limited to three? Are there any limitations?\n\n- Why you select CLIP, OpenCLIP, and MetaCLIP to approximate the incomplete semantic factor? What model can be used and what model cannot?\n\n- The final equation pretty sounds like a simple mixture of the large FMs, like MoE, so what's the benefit of the proposed method compared to typical MoE methods? How about compared with the MoE results of CLIP, OpenCLIP, and MetaCLIP? For example, you can use the inference results from the finetuned CLIP, OpenCLIP, and MetaCLIP and make a voting machine from their results (without further training) and compare it to your model.\n\n- You define the imbalanced factor, then how and where it can be used? Will this score get significantly changed on different datasets even if they have the same categories?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission rethinks the inherent bias of foundation models on downstream tasks. Specially, the authors examined the imbalance bias from foundation models as parameter imbalance and data imbalance, and found the latter can be alleviated by the conventional methods, while the former cannot. Based on this observation, the authors proposed a new method from the perspective of causal learning, which is verified with about 1.67% improvement on each dataset.

Three reviewers reviewed this submission and gave the ratings of 4, 3, 4. After the rebuttal, one reviewer confirmed the authors' substantial effort while the other two reviewers keep silence. After AC carefully checked the reviewers' comments and the authors' feedback on improving the clarification and more experiments, it seems that most questions about this submission have been addressed. Therefore, based on the reviewers' suggestion and the authors' rebuttal, AC tends to recommend acceptance.
Decision: Accept (poster)