Rethinking Classifier Re-Training in Long-Tailed Recognition: Label Over-Smooth Can Balance
Authors: Siyu Sun
Han Lu
Jiangtong Li
Yichen Xie
Tianjiao Li
Xiaokang Yang
Liqing Zhang
Junchi Yan
Venue: ICLR 2025 Poster (2025)
Keywords: Long-Tailed Recognition and Decoupled Training
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: In this study, we propose a new logits-based metric for classifier re-training in long-tailed recognition, introducing a simple retargeting method "Label Over-Smooth" that achieves state-of-the-art performance on imbalanced datasets.

Abstract: In the field of long-tailed recognition, the Decoupled Training paradigm has shown exceptional promise by dividing training into two stages: representation learning and classifier re-training.  While previous work has tried to improve both stages simultaneously, this complicates isolating the effect of classifier re-training.  Recent studies reveal that simple regularization can produce strong feature representations, highlighting the need to reassess classifier re-training methods.  In this study, we revisit classifier re-training methods based on a unified feature representation and re-evaluate their performances. 
We propose two new metrics, Logits Magnitude and Regularized Standard Deviation, to compare the differences and similarities between various methods. 
Using these two newly proposed metrics, we demonstrate that when the Logits Magnitude across classes is nearly balanced, further reducing its overall value can effectively decrease errors and disturbances during training, leading to better model performance. 
Based on our analysis using these metrics, we observe that adjusting the logits could improve model performance, leading us to develop a simple label over-smoothing approach to adjust the logits without requiring prior knowledge of class distribution.
This method softens the original one-hot labels by assigning a probability slightly higher than $\frac{1}{K}$ to the true class and slightly lower than $\frac{1}{K}$ to the other classes, where $K$ is the number of classes.
Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist2018.

In the field of long-tailes recognition, the Decoupled Training paradigm has shown exceptional promise by dividing training into two stages: representation learning and classifier re-training- While previous work has tried to improve both stages simultaneously, this complicates isolating the effect of classifier re- re-training. Recent studies reveal that simple regularization can produce strong feature representations, highlighting the need to reassess classifier re- re-training methods. In this study, we revisit classifier re-training methods based on a unified feature representation and re-evaluate their performances. We propose two new metrics, Logits Magnitude and Regularized Standard Deviation, to compare the differences and similarities between various methods. Using these two newly proposed metrics, we demonstrate that when the Logits Magnitude across classes is nearly balanced, further reducing its overall value can effectively decrease errors and disturbances during training, leading to better model performance. Based on our analysis using these metrics, we observe that adjusting the logits could improve model performance, leading us to develop a simple label over-smoothing approach to adjust the logits without requiring prior knowledge of class distribution. This method softens the original one-hot labels by assigning probability slightly higher than 1 K to the true class Published as a conference paper at ICLR 2025
RETHINKING CLASSIFIER RE- RE- RE-TRIINI IN LONG- TAILED RECOGNITION:
LABEL OVER-SMOOTH CAN BALANCE
Siyu 1.2# Han Lu Jiangtong Li Yichen Xie", Tianjiao Li² Xiaokang Yang', Liqing Zhang', Junchi Yan DDpartment of CSE & MoE Key Lab of AI, Shanghai Jiao Tong University
Bilibilii Inc "Tongji University 4UC Berkeley (sunsiyu, sjtu,luhan, xkyang, yanjjunchijesstuu.dduu j
zhang--gdfcs.sjjuueeu..o
yichen xww@berkeley.coooooooooo .ddu, 1ttaanjae001111111111111
Code: http://gthhub.cccm/ttnikaaa---ss
ABSTRACT
and slightly lower than 1 K to the other classes, where K is the number of classes. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR 100-LT, ImageNet-LT, and 1Ntuuralis20188.
INTRODUCTION
Real-world datasets often exhibit long-tailed distributions, where the majority of samples belong to a few major classes, while the remaining samples are spread across many rare classes (Buda et al.
20T8,Zhang et al. 2023). Although traditional classification methods (He et al.) 2016 Redmon & Farhadi 2017) perform well on balanced datasets, they may not be effective on imbalanced datasets as they tend to prioritize majority classes and neglect minority ones (Buda et al. 2018 He & Garcial
2009: Van Horn & Peronal 2017). Long-tail recognition (LTR) has emerged as a challenging and important task, where models are trained on imbalanced datasets and are expected to perform well across both majority and minority classes (Zhang et al.] 2023).
Various techniques developed for LTRR, including class re- reieeigg (Cui al. 2019 Lin et al.
2017 Ren et al. 2020), class re-sampling (Kang et al. 2019 Wang et al. 2020a), and others. Among these techniques, the Decoupled Training paradigm (Kang al. 2019)
shows remarkable efficacy.
'Han Lu and Siyu Sun contribute equally. Junchi Yan is the correspondence author. Work was in part supported by NSFC (62222607, 62402341), Shanghai Municipal Seience and Technology Major Project (201SHHDDZX0102), and the Postdoctoral Fellowship Program of CPSF (G7C20241225).

---

Published as a conference paper at ICLR 2025
Many [Class 0:34]
Modium [Class 35-69]
Few [Class 70-99]
0.6
0.7
Non-Tuue
True
0.6
True
Non-True
677
Thue Non-True
Probability Density
Procabiitty Dersitty
Probability Dansey
0.5
8.5
0.4
(0.67, 0.24)
0.3
19 0.201
0.1
51
6.0
0.0
50
Logits
Logits
Logits
(a) Vanilla Cross-Entropy (CE). "Many" class logit 2.58 is larger than the ground truth "Few" class logit 1.69, leading to the misclassification.
Many [Class 0-34]
Medium [Cless 35-69)
Few [Class 70-99)
True
10
True
2.0
True
2.0
Non-True
Non-True
Non-True
Probabbiltt Densty
Densly
0.8
Probabilityy Donety
0.8
0.6
4.6
0.4
0.4
(0.52. 0.30)
(1.57, 0.27)
(0.37,
6.2
-56
2.5
5.
7.5
0.0
59
-2.5
Logits
Logits
Logits
(b) LOS (ours). "Few" class logit 1.57 outperforms the other classes.
Figure 1: An Intuitive View of Logits Magnitude Influence. The two models are trained on CIFARIO0 with IR=100. The blue columns represent the logits of the true samples of current class, while the red columns correspond the logits of the non-true samples. We take the instance indexed 7202 in test set as an example (the point). Its true class is 75, but it is erroneously classified as class 3 in Many class by Vanilla CE Our method LOS can obtain the more balanced and smaller logits magnitudes (defined for each class as the difference between the mean logits assigned to the true class and the mean logits assigned to the other non-true classes), helping discrimination in a multi-class classification scenario.
This approach divides the training process into two different stages: representation learning and classifier retraining. In the first stage, the model is trained to learn a general feature representation without considering class imbalance. In the second stage, the classifier is retrained using the acquired feature representation, employing techniques address class imbalance.
Previous works (Alshammari et al., 2022 Desai et al. 2021 Ren et al. 2020 Zhong et al. 2021 on Decoupled Training often improve both representation learning and classifier retraining simul- taneously, which can obscure the individual impact of classifier retraining. LTWB (Alshammari le al. 2022) shows that simple regularization techniques can outperform the original representation learning methods. These findings highlight the need to evaluate classifier retraining separately to isolate its effects and understand its true contribution model performance. By using unified feature representations, we can better identify effective factors and make improvements.
Therefore, we revisit classifier retraining methods, including re-weighting (Kang et al. 2019 Cao et al.) 2019 Ren et al. 2020; Lin et al. 2017; Cui et al. 2019), re-sampling (Shen et al. (016).. parameter regularization (Kang al. [2019] (01s et al. 2022). and direct logits modification during inference (Kang et al.) 2019 Menon et al., (021) ). For the fair comparison of these baselines, we design unified formulation and use consistent and high-quality first-stage feature representations to conduct accuracy experiments. The performance of the re-evaluated methods, shown in Tab. [] far exceeds those claimed in previous literature, providing a new, fair benchmark for further analysis.
Based on our experimental results, we conduct further analysis. Previous works (Alshammari et al.) 2022; Kang et al.) 2019) use classifier weight norm to show that their models do not overfit majority classes, but this metric is unreliable capacity. We propose a more reliable metric: Logits Magnitude (LoMa), defined for each class as the difference between the mean logits assigned to the true class and the mean logits assigned to
it ignores feature magnitude and limits representational
2

---

Published as a conference paper at ICLR 2025
other classes. We observe that performance correlates positively when the LoMa is balanced across all classes. Since directly optimizing LoMa is impractical, we introduce the Regularized Standard Deviation, calculated as the standard deviation of the logits divided by the LoMa. By reducing the LoMa, particularly when it is balanced across classes, we can minimize training disturbances and mitigate the effects of class imbalance, which inspired the development of our method.
We propose "Label Over-Smooth" (LOS), simple approach similar in form to label smooth- ing (Huang et al. 2019) that reduces LoMa while maintaining class balance without needing prior knowledge of class distribution. Our LOS method softens the original one-hot labels by assigning a probability slightly higher than K the true class and slightly lower than * to the other classes, where K is the number of classes. By over-smoothing the labels, LOS effectively reduces the LoMa and maintains class balance. Both theoretically and experimentally, we show that higher non-true label probabilities improve results. As illustrated in Fig. [ LOS maintains the discriminative power between the true class and other classes within each sample. It reduces the influence of dominant classes and diminishes majority class dominance, enhancing sample discernibility. Our method achieves state-of-the-art performance on various mainstream imbalanced datasets, such as CIFAR 100- LT, ImageNet-LT, and INaturalis22188. Extensive ablation studies also verify the robustness of our method to hyperparameter variations. Additionally, LOS can serve as gelug-and-pla retraining method that effectively enhances the performance of existing classifiers.
Our contributions can be summarized as follows:
We revisit previous classifier retraining methods for long-tailed recognition, re-evaluating them under unified framework to enable fair comparisons and further analysis.
We propose two metrics, Logits Magnitude (LoMa) and Regularized Standard Deviation, to effectively compare the distributions of logits values across different methods and classes. These metrics provide insights into performance and highlight conditions for achieving improved results.
We introduce a simple Label Ever-Smoothing approach (LOS) that does not require prior knowl- edge of class distribution. LOS mitigates overfitting and reduces bias caused by class imbalance in the training set, achieving state-of-the-art performance on three commonly used datasets.
RETHINKING CLASSIFIER BE-TRAINING
2.1 REVISITING CLASSIFIER RE- TRAINING METHODS
In long-tailed recognition, the Decoupled Training Paradigm <Kng et al. 2019) can effectively improve model performance. Various training paradigms for classification heads have been explored, including both joint training from scratch (Cao et al. 2019 Ren et al., 2020 Shen al. 2016) and
fine-tuning only (Kang et al. 2019 Lin et al. 2017] Cui et al. 2019 Alshammari et al. 2022 Menon
ett al., 2021). However, there is still a lack of systematic studies determine the most appropriate method for classifier retraining, as previous works (Alshammari et al.) 2022: Cui et al.) 2019 Ren
cet al., 2020 Zhong al. [221)) often use different backbone feature extractors, making comparisons difficult. To provide a fair comparison, we conduct an analysis of these methods using the same strong feature representations trained in the first stage by LTWB (Alshammari et al. 2022). During the retraining process, we freeze the weights of the backbone and train only the classification head.
Notation.
Suppose there are K classes with n, denoting the sample number of class i. The deep feature extracted from an image x is represented by f(x), while the classifier weight vectors are denoted as W [ur,..., wx).
Inspired by (Wu et al. 2021 we construct a unified formula for the different methods:
c.2y
((x)((x),)) -r log
Σ, cz.
where z, = 9: (w,, f(x)).
Given input feature f(x) and the label y, (w) denotes the re-weighting factor for current label categories and z, = ==((w; (; (x)) represents the calculated logit after classifier. The common used classifiers like Vanilla FC are (x)) = W f(x) + b. For simplicity,
omit the bias b in
the table. The methods Vanilla Cosine, Learnall Weight Scaling (LWS) (Kang et al. 2019), Label- Distribution-Awee Aware Margin Loss (Cao et al. 2019). Balanced-Soffe Softmmax Loss (Ren al. 2020 focus
3

---

Published as a conference paper at ICLR 2025
Table I: Study of the classifier re-training methods in decoupled learning scheme in LTR. Vanilla FC trains a general representation, performing itself in role as both the baseline and the backbone feature extractor. Experiments are conducted on CIFAR 100-LT with imbalanced ratio 100 and constant 7, 3, 8, r. See details for the listed formulation in Sec. 2.1
Methods
Formulation
All
Many
Accuracy
Medium
Few
Vanilla
yv W. S(x)
48.44
78.74
47.37
14.33
Vanilla Cosine
W /IW foo/Il 1/r
52.20
77.21
50.52
24.98
Learnalle Weight Scaling KKng et al
2019
Gi e, W. f(x), where cy learnable
48.99
78.51
47.51
16.26
LDAM
Cao
9i W. f(x) 148 y) C/n?
52.94
72.37
54.31
28.67
Salanced-Softmax Loss [Ron al.
2020
W.T (TT) + log (ne)
52.23
77.63
50.74
24.33
Ro-Sampling (Shen all
2016
r.(8) x 1/tiuin (e) the sample ratio class
48.44
78.74
47.37
14.33
Focal Loss Lin al.
2017
r(y) (1 pul". is the predicted probubility of
50.94
78.00
49.97
20.50
Clast-Bllanced Loss (Cui al.
2019:
r(y) = (1 BB (1 PC) applied with RCE loss
53.31
70.88
50.25
36.41
MaxNorm (Alshammari et
[Wilß s
51.64
77.60
49.37
23.99
--normalized Kang et al.
2019
Iy (W./ ([WA") 100)
53.01
74.74
47.31
34.30
Post boc Logit Adjustment Menon
2021
W. f(x) rlog (nu)
52.45
75.57
47.71
31,00
on designing different classifier function g. Re-Sampling (Shen et al. 2016) guarantees the same sampling probability of each class and use T. (i) as the sampling ratio, acting the supplementary of the equation. Focal Loss (Lin et al. (017)) and Class BalsaceB Loss (Cui et al.) (2019) design the
class weights (y) according class label y. In addition the general formulation in Eq. there are other perspectives. MaxNorm (Alshammari et al. 2022) regularizes the weight norm through Projected Gradient Descent (PGD) in practice. 7-normalized (Kang et al., (219)) and Logit Adjusted Loss (Menon et al. [2021) use h; as the logit modification function in the inference stage. We conduct experiments on CIFARTOO-LT dataset with imbalanced ratio 100 and report the results in Tab. T
2.2 INSIGHTS
Before delving into these methods, we will first present several propositions that aim to offer intuitive insights and foundational understanding relevant to the Long-tail problem.
Consider the general form of CE without regularization, assuming the label vector y instead of one-hot label in Eq.
6::
L(W,b; b: f()))))))))
Σ y. log
E E
(2)
where W fxx) b and define the final predicted probability that s, = /
Proposition 1 (Bias Convergence). Given a deterministic matrix W, loss function L. is a convex function with respect to b and will reach the global minimum as the corresponding bias converges.
Proof. Calculate the Hessian matrix of L w.r.t. b as H
H = (: - si)
ifi= ij
(3)
The Hessian matrix H is positive semi-definite matrix because x xH > 0 for any æ. (Refer to detailed proof in Appendix B
Therefore, our focus can shift to the optimization problem concerning the weight matrix W. In all cross-entropy based methods, the aforementioned conclusion remains valid.
Proposition 2 (Arbitrary Length). Take the basic cross entropy the loss function, we defined Eq. P If there exist optimal parameters W and by such that convergence is achieved, then there exist a series of convergence points (W', b²) where W can have arbitrary magnitudes.
Let w = w. + we have
e² E5 W3 = Exer e² pas-xx
s
= =Si,
4

---

Published as a conference paper at ICLR 2025
(a) Many Classes
(b) Few Classes
0.8
true samples
Probability Density
samples
0.6
Average Zn
Average of Zp
0.4
0.2
Logits
Logits
Figure 2: Visual interpretation of Logits Magnitude and Standard Deviation. The LoMa L is defined as the difference between the mean logits of true class (dashed blue line) and other non-true classes (dashed red line). The distribution simulates a normal distribution.
L(W bb; f(x))x L(W", bb; fx).)
(5)
The predicted probability is not affected by 6 as the length ||w:||2 of each w, could be changed.
Proposition highlighss that the magnitude of weight vectors can be arbitrary without affecting optimal prediction. This suggests that directly comparing || l| across classes is meaningless. To address this, propose Logits Magnitude, which enables unified comparision across classes.
2.3 LOGITS MAGNITUDE AND REGULARIZED STANDARD DEVIATION
Previous techniques such as MaxNorm (Alshammari et al. 2022) and 7-Norm (Kang et al. 2019) have been employed to constrain the length of each vector in W. However, these Weight Norm constraint methods cannot effectively characterize the relationships between different classes. Therefore, propose more reasonable metric directly from the perspective logits and compare it with the commonly used Weight Norm follows:
Definition (Logits Magnitude). Suppose the number of classes is K, Zp; represents the logits of samples of class and zns represents the logits of the samples for non-true classes (excluding class i). For each true class, the LoMa L E R.K is defined as the difference between the mean logits of true class and other classes.
L E(zpd] E[zw:]
(6)
Fig. 2 provides an intuitive visual interpretation of LoMa. We compute the mean and variance the logits for the true and non-true classes, and fit them to a normal distribution. We display schematic diagrams for the "Many" and "Few" classes as a whole for comparison.
LoMa is effective in characterizing the performance. As observed in Fig. 3a where the methods are listed increasing order of performance, we have empirically found that achieving more uniform distribution of LoMa tends to yield improved performance. Fig. iillstrates the intuitive advantage balanced LoMa on performance, particularly for the few classes.
Previous methods such as (Alshammari al. 2022 Kang et al. 2019) often incorporate regularization as crucial parameter. Regularization helps align each Lj, but decreases the capabilities of W. LoMa shows reduced bias susceptibility and a strong correlation with classification accuracy, unlike vector length magnitudes. However, it is challenging to optimize directly during training. To enhance performance under balanced LoMa, introduce Regularized Standard Deviation, evaluate and overcome these challenges.
metric to
Definition 2 (Regularized Standard Deviation). Given the logits z, the standard deviation (z) and logits magnitude L. The regularized standard deviation R E TRK is defined as:
(2,) L
R.
The corresponding R, increases gradually as the classes transition from head to tail shown in Fig. 3b This observation in line with general intuition, classes with more training samples typically
5

---

Published as a conference paper at ICLR 2025
(a) Logits Magnitude
(b) Regularized Standard Deviation
Varilla PC
Logts Magntudde
tocal
Balances Sortmee
LWS
teu-hom
Van Coain
Markum
130H
Caee-Balaredd
Requi Requlnnze annuee Standrd
0.5
LDAM
Naxtaom
UAS
Claos Belanced Loss
Focal
Verill Cocne
Nomm
Pnlenoss-ootmme
LOStearst
9.4
LOS
0:230
21-26
21-30
31-41
diusd
31-50
21-70
22-50
21-56
Cosa $1-300
cuss 1-10
21-20
31-40
-1-00
51--0
82-18
72-40
22-90
cour 91-103
Figure 3: Overview of our proposed metrics. Left: Overview of Logits Magnitude for various
methods on CIFAR100-LT with an imbalanced ratio of 100. Classes are grouped into segments of 10, and mean values computed for comparative analysis. The methods in the legend are sorted in ascending order of performance from top to bottom. The difference in means between samples from true and non-true classes is evaluated for each class in the test set. Magnitude regularization using the I-norm is employed to enhance comparability. Right: Overview of Regularized Standard Deviation. The true distribution of logits for each class not available considering the lack of the samples, so the displayed results represent computations on the test set.
exhibit better feature extraction. Importantly, the computed R, values are quite similar for models derived using different optimization techniques. Therefore, the Regularized Standard Deviation can serve as an approximate invariant for further analysis.
In conclusion, we introduce LoMa as an effective metric for characterizing model performance, demonstrating that a more uniform distribution correlates with improved outcomes, particularly for minority classes. Additionally, Regularized Standard Deviation exhibits minimal variation across different models, with consistent trends.
METHOD
3.1
DEEP DIVE INTO LOGITS MAGNITUDE (LoMA)
In classification problems, minimizing the loss L. and maximizing classification accuracy are not completely equivalent. Based on the above observation, we find that the computation of gradients for parameters associated with the "Few" classes is significantly influenced by sampling and process perturbations, whereas the impact on "Many" classes is relatively mild. This manifests as the model often being overconfident in the "Many" classes while exhibiting unstable predictions for the "Few" classes. We discuss this process in the following.
Suppose we achieve relatively balanced magnitudes for each class, observed in Sec. 2.33 denoted as L2 R L2 R 2 Lg. Despite this balance, it is inevitable to encounter variations in the standard deviations (2,) R,L, across different classes. These variations significantly impact the training dynamics and the computation of gradients during the training process. Although it is not feasible to completely eliminate the disparity in R;L; caused by the data distribution, the impact can be mitigated by significantly reducing the magnitudes L.
Consider random perturbations A, that independently affect each class z, w.ff + b, and are proportional to the standard deviation of their logits. i.c., z z; + A, and the perturbation A, ~ R,L,e. Here, is random variable with an expected value of Eje] perturbations, we can better understand their impact on training dynamics and develop strategies to mitigate adverse effects. These perturbations are introduced during the training process, particularly due to factors such as overfitting in the initial stage and bias in the sampling of the training set.
By modeling these
When comparing the use of z; and z for subsequent calculations, the impact is minimal for balanced datasets but be significant for imbalanced datasets.
Proposition 3.

---

Published as a conference paper at ICLR 2025
The expected value of probability s can be expressed as:
1
Eis
E Σg
E
(8)
el (2)
In the balanced datasets, expected with R, R2 RR. (RyLı R,L,))
close to zero.
E(s:]
E E|s Elexp (A, - A:) Σ,
ex c Elexp (A, - Ar)
where Ejexp (A, A,)] can be considered as a constant &. See the Appendix C for a detailed analysis of the approximation. For balanced datasets, the impact of disturbances is consistent across different classes i.
E[s|] E(s.] 1/E.
(10)
However, for imbalanced datasets, Eq. no longer holds. There is significant disparity in the values of exp ( A,) for different class J. R R,, Ejexp (A, A,)] < E/exp (A: A.)]. For instance, when follows normal distribution we have
[[e - -A))] = exp((o(A, - A,)))/2).
(1I)
where (4, - A,) represents the standard deviations the random variable (A, - A.). ). As result, the impact of perturbations is directly correlated with the differences in standard deviations between classes, leading to E[s[] / [[[s:] not consistent across classes i, where E[s{]] / TE[s,] * 1/E. The expectation [[[s:] plays crucial role as it not only relates to the final predictions but also has direct correlation with the derivatives obtained from the gradient descent process. Specifically, the expectation of the derivative EI ( [E[(s y:)] can be computed the difference between the
expectations [s: and []
Therefore, in the long-tailed scenario, the perturbations A introduced in training process will have significant impact on model training, ultimately compromising the final classification accuracy.
Deliberately reducing the logits magnitudes holds promise for addressing the effects caused by these perturbations. Previous methods (Alshammari et al., 2022 Kang et al.) 2019) can achieve implicit reduction of LoMa by regularizing the weights of the prediction head. However, they also limit the expressive power of the classification head. Other methods (Cao et al. 22199 Ren et al.) 2020) Cui et al., 2219;; Menon et al. 22021) also fail to recognize the importance of deliberately optimizing this effective objective. To overcome these limitations, we propose "Label Over- wver-Smotthing" approach (LOS) soften the original one-hot labels and achieve balanced logits magnitude.
3.2 LABEL OVER-SMOOTH
Based on the above analysis, we can observe that when the value of s; is larger, the impact of perturbation smaller, and the differences between classes are also smaller. So, intuitive idea is to make as large possible during the process.
In our method, we softens the original one-hot labels by assigning a probability slightly higher than 1 I to the true class and slightly lower than π to the other classes, where K is the number of classes. R
is essential to ensure that the probability of the true class is higher than the probability of each non-true class for model convergence and discriminability in the true class.
Suppose the number the categories K. Given the input image x and label y, feature extractor backbone f(-) and classifier 0 (W, b), the formulation of our "Label Over-Smooth" is as follows:
c
C(O: ((x)))
Y: log
Σ, e
(12)
1 8- K
i
8.
K
i i y
where E [0, is a constant value to control the overall non-true class probability. Interest- ingly the above formulation is fundamentally consistent with the conventional Label Smoothing
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison for CIFARIO0-LT, ImageNet-LL and iNaturalist2018 Benchmarks. Top-1
accuracy (%) is reported and CIFARI00-EE consists of three imbalanced ratio (IR) 100/50/10. Our model outperforms all the baselines, showing satisfactory accuracy in the few-class categories.
Method
Reference
CIFAR100-L.
ImageNet-Li
INoturalist2018
IR=100
IR=50
IR=10
Many
Medium
Few
All
Many
Medium
Few
All
38.3
65.9
37.5
44.4
72.2
63.0
61.7
1OCV17
38.4
44.3
55.7
29.9
16.0
30.5
DDAMEDRWEC Con ml. 2019
NIPS19
42.0
46.6
58.7
48.88
eRT Kanr mi 2019
ICLRV
61.8
46.2
27.3
49.6
69.01
66.0
63.2
65.2
2119
ICLR19
47.7
52.5
63.8
59.1
46.9
30.7
49.4
65.6
65.3
65.5
65.6
CB-CE
ca 201V
CVPR19
39.5
45.3
58.0
39.6
32.7
16.8
33.2
53,4
54.8
53.2
54,0
Do-confiint Tanr o1 20721
NIP520
44.1
50.3
59.6
62.7
48.8
31.6
51.8
BAI MStRen a 20221
NIPS20
50.8
63.0
61.1
48.5
31.8
50.9
65.5
67.5
67.5
67.2
Zhog el
CVPR20
42.6
70.8
66.3
LogitAjaar
K21
42.0
47.0
57.7
69.4
DisAlign C-P Mi211
CVPR21
61.3
52.2
31.4
52.9
69.0
71.1
70.2
70.6
LTWB
2022
CVPR22
5344
57.7
68.7
62.5
50,4
41.5
53.9
71.2
70,4
69.7
70.2
AREA
IOCV23
68.4
RBI. P
ICML22
53.1
64.8
49.6
34.2
53.3
EWB-FDK Masqgina & Sato 2024
KCL244
53.0
63.4
50.0
35.1
53.2
1.08 (ours)
$4,9
58.8
69.7
63.2
50.7
42.3
54.4
69.2
70.7
71,3
70.8
approach (Szegedy al.) 2016) and we can name the tlyper-paramete 5 as "Smooth Factor". The primary distinction lies in the ó, which can be set as a quite large value in our approach.
To best knowledge, Label Smoothing is typically employed in lightweight manner. For example, 5 is usually set as 0.2. Our method represents a novel approach in the LTR field, setting 5 as 0.98 or 0.99 to achieve SOTA results. As theoretically analysis in Sec. Scc. a larger label smoothing value can reduce the LoMa to achieve better performance. From this perspective, d approaching 1 can yield the best results. However, our method essentially acts as an auxiliary regularization technique to mitigate biases among different class samples and assist the model in converging. The model needs to exhibit both convergence and discriminability, meaning that the ground truth labels should remain distinct from the other labels. We believe this represents trade-off, and we demonstrate this phenomenon experimentally in Sec.
Compared to previous methods, LOS directly focus on optimizing the fundamental objective LoMa. Our method not only achieves outstanding experimental performance but also offers case of implemen- tation and independence from prior information about class distribution. This represents significant improvement both theoretical analysis and empirical performance over existing methods.
4 EXPERIMENTS
We conduct experiments on long-tailed image classification benchmarks: CIFARI00-LT (Krizhessky al. 2009), ImageNet-LT (Liu et al.] 2019) and iNauuralss20018 (Van Horn et al., 2018). All experiments are implemented using Py ToTch and run on GeForce RTX 3090 (24GB) and A100-PCIE (40GB). Source code will be made publicly available.
4.1 DATASETS AND IMPLEMENTATION DETAILS
We define the class number as K and imbalanced ratio (IR) of the long-tailed datasets IR- Tmazz/ Pmaz//Tmmn Roméns where Nmaz and mmmn is the number of training samples in the largest and smallest class. respectively. In line with the previous SOTA work (Alshammari al. (022), use ResNet34 (He et al. 2016) in CIFAR 100-LT, Re50eXt50 (Xie et al. [011]] in ImageNet-L. and ResNet50 (He et al.
2016) in NNtuuralist2018. 8. For smooth factor, we use 0.98 in CIFARI00-LT and 0.99 in ImageNet-LT and NNtuuralis20188. 188. More details of the datasets and implementation are included in Appendix D
Evaluation Metrics. We evaluate the models on corresponding balanced test dataset and report top-1 accuracy. In line with the protocol in literature (Liu et al. (001)). we give accuracy on three different splits of classes with varying numbers of training data: many (over 100 images), medium (20~100 images), and few (less than 20 images).
4.2 BENCHMARK RESULTS
Compared Methods. We selected the popular methods in LTR and split them into two categories. (i) Baseline models: Some of these models modify the loss directly, such cross entropy (CE) loss, class
8

---

Published as a conference paper at ICLR 2025
Table Plugin for CIEARIO0-LT, ImageNet-LT. Top-1 accuracy (%) is reported and CIFARI00- LT consists of three imbalanced ratio (IR) 100/50/10. Our method can be directly plugged into non-conflicting upperbound methods by finetuning the classifier and can significantly improve results.
Method
Reference
IR=100
CIFARIOU-LT
IR=10
Many
Medium
ImtgeNet-LT
Few
Vallina CE + LOS (ours)
54,9
58.8
69.7
63.2
50.7
42.3
54.4
RIDE
Wang et al. 2020b
ICLR20
49.1
67.9
52.3
36.0
56.1
ICCV21
46.0
62.3
53.1
35.4
PaCo
2021
ICCV21
52.0
$6.0
64.2
63.2
51.6
39.2
54.4
7hu al
CVPR22
51.0
64.4
53.5
36.3
NCL
e Ml. 20224
CVPR22
53.3
$6.8
57.4
GMI
& Sen
53.0
65.7
ProCo
Di et al. 2024
TPAM124
52.8
57.1
65.5
58.0
RIDE-TOS((UUS)
50.2(+1.1)
66.9(-1.0)
53.2(+0.9)
37.9(+1.9)
56.5(+0.4)
SSD+I.0S(our)
1..3(+1.3)
51.4(+0.9)
(3-1(-))8)
66 66,2(-0.6)
53.8(40,7)
36.7(-1.3)
56.3(+0.3)
PaCo+-LOS(ours)
52.9(+0.9)
56.8(+0.8)
65.3(-1.1)
62.7(-0.5)
52.5(+0.9)
41.2(+2..)
55.00+0.6]
BCL+LOSS(urs))
12.2(+1.2)
56.0(+1.1)
65, (6.1(-0.7)
64.7(-0.6)
54.1(-0.8)
28. (8-1())8))
56.1(+0.5)
NCL+L.OSrurs))
54.1(+0.8)
57.5(+0.7)
57.7(+0.3)
GML+LLOS(ourS)
54.0++10)
58.4(+0.8)
66.6(-0.9)
68.00-0.7
55.7(40.6)
40.7(+2.1)
58.7(+004)
ProCo+LOS(ours)
53.9(+1.1)
58.0(+0.9)
66.7(-1.2)
58.5(+0.5)
Table 4: Improvement for different feature representation backbones (in different rows) on CIFAR IO with IR=100. LOS outperforms all finetuning methods (across columns).
Backbone Training
Classitier Finetune Methods
Vanilla
CE
Focal
LDAM
BS
CB-BCE
LOS(ours)
CE
41.59
41.67(+0.08)
44.18(42.59)
43.87(42.28) 44.03(+344)
45.01(43.42)
45.18(+3.59)
CE-DRW
42.38
42.38(+0.00)
42.55(+0.17)
42.41(+0.033
44.27((1.89)
44.39(+2.01) 44.88(+2.50)
LDAM DRW
44.43
44.45(+0.02)
44.58(+0.15)
45.34(-0.91) 44.47(+0.04)
45.52(+1.09) 45.86(+1.43)
BS
45.90
45.91(+0.01)
46.04(+0.14)
45.90(+0.00)
46.81(-0.91) 46.76(+0.86) 47.08(+1.288)
balanced CE loss (CB-CE) (Cui et al.
2019) or focal loss (Lin al. |
2017). RBL (Peifeng et al. 2023)
focuses the perspective of feature learning. De-confound (Tang al. 2020), LogitAjust (Menon
et al. 2021), BALMS (Ren et al. 2020), AREA (Chen et al.] 2023) correct the classifier from different analysis aspects. BBN (Zhou al.) 2020 LDAM-DRW (Cao et 2019) leverage learning rate annealing strategy to balance the feature learning process and the re- re-weighting procedure. The more straightforward method is decoupled learning (Kang al. 2019). consisting of cRT (Kang ct al. 2019), 7-norm (Kang et al. 2019). DisAlign (Zhang et al. 202 and LTWB (Alshammari 2022 Among them, LTWB the most competitive model, whose main contribution is to use simple regularization to learn better robust features in the first stage. EWB-FDR (Hasegawa & Sato 2024) delved into the success of LTWB by exploring Fisher's 's Discriminant Ratio in depth and streamlining the process, albeit at the expense of reduced performance. (ii) Upper Bound models:
al.
These models tend to make complex design or need more computational resources. They include self-supervised pretraining method PaCo (Cui et al.) 2021). BCL (Zhu et al. 2022). GML (Suh & Seo (023). ProCo (Du et al. 2024), additional augmentation training data method OPeN (Zada et al.) 2022), NCL (Li al.) 2022a) multi-experts ensemble method RIDE (Wang et al., (220b)), transfer learning method SSD
et al 2021). We utilize the results reported these original works.
Results and Analysis. Tab. presents the results for taiggt dataset recognition across three benchmarks. Our method LOS achieves state-of-the-art performance on all the benchmarks. Com- pared the origin SOTA model LTWB (Alshammari al. 2022) on CIFAR100-LY, LOS achieves an improvement of 1% ~ 1.5% in CIFARTOO-LT dataset, 0.5% in ImageNet-LT and 0.6% in iNat- uralist2018. In the long-tail tasks without bells and whistles, the improvement significant. On ImageNet-LT, LOS achieves a significant improvement in classification accuracy for minority classes compared to previous methods. In Tab. 3 we present the performance the Upperbound models. On the CIFAR 100-LT dataset, our method LOS, significantly outperforms others, whereas on ImageNet, these methods take the lead. We attribute this phenomenon to a general gap in feature learning on more challenging datasets. In CIFAR 100-LT, even simple models can overfit the imbalanced training set, making the main goal to ensure feature generalization and robustness, while fine-tume tenint the classifier to adapt better to balanced data. As result, LOS excels over other methods. However, for larger datasets, there remains considerable room for improvement in the backbone network's 's feature representation. Hence, more advanced methods hold clear advantage in this aspect. Consequently,

---

Published as a conference paper at ICLR 2025
80
IR=10
75
IR~100
70
Accuracy
65
60
55
50
Smooth
Figure 4: Effect of Smooth Factor. The overall accuracy will be improved with the growth of smooth factor. As the imbalance ratio (IR) increases, the improvement brought by our method becomes more sigmificant. For fully balanced dataset (IR=1), our method has no effect. Moreover, it can be observed that the improvement between 0.2 and 0.3 is very gradual.
we leverage our method plugin to assist these methods in fine-tuning the classification head, yielding notable improvements, as shown in Tab.
4.3 ABLATION STUDY
We further conduct ablation study on four aspects: 1) Effect of Smooth Factor, 2) Improvement for different backbone. See the Appendix E to view sensitivity of learning rate and weight decay.
Effeet of Smooth Factor. The smooth factor the only adjustable parameter LOS. Sec. we suggest using large label smooth factor, and further experiments exploring the impact of different values. We evaluate LOS using a range smooth factors. The value 0.0 represents the Vanilla CE loss and the value 0.2 represents the common used label smoothing method (Szegedy et al., 2016 which uses soft labels to reduce overfitting. Due to the very gradual changes within the commonly used range of label smoothing, investigation. Fig. shows overall accuracy improvement with the increase of smooth factor. The best overall accuracy achieved with 0.98 and the adjacent accuracy is close, within the allowable range of disturbance. Therefore, we can choose the large smooth factor reassuringly.
believe this has led previous researchers not pursue further
Improvement for Different Backbone. Our method has achieved excellent results based on strong feature representation, and we are interested in exploring the potential for improving models with arbitrary representations. Therefore, we first train traditional methods, freeze the trained backbone, and then finetune the classifier using different finetuning methods. In Tab. 4 our method exhibits the strongest ability to improve performance compared to the other fine-tuning methods evaluated. We find that the first-stage features without prior imbalanced information are more malleable, and different methods can obtain significant improvements, while imbalanced feature representations not often be effectively improved. Interestingly, the BS finetuning method applied to a BS- trained backbone can still result in significant improvements, indicating the importance of second stage training. As plug-and-play adjustment method, our approach is capable of effectively improving the performance of other models by finetuning the classifier.
5 CONCLUSION
In this paper, we conduct thorough analysis of current classifier retraining methodologies and introduce two innovative metrics: "Logits Magnitude" and "Regularized Standard Deviation." These metrics offer fresh insights into model performance and highlight the key requirements for achieving enhanced accuracy. Consequently, we propose a "Label Over-Smooth" (LOS), which systematically adjusts the optimization targets for logits and achieves more balanced Logits Magnitude. Our extensive experimental evaluation demonstrates that LOS achieves tfate-ofthhe-e performance on various long-tain loig-tailed recognition datasets. At the same time, LOS not only adapts to theoretical analysis but also serves as regularization constraint on the model's capacity. Therefore, there may be opportunities in the future explore more suitable methods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In IEEE/ EVEE//VF Computer Vision and Pattern Recognition Conference (CVPR), 2022.
Mateusz Buda, Atsuto Maki, and Macicj A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 2018.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In International Conference Computer Vision (ICCV), 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with labelldissributtin-aaaa margin loss. Advances Neural Information Processing Systems (NeurIPS), 2019.
Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang. Area: adaptive reweighting via effective area for long- on Computer Vision (ICCV), 2023.
long-tailed tailed classification. In International Conference
Hsin-Ping Chou, Shih-Chieh Chang, Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference on Computer Vision Workshop (ECCV Workshop), 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In International Conference on Computer Vision (ICCV), 2021.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In JEEE/ CVF Computer Vision and Pattern Recognition Conference (CVPR), 2019.
Alakh Desai, Tz- Yz-Y Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations: The devil is in the tails. In International Conference on Computer Vision (ICCV), 2021.
Chaoqun Du, Yulin Wang, Shiji Song, and Gao Huang. Probabilistic contrastive learning for long-tailed visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
Naoya Hasegawa and Issei Sato. Exploring weight balancing on long-tailed recognition problem. In International Conference on Learning Representations (ICLR), 2024.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions Knowledge and Data Engineering (TKDE), 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2016.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for tailed visual recognition. In IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2021.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. IEEE/ CVF Computer Vision and Pattern Recognition Conference (CVPR), 2016.
Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural net- works using pipeline parallelism. Advances in Neural Information Processing Systems (NeurIPS), 2019.
11

---

Published as a conference paper at ICLR 2025
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations (ICLR), 2019.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations (ICLR), 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, University of Tront, 2009.
Jun Li, Zichang Tan, Jun Wan, Zhen Lei, and Guodong Guo. Nested collaborative learning for long- tailed visual recognition. In JEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022a.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022b.
Tianhao Li, Limin Wang, and Gangshan Wu. Self supervision to distillation for long-tailed visual recognition. In International Conference on Computer Vision (ICCV), 2021.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In International Conference on Computer Vision (ICCV), 2017.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXiv preprint arXiv:2110.0500 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale long-tailed recognition in an open world. In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /608 /608.03833 2016.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision (ECCV), 2018.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representations (ICIR), 2021.
Gao Peifeng, Qianqian Xu, Peisong Wen, Zhiyong Yang. Huiyang Shao, and Qingming Huang. Feature directions matter: Long- Longetailed learning via rotated balanced representation. In International Conference on Learning Representations (ICLR), 2023.
I. Perez. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint 17Xiv:1712.046221 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In JEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2017
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems (NeurIPS). 2020.
Lixiang Ru, Xin Guo, Lei Yu, Yingying Zhang, Jiangwei Lao, Jian Wang. Jingdong Chen, Yansheng Li, and Ming Yang. Parameter-efficien complementary expert learning for long-tailed visual recognition. In ACM International Conference on Multimedia (MM), 2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European Conference Computer Vision (ECCV), 2016.
12

---

Published as a conference paper at ICLR 2025
Jiang-Xin Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin Han, and Yu- Yu-Feng Li. Long-tail tail learning with foundation model: Heavy fine-tuning hurts. In International Conference on Machine Learning (ICML), 2024.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of big data, 2019.
Min-Kook Suh and Seung- Woo Seo. Long-tailed recognition by mutual information maximization between latent features and ground-truti labels. In International Conference on Machine Learning (ICML), 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe. Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE CVF Computer Vision and Pattern Recognition Conference (CVPR), 2016.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the had momentum causal effect. Advances Neural Information Processing Systems (NeurIPS), 2020.
Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint 1.X9v:170.0014440 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In IEEE/ / CVF Computer Vision and Puttern Recognition Conference (CVPR), 2018.
Binwu Wang, Pengkun Wang, Wei Xu, Xu Wang. Yudong Zhang, Kun Wang, and Yang Wang. Kill two birds with one stone: Rethinking data augmentation for deep long-tailed learning, In International Conference on Learning Representations (ICLR), 2024.
Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng. The devil is in classification; simple framework for long-tail instance segmentation. In European Conference on Computer Vision (ECCV), 2020a.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by rout- ing diverse distribution-aware experts. International Conference on Learning Representations (ICLR), 2020b.
Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long-tailed taile distribution. In JEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.. Aggregated residual transformations for deep neural networks. In IEEE/ CVF Computer Vision and Pattern Recognition Conference (CVPR), 2017.
Shiran Zada, Itay Benou, and Michal Irani. Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images. In International Conference on Machine Learning (ICML), 2022.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In JEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2021a.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In International Conference Computer Vision (ICCV), 2017.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint arXiv:2107.092: 2021b.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long- learning: A survey. IEEE Transactions Paltern Analysis and Machine Intelligence (TPAMI), 2023.
13

---

Published as a conference paper at ICLR 2025
Yan Zhao, Weicong Chen, Xu Tan, Kai Huang, and Jihong Zhu. Adaptive logit adjustment loss for long-taiin tailet visual recognition. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In JEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-brance network with cumulative learning for long-tailed visual recognition. In JEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2020.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022.
Linchao Zhu and Yi Yang. Inflated episodic memory with region self-attention for long-tailed visual recognition, JEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2020.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Long-Tailed Recognition (LTR). In real-world scenarios, the collected data exhibits long-tail distribution, wherein a small fraction of classes possess a substantial number of samples and the remaining classes associated with only few samples (Zhang al. 2023). To address oong-tailed class imbalance and the failure of the vanilla empirical risk minimization methods, massive studies have been conducted recently. (i) Re-sampling aims to balance the training samples of different
class (Kang et al. 2019; Wang et al. [2000a] Mahajan et al. 2018 Ren et al.) 2020) by over-sampling the data from the tail class and under-sampling the data from the head class. (ii) Cost-sensifive learning adjusts loss values for different classes to make a balance, such as focal loss (Lin et al.
2017). class-balanced loss (CB) (Cui al. 2019). balanced softmax loss (BS) (Ren al. (0200)
LADE (Hong et al. (202T). LDAM (Cao et al.) 2019) and logit adjustment loss (Zhao et al. 2022
ct al.
2022b). (iii) Representation Learning focuses on improving the feature extractors, consisting
some paradigms, i.e., metric learning (Huang et al. 2016,] Zhang et al.] 2017), prototype learning Liu
ct al.
2019 Zhu & Yang 2020), and transfer learning (Zhong et al. 2021 Li et al. 2021). (iv)
Classifter Design replaces the linear classifier with well-designed classifiers, such as scale-invariant
cosine classifier (Wu et al., 202I) and 7-normalized classifier (Kang et al., 2019). (v) Decoupled
Training decouples the learning into representation learning and classifie retraining. rather than
training them jointly. (vi) Other methods use data augmentation (Zada et al. 2022 Zhong et al.
2021 Perez, 2017: (Shorten & Khoshgoftaar [2099] Chou et al. 22020) (Li al. 2022a Wang et al.
2024). self-supervised learning (Liu et al. 2021 Cui et al. (022)) (Suh & Sco
(002)).
). (Du et al.
2024). multi-expert ensemble (Cai et al., 2021| Cui et al.) 2022 Zhang et al.)
[2021b), pretrained
models (Ru et al., 2024 Shi et al. 2024) and so on (Zhang et al., (202)). These models require more
data or computation resources and are inclined to achieve high accuracy. In the paper, we report them as the upper bound baseline.
Decoupled Training. The Decoupled Training scheme (Alshammari et al. 2022 Cui et al., 2019
Zhang et al. [021a] |en et al. 2020) in LTR separates the joint learning process into feature learning and classifier learning stages. Decoupling (Kang et al.) (019)) is the pioneering work to experimentally evaluate different sampling strategies in the first stage and classifier design (e.g. cRT and 7-norm) in the second stage. KCL (Kang et al. 202T) empirically finds a balanced feature space is important for LTR. MiSLAS (Zhong al. (021) observes that data mixup benefits the feature extractor in the first stage but harms classifier performance in the second stage. DisAlign (Zhang et al. 2021a) develops the distribution alignment strategy with a generalized re-weight method. (LTWB) B) (Alshammari et al. 2022) emphasizes parameter regularization in both feature learning and classifier finetuning, achieving state-of-the-art performance.
B PROOF OF PROPOSITION 1 BIAS CONVERGENCE
Proof. The loss function is
E² E,eet ca
L -=Σ y, -...oog
(13)
Using the Chain Rule to calculate derivatives,
2c
OL 0s;
Σ
Oz:
Us, Oz.
OC as:
Σ OC as;
+
0s: Oz.
0is Oz.
jti
yi (s 8s)) Σ
Yi (-8,8)
(14)
8;
8,
j&i
= 1)+ + Σ y,s:
j+1
=i - -y:
15

---

Published as a conference paper at ICLR 2025
Therefore, the Hessian matrix of L with respect to b as H is as follows:
s 8:)
if
H 1
(15)
5,8j
j
And for any vector x,
xHx = Σ sxx? -
Σ s;x,
(16)
Since g(x) = =x2 is convex function, with 8; and Σ S: by Jensen's Inequality:
Σ s >
Σ sixi
(17)
xHx
Therefore, the Hessian matrix H is positive semi-definite matrix.
C ANALYZING THE APPROXIMATION IN EQ. 9
The fomulation:
1
E
ES ,ld - d,) -2,]
~
Ele
A.
E
Σ
e
(18)
For convenience, let P3 A Aj, Gj 2j Zi.
1
E
23
E
(19)
Σ
e
eoo
Elep3
e9
E
(20)
Σ,
E pps
69
Ee(er
E(eps) Deas
Assuming that the fluctuations ePs approximation the reciprocal:
[ are small compared to E[ees), we can use the first-order
1
82
(21)
A
Aa
8
(22)
A
1
E, E(ePss)ee
E
E E
E
(23)
Σ, ePs egi
ElePs
E
E,
(BlePs) E, e:0 )2
Since BlePs [F[eP]] 0,
1
E
Ri E
(24)
Σ, cpic
E[eP' Σ, E9
16

---

Published as a conference paper at ICLR 2025
D DATASETS AND IMPLEMENTATION DETAILS
We define K' as class number and imbalanced ratio (IR) of the long-tailed datasets as IR = Aemasr Mminn where Amax and Remin is the number of training samples in the largest and smallest class, respectively.
Datasets. 1) CIFAR 100-LT: The original balanced CIFARIO0 (Krizhevsky et al. 2009) consists of 50,000 training images and 10,000 test images of size x2 with 100 classes. Following (Cao et al.. 2019), the cong-tailed version CIFARI00-EE was constructed by exponentially decaying sampling for the training images of each class, maintaining the balanced test set unchanged simultaneously. Setting the imbalanced factor IR to 10/50/100, we create three long-tailed training datasets as (Alshammari al. 2022). 2) ImageNet-LT is built in (Liu et al., 2019) from ImageNet dataset (Russakovsky et al., 2015) with 1000 classes. The number of the images for each class ranges from 5 to 1280, which means the imbalanced factor IR-256. 3) iNaturalist2018 (Van Horn et al. 2018) is a roal-world, large-scale dataset for species identification of animals and plants with 438K images for 8142 classes. The imbalanced factor is extremely large as 500.
Implementation. In line with the SOTA method (Alshammari et al. 2022 we use ResNet34 (He ct al. 2016) in CIFARIO0-LT, ResNeXt50 (Xie et al. 2017) in ImageNet-LT and ResNet50 (He
et al. 2016 in Naturalist20188 For each model,
SGD optimizer with momentum 0.9 and cosine learning rate scheduler (Loshchilov & Hutter (016) from given initial learning rate to zero.
We use Decoupled Training Scheme which consists of training the feature extractor and finetune the classifier separately. The feature extractor is trained with cross-entropy (CE) loss and suitable weight decay follows (Alshammari al., 2022) and the classifier is finetuned using our proposed method. On CIFAR 100-LT, We train each model for 200 epochs. with batch size 64 and initial learning rate 0.01. On ImageNet-LT iNaturalist2018 we train for 200 epochs, with batch size as 128 256 and initial learning rate 0.03 / 0.1. We use random left-right flipping and cropping as our training augmentation, left-right flipping as testing augmentation for the three datasets with the same configuration
(Alshammari et al., 2022). In ImageNet-LT, following (Cui et al. 2022). For smooth factors, we use 0.98 in CIFARIOO I00-LT and 0.99 in ImageNet- LT and INaturalist2013 The classifier finetune epoch is 20 with adequate learning ratio and weight. We report the best result in five trails. Results in Tab. for CIFAR100LA with IR=100 also follow this implementation.
also use the color jitter
E SENSITIVITY OF LEARNING RATE AND WEIGHT DECAY
Best Accurary Learning Rate and Weight Decay
g
0:00
Learning Rate
54.25
59
6,03
25.00
V1.u
k1
35.66
14.4
61.45
$3.54
54,14
$4.77
54.23
54,54
54:4
54.50
0,003
31.33
31.35
11.04
11
11.03
13.45
31.51
Weight Decay
Figure 5: Sensitivity Comparison of Learning Rate and Weight Decay.
The adjustable hyper- parametrrssss in classifier finetuning are learning rate and weight decay. We use grid search to find the best accuracy and report the sensitivity using heat map. According to Fig. with the smooth factor as 0.99, it is evident that our method is robust and can achieve a result close to the optimal solution under different combinations. Therefore, our proposed model exhibits stability and ease of convergence.
17


Reviews:
[{"review_id": "5vIaMUMWRG", "reviewer": "Reviewer_sTG2", "summary": "This paper revisits the two-stage learning paradigm in long-tailed recognition and proposes two metrics, Logits Magnitude, and Regularized Standard Deviation, to compare different classifier re-training methods. Moreover, it develops a label over-smoothing approach to improve model performance. Experimental results on multiple long-tailed datasets demonstrate the effectiveness of the proposed method.", "questions": "See \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper reveals some shortages in previous metrics, such as the ignoration of feature magnitude.\n\n- The proposed label over-smoothing is interesting. Both theoretical and empirical results demonstrate the effectiveness.\n\n- The proposed LOS outperforms most compared methods on multiple datasets. The ablation studies are well done.", "weaknesses": "- Figure 1 is hard to follow. What do the red and blue lines mean? Why do they obey Gaussian distributions? More explanations are required.\n\n- Since the feature magnitude and the class weight norm both have effects on prediction results, why not directly use the Cosine classifier? The Cosine classifier is quite simple and effective, which calculates Cosine similarities between the normalized features and the normalized class weights, and then divides it by a temperature to obtain the final logits. Therefore, it can directly mitigate the impact of feature magnitude. (Well, maybe the logit magnitude is different from the feature magnitude, but I suggest the authors include the Cosine classifier as a baseline.)\n\n- Despite the superior performance. Some recent works are ignored and not compared. For example, RIDE, BCL, NCL, GML.\n\n- The references are outdated. Among the 50 cited papers, there are no papers from 2024, only 3 papers from 2023, and 4 papers from 2022.\n\n- Please pay attention to the format of the references. Some do not include the source (e.g. Peifeng et al., 2023); some have a URL (Huang et al., 2018) but others do not; some source names are inconsistent (e.g. \"Computer Vision-ECCV ...\" or \"Proceedings of ... (ECCV)\"? and \"arXiv preprint\" or \"ArXiv, abs/...\" or \"arXiv e-prints\"?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CuLyWsivHn", "reviewer": "Reviewer_g4ig", "summary": "This paper revisits the paradigm of classifier re-training in the context of long-tailed recognition and proposes two metrics for analyzing the efficacy of classifiers: logits magnitude (LoMa) and regularized standard deviation. Various existing methods for LTR are re-evaluated with a unified feature representation and with respect to the proposed metrics. They further use a variant of label smoothing where very strong smoothing is used to achieve better performance. Results and ablation studies are reported on multiple benchmark LTR datasets.", "questions": "Please address the points in weaknesses above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The performance of various methods seems to somewhat correlate to the proposed metrics.\n2. The proposed method outperforms prior arts on multiple benchmarks in most of the evaluated settings.\n3. The over-smoothing technique can be used as a plugin on top of existing methods.", "weaknesses": "1. The proposed LoMa metric is hard to understand due to ambiguous language and notation. The authors should use two indices in the subscript of the logits to differentiate between sample and class. In Fig. 2 it is unclear what true samples and non-true samples are. \n2. While the authors claim it is hard to optimize directly for LoMa or Regularized Standard Deviation, they don't propose any way to do this. The suggested label smoothing approach doesn't directly fit into the story of achieving uniform LoMa.\n3. There are wild claims made, for instance, that classifier weight norm cannot capture a method's efficacy due to potential arbitrary lengths (Proposition 2). It is relative weight norm that matters, not absolute, thus rendering Prop. 2 meaningless. Section 3.1 on the deep dive into LoMa seems like a rambling section without concrete takeaways. \n4. The math in the paper could be improved. For instance, in Eq. 9, the expectation can't be decomposed like that, it looks artificial. \n5. The proposed (over) label smoothing technique is just a minor variant of prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oNVhqh5hbT", "reviewer": "Reviewer_aP5D", "summary": "This paper proposes a simple but effective method for long-tailed problem. They base their method in decoupled ones and first propose two new metrics, Logits Magnitude and Regularized Standard Deviation, to compare the differences and similarities between various methods. Then, they propose a simple method by softening the original one-hot labels by assigning a probability slightly higher than $1/K$ to the true class and slightly lower than $1/K$ to the other classes, where $K$ is the number of classes. In the method, they also provide the detailed analysis how the method is designed. In the experiments, the results validate the effectiveness of proposed method.", "questions": "I tried to understand the logic and the details of this paper, so some of the points in the paper require further clarification:\n\n(1) In Proposition 1. It is said \" In all cross-entropy based methods, the aforementioned conclusion remains valid\", in deep learning networks, the objective is always a non-convex optimization problem, how the conclusion is still valid in this case?\n\n(2) The Proposition 1 is used to exclude the effect of b and to introduce Proposition 2 which illustrates that the weight norm-based method may not be reliable in some cases, am I right? But how the equation (4) is obtained? from s' to s.\n\n(3) From line 313-314, how the assumption is determined\" Consider random perturbations ∆i that independently affect each class zi and are proportional to the standard deviation of their logits, i.e., z′\"?? It is said \"These perturbations ∆ are commonly introduced during the training process, particularly due to factors such as overfitting in the initial stage of training and bias in the sampling of the training set.\", can you clarify more on this point?\n\n(4) if the point lies in $\\Delta$ and $\\sigma$ in section 3.1, what is the point to introduce definition 2?? The point in definition 2 shoule be $R_i$, right?\n\n(5) Figure 1 is also not easy to understand for the first time until figure 2 is given.\n\nTypo: (1) Is the parenthesis missed in line 107?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The structure of this paper is clear, starting from the definition of problem, the background motivation and analysis to the final implementation.\n\n(2) The experiments are good. By applying the proposed method to some existing methods, their performances can be further improved.", "weaknesses": "Although the structure is clear, the transition from last part to the next can be further improved. \n\nSome of the explanations are also missed. See below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "laQg5bEX2o", "reviewer": "Reviewer_o1sQ", "summary": "This paper revisits the classifier re-training in long-tailed recognition, proposing a simple label over-smoothing approach to balance the logits magnitude across classes. The method achieves state-of-the-art performance on various imbalanced datasets.", "questions": "See the weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is simple, using basic label smoothing to effectively reduce overfitting.\n1. Theoretical analysis and experimental validation are comprehensive, demonstrating the method's effectiveness across various datasets.", "weaknesses": "There are no significant methodological drawbacks. It is recommended that the authors include comparisons with the latest SOTA methods^[1] and discuss related works^[2,3].\n\n[1] Probabilistic Contrastive Learning for Long-Tailed Visual Recognition. TPAMI 2024.\n\n[2] Adaptive Logit Adjustment Loss for Long-Tailed Visual Recognition. AAAI 2022.\n\n[3] Long-Tailed Visual Recognition via Gaussian Clouded Logit Adjustment. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)