Rethinking Classifier Re-Training in Long-Tailed Recognition: Label Over-Smooth Can Balance

Authors: Siyu Sun, Han Lu, Jiangtong Li, Yichen Xie, Tianjiao Li, Xiaokang Yang, Liqing Zhang, Junchi Yan
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Long-Tailed Recognition and Decoupled Training

Abstract:
In the field of long-tailed recognition, the Decoupled Training paradigm has shown exceptional promise by dividing training into two stages: representation learning and classifier re-training.  While previous work has tried to improve both stages simultaneously, this complicates isolating the effect of classifier re-training.  Recent studies reveal that simple regularization can produce strong feature representations, highlighting the need to reassess classifier re-training methods.  In this study, we revisit classifier re-training methods based on a unified feature representation and re-evaluate their performances. 
We propose two new metrics, Logits Magnitude and Regularized Standard Deviation, to compare the differences and similarities between various methods. 
Using these two newly proposed metrics, we demonstrate that when the Logits Magnitude across classes is nearly balanced, further reducing its overall value can effectively decrease errors and disturbances during training, leading to better model performance. 
Based on our analysis using these metrics, we observe that adjusting the logits could improve model performance, leading us to develop a simple label over-smoothing approach to adjust the logits without requiring prior knowledge of class distribution.
This method softens the original one-hot labels by assigning a probability slightly higher than $\frac{1}{K}$ to the true class and slightly lower than $\frac{1}{K}$ to the other classes, where $K$ is the number of classes.
Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist2018.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730553930596

thread_type:
official_review

<thread_object>
creator: reviewer_o1sq

<content>
summary:
This paper revisits the classifier re-training in long-tailed recognition, proposing a simple label over-smoothing approach to balance the logits magnitude across classes. The method achieves state-of-the-art performance on various imbalanced datasets.

strengths:
1. The method is simple, using basic label smoothing to effectively reduce overfitting.
1. Theoretical analysis and experimental validation are comprehensive, demonstrating the method's effectiveness across various datasets.

weaknesses:
There are no significant methodological drawbacks. It is recommended that the authors include comparisons with the latest SOTA methods^[1] and discuss related works^[2,3].

[1] Probabilistic Contrastive Learning for Long-Tailed Visual Recognition. TPAMI 2024.

[2] Adaptive Logit Adjustment Loss for Long-Tailed Visual Recognition. AAAI 2022.

[3] Long-Tailed Visual Recognition via Gaussian Clouded Logit Adjustment. CVPR 2022.

questions:
See the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626095429

thread_type:
official_review

<thread_object>
creator: reviewer_ap5d

<content>
summary:
This paper proposes a simple but effective method for long-tailed problem. They base their method in decoupled ones and first propose two new metrics, Logits Magnitude and Regularized Standard Deviation, to compare the differences and similarities between various methods. Then, they propose a simple method by softening the original one-hot labels by assigning a probability slightly higher than $1/K$ to the true class and slightly lower than $1/K$ to the other classes, where $K$ is the number of classes. In the method, they also provide the detailed analysis how the method is designed. In the experiments, the results validate the effectiveness of proposed method.

strengths:
(1) The structure of this paper is clear, starting from the definition of problem, the background motivation and analysis to the final implementation.

(2) The experiments are good. By applying the proposed method to some existing methods, their performances can be further improved.

weaknesses:
Although the structure is clear, the transition from last part to the next can be further improved. 

Some of the explanations are also missed. See below.

questions:
I tried to understand the logic and the details of this paper, so some of the points in the paper require further clarification:

(1) In Proposition 1. It is said " In all cross-entropy based methods, the aforementioned conclusion remains valid", in deep learning networks, the objective is always a non-convex optimization problem, how the conclusion is still valid in this case?

(2) The Proposition 1 is used to exclude the effect of b and to introduce Proposition 2 which illustrates that the weight norm-based method may not be reliable in some cases, am I right? But how the equation (4) is obtained? from s' to s.

(3) From line 313-314, how the assumption is determined" Consider random perturbations ∆i that independently affect each class zi and are proportional to the standard deviation of their logits, i.e., z′"?? It is said "These perturbations ∆ are commonly introduced during the training process, particularly due to factors such as overfitting in the initial stage of training and bias in the sampling of the training set.", can you clarify more on this point?

(4) if the point lies in $\Delta$ and $\sigma$ in section 3.1, what is the point to introduce definition 2?? The point in definition 2 shoule be $R_i$, right?

(5) Figure 1 is also not easy to understand for the first time until figure 2 is given.

Typo: (1) Is the parenthesis missed in line 107?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679604290

thread_type:
official_review

<thread_object>
creator: reviewer_g4ig

<content>
summary:
This paper revisits the paradigm of classifier re-training in the context of long-tailed recognition and proposes two metrics for analyzing the efficacy of classifiers: logits magnitude (LoMa) and regularized standard deviation. Various existing methods for LTR are re-evaluated with a unified feature representation and with respect to the proposed metrics. They further use a variant of label smoothing where very strong smoothing is used to achieve better performance. Results and ablation studies are reported on multiple benchmark LTR datasets.

strengths:
1. The performance of various methods seems to somewhat correlate to the proposed metrics.
2. The proposed method outperforms prior arts on multiple benchmarks in most of the evaluated settings.
3. The over-smoothing technique can be used as a plugin on top of existing methods.

weaknesses:
1. The proposed LoMa metric is hard to understand due to ambiguous language and notation. The authors should use two indices in the subscript of the logits to differentiate between sample and class. In Fig. 2 it is unclear what true samples and non-true samples are. 
2. While the authors claim it is hard to optimize directly for LoMa or Regularized Standard Deviation, they don't propose any way to do this. The suggested label smoothing approach doesn't directly fit into the story of achieving uniform LoMa.
3. There are wild claims made, for instance, that classifier weight norm cannot capture a method's efficacy due to potential arbitrary lengths (Proposition 2). It is relative weight norm that matters, not absolute, thus rendering Prop. 2 meaningless. Section 3.1 on the deep dive into LoMa seems like a rambling section without concrete takeaways. 
4. The math in the paper could be improved. For instance, in Eq. 9, the expectation can't be decomposed like that, it looks artificial. 
5. The proposed (over) label smoothing technique is just a minor variant of prior work.

questions:
Please address the points in weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730785414021

thread_type:
official_review

<thread_object>
creator: reviewer_stg2

<content>
summary:
This paper revisits the two-stage learning paradigm in long-tailed recognition and proposes two metrics, Logits Magnitude, and Regularized Standard Deviation, to compare different classifier re-training methods. Moreover, it develops a label over-smoothing approach to improve model performance. Experimental results on multiple long-tailed datasets demonstrate the effectiveness of the proposed method.

strengths:
- This paper reveals some shortages in previous metrics, such as the ignoration of feature magnitude.

- The proposed label over-smoothing is interesting. Both theoretical and empirical results demonstrate the effectiveness.

- The proposed LOS outperforms most compared methods on multiple datasets. The ablation studies are well done.

weaknesses:
- Figure 1 is hard to follow. What do the red and blue lines mean? Why do they obey Gaussian distributions? More explanations are required.

- Since the feature magnitude and the class weight norm both have effects on prediction results, why not directly use the Cosine classifier? The Cosine classifier is quite simple and effective, which calculates Cosine similarities between the normalized features and the normalized class weights, and then divides it by a temperature to obtain the final logits. Therefore, it can directly mitigate the impact of feature magnitude. (Well, maybe the logit magnitude is different from the feature magnitude, but I suggest the authors include the Cosine classifier as a baseline.)

- Despite the superior performance. Some recent works are ignored and not compared. For example, RIDE, BCL, NCL, GML.

- The references are outdated. Among the 50 cited papers, there are no papers from 2024, only 3 papers from 2023, and 4 papers from 2022.

- Please pay attention to the format of the references. Some do not include the source (e.g. Peifeng et al., 2023); some have a URL (Huang et al., 2018) but others do not; some source names are inconsistent (e.g. "Computer Vision-ECCV ..." or "Proceedings of ... (ECCV)"? and "arXiv preprint" or "ArXiv, abs/..." or "arXiv e-prints"?)

questions:
See "Weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732190655856

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We express our gratitude to all reviewers for their valuable time and insightful comments. ***We appreciate the unanimous recognition of our method's effectiveness across multiple benchmarks (sTG2, g4ig, aP5D, o1sQ), the acknowledgment of our comprehensive theoretical and experimental validation (sTG2, aP5D, o1sQ), and the appreciation of our simple yet practical approach (sTG2, g4ig, o1sQ).***

The reviewers have raised several questions and constructive suggestions, which we will ***address point by point in our rebuttal***. Meanwhile, based on the reviewers' feedback, we have made  revisions to the paper. The changes include enhanced readability, expanded discussions incorporating recent literature, and additional experimental results. ***We have uploaded the revised PDF version with all modifications highlighted in blue for your reference.***
</content>
</thread_object>

</thread>
<thread>
creation date:
1732370187761

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Inquiry for post-rebuttal comments

comment:
Dear Reviewers:

Thank you again for your wisdom and valuable comments. We have provided complete explanations for all the questions. Since the discussion period has been underway for some time, we would be glad to hear from you whether our rebuttal has addressed your concerns. Feel free to comment on our rebuttal if you have further questions and considerations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733245387215

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Author-Reviewer Discussion Phase

comment:
Dear PCs, SACs, and ACs,

We sincerely thank the reviewers for their time and valuable feedback. As the discussion phase concludes, we summarize the reviewers' comments and outline the improvements made during this period. The reviewers generally regarded our method as "Effective," "Simple," "Well-experimented", featuring "Comprehensive Theoretical Analysis," "Plugin-Compatible," offering "Enhanced Evaluation Metrics," having a "Clear Structure," conducting "Comprehensive Ablation Studies," and providing "Extensive Analysis Across Various Datasets."

We summarize the reviewers' concerns and our corresponding responses as follows:

| | Reviewers' Concerns | Author's Responses |
| - | - | -|
| Common Concern | The explanation for Fig.1 is not clear. | We have provided a more detailed explanation of the illustrative figures and revised them in the updated version. These enhanced descriptions facilitate a better understanding of our subsequent analysis and ensure that the figures effectively convey the intended concepts.|
|| Why not compare with/use specific methods?| In response to all methods suggested by reviewers for comparison and integration, we have conducted additional experiments and included the corresponding results in the revised manuscript. By incorporating these relevant methods into our study, we can enhance the completeness and robustness of our work. This ensures that our comparisons are comprehensive and that our proposed method is evaluated against a broader range of existing techniques.|
|| Some of the points in the paper require further clarification.| We have provided more detailed explanations for the points that were unclear and have revised the manuscript based on the reviewers' suggestions to improve overall clarity. This includes elaborating on complex concepts, refining our arguments, and ensuring that all sections are well-articulated to facilitate better understanding by the readers.  |
| Reviewer sTG2  | The references are outdated and the format of the references needs improvement. | We have promptly supplemented the revised manuscript with a substantial amount of the latest research and analyzed that these papers focus on feature learning based on more complex paradigms, which is orthogonal to our emphasis on classifier learning methods. Therefore, we have not only included analyses of these studies but also employed our method as a plugin to adjust the classifier based on these method backbones, achieving significant performance improvements. Additionally, in the revised version, we standardized the formatting of each cited article to more clearly and uniformly display the publication conferences/journals and their dates. |
| Reviewer g4ig  | The suggested label smoothing approach doesn't directly fit into the story of achieving uniform LoMa. | We clarified our overall motivation to demonstrate how to derive the conditions that an effective method should have through our analytical process. To meet these conditions, our approach can be naturally thought of.  |
| | The math in the paper could be improved.| We believe there may have been a misunderstanding of the equation we performed. To address this, we have added more detailed explanations and comprehensive proofs in the revised manuscript. These additions are included in the appendix to provide clarity and ensure that the mathematical formulations and derivations are transparent and easily understandable. This enhancement aims to eliminate any confusion and strengthen the theoretical foundation of our work.|
|  | The proposed method is just a minor variant of prior work. | Although our proposed method is a variant of prior work, its application is entirely different. We not only introduce the concept of over-label smoothing but also provide a theoretical analysis of its impact on the model learning process. Label smoothing has not established a dominant position in the long-tailed (LT) field, and no one has attempted over-smoothing in other domains. The difference between values such as 0.99 and 0.2 goes far beyond mere formal consistency. Our method is simple yet effective and can be extremely beneficial in the LT domain, constituting a novel contribution. |
| Reviewer aP5D  | How the equations and claims is obtained? | We reiterate our initial definitions and overall reasoning to enhance understanding. By providing additional details and clarifying the processes, reviewers can better comprehend how we derived these equations and formulated our claims. |
| Reviewer o1sQ |  Recommend to include comparisons with the latest SOTA methods  and discuss related works. | We performed additional experiments and reported the results. The new experiments are included in our revised manuscript. |
</content>
</thread_object>

</thread>
<thread>
creation date:
1741025731156

thread_type:
public_comment

<thread_object>
creator: ~yi-hang_zhu1

<content>
title:
About reproducing the results

comment:
(1) In the paper, appendix D, you mention that "The classifier finetune epoch is 20 with adequate learning ratio and weight." Would you please elaborate on the experimental setup for imagenet-LT and iNaturalist2018? 

(2) If I understand correctly, to implement the label smoothing, I simply need to use 
torch.nn.CrossEntropyLoss(label_smoothing=args.label_smooth)
during the classifier retraining, right? 

(3) The label smoothing does not use the information of the number of training samples per class. Could you explain why it has the reweighting/rebalancing effect? 

Thanks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes two new metrics, Logits Magnitude and Regularized Standard Deviation, to compare the differences and similarities between various methods for classifier re-training based long-tailed recognition. Overall, the reviewers acknowledged the contribution of this paper. After rebuttal, most reviewers tend to accept this paper, yet one reviewer assumes the method as a patchwork with some takeaways. AC carefully checked the feedback and discussions and assumes the pros outweigh the cons for this work, thus AC recommends accepting this paper for publication in ICLR 2025. However, the authors should revise their final manuscript according to the valuable comments from all the reviewers.