Title: Long-Tailed Recognition via Information-Preservable Two-Stage Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long-Tailed Recognition

Two-Stage Learning

Information Theory
Abstract: The imbalance (or long-tail) is the nature of many real-world data distributions, which often induces the undesirable bias of deep classification models toward frequent classes, resulting in poor performance for tail classes. In this paper, we propose a novel two-stage learning approach to mitigate such a majority-biased tendency while preserving valuable information within datasets. Specifically, the first stage proposes a new representation learning technique from the information theory perspective. This approach is theoretically equivalent to minimizing intra-class distance, yielding an effective and well-separated feature space. The second stage develops a novel sampling strategy that selects mathematically informative instances, able to rectify majority-biased decision boundaries without compromising a model’s overall performance. As a result, our approach achieves the state-of-the-art performance across various long-tailed benchmark datasets, validated via extensive experiments. Our code is available at https://github.com/fudong03/BNS_IPDPP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of long-tailed recognition by proposing a novel two-stage learning framework that preserves valuable dataset information while mitigating class imbalance. In the first stage, the authors introduce Balanced Negative Sampling (BNS), a representation learning strategy grounded in information theory. BNS is shown to maximize mutual information between augmented views, which is theoretically equivalent to minimizing intra-class distances, thus leading to well-separated feature spaces. In the second stage, the authors propose Information-Preservable Determinantal Point Process (IP-DPP), a novel undersampling technique that prioritizes mathematically informative samples to correct majority-biased decision boundaries without degrading overall model performance. Extensive experiments on four benchmark datasets validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The authors provide a theoretical analysis of the proposed method, enhancing its soundness and credibility.\n- Extensive experiments are conducted, including performance comparisons and ablation studies, to validate the effectiveness of the method, which strengthens its overall reliability.\n- The results demonstrate that the proposed method is effective, achieving substantial performance gains.\n\n\n\n**Weaknesses:**\n\n- The paper would benefit from more explanation of why the BNS method is particularly effective for learning representations under long-tailed scenarios. It would be helpful to elaborate on the differences and advantages of the proposed implementation compared to standard contrastive learning (CL) methods, as well as other CL-based long-tailed learning methods such as KCL, TSC, and SBCL.\n- More representative long-tailed learning methods, such as PaCo$^1$, BCL$^2$, DDC$^3$, GPaCo$^4$ and DirMixE$^5$, should be included in the related work for a more comprehensive comparison.\n- There are also some typos, such as in line 161, where \"presentation space\" should be \"representation space\".\n\n-----\n\n$^1$ Parametric contrastive learning, ICCV 2021 \n\n$^2$ Balanced Contrastive Learning for Long-Tailed Visual Recognition, CVPR 2022\n\n$^3$ A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning, NeurIPS 2023\n\n$^4$ Generalized Parametric Contrastive Learning, TPAMI 2023\n\n$^5$ Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition, ICML 2024\n\nQUESTIONS:\nPlease see the \"Strengths And Weaknesses\" part.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo solve the long-tailed imbalance classification problem, the authors propose a novel two-stage learning approach to mitigate such a majority-biased tendency while preserving valuable information within datasets.\n\nIn the first stage, the work aims to learn an effective and well-separated feature space for long-tailed recognition, by maximizing mutual information between instances sharing the same label—a process mathematically equivalent to minimizing intra-class distances. In the second stage, the authors propose Information-Preservable Determinantal Point Process (IP-DPP), aiming to rectify majority-biased classification decision boundaries while maintaining the model’s overall performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This work has addressed the challenging long-tailed data classification problem, by proposing a novel information-preservable two-stage learning approach. \nWeaknesses: It would be better if the authors can provide a comparison with the clip-based long-tailed methods since the vision encoder is learned in a contrastive way.\n\nIt would be better if the authors can discuss the limitations of their proposed work.\n\nQUESTIONS:\nWhat is the relationship between the first and second stages? Is there any interaction between the two stages? What parameters are trained in the second stage? What is the L-ensemble as stated in the second stage?\n\nHow it performs if the authors compare the method with the clip-based long-tailed methods? Since CLIP is pretrained based on the contrastive learning, it may provide a powerful backbone.\n\nWhat type of long-tailed methods can be combined with the proposed method except for the representation-based methods?\n\n\nAs shown in Table2, the few-shot results in iNaturalist 2018 even outperforms the Many-shot and Medium-shot, what's the reason? How to achieve a balance between these groups in this method?\n\nIn terms of Table 5, why experimental results on ImageNet-LT and iNaturalist 2018 using various model architectures seem similar?\n\nLIMITATIONS:\nThe second stage is somewhat difficult to understand. It would be better if the authors can provide an overall training workflow and the discussion in the end of method. Please see weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper addresses the challenge of class imbalance in real-world datasets by introducing a novel two-stage learning framework. The first stage employs an information-theoretic representation learning method that effectively minimizes intra-class distances to create well-separated feature spaces. The second stage introduces an innovative undersampling technique that selects informative samples to correct biased decision boundaries caused by majority classes, without sacrificing overall model performance. Extensive experiments on long-tailed benchmark datasets demonstrate that this approach achieves state-of-the-art results.

STRENGTHS AND WEAKNESSES:
Strengths :
The author's reasoning is very clear and easy to understand, with a well-defined motivation. The overall flow is smooth, and the ablation studies are comprehensive.

Weaknesses:

Tables 1 and 2 do not show results under different class balance ratios. The appendix only reports overall performance on CIFAR datasets. It would be beneficial to present more detailed results showing the trade-off between head and tail class performance at varying balance ratios.

The effect of the initial value of the temperature parameter τ on results is not investigated (e.g., how changes in τ impact accuracy).

QUESTIONS:
1.How do the limitations of traditional representation learning manifest? Why can't the feature learning stage effectively distinguish between head and tail classes? 

2.Is it possible to visualize how Balanced Negative Sampling (BNS) improves representation specifically for tail classes?

 If the authors are able to address the above issues, I would be willing to raise my score.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel two-stage learning framework to address the challenges of long-tailed data distributions in classification tasks. The authors identify two major issues in long-tailed settings: representation learning bias and classifier bias. To tackle these, they introduce:\n\nBalanced Negative Sampling (BNS): A contrastive representation learning strategy that constructs semantically rich and balanced feature spaces by carefully selecting informative negative samples. BNS captures both instance- and class-level semantics, enabling better feature separation, especially for medium- and few-shot classes.\n\nInformation-Preservable Determinantal Point Process (IP-DPP): A sampling method used during classifier training to select a diverse and representative subset of the training data. IP-DPP maintains a balance between classes while preserving informative instances and rectifying biased decision boundaries.\n\nExtensive experiments are conducted on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 datasets. The proposed method achieves state-of-the-art performance and shows robustness across different imbalance levels and model architectures. The paper also includes theoretical insights, ablation studies, computational cost analysis, and discussions on hyperparameter sensitivity. The framework’s modularity and compatibility with existing methods make it a promising solution for real-world imbalanced learning scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper proposes a novel two-stage learning framework to address long-tailed recognition, integrating Balanced Negative Sampling (BNS) for improved representation learning and Information-Preservable Determinantal Point Process (IP-DPP) for balanced classifier training. The approach is both theoretically grounded and empirically validated. The authors provide rigorous mathematical derivations, detailed ablation studies, and comprehensive evaluations on multiple standard benchmarks, including CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018. Notably, the method consistently outperforms strong baselines and can be easily combined with existing techniques, demonstrating broad applicability. The paper is well-written and clearly structured, with useful visualizations and thorough supplementary material that supports the main contributions. The use of IP-DPP for informative and class-balanced subsampling is particularly innovative, and the combination of DPP theory with long-tailed learning is a valuable contribution.\n\nWeaknesses:\n\nWhile the theoretical contributions are strong, some derivations (e.g., those involving eigenvalue distributions in DPP) rely on simplifying assumptions (e.g., uniform distribution) that may not always hold in practice, limiting their generality. Some notation in the theoretical sections is dense, and parts of the methodology (e.g., details of the BNS selection mechanism) could benefit from more intuitive explanation.\n\nOverall Assessment:\n\nThis is a high-quality, well-motivated, and thorough piece of work that offers solid improvements and insights into long-tailed recognition problems.\n\nQUESTIONS:\n1. Clarification on the Statistical Assumption in DPP Derivation\n\nIn Theorem A.5, the expected sample size of a DPP is derived under the assumption that eigenvalues follow a uniform distribution in [0, 1]. This is a strong assumption and may not reflect the actual eigenvalue distribution in practice. Could the authors provide empirical evidence (e.g., histogram of eigenvalues) to support the validity of this assumption on real datasets? Alternatively, could the authors comment on how sensitive the performance is to this assumption?\n\n2. Ablation on the Interaction Between BNS and IP-DPP\n\nThe paper presents ablations for BNS and IP-DPP independently. However, it is unclear how these two components interact. Could the authors provide an ablation showing the incremental benefit of BNS and IP-DPP when added sequentially (e.g., base → base+BNS → base+BNS+IP-DPP)? This would help isolate their individual and combined contributions more clearly.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}