Title: Long-tailed Recognition with Model Rebalancing
Venue: NeurIPS 2025
Accepted: yes
Keywords: machine learning

imbalanced learning
Abstract: Long-tailed recognition is ubiquitous and challenging in deep learning and even in the downstream finetuning of foundation models, since the skew class distribution generally prevents the model generalization to the tail classes. Despite the promise of previous methods from the perspectives of data augmentation, loss rebalancing and decoupled training etc., consistent improvement in the broad scenarios like multi-label long-tailed recognition is difficult. In this study, we dive into the essential model capacity impact under long-tailed context, and propose a novel framework, Model Rebalancing (MORE), which mitigates imbalance by directly rebalancing the model's parameter space. Specifically, MORE introduces a low-rank parameter component to mediate the parameter space allocation guided by a tailored loss and sinusoidal reweighting schedule, but without increasing the overall model complexity or inference costs. Extensive experiments on diverse long-tailed benchmarks, spanning multi-class and multi-label tasks, demonstrate that MORE significantly improves generalization, particularly for tail classes, and effectively complements existing imbalance mitigation methods. These results highlight MORE's potential as a robust plug-and-play module in long-tailed settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MORE (MOdel REbalancing), a novel method for tackling long-tailed visual recognition by rebalancing the model’s parameter space. MORE decomposes the weights of each model layer into general and minority-specific components using a low-rank parameterization, guided by a tailored discrepancy-based loss and sinusoidal reweighting schedule. This design allows the model to allocate dedicated capacity to underrepresented classes without increasing inference complexity. Theoretical analysis demonstrates that this decomposition tightens the generalization bounds. Extensive experiments across multi-class and multi-label benchmarks validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper is generally well-written and easy to follow.\n- The proposed method is novel, addressing the long-tailed learning challenge from a new perspective by rebalancing the model's parameter space.\n- The authors provide a theoretical analysis demonstrating that the proposed decomposition leads to a tighter generalization bound, which enhances the soundness and credibility of the approach.\n- The method is easy to implement and can be seamlessly integrated with other long-tailed learning techniques.\n\n\n\n**Weaknesses:**\n\n- For the multi-class image recognition experiments, the authors primarily evaluate on CIFAR100-LT and Places-LT. However, ImageNet-LT and iNaturalist are widely used large-scale long-tailed benchmarks. Including results on these datasets would better validate the generalizability and effectiveness of the proposed method.\n\n- The paper should be better to clarify which modules of ResNet and CLIP are decomposed using LoRA, respectively. Additionally, it would be helpful to specify the chosen rank $r$ for LoRA and compare the number of learnable parameters with and without LoRA during training.\n- It would be valuable to report the values of $M(x; \\theta)$ across samples from different classes in the final trained model. Furthermore, presenting the difference $f_y(x; \\theta^g \\oplus \\theta^t) - f_y(x; \\theta^g)$ across class samples would help support the effectiveness of the proposed loss $L_{\\text{MORE}}$, as increasing the ground-truth logit is key to improving model performance.\n- There are also some typos, such as in line 40, where \"Rachemacher complexity\" should be \"Rademacher complexity\".\n\nQUESTIONS:\nPlease see the \"Strengths And Weaknesses\" part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes MORE, a method designed to rebalance the model parameter space in imbalanced learning scenarios. The authors introduce a tailored loss function that incorporates logit-level contrastive supervision for space reallocation, along with a dynamic weighting scheme for model rebalancing. Experiments and ablation studies across different datasets are conducted to demonstrate the effectiveness of the proposed approach.

STRENGTHS AND WEAKNESSES:
Pros

- The paper is generally well written, and the motivation is clear and well justified. The authors build on meaningful observations to guide their design.

- The proposed method is simple yet effective. Ablation studies are thorough, and the approach shows solid performance across various long-tailed learning scenarios.


Cons


- The baselines are relatively weak. The paper lacks comprehensive comparisons against recent state-of-the-art methods in long-tailed single-label classification. Including stronger baselines would help solidify the claimed improvements.

QUESTIONS:
- The connection between the theoretical analysis in Section 3.3 and the methodology in Section 3.4 is unclear. This relationship should be clarified in the rebuttal. It would also be helpful if Section 3.4 included a brief remark explicitly referencing how the theory informs or motivates the method.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ***MORE*** (MOdel REbalancing). The core idea is to rebalance the model's parameter space through a low-rank parameter decomposition and incorporates a difference-based loss function and a sinusoidal re-weighting schedule to achieve this. Experimental results show that ***MORE*** significantly improves the model's generalization on tail classes across several multi-class and multi-label long-tailed benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The design of the $L_{more}$ loss is particularly insightful. By penalizing the activation of the tail-specific parameters ($θ^t$) on head-class samples, the framework effectively incentivizes these parameters to specialize in learning features pertinent to the tail classes. This is a clever and direct mechanism to encourage parameter specialization.\n\n2. The sinusoidal re-weighting schedule is a simple yet elegant mechanism. It effectively creates a dynamic curriculum, allowing the model to alternate its focus between learning generalizable features and specialized features (for tail classes) at different training stages. This simple periodic function achieves an effect similar to more complex, hand-crafted curriculum learning strategies.\n\n3. The use of low-rank parameterization for the tail-specific parameters is well-justified. This design choice significantly reduces the number of parameters dedicated to the tail classes, which inherently acts as a strong regularizer.\n\nWeaknesses:\n\n1. While the paper correctly claims no increase in inference cost, the impact on training cost is not addressed. The formulation, particularly as implied by Equation (2), suggests that the training process requires two forward passes per iteration. The lack of empirical data (GPU hours) on this training overhead is a significant omission. This information is crucial for assessing the practical feasibility of the method, especially for training on large-scale datasets.\n\n2. I have some reservations about the sensitivity of the method to its key hyperparameters (the rank r and the balancing coefficient α). A critical trade-off exists for the rank r: if it is too small, the representational capacity of the tail-specific parameters $θ^t$ may be insufficient; conversely, if it is too large, the benefits of the low-rank assumption are diminished. It is unclear how these hyperparameters should be set for new datasets or different backbone architectures without extensive tuning. Furthermore, the assumption that a single shared low-rank space $θ^t$ can effectively serve all tail classes might be a limitation. If certain tail classes are semantically distinct and require features that are largely \"orthogonal\" to those of other tail classes, this shared space may fail to adequately capture their unique characteristics.\n\n3. The paper could further explore the relationship between the general ($θ^g$) and tail-specific ($θ^t$) parameters. While the loss function applies them to different objectives, they are still optimized jointly. Have the authors considered enforcing a stricter decoupling, for instance, through an orthogonality constraint between $θ^g$ and $θ^t$? This could prevent representational overlap and lead to a more modular model.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MOdel REbalancing (MORE), an approach to long-tailed recognition that rebalances the model’s parameter space instead of data or loss. By decomposing model weights into general and low-rank components, MORE allocates capacity specifically for tail classes. A discrepancy-based loss and sinusoidal reweighting guide this process during training. The method improves generalization for underrepresented classes without increasing inference cost, and achieves strong results across both single- and multi-label long-tailed benchmarks, including CLIP finetuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe method does not introduce additional inference cost, making it appealing for real-world deployment. Besides, MORE is plug-and-play and can be combined with existing reweighting or contrastive learning methods.\n\n**Weaknesses**\n\n1. Lack of conceptual differentiation from LoRA. Although the authors claim their low-rank decomposition is “fundamentally different” from LoRA, the distinction is not clearly articulated in terms of optimization behavior or empirical ablation. Figure 3 alone is **insufficient** to convincingly demonstrate that LoRA is ineffective in the long-tail setting. Without a sharper contrast, the contribution risks being viewed as an application of existing adaptation techniques to long-tailed learning.\n\n 2. Evaluation breadth. Although the experiments cover a diverse range of tasks, they are limited to relatively small- or medium-scale datasets. Common benchmarks in long-tail learning—such as ImageNet-LT and iNaturalist—are notably absent. This limits the ability to draw conclusions about the method’s scalability and robustness in more realistic or challenging scenarios.\n\n3. The paper does not explore strong baselines that manipulate model capacity, such as mixture-of-experts methods [1, 2] or self-supervised learning approaches [3], which offer alternative solutions to tail-class underfitting. Moreover, using extra parameters is a common and effective strategy—parameter-efficient finetuning approaches like [4] have also shown strong performance in such scenarios. Including comparisons or ablations against these baselines would help clarify whether the performance gains come from the specific design of MORE or from the general benefit of added parameter capacity.\n\n4. Modest overall gains in some settings. On certain benchmarks (e.g., NUS-WIDE, CIFAR100-LT and Places-LT), the overall gains are relatively small , raising questions about trade-offs and consistency across datasets.\n\n[1] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts\n\n[2] Balanced Product of Calibrated Experts for Long-Tailed Recognition\n\n[3] Generalized Parametric Contrastive Learning\n\n[4] Parameter-Efficient Long-Tailed Recognition\n\nQUESTIONS:\n1. Can the authors clarify how their method is conceptually and empirically different from existing low-rank adaptation techniques? What evidence shows that this difference is critical in the long-tailed setting?\n\n2. Can the authors evaluate the method on larger and more standard long-tailed benchmarks to better demonstrate its scalability and generalization?\n\n3. Can the authors include comparisons or ablations against alternative capacity-based baselines to isolate the source of performance improvements?\n\n4. How consistent are the improvements across different datasets, and can the authors explain the relatively modest gains observed in some settings?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}