Title: LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups
Venue: NeurIPS 2025
Accepted: yes
Keywords: imbalance classification

foundational model

fine-tuning

model merging
Abstract: Real-world datasets typically exhibit long-tailed (LT) distributions, where a few head classes dominate and many tail classes are severely underrepresented. While recent work shows that parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer preserve tail-class performance on foundation models such as CLIP, we find that they do so at the cost of head-class accuracy. We identify the head-tail ratio, the proportion of head to tail classes, as a crucial but overlooked factor influencing this trade-off. Through controlled experiments on CIFAR100 with varying imbalance ratio ($\rho$) and head-tail ratio ($\eta$), we show that PEFT excels in tail-heavy scenarios but degrades in more balanced and head-heavy distributions. To overcome these limitations, we propose LT-Soups, a two-stage model soups framework designed to generalize across diverse LT regimes. In the first stage, LT-Soups averages models fine-tuned on balanced subsets to reduce head-class bias; in the second, it fine-tunes only the classifier on the full dataset to restore head-class accuracy. Experiments across six benchmark datasets show that LT-Soups achieves superior trade-offs compared to both PEFT and traditional model soups across a wide range of imbalance regimes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces LT-Soups, a two-stage model soups framework designed to address long-tailed (LT) class imbalance in machine learning. It focuses on the problem where head classes dominate, and tail classes are severely underrepresented, a scenario common in real-world datasets. Previous work on parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer show improved tail-class performance but often at the cost of head-class accuracy. The authors propose LT-Soups, which averts this trade-off by combining model soups and tailored fine-tuning techniques. The framework works in two stages:\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nOriginality: The introduction of LT-Soups as a two-stage framework that balances performance across both tail and head classes is novel. It builds on the understanding that class imbalance is a complex, multi-dimensional problem and uses a dual-axis framework for imbalance characterization.\nRobustness: The method shows significant improvement across both synthetic and real-world long-tailed datasets, including scenarios with varying imbalance ratios and head-tail class distributions.\nPractical Application: The proposed method shows better adaptability and performance stability across diverse datasets, making it applicable to real-world tasks, especially those with imbalanced data distributions.\n\nWeaknesses:\nComplexity: While the approach is robust, it involves two stages of fine-tuning, which could increase computational costs and complexity compared to simpler methods like PEFT or traditional model soups.\nLimited Statistical Analysis: The paper lacks error bars or statistical significance tests in the experimental results, which could make it difficult to assess the variability and robustness of the reported results fully.\nAssumptions: The reliance on the head-tail ratio and imbalance ratio, while insightful, might oversimplify the complexity of real-world class distributions. A more flexible or generalized model could be explored to account for other potential distributional shifts.\n\nQUESTIONS:\nReal-World Generalization: How does LT-Soups perform when tested on highly skewed datasets or under conditions with noisy labels or missing data? Would the approach still provide strong results?\n\nStatistical Robustness: The paper lacks statistical significance measures such as error bars. Could you provide additional experiments or statistical tests to support the stability of your findings across multiple runs or data splits?\n\nAlternative Subsampling Strategies: Have you considered experimenting with more advanced or adaptive subsampling techniques, such as those based on active learning or dynamic class balancing during training? Could they offer better performance than the current progressive subsampling strategy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey authors propose LT-Soups, a two-stage procedure, where:\n\n1. Progressive subsampling + model-averaging. Multiple CLIP copies are fully fine-tuned (in parallel) on subsets whose imbalance ratios grow geometrically, then exponentially averaged to couple \"tail-friendly\" and \"head-friendly\" specialists\n\n2. Classifier re-tuning. Only the final linear head is re-trained on the full data with logit adjustment to regain head-class sharpness\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- it's a very simple approach\n- it's efficient since it can be parallelized\n- consistently outperforms PEFT and classical model-soup baselines on overall, head and tail accuracies\n- i really admire that the authors released code in the supplementary\n\n\nWeakness:\n- although the method works well, it's currently hard for me to pinpoint exactly where are the gains coming from -- look at questions section, some ablations will help disentangle the multiple components of the method\n\nQUESTIONS:\n1. i'd like to see some small ablations if the authors can provide these :)\n- Performing PEFT, followed by performing the same final layer only tuning with logit adjustment that is performed in LT-Soups -- would this help in closing gap in head accuracy for PEFT ?\n- Performing Model Soups as done in Wortsman et al. [2022], followed by performing the same final layer only tuning\n\nAbove two ablations would tell if its the new model soup strategy which is driving the gains or the final logit adjustment.\n\n\n2. perhaps one more thing to test would be:\n- take a subsampled dataset that has the least imbalance, perform vanilla model soups (Wortsman et al. [2022]) on it or PEFT, followed by final layer only tuning -- this ablation would tell us that does one need multiple models at different levels of imbalance in the dataset, or does the least imbalanced dataset is enough\n\nlet me know if these make sense.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the trade-off between head- and tail-class performance when fine-tuning large pre-trained models on long-tailed (LT) datasets. The authors begin by introducing a new metric, the head-tail ratio (η), which they use alongside the standard imbalance ratio (ρ) to more comprehensively characterize the structure of long-tailed distributions. Through experiments, they show that Parameter-Efficient Fine-Tuning (PEFT) methods excel in tail-heavy scenarios, while full fine-tuning is more advantageous when head classes are dominant.\n\nTo resolve this issue, the authors propose LT-Soups, a two-stage model merging framework. In the first stage, instead of training models on the entire imbalanced dataset, LT-Soups creates a series of data subsets with progressively increasing imbalance ratios. It fine-tunes a full model on each subset and then merges the weights of these \"specialist\" models via recursive weighted averaging to form a unified feature extractor. In the second stage, to recover head-class information potentially lost during subsampling, the framework freezes the merged backbone and retrains only the classifier layer on the full training set. Experimental results across several synthetic and real-world long-tailed datasets demonstrate that LT-Soups achieves a superior performance balance compared to PEFT and traditional model soups across diverse long-tailed regimes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper's analysis of the head-tail performance trade-off is insightful. The introduction of the head-tail ratio (η) as a new analytical dimension provides a more fine-grained framework for understanding the complexities of long-tailed recognition.\n\n2. The paper's experimental section is robust. The authors validate their method on a wide range of standard benchmarks and use well-designed ablation studies to demonstrate the contribution of each component. The visualizations effectively support the core claim that different methods are optimal for different imbalance structures.\n\n**Weaknesses**\n1. While LT-Soups is an effective framework, its core ideas are more of an intelligent combination of existing techniques—Model Soups, data subsampling, and two-stage training—rather than a fundamentally new concept. The innovation lies in the specific recipe, which, while successful, may not meet the high bar for conceptual novelty at a top-tier venue.\n\n2. The framework requires training N×M models in the first stage. Although parallelizable and trained on smaller subsets, the total computational budget is significantly higher than for single-run methods like PEFT or Full-FT. A direct comparison of total GPU hours is missing, making it difficult to assess the practical cost-benefit trade-off.\n\n3. The recursive, ordered merging strategy is shown empirically to be superior to alternatives. However, the paper lacks a deep, intuitive explanation for why this curriculum-like fusion (from balanced to imbalanced) is the optimal approach. It remains unclear if this is a robust principle or an empirical finding specific to this setup.\n\n4. Lack of discussion on recent literature. For ensemble long-tail recognition: Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition, Mdcs: More diverse experts with consistency self-distillation for long-tailed recognition. For recent literature: LTRL: Boosting Long-tail Recognition via Reflective Learning, DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge, Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.

STRENGTHS AND WEAKNESSES:
As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.

QUESTIONS:
As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.

LIMITATIONS:
As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors conduct experiments to show that previous methods for the long-tail problem cannot achieve better performance both on head and tail classes. Therefore, the authors propose to adapt the model soup for the problem by training different models on sub-datasets with different class imbalance ratios and averaging their weights to obtain better representation.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The authors identify the problem of head-tail imbalance ratio which is neglected by previous works.\n2. The motivation of the paper is stated clearly by conducting preliminary experiments.\n\nWeakness:\n1. The writing is not proper enough. For example, throughout the paper, how the Logit Adjustment is performed is not mathematically formulated. Although it has been proposed in previous works, rephasing it in the current paper helps others understand it better. The caption of Figure 5 is neglected as well. The momentum coefficient $\\mu$ in line 210 is not defined either. If it is the same to $\\lambda$ above, then the last weight of the averaged model is almost the same as the first model so that the proposed methods is meaningless.\n2. It seems that the proposed method cannot solve the targeted problem. From the experiments, the proposed methods can improve the performance of model soup on tail classes, but the performance on head classes still degrades which is the problem that the authors stated to solve.\n3. The experimental setting is not comprehensive enough. For example, one of the setting of imbalance ratio or head-tail ratio in Table 1 and 2 should be kept when the other one is changing. Moreover, the results illustrated in Figure 2 is not convincing enough because they are averaged on head-tail ratio which will not be able to show the detailed results when the head-tail ratio is high or low.\n4. Can you show how the performance changes when the total number of models changes?\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nPlease refer to the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-09"}