LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups
Authors: Masih Aminbeidokhti
Subhankar Roy
Eric Granger
Elisa Ricci
Marco Pedersoli
Venue: NeurIPS 2025 poster ()
Keywords: imbalance classification
foundational model
fine-tuning
model merging
Primary area: deep_learning
TL;DR: LT-Soups merges CLIP models fine-tuned on balanced subsets and retrains the classifier on the full dataset, achieving SOTA head/tail accuracy trade-offs across five benchmarks.

Abstract: Real-world datasets typically exhibit long-tailed (LT) distributions, where a few head classes dominate and many tail classes are severely underrepresented. While recent work shows that parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer preserve tail-class performance on foundation models such as CLIP, we find that they do so at the cost of head-class accuracy. We identify the head-tail ratio, the proportion of head to tail classes, as a crucial but overlooked factor influencing this trade-off. Through controlled experiments on CIFAR100 with varying imbalance ratio ($\rho$) and head-tail ratio ($\eta$), we show that PEFT excels in tail-heavy scenarios but degrades in more balanced and head-heavy distributions. To overcome these limitations, we propose LT-Soups, a two-stage model soups framework designed to generalize across diverse LT regimes. In the first stage, LT-Soups averages models fine-tuned on balanced subsets to reduce head-class bias; in the second, it fine-tunes only the classifier on the full dataset to restore head-class accuracy. Experiments across six benchmark datasets show that LT-Soups achieves superior trade-offs compared to both PEFT and traditional model soups across a wide range of imbalance regimes.

LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups
Masih Aminbeidokhti Subhankar Roy²
Eric Granger!
Elisa Ricci34
Marco Pedersoli
"cool de technologie supérieure, University of Bergamo University of Trento, "Fondazione Bruno Kessler (FBK)
Abstract
Real-world datasets typically exhibit long-tailed (LT) distributions, where few head classes dominate, and many tail classes are severely underrepresented. While recent work shows that parameter-efficieet fine-tuning (PEFT) methods like LoRA and AdaptFormer preserve tail-class performance on foundation models such as CLIP, find that they do the cost of head-class accuracy. We identify the head-tail ratio, the proportion of head to tail classes, as a crucial but overlooked factor influencing this trade-off. Through controlled experiments with varying imbalance ratio (p) and head-tail ratio (7)). we show that PEFT excels in tail- tail-heavy scenarios but degrades in more balanced and head-heavy distributions. To overcome these limitations, we propose LT-Soops, a two-stage model soups framework designed generalize across diverse LT regimes. In the first stage, LT-Soups averages models fine-tuned on balanced subsets to reduce head-class bias; in the second, it fine-tunes only the classifier on the full dataset restore head-class accuracy. Experiments across six benchmark datasets show that LT-Soups achieves superior trade-offs compared to both PEFT and traditional model soups across a wide range of imbalance regimes.
CIFAR 100
Introduction
In machine learning, balanced class distributions are often assumed in both theory and prac- tice [3, 28). However, real-world datasets frequently deviate from this assumption, exhibiting severe class imbalance where a few head classes dominate and tail classes remain significantly underrepresented [56] 19 34 This imbalance poses a fundamental challenge: models must learn effectively from limited tail-class data while preserving overall robustness
Recent advances in vasion-language foundation models, particularly CLIP have introduced promising tools for addressing class imbalance. Trained on large-scale, diverse datasets, CLIP demonstrates strong robustness to distributional shifts and has become a popular backbone for
55 35). Building on this, Shi al. (50) achieve sate-of-the-ane results by applying parameter-efficiet fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) [20] and AdaptFormer (8) in combination with logit adjustment (LA) loss 40 46), which incorporates class priors by adding class class-dependent offset to the logits. While this PEFT-based approach improves overall and tail-class accuracy, they observe that it still underperforms full fine-tuning in certain regimes.
long-tailed recognition [59 58 37
"Correspondence to: masih. aniiiiaaiibeii @ens. etsmtl.ca. Code at httt:://gtthbbcoom//sssst//S//Ss
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Hoad
Tail
85.0
90.0
80.0
A-500
80.0
85.0
70.0
80.0
60.0
Accuracy
85.0
80.0
80.0
75.0
80.0
60.0
85.0
90.0
80.0
85.0
70.0
80.0
75.0
0.67 0.25
0.05
19.0
4.0
1.5
0.67
0.25 0.05
19.0
0.67 0.25 0.05
Head- ration ig)
Full Model Soups
PEFT
Soups (Ours)
Figure 1: Performance of baselines and LT-Soups on the CIFAR100 benchmark varying p and While full fine-tuning generally outperforms PEFT on head classes, PEFT demonstrates superior performance on tail classes. In contrast, our approach maintains robust accuracy across all imbalance settings, showing resilience to shifts in both the sample distribution and class structure.
These observations motivate deeper investigation into when and why PEFT is effective. To this end, we construct controllable long-tailed benchmark based on CIFAR 100 that allows systematic variation in both the sample counts across classes, quantified by the imbalance ratio (p), and the number of classes above or below a sample threshold, defined as the head-tail ratio (1)) This setup enables a more fine-gratd gra-ira analysis of imbalance structures. Within this framework, we compare two full fine-tuning strategies against PEFT baseline: full fine-tuning with logit adjustment (LA) and model soups [60]. which averages weights of multiple LA-trained models initialized with different seeds and hyperparameters. Results from this benchmark confirm previous findings: average, PEFT improves overall accuracy (80.8 vs. 81.2) and tail-class accuracy (76.4 vs. 70.2) compared to full fine-tuning, but at the cost of degraded head-class performance (87.0 84.3). A detailed breakdown in Figure shows that PEFT is especially effective in tail-heavy scenarios, where classes dominate (n) < 1), but its performance declines the class structure becomes more balanced or head-heavy, highlighting its limited robustness to shifts in class structure.
This highlights key trade-off: PEFT helps prevent overfitting and supports tail classes, but lacks adaptability in more balanced and head-heavy settings. Conversely, full fine-tuning offers stronger adaptation but requires careful regularization. Model soups offer a middle ground by averaging models trained with different seeds and hyperparameters (29) 27 our experiments show that traditional soups, built from models trained the same imbalanced dataset, still underperform in tail- mal--eavy cases, as they tend to overemphasize head-class performance due to the dominance of high imbalance ratios.
To address these limitations, we introduce LT-Soups, two-stage model soups framework designed to deliver robust performance across diverse imbalance scenarios, jcenarios,joinlly characterized by the imbalance ratio (p) and the head-tail ratio (7). Unlike PEFT, which performs well primarily in tail-heavy settings, LT-Soups consistently achieves strong results across tail-heavy, balanced, and head-heavy class structures (Figure the first stage, L.T-Soups constructs a weight ensemble by averaging multiple fully fine-tuned models, each trained on subset exhibiting distinct imbalance ratio, collectively spanning a spectrum of imbalance ratios. The aim is to create models that "specialize" for each of these imbalance ratios, when averaged, promote balanced representation that performs well on both the head and the tail classes. To recover any head-class information lost during subsampling, the second stage fine-tunes only the classifier on the full dataset using class-balancing techniques. By seamlessly combining the adaptability of full-rank optimization, favouring the head-classes, and the robustness of weight ensembling for the tail-classes, LT-Soupe strikes better trade-off than PEFT and model soups, and thus bridges the head and the tail classes.
Our contributions are threefold: (1) We introduce a dual-axis framework for characterizing class imbalance using both the imbalance ratio (p) and head-tail ratio (7)), and show how they jointly affect performance. (2) We propose LT-Soups, novel two-stage approach that mitigates representation

---

bias and adapts effectively across broad range of imbalance structures. (3) We conduct extensive experiments on five benchmark datasets and show that while existing LT methods perform well only under specific imbalance configurations, our approach consistently delivers robust, all-around performance across a wide range of imbalance scenarios.
2 Related Work
Imbalanced Learning. Class imbalance has traditionally been tackled through oversampling minority classes, undersampling majority classes, or applying reweighted loss functions such as focal loss or logit adjustment (LA)
40 While effective in certain settings, these techniques often
struggle under overparameterized models [66]. Decoupled training frameworks further refine this process by separating representation learning and classifier training [25] 67], assuming biases lie
primarily in the classifier layer. However, this assumption breaks down when adapting foundation
models, as full fine-tuning can lead to catastrophic forgetting and degraded generalization for both
head and tail classes [1 [5].
Ensemble-based methods address class imbalance by combining experts trained on diverse data distributions 4 (53) Examples include BBN 774 and RIDE (57), which use architectural branching or dynamic routing, and LFME 62], which employs group-wise distillation. Mixture-of- -f-eeeerts approaches such as SADE 688 Mdes [69] and DirMixE [65] merge experts trained with different logit adjustments (e.g., uniform, long-tail, inverse long-tail). Unlike these methods that require all experts at inference, LT-Soups collapses multiple fine-tuned models into single network via weight averaging, offering an inference-efficient alternative. While prior works rely on specialized architectures and heuristic expert definitions, our approach retains architectural simplicity by using parallel fine-tuning on controlled subsamples and model averaging to preserve both the generalization and efficiency of the foundation model.
CLIP and other vision-language models exhibit inherent robustness to class imbalance, largely due to the diversity of their pretraining data [59]. This robustness has been further extended through techniques such as prompt tuning [13. retrieval-basee augmentation [35,, and joint vision- lisoon-lagguage
training paradigms [37 58). While these methods improve adaptation to long-tai tailed distributions,
Shi et al. [50] show that PEFT combined with logit adjustment (LA) loss achieves state-of- ofte-offthhe-art performance by selectively adapting CLIP's pretrained features. However, they also observe that this comes at the cost of reduced head-class accuracy.
In this work, we demonstrate that PEFT is particularly effective in tail-heavy scenarios, but its performance diminishes as the class structure becomes more balanced or skews toward head-class dominance. To overcome this limitation, we propose a method designed to maintain robust perfor- mance across the entire long-tail distribution spectrum. Our approach merges models trained on a subset exhibiting distinct imbalance ratio, collectively spanning a spectrum of imbalance ratios, enabling the final model to achieve balanced accuracy across both head and tail classes.
Model Merging. Methods based on model merging, weight averaging, has emerged as prac- tical strategy for reducing communication overhead in federated and distributed settings [3 [5], improving robustness to distribution shifts [60], and enhancing generalization through techniques like SWA and EMA 23 [54] Recent efforts also apply merging for continual learning and RLHF
fine-tuning 45. Yet, to our knowledge, model merging has not been explored for imbalanced classification.
3
A Closer Look at Imbalanced Learning with Foundation Models
3.1 Preliminaries
Given training data D {xx y
) where 2; denotes input sample and ys C is its corresponding class label from a set of K [C] classes. Let ny denote the number of training samples for class j. and let the total number of training samples be N Σド= Nz. Without loss of generality, we assume that classes are sorted in decreasing order of frequency, i.e., if < j. then n; 2 ny. the imbalanced setting considered here, the most frequent class is significantly larger than the rarest one, such that ni ng. To quantify this imbalance, define the imbalance ratio as
3

---

p= ==nk/nn Following 34
we categorize classes with more than 100 training samples (n, > 100) as head classes, and the rest as tail classes Since we aim to achieve balanced performance across all classes, we report BalAcc class.
c EeEc
Accuracy(c) which equally weights performance on each
Our model is composed of two main components: a feature extractor and classification head. For feature extraction, we adopt the CLIP vision encoder, implemented using a Vision Transformer (ViT) [14] and parameterized by 0. The representation for input is given by fr(2; 0) = 2 where x the extracted feature vector. The final class prediction is computed as arg max ((z; w) where g denotes the prototypical classification head with parameters constructed from the CLIP text encoder. (see Appendix D for the full details).
Previous work suggests that training with standard Cross-Entropy loss with instance-balanced sam- pling often leads to head-class bias due class imbalance [5] 17] Logit Adjustment (LA) [40] addresses this by adding a class-dependent offset to the logits, thereby correcting for prior class frequencies as follows:
exp(gy(z) + log 1yg Eyec exp (2) + log Tyy)
(L((,,(())) -log
where 9y denotes the predictive logit of model on class y and π E Ay are estimates of the class priors P(y) based the empirical class frequencies the training data D. However, Shi al. [50] observed that when fine-tuning CLIP models from pretrained weights using LA (referred to as Full-FT), the resulting condittooi distributions can become inconsistent, particularly for tail classes. To mitigate this, they advocate for methods that preserve proximity to the pretrained initialization, leveraging PEFT strategies such as LoRA and AdaptFormer.
3.2 Characterizing Imbalanced Distribution with Head-Tail Ratio
In practice, class imbalance manifest in diverse structural forms. While the imbalance ratio (p) is a standard metric for quantifying distributional skew, we show that it is insufficient to fully capture the complexity of long-tailed distributions. As a complementary measure, we introduce the head-tail ratio (1)), which reflects the relative number of head versus tail classes and emphasizes the underlying class structure.
Definition 1.
Let H = {cl n. > r and T= (c n. < 5T} denote the sets of head and tail classes, respectively, based on sample threshold r. Let H = HH] and T |T] be the number of head and tail classes. The head-tail ratio is then defined as T
To investigate the joint effect of p and 17 on model performance, we construct synthetic benchmark based CIFARION 100, where both parameters are systematically varied. For fixed 7,, classes are partitioned into head and tail groups, and within each group, sample sizes are assigned following an exponential decay distribution. This procedure is repeated across 11 values of 7) ranging from 19 -eavy)) to 0.05 (tail-heavy), and for p e (50, 100, 250). these configurations, head-class sample sizes range from 500 to 101, and tail-class sizes from 100 to 2 (see Figure 7a in the Appendix for visualization).
Figure 2 presents performance trends marginalized over varying p. 7), and their joint effects. The results reveal that no single method consistently dominates; instead, the best- pert-perrorming approach shifts depending on the imbalance configuration. In tail-heavy regimes (low 11), PEFT methods excel due to their ability to retain generalizable pretrained features for underrepresented classes. Conversely, in head-heavy settings (high 1)), full fine-tuning becomes more advantageous, leveraging its flexibility to fit the dominant head-class structure. These trends underscore the need for methods that can adapt effectively across the full spectrum of imbalance scenarios.

---

0-30
p-100
p-250
Average across p and
88
87.5
87.5
Accuracy (86
86
85.0
82.5
80
82.5
82
80.0
80
77.5
75
77.5
78
76
72.5
70.0
65
70.0
All
Head
Tail
All
Head
Tall
All
Head
Tail
All
Head
Tail
Ful-FT
Model Soups
PEFT
LT-Soups (Ours)
Figure 2: Marginalized performance of baselines, including L.T-Souss, on CIFAR 100 across varying P and n. The first three columns average over for each p; the last column averages over all configurations. Refer to Figure I for the detailed results.
(0,w)
(0, w)
(0,w> }²
Merge
{0,
Merge
(0,w)
(0.w)
(a) Each model within Model Soups fine-tunes the full training set.
(b) Each model within LT-Soups fine-tunes a subset of less imbalanced data from the full training set.
Figure 3: Comparison between Model Soups and LT-Soups. (a) Model Soups merges models fine- tuned on full, severely imbalanced training data. (b) LT-Soups merges models fine-tuned subsets with increasingly higher imbalance ratios to preserve pretrained features while adapting to class distribution shifts.
4 LT-Soups: Imbalanced Learning by Subsampled Model Averaging
The preceding toy experiment illustrates that the optimal method depends on the underlying class structure. In head-heavy distributions, full fine-tuning is particularly effective, as it adjusts all model parameters to capture the rich structure of frequent classes. In contrast, when tail classes dominate, PEFT approaches like LoRA and AdaptFormer (as used in LIFT (50) perform better by preserving pretrained representations that generalize well under limited supervision. Motivated by this trade-off, our goal is to design method that maintains balanced performance across both extremes, regardless of the imbalance pattern.
Ensemble methods such model soups (60] (Figure 3a have demonstrated effectiveness in improving both overall and minority-class performance [29] 27 However, as shown the previous section, traditional soups, while outperforming single- model fine-tuning, remain suboptimal for the full spectrum of long-tailed distributions. This especially true in tail-heavy scenarios, where they tend overemphasize head-class performance due the dominance of high imbalance ratios in the training data. We address this limitation with LT-Soups, a soups-based framework specifically designed for long-tailed distributions. Each model in LT-Soups is
on subset of the training data with distinct, reduced imbalance ratio (Figure 3b While such subsampling enhances tail-class learning [6]. it may omit valuable head-class information [26). To balance this, train models on subsets with gradually increasing imbalance levels, allowing each model to specialize in different regions of the long-tail spectrum. The final model is obtained by averaging these specialized models. To further recover any lost head-class information, we introduce second stage where only the classifier head is fine-tuned the full dataset using a -las-balaaccd objective (e.g.. LA loss).
For simplicity, we initially group all low-resource classes into a single tail category. In the experimental section, further subdivide the tail into medium-shoi shot and few-shot groups for consistency with prior work.
5

---

Algorithm LT-Soups (Parallelizable Pseudocode)
1: Input: On pre-trained weights, full training data D, (D,,, J.N subsets with N imbalance ratios and M bootstrapping per Pn. À merging interpolation.
2: Training: for all 1 to N in parallel do
3:
n k Fune (Bu. Dp..)
4: Prepare models: Sort models in an increasing order of Pn
5: Weight Averaging: An to N. f (1 X)O,. + 10n-1
6: Re-train final classifier on full D
Pscudocode for LT-Soups is provided in Algorithm ! and the remainder of this section details the full procedure.
Balanced representation. Subsampling common strategy for addressing class imbalance by re- ducing overrepresented head-class samples [17 However, aggressive subsampling can discard use- ful head-class information, degrading overall performance (7 To address this, propose pro- gressive subsampling, which incrementally increases the imbalance ratio across subsets. Each model fine-tuned on subset with a specific ratio, preserving tail-class data while managing head-class underutilization. We construct the subset sequence as {Dp, P; 2', E {0, 1, 2, ..... [logy (p) } where pp, is a dataset with imbalance ratio p:. This yields a sparse sequence of subsets with expo- nentially increasing imbalance ratios, ensuring broad coverage while limiting the number of models in the soups procedure. In practice, we retain only the first N subsets, as extremely high imbalance ratios tend to overly favor head classes and degrade tail-class performance.
The resulting N models, each trained different subset, merged using recursive interpolation strategy. Given weights [0x) where 9g is the pretrained model, L.T-Soups recursively combines models via 0, +
The interpolation coefficient À controls knowledge retention from previous stages. This procedure ensures (1) proximity to the pretrained model 0g preserving CLIP's zero-shot capabilities 61, and (2) smooth integration of head and tail class representations 774. As shown in Section 5.3 LT-Soups exhibits partial insensitivity to the choice of loss function, owing to the balance introduced by subsampling and model averaging. However, since each subset remains mildly imbalanced, albeit less so than the full training set, applying LA loss during fine-tuning further mitigates the effects of label distribution shifts.
Variance reduction. While weight averaging and subsampling help mitigate head-class dominance, fine-tuning large pretrained models can still lead to degradation in tail-class performance (59). To address this, we maintain an exponential moving average (EMA) of model weights via Oema (1 p4) fema + 0, with momentum coefficient = 0.99. EMA acts as a regularizer during training [21]. promoting convergence flatter minima (23). which has been shown enhance generalization, particularly for underrepresented classes.
Since subsampling reduces data per subset and introduces variance, we adopt a bootstrapping strategy inspired by bagging [2]: for each subset, we train M models on different bootstrap samples and uniformly average their weights. This stabilizes learning and yields more robust representations.
Classifier re-training. To further recover head-class information lost during subsampling, perform final fine-tuning stage on the classifier head using the full training set. The backbone is frozen preserve merged representations, and LA loss is applied to adjust decision boundaries based on label frequencies. This step improves head-class accuracy without harming tail-class performance, similar to calibration in two-stage LT methods 25
Experiments
5.1
Datasets and evaluation protocol.
We evaluate our method on both synthetically constructed and naturally occurring long-tailed (LT) datasets. For synthetie benchmarks, we use CIFAR-100-ITT, ImageNet-LT, and Places-ITT-long- tailed variants derived from their balanced counterparts by sampling class instances according

---

to Pareto or exponential decay distributions [34]. These datasets exhibit sample counts ranging from 1,280 to as few as 5 images per class. For real-world evaluation, we include iNaturalist 2018 18.142 classes, 437.5K images) and NIH-CXR-L.T (20 classes, 88.5K images), which reflect different imbalance structures, with approximately 10% and 90% head classes, respectively. To assess performance across the long-tail spectrum, we also report the average accuracy across all five datasets. Following [34], we evaluate separately on many-shot (> 100 samples), medium-shot (20-100), and few-shot (<20) class subsets. For ablation analysis, we use TinylmageNet-LI. which contains 200 classes with sample counts ranging from 500 in head classes to tail classes. To conserve space, present only CLIP-based results in the main text; additional implementation details and extended results are included Appendix D
Table 1: Comparison with state-of-the-ar methods on synthetic LT distributions.
CIFARIO0-LT
Places-LT
ImageNet-LT
Methods
p=100 Many
y=0.54
9=99
1=0.55
p=256
7=0.62
All
Med.
Few
All
Many
Med.
Few
All
Many
Med.
Few
BALLAD 37
49.5
49.3
50.2
48.4
75.7
79.1
74.5
69.8
Decoder [58
46.8
73.2
LPT [13
50.1
49.3
52.3
46.9
Linear Probing
70.0
77.2
71.1
60.4
48.8
48.8
49.7
47.1
74.2
77.8
73.3
67.4
Full-FT
79.6
88.1
79.9
69.3
46.6
49.9
46.3
41.4
73.9
79.8
71.9
63.9
cRT 25
78.8
89.7
79.7
65.1
44.4
51.0
43.1
35.4
72.6
81.1
70.6
56.1
PEFT 50
81.3
85.2
80.9
77.1
51.5
51.3
52.2
50.5
77.0
80.2
76.1
71.5
Model Soups 600
82.1
89.9
82.2
73.0
49.4
51.7
50.0
43.7
76.0
81.5
74.5
65.5
I.T-Soups (Ours)
83.5
88.2
83.5
78.0
51.7
51.2
52.8
50.3
77.4
81.2
76.1
70.7
Table Comparison with state-of- -art methods on real-world I.T' distributions.
NIH-CXR-LT
iNaturalist 2018
Methods
p=64
5.6.66
p=500
0=1.11
All
Many
Med.
Few
Many
Med.
Few
BALLAD
37
34.5
36.7
38.9
20.8
49.5
49.3
50.2
48.4
Decoder
59.2
LPT [2
79.3
Lincar Probing
17.5
13.3
21.1
16.7
60.4
60.0
63.9
Full-FT
38.0
43.8
41.5
20.0
76.1
75.7
76.9
75.3
cRT 25
37.7
42.9
39.3
25.0
44.4
51.0
43.1
35.4
PEFT [50
38.5
43.3
40.4
25.5
79,1
72.4
79.0
81.1
Model Soups (60
38.0
45.6
40.2
20.0
76.4
77.1
76.8
75.6
L.T-Soups (Ours)
39.3
42.4
40.7
30.9
78.2
76.7
78.5
78.2
5.2 Main results
Synthetic LT datasets. Table Table presents the accuracy of L.T-Soups on three benchmark datasets with synthetically induced long-tail distributions: CIFAR100-I.T, Places-LTT, and ImageNet-LTT. Our method outperforms all state-of-the-arr baselines in overall accuracy on every dataset. Notably, I.T-Soups surpasses Model Soups and PEFT, the most competitive baselines. While PEFT achieves competitive performance on tail classes through low-rank adaptation, it often does so at the cost of many-shot accuracy, especially in 100-T, where LT-Soups maintains strong tail accuracy (78.0) without sacrificing performance on many-shot (88.2) or medium-shot (83.5) categories. Model Soups, on the other hand, tends to overfit many-shot categories (e.g., 89.9 CIFAR 100-LT) but underperforms on few-shot classes due to averaging independently tine-tuned models without accounting for class imbalance.
Real-world LT datasets. In Table 2 we evaluate LT-Soups on two naturally imbalanced datasets NIH-CXR-LT and iNaturalist 2018 10018-hiichh present distinct challenges. (1) NIH-CXR-LT consists primarily of head-class (many-shot) samples but diverges significantly from CLIP's pretrain- ing domain, as comprises medical X-ray images.
(2) iNaturalist 2018 heavily skewed toward medium- and few-shot categories and is more closely aligned with CLIP's natural image priors. On NIH-CXR-LTT, LT-Soups achieves the highest overall accuracy (39.3%), outperforming PEFT (38.5%) and delivering substantial gains in the few-shot regime (+5 points over PEFT and +10 over
7

---

Model Soups). On iNaturalist, where PEFT performs strongly (79.1% overall), LT-Soups remains competitive (78.2%) while offering more balanced accuracy across many-, medium-, and few-shot subsets.
Full spectrum results. The datasets in our benchmark exhibit diverse long-tail characteristics, with imbalance ratios ranging from 100 6,491 and head-tail ratios span- ning 0.11 to 5.66. To abstract away the effects of individ-
67.5
E 65.0
62.5
ual dataset characteristics, Figure 4 reports the average accuracy of Full-FT, Model Soups, PEFT, and LT-SSops across all five benchmarks. While PEFT performs well on medium-shoe and few-shot splits, and Model Soups cels on many-shot classes, LT-Soups consistently achieves strong performance across all splits, demonstrating its bustness across the long-tal tail spectrum.
60.0-
52.5
Many
Mode Soups
PEFT
LF-seups (Durs)
Figure Average performance across LT benchmarks.
5.3 Empirical analysis of LT-Soups
In this section, we provide comprehensive analysis of LT--Sous from multiple perspectives (due to space constraints, some of the analysis is provided in the Appendix A
Effect of subsampling. LT-Soups averages the weights of NM models, where N is the number of subsets used during each fine-tuning and is the number of bootstraps per subset. For Tiny Imageett- LT (imbalance ratio 100 and head-tail ratio 0.3), we use N and M=2. To show the impact of our proposed weight averaging scheme, we compare this with Soups-P> baselines that follow the same two-stage framework LT-Soups, except in the first stage, they average 16 models, all trained on subsets with the same imbalance ratio Pn- Notably, Soups-100 aligns with the traditional model soup approach [60], where the weights of 16 fully fine-tuned models the entire dataset are averaged. As shown Table 3 all of the soups baselines consistently outperform full fine-tuning, regardless the subset choice. However, results show that different imbalance ratios yield varying outcomes across head and tail categories. For example, Soups-8 8 achieves the highest tail accuracy of 75.0, whereas Soups- 100 reaches the highest head accuracy of 85.9. Rather than optimizing for single imbalance ratio, LT-Soups applies weight averaging across the full spectrum, effectively merging the advantages balanced overall trade-off. (See Table 10 in the Appendix for
of both approaches achieve similar analysis on PEFT.)
Table 3: Comparison of LT-Soups and Soups-p each with total of 16 models across All, Head, and Tail accuracy.
FullFTT
Soups-1
Soups-2
Soups-4
Soups-8
Soups-16
Soups-32
Soups
Soups-
L.T.Soups
All
73.2
77.1
71.7
75.9
76.0
77.2
77.2
77.3
77.9
77.6
78.6
Head
834
83.0
74.6
78.6
78.7
81.0
82.8
84.7
85.5
85.9
85.0
Tail
67.7
70.1
74.4
74.6
75.0
74.1
73.3
73.7
73.0
75.2
Effect of classifier re-training (CR). We found that additional final-layer tuning with logit ad- justment on PEFT and Model Soups has little to no effect. Table 4 summarizes the results on TinyImageNet-LT. We hypothesize that, unlike these baselines, LT-Soups does not fully exploit the entire training set, due to the downweighting effect introduced by weight averaging. Conse- quently, fine-tuning the final layer helps LT- LT-Soups recover head-class sharpness and improves overall performance.
Component analysis. LT-Soups is designed to balance effective task adaptation with minimal deviation from pretrained weights. Figure 5 shows the cumulative effect of its components on accuracy and weight change. Starting from Full Fine-Tuning, which causes the largest deviation from the CLIP zero-shot model (35.4), each component incrementally improves performance while reducing or controlling weight deviation. EMA offers modest accuracy boost with minimal impact on weight shift. Subsampling and model merging significantly improve tail accuracy (+6.3) and reduce weight change to 12.7, highlighting the benefit of balanced training. Bootstrapping stabilizes training further, slightly improving head accuracy. Classifier re-training refines decision boundaries,

---

Table Comparison of baseline methods in- cluding LT-Soups with and without classifier re-training (CR).
Table 5: Comparison of our merging strategy with uniform merging across two datasets ex- hibiting distinct long-tailed distributions.
Method
All
Head
Tail
TinyImageNer-LIT
iNaturalist 2018
PEFT
77.1
83.0
73.9
Ours
Unifrom
Ours
Unifrom
PEFT CR
77.0
83.0
73.8
All
78.6
78.5
78.2
74.7
Model Soups
77.6
85.9
73.0
Many
85.0
83.4
76.7
67.4
Model Soups CR
77.6
85.5
73.4
Med.
78.3
78.4
78.5
75.8
SouSo Stage
78.1
84.9
74.5
Few
71.5
72.9
78.2
75.3
LT-So
78.6
85.0
75.2
Head
[[А!]]
All
RIll-FT
73.2
81.4
35.4
62
+EMA
74:
35.1
2
80-
+Sub.
78.0
84.6
12.7
+Bootst.
74.5
78.1
84.9
11.9
+Olassifer
75,2
78.6
85.0
12.1
72
70
PER1
10.3
Subests
Figure Performance and weight change comparison across different stages of LT- Soups.
Figure 6: Performance across number of sub- sets, NN, each with increasing imbalance ratios on TinylmageNet-LT.
yielding the highest overall and head accuracy. Compared to PEFT, L.T-Soups shows slightly higher weight change (12.1 1 vs. 10.3) but delivers better accuracy across all class groups. This reflects its ability to adapt meaningfully while preserving pretrained knowledge.
Number of subsets. Figure illustrates the impact of the number of subsets N, each with in- creasing imbalance ratios, used in LT-Soups during fine-tuning, In this experiment, the interpolation weight À and M are fixed at 0.7 and 2, respectively. As N increases, head-class accuracy steadily improves frevees-from- 74.6 at N=1 to 85.0 at N=8 while The best overall trade-off is observed at N=8, indicating it as the most balanced configuration.
-hile tail-class accuracy peaks at 76.1 1 when N=3.
Merging strategies. LT-Soups recursively merges models trained subsets with progressively higher imbalance ratios. One intuitive way to think about this merging procedure is to interpret as an exponential moving average (EMA) over fine-tuned models sorted by increasing imbalance severity, with a tunable parameter that adjusts the influence of more balanced (but smaller) versus less balanced (but larger) subsets. In this section, we compare this strategy against uniform WA, which applies a simple arithmetic mean, giving equal weight to all models regardless of their imbalance level.
Table our hypotheses. In particular, we compare recursive WA and uniform WA across two datasets with different similarities compared CI.IP-rretraine weights (according to the zero-shot performance). On l'iyIlmagNNee there is little no difference between the averaging schemes. However, for datasets that require significant adaptation, such as iNaturalist2018. 18, recursive WA yields clear benefits by leveraging information from more data-rich subsets.
which already well-aligned with CL.I--rrtrriine features,
Following this intuition, in all our experiments in the paper, we use only two values for X: 0.3 and 0.7, corresponding to high and low adaptation needs, respectively. Intuitively, when the target dataset close to the pre- pre-trainnng weights, the value of the À becomes less important as even small datasets enough for adaptations. However, when the shift becomes larger, subsets with more data (albeit biased towards head classes) become crucial.
Effect of class-balance strategies. By default, LT-Soups exhibits partial insensitivity to the choice of loss function, due to the balance introduced through subsampling and model averaging. However, in the first stage, each subset remains mildly imbalanced, though significantly less so than the full training set. Table o the impact of different class-balancing strategies used during this

---

Table Comparison of PEFT and LT-Soups under different loss functions (CE, CB, LA).
CE
CB
LA
Methods
All
Head
Tail
All
Head
Tail
All
Head
Tail
PEFT
72.6
85.1
65.9
75.3
81.0
72.2
77.1
83.0
73.9
LT-Soups
76.3
84.5
71.9
78.2
84.5
78.4
78.6
85.0
75.2
stage, including logit adjustment loss (LA), cross-entropy loss (CE), and class-balanced sampling (CB) [25]. Unlike PEFT, which heavily depends on LA loss for optimal performance, LT-Soups is only moderately affected by the choice of class-balancing strategies, owing to the structural regularization introduced by training on multiple, complementary subsets.
Computational analysis. LT-Soups involves a total of NM + 1 training runs: M models are trained at each the N subsampling levels in the first stage, followed by a single classifier trained on the full dataset in the second stage. Two key factors mitigate the computational burden of this procedure. First, each Stage model is trained on a heavily subsampled dataset, significantly smaller than the full training set. For instance, under maximum imbalance ratio of 64, each model sees only about 65% of the full dataset (see Table 11 in the Appendix for precise values), which leads to substantially faster training times compared to full-data training. Second, all models in the first stage are trained independently, enabling full parallelization. As a result, the overall wall-clock time bounded by the longest individual training job. typically the using the most imbalanced subset (e.g., imbalance ratio 64). This parallel-friendly design allows LT-Soups to scale efficiently and deliver competitive performance with minimal overhead. Appendix provides full breakdown of computational cost.
Limitations and Future Work
While we define the head-tail ratio using fixed sample threshold, this binary split may oversimplify the class distribution structure. A more nuanced approach could leverage the generalized Pareto distribution [7 to model imbalance with controllable location, scale, and shape parameters. We leave exploration of such parameterized formulations for future work.
7 Conclusion
This work introduces L.T-Souss, novel two-stage model merging framework tailored for long-tailes distributions. We identify the head-tail ratio (1)) as critical yet underexplored factor influencing model performance alongside the commonly studied imbalance ratio (p). Through comprehensive experiments, we demonstrate that existing approaches, particularly PEFT and traditional model soups, fail to generalize across the full spectrum imbalance scenarios. In contrast, LT-Soups builds weight ensemble by averaging fully fine-tuned models trained subsets with varying imbalance ratios, enabling specialization across the imbalance spectrum while preserving robust representations. Extensive experiments five benchmarks and ablations Tiny-ImageNer confirm its consistent performance across long-tailed scenarios.
This work was supported by Distech Controls Inc., the Natural Sciences and Engineering Research Council of Canada, the Digital Research Alliance of Canada, and MITACS. This work was also supported in part by the project SERICS (PE00000014) under the NRRP MUR program funded by the EU NGEU.
Acknowledgements.
References
[1] Anton Alexandrov, Veselin Raychev, Mark Niklas Mueller, Ce Zhang, Martin Vechev, and Kristina Toutanova. Mitigating catastrophic forgetting in language transfer via model merg- ing. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, EMNLP, pages 17167-17186, Miami, Florida, USA, November 2024. Association for Computational Lin- http:s::///aaaanoologo.ooogg
guistics. doi: 10.18653/v12024, finni//-24.inniinn-eemm...... 1000. 2024.findimg fww.iiinnng--i. emnle. 1000/
10

---

[2] Leo Breiman. Bagging predictors. Machine learning. 24:123-140 1996.
[3] Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
[4] Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In ICCV, pages 112-121, 2021.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distriutiio--vaar margin loss. NeurlPS, 32, 2019.
[6] Kamalika Chaudburi, Kartik Ahuja, Martin Arjovsky, and David Lopez- saee-PPz. Why does throwing away data improve worst-group error? ICML, pages 41 144-44 4188. PMLR, 2023.
[7] Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16: 321-357, 2002.
[8] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS, 35: 16664-16678, 2022.
[9] Wuxing Chen, Kaixiang Yang. Zhiwen Yu, Yifan Shi, and CL Chen. A survey on imbalanced learning: latest research, applications and future directions. Artificial Intelligence Review, 57 (6):1-51, 2024.
[10] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 715-724, 2021.
[11] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, pages 9268-9277 2019.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255 Icce, 2009.
[13] Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Lpt: Long-tailed prompt tuning for image classification. In ICLR, 2022.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. htttt:t:/ www.aarowiionnet/oouuimiiiii
YicbFaNTTy
[15] Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Mare' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training language models. arXiv preprint arXiv:2311.08105 2023.
[16] Jintong Gao, He Zhao, Zhuo Li, and Dandan Guo. Enhancing minority classes by mixing: An adaptative optimal transport approach for long-tailed classification. Advances in neural information processing systems, 36:60329-60348, 2023.
[17] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 1199::263-111 2009.
[18] Yin- Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recog- nition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 235-244, 2021.
[19] Gregory Holste, Song Wang, Ziyu Jiang, Thomas Shen, George L. Shih, Ronald M. Summers, Yifan Peng, and Zhangyang Wang. Long-tailed classification of thorax diseases chest x-ray: new benchmark study. MICCAI workshop, DALI (Workshop), 13567:22-32, 2022. URL. https://apg semanticscholar. wttpeecchooarroooooootooouu org/CorppusD::25190388.

---

[20] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL https::/openrevieee wwt.:/ooounn.dddeeeeeeee.e...
[21] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. In ICLR, 2017. URL htt.:httpp///ee //opr//v//eeeevee
net/foruu7iddBB3ewo001
[22] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023. URL
[23] Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, 2018. URL /ttt::::/aap...smmttti..o..
org/
:
[24] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurious correlations. NeurIPS, 35:38516-38532, 2022.
[25] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020. URL WRLLattps://pperrrriie http://wwwweetttttoooooooodd////
[26] Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In CVPR, pages 13896-13905, 2020.
[27] Wei-Yin Ko, Daniel D' D'souza, Karina Nguyen, Randall Balestriero, and Sara Hooker. Fair-ensemble: When fairness naturally emerges from deep ensembling. arXiv preprint arXiv:2303.00580 2023.
[28] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[29] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 30, 2017.
[30] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https //api. www.nticcchooa.r org/CorpusIID:ao..oo ID: : 16664790
[31] Jun Li, Zichang Tan, Jun Wan, Zhen Lei, and Guodong Guo. Nested collaborative learning for long-tal tailed visual recognition. CVPR, pages 6949 (949-6958, 2022.
[32] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint ar 2309 2023.
[33] Xu- Yung Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2): 539-550, 2008.
[34] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition an open world. In CVPR, pages 2537-2546, 2019.
[35] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In CVPR, pages 6959-6969, 2022.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. URL https://openreviie ottt://foruurr.--Bgg.g.......
[37] Teli Ma, Shijie Geng, Mengmeng Wang. Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple long-tailed recognition baseline via vision-language model. arXiv preprint
arXiv:2111.14745, 2021.
12

---

[38] Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, and Lingling Li. Curvature- balanced feature manifold learning for long-tailed tailed classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15824-15885, 2023.
[39] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. ceapiuniicatoofffffiieeee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[40] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021. URL
[41] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308. 13320, 2023.
[42] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, volume 29, 2015.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages $748-8763 PMLR, 2021.
[44] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gal- linari, and Matthicu Cord. Diverse weight averaging for out-of-distributiin generalization. NeurIPS, 15:1011110 2022.
[45] Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies. arXiv preprint wwXi::2406.1666,, 2024.
[46] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. NeurlPS, 33:4175-4186, 2020.
[47] Holger Rootzén and Nader Tajvidi. Multivariate generalized pareto distributions. Bernoulli, 12 (5)-917-330, 2006.
[48] Jie Shao, Ke Zhu, Hanxiao Zhang, and Jianxin Wu. Diffult: Diffusion for long-tail recognition without external knowledge. Advances in Neural Information Processing Systems, 37:123007- 123031, 2024.
[49] Jiang-Xis Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learning? NeurlPS, 36, 2023.
[50] Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yan Han, and Yu-Feng Li. Long-tail learning with foundation model: Heavy fine-tuning hurts. In ICML,, 2024.
[51] Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228. 2024.
[52] Min-Kook Suh and Seung-Wor Seo. Long-tailed recognition by mutual information maximiza- tion between latent features and ground-truth labels. In International conference on machine learning, pages 32770-32782. PMLR, 2023.
[53] Yingfan Tao, Jingna Sun, Hao Yang, Li Chen, Xu Wang, Wenming Yang. Daniel Du, and Min Zheng. Local and global logit adjustments for long-tailed learning. In ICCV, pages 11 11792-11792, 2023.
[54] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NewrIPS, 30, 2017.
[55] Changyao Tian, Wenhai Wang, Xizhou Zhu, Jifeng Dai, and Yu Qiao. VI-Itr: Learning class- wise linguisic representation for long-tailed visual recognition. In ECCV, pages 73-91. Springer, 2022.
13

---

[56] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard. Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, pages 7778, 2018.
[57] Xudong Wang. Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. loinettailed recognition by routing diverse distribution-awan experts. In ICIR, 2021. URL. httpps://openrreiiieeeeee/ net/ Jorum?id=D9I3drBae
[58] Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. Exploring vision-language models for imbalanced learning. IJCV, 132(1):224-233)) August 2023. ISSN 0920-5691. doi: 120777/11166-02---00 http::://doo..og////// 10000 023-00022--1----
URL
[59] Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, and XIAOJUAN QI. What makes CLIP more robust to long-tailed pre-training data? a controlled study for transferable insights. In NeurlPS, 2024. URL httpap/pesppppeeeeeeeeeeee/r//rp wttp:/orrmmddddayyoooo.o...
[60] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rehecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, pages 23965-239988. PMLR, 2022.
[61] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Korublith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959-7971, 2022.
[62] Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In ECCV, pages 247-263. Springer, 2020.
[63] Zhengzhuo Xu, Ruikang Liu, Shuo Yang, Zenghao Chai, and Chun Yuan. Learning imbalanced data with vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15793-15803, 2023.
[64] Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look at subpopulation shift. In ICML, 2023.
[65] Zhiyong Yang. Qianqian Xu, Zitai Wang, Sicong Li, Boyu Han, Shilong Bao, Xiaochun Cao, and Qingming Huang. Harnessing hierarchical label distribution variations in test agnostic long-tail recognition. ICML, 2024. URL https://::/// http:::://ppeeevieess wttp://eww..eeeooorrrreeeeeeeeee
[66] Runtian Zhai, Chen Dan, J Zico Kolter, and Pradeep Kumar Ravikumar. Understanding why generalized reweighting does not improve over ERM. In ICLR, 2023. URL https
/ //openreview.neteee net/forummiddashhee.. _BF-
[67]
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In CVPR, pages 2361-2370, 2021.
[68] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in neurul information processing systems, 35:34077-34090, 2022.
[69] Qihao Zhao, Chen Jiang, Wei Hu, Fan Zhang, and Jun Liu. Mdes: More diverse experts with consistency self-distillation for long-tailed recognition. Proceedings of the IEEE/CV International Conference on Computer Vision, pages 1 11597-11608, 2023.
[70] Qihao Zhao, Yalun Dai, Shen Lin, Wei Hu, Fan Zhang, and Jun Liu. Lirl: Boosting long-tail recognition via reflective learning. In European Conference Computer Vision, pages 1-18. Springer, 2024.
[71] Yan Zhao, Weicong Chen, Xu Tan, Kai Huang, and Jihong Zhu, Adaptive logit adjustment loss for long-tailed visual recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 3472-3480, 2022.
14

---

[72] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-taile saggtailed recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16489-16498, 2021.
[73] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. PAMI, (016):1422-1464 2018. doi:
10.1 1109T//A...10..2272222000
[74] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: branchh network with cumulative learning for long-tailed visual recognition. In CVPR, pages 9719-9728, 2020.
[75] Jianggang Zhu, Zheng Wang. Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Bal- anced contrastive learning for long-tailed visual recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6908 6917, 2022.
Broader Impact
Our work advances the field of long-tailed recognition by improving model performance across imbalanced datasets, which are prevalent in real-world applications such as medical imaging, wildlife monitoring, and autonomous driving. By enhancing accuracy for both head and tail classes, method promotes fairness and inclusivity in Al systems, reducing biases toward dominant categories.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Empirical validation on the synthetic CIFARI00 benchmark is partially pre- sented in the Introduction and Section 3 The remaining experimental results are detailed in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Section 5 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All of the datasets publicly available. And the implementation details provide in Appendix [
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Coeeeuuiiis casssiinnooyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccpblccgguui.ceeee wtteeee//ooSSSeesse..ee.. www.ssssiicggsooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix D]and Appendix A Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due resource constraints, all experiments were conducted using a single random seed. For model soups, however, multiple seeds were used during training, as required by the method, but only the averaged results are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work is based on public data.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We only use public datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We used LLM only for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
22

---

A Additional ablations
Model calibration analysis. An inherent advantage of model merging methods is their ability to improve prediction calibration metrics. We evaluate LT-Soups against PEFT and Full-FT by measuring Negative Log-Likelinne
(NLL), Expected Calibration Error (ECE) 422 and Brier score 3 For NLL and Brier scores, we also provide category-wise results. All metrics computed after
temperature tuning on the validation set. As shown in 7 LT-Soups consistently outperforms the other methods TnylmmageNet-LTT in terms of calibration.
Table 7: Calibration metrics on TinylmageNet for Full-FT, PEFT, and our LT-Soup.
Method
Metric
Overall
Head
Tail
ECE
1.97
Full-FT
Brier Score
0.36
0.21
0.40
NLLI
1.03
0.63
1.25
ECE
1.95
PEFT
Brier Score
0.32
0.23
0.35
NLL
0.89
0.68
0.99
ECE
1.36
LT-Soups
Brier Score
0.30
0.20
0.33
NLL
0.83
0.59
0.97
Table Performance across different values of À with fixed M=2, on TinylmageNet-LT.
Table 9: Performance across different values of M with fixed A=0.7, TinylmageNet- LT.
A=0.3
A=0.7
M=1
M=2
M=12
Acc
78.3
78.6
Head
84.6
85.0
Acc
78.2
78.6
78.8
Tail
75.0
75.2
Head
84.8
85.0
85.5
Tail
74.6
75.2
75.2
PEFT compatibility. A natural question whether PEFT methods can replace the full fine-tuning process in LT-Soups. To investigate this, LoRA representative approach. In the first stage, we freeze the CLIP pre-trained weights and tune LoRA parameters using the same subsets LT-Soups. The LoRA parameters combined with the pre-trained weights before applying our merging schema. Finally, we retrain the classifier using the LA loss. The performance on Tinylmagenet-LT dataset (77.1 77.2) matches that of end-to-end LoRA training. We hypothesize this outcome is due to a phenomenon observed in LLM literature (51). where LoRA introduces high-ranking singular vectors (intruder dimensions) that are absent in full fine-tuning. While these models achieve comparable task performance, they adapt less robustly to sequential tasks and diverge from the pre- pre-training distribution.
Table 10: Effect of subsampling and classifier re-training in conjunction with the PEFT method. Each column reports PEFT fine-tuning performance on given subsample ratio p after classifier re-training on TinyImageNNetL.TT.
8
16
100
All
73.9
74.3
74.8
76.2
76,4
77.1
77.0
77.0
Head
75.9
75.8
77.0
78.2
80.4
81.8
81.9
83.0
Tail
72.8
73.5
73.6
75.1
74.2
74.5
74.4
73.8
Hyperparameter sensitivity.
addition to the number of subsets used during LT-Soups fine- tuning, two other hyperparameters impact performance: (1) M: the number of models trained per subset Dp., with each model bootstrapped from the same imbalance ratio ps. (2) X: the interpolation coefficient used during recursive weight averaging.
23

---

Table that increasing M on TinyImageNet-LT improves overall, head-, and tail-class accuracy, highlighting the benefits of ensembling across bootstrapped models. To ensure computational feasibility across all five datasets, we fix M = 2 in the main experiments.
Table 8 compares performance for À 0.3 and À 0.7. We observe that datasets closely aligned with CLIP's pretraining domain benefit from larger A, which retains more pretrained knowledge. In contrast, datasets with significant domain shifts-such smaller X, allowing greater adaptation during model merging.
CECCX---T-eefonne better with
Table 11: Number of subsampling rounds (N), size of the largest subset relative to the full training set and À used for each dataset.
Dataset
N
Relative size of largest subset
X
CIFAR100-LTT
Places-LT
5
ImageNet-LT
7
79
0.7
iNaturalist
8
90
0.3
NIH-CXR-LT
8
24
0.3
B Full Computational analysis
Table 12:
Comparison of methods across ImageNet-LT and CXR-LT in terms of training time, iterations, model size, memory, and accuracy.
Wall-clock Time (H-M-S)
Parameters (M)
Memory (G)
Method
Training Iterations
ImageNet-LT
Full-FT
1:37:56
9060
87.0
14.5
Model Soups
1:37:56
9060
87.0
14.5
LoRA (rank 641
1:25:33
9060
9.0
13.3
LTTSSous Stage 1
1:15:38
8050
87.0
14.5
LT-Soups Stage 2
0:30:00
9060
0.7
2.6
Full I.T-Soups
1:48:38
NIH-CXR-LT
Full-FT
0:53:43
5320
87.0
14.5
Model Soups
0:53:43
87.0
LoRA (rank 64)
2:14:32
13300
9.0
13.3
LT Lo-Soups Stage
0:12:51
1300
87.0
14.5
SouSs Stage
0:19:26
5320
0.7
2.6
Full LT-Soups
0:32:17
Table 12 compares the computational costs of Full Fine-Tuning (Full-FT), LIFT (which employs LoRA adapter with rank 64 applied to all MLP layers), Model Soups, and LT-Soups the ImageNet- LT and WIH-XXR-LB datasets. All models were trained convergence using batch size of 128 and mixee-precision training with NVIDIA RTX 3090 GPUs (24GB VRAM), using Python 3.9.15, PyTorch 2.4.0, and CUDA 1.8. For LT-Soups, we break down the computational cost into two stages. Stage involves training models independently and in parallel on subsets with different imbalance ratios. Since the subset with the highest imbalance ratio contains the most training samples, dominates the overall wall-clock time. Stage retrains only the linear classifier on the full dataset-a highly efficient step, as it updates just a single lincar layer. In our experiments, we used the same number of epochs for both stages of LT-Soups.
The computational overhead of LT- LT-Soups compared to existing methods depends heavily on dataset characteristics, particularly the original imbalance ratio. For example, in ImageNet-LT, which has an imbalance ratio of 256, the largest subset used Stage accounts for 89% of the full training data, resulting in relatively higher wall-clock time. In contrast, on NIH-CXR-LT, with much more extreme imbalance ratio of 6401, the largest Stage subset represents only 24% of the dataset, leading to 4.4x reduction in training time compared to Full-FT (Table IT) Additionally, while full-rank
24

---

methods like Full-FT and LT-Soups typically converge within 10 epochs on CXR-LT, LIFT required 50 epochs-substtantally increasing its wall-clock time despite its parameter-effcien design.
#:100
82
headiTail Radoo (m)
p1
p4
p:6
19.00
2.33
0.67
0.11
4.00
1.30
0.25
0.05
500
#Samples (log scale)
Number of samples
400
102
300
200
100
101
0
20
40
60
80
100
20
80
100
Class index
Class Index
(a) Visualization of imbalance distributions in
(b) Example of subsampled distributions used in Soups, with the x-axis shown on a logarithmic scale.
CIFARI00-LT with varying values 17.
C Extended Related Work
C.1 Imbalanced Classification
We can roughly divide progress on imbalanced classification into three groups.
Re-samplin/RR--eiighin Class imbalance mitigation strategies broadly involve oversampling minority classes [7], subsampling majority classes (33), or reweighting losses [17]. Generative approaches such as DiffuL.T (48] train a diffusion model on a long-tailed distribution and then uses it generate balanced proxy dataset for training the final classifier. Subsampling risks losing majority-class discriminative patterns, oversampling may overfit minority classes [74], and reweighting struggles in overparameterized networks (66). Recent advances like balanced softmax [46] and its generalization, logit adjustment loss (LA) 44 address these issues by enforcing larger margins for tail classes, bridging data imbalance with geometric regularization.
Decoupled learning. Decoupled learning frameworks address class imbalance through sequential training phases: representation learning via instance-balancee sampling followed by classifier re- finement using class-balanced strategies [25 671. This paradigm assumes model biases primarily reside in the classifier layer. positing that head-tail performance gaps can be resolved through post- hoc classifier calibration 22 (4).. However, (50 show empirically that this assumption becomes invalid when fine-tuning foundation models, neglecting a tailored strategy for representation learning, degrades both head and tail class performance due catastrophic forgetting of pre-trained features [1
Ensemble learning. Ensemble methods address data imbalance by combining specialized experts trained on complementary distributions 31). Notable approaches include: BBN's duall archi- tecture balancing original and re-sampled distributions [74]; RIDE's dynamic routing of instances to distribution-awer experts (57]; and LFME's multi-teacher distillation across many/mediimmffe-shoe groups 62); Reflective Learning (70 promotes consistency across training iterations by minimizing KL divergence between predictions and soft labels induced from feature similarity. While effec- tive, these methods rely on heuristic expert specialization rules and often result in cumbersome architectures that hinder adaptation to foundation models, increase training complexity, and limit inference speed. Our work circumvents these limitations through two key innovations: (1) replacing specialized expert design with parallel fine-tuning of foundation models on controlled subsamples, and (2) employing model averaging and EMA instead of complex aggregation mechanisms. This pre-
25

---

serves the ensemble's s variance-reduction benefits while maintaining the original foundation model's architectural simplicity and computational efficiency.
C.2
Model Merging
Model merging, also sometimes referred to to weight averaging, has gained significant attention in recent years promising research direction [32]. focusing on reducing communication costs in federated learning [39] and distributed training [15], enabling the efficient combination of multi- ple models without additional training [22], and enhancing model robustness in of--oistrriuutiion scenarios [60] [4]. Early approaches like Exponential Moving Average (EMA) [54] and Stochastic Weight Averaging (SWA) [23] have been widely adopted accelerate training convergence, stability, and enhance the generalization capabilities of deep neural networks. Recent work extends merging to sequential adaptation: Alexandrov et T mitigates catastrophic forgetting continual pretraining via iterative merging, while Ramé et al. 45 align LLMs through multi-stage averaging during RLHF. To our knowledge, no prior work applies model merging to imbalanced recognition. Unlike existing sequential merging approaches, our framework trains multiple models in parallel on complementary subsampled distributions, a critical design choice for handling long-tailed data. We propose the first schema specifically tailored for imbalance, integrating subsampling (to retain tail-class discriminabil- ity) and bootstrapping (to stabilize head-class representations). This parallelized merging strategy directly addresses feature-space asymmetry in long-tain taigttilled distributions while maintaining compu- tational efficiency, enabling foundation models to adapt to extreme imbalance without sacrificing pre-trained generalization.
D Baselines and implementation details.
We use CLIP with the ViT-B/16 backbone. Following 43 we adopt a prototypical classification head for g. where both features and classifier weights are l2 normaaall and temperature is applied to the logits. The parameters w are initialized by generating text. We use descriptive prompts such as photo of cat" "a photo of a dog" to represent each class 45 for the classes and extracting corresponding textual features using the CLIP text encoder.
We optimize the model using the Adam W optimizer (36). The batch size set to 128, with learning rates of 3e - 4 for both the representation and the classification stage. A cosine decay learning rate scheduler is employed, gradually reducing the learning rate to 0.1 -mar Jr after warmup period spanning max(110) 0.01 rotal_steps) steps. The validation set of each dataset is used to select the used for each dataset. We select V and
best checkpoint. Table I shows the hyperparameters based on the validation set of each dataset and fix ==2 across all experiments. We report all baseline results without test-time augmentation, which offers orthogonal gains.
Table 13:
Dataset details used in our work.
Dataset
Classes
Total samples
Max samples
Min samples
p
T
CIFAR100-LT
100
10.8k
500
100
0.54
TinyImageNet-
30
200
21.5k
500
5
100
0.53
Places-LT
365
62.5k
4980
996
0.55
ImageNet-LT
34
1000
115.8k
1280
5
256
0.62
iNaturalist [56
8.142
437.5k
1000
500
0.11
NIH-CXR-LT
19
20
88,637
53260
12
6491
5.66
D.1 Full results
26

---

Table 14: Comparison of methods for training on CIFARTOO-LT.
Methods
Backbone
Overall
Many
Medium
Few
Training from scratch
LDAM [5]
ResNet-32
42.0
BBN 77
ResNet-32
42.6
DiVE [8]
ResNet-32
45.4
MiSLAS 72
ResNet-32
47.0
BS [46
ResNet-32
50.8
PaCo 10
ResNet-32
52.0
BCL 75
ResNet-32
51.9
Fine-tuning CLIP
Linear Prob (LA)
ViT-B/16
70.0
77.2
71.1
60.4
Full-FT (LA)
ViT-B/16
79.6
88.1
79.9
69.3
cRT 25
VIT-B/16
78.8
89.7
79.7
65.1
PEFT 50
VIT-B/16
81.3
85.2
80.9
77.1
Model Soups 600
VIT-B/16
82.1
89.9
82.2
73.0
LT-Soups (Ours)
ViT-B/16
83.5
88.2
83.5
78.0
Table 15: Comparison methods for training on Places-LT.
Methods
Backbone
Overall
Many
Medium
Few
Training from ImageNet-1K pre-trained backbone
OLTR 34
ResNet-152
35.9
44.7
37.0
25.3
CRT 25
ResNet-152
36.7
42.0
37.6
26.4
LWS 25
ResNet-152
37.6
40.6
39.1
28.6
MiSLAS
72
ResNet-152
40.4
39.6
43.3
36.1
DisAlign
ResNet-152
39.3
40.4
39.4
32,9
ALA
ResNet-152
41.2
36.1
47.9
35.3
PaCo
10
ResNet-152
40.5
33.7
44.4
35.3
LiVT
63
ViT-B/16
40.8
48.1
40.6
27.5
Fine-tuning CLIP
Linear Prob (LA)
ViT-B/16
48.8
48.8
49.7
47.1
cRT 25
VIT-B/16
44.4
51.0
43.1
35.4
BALLAD
ViT-B/16
49.5
49.3
50.2
48.4
Decoder [5
VIT-B/16
46.8
LPT 13
ViT-B/16
50.1
49.3
52.3
46.9
Full-FT (LA)
VIT-B/16
46.6
49.9
46.3
41.4
cRT 25
ViT-B/16
44.4
51.0
43.1
35.4
LIFT [50]
VIT-B/16
51.5
51.3
52.2
50.5
Model Soups 60
ViT-B/16
49.4
51.7
50.0
43.7
LT-Soups (Ours)
ViT-B/16
51.7
51.2
52.8
50.3
27

---

Table 16: Comparison of methods for training on ImageNet-LT.
Methods
Backbone
Overall
Many
Medium
Few
Training from scratch
cRT 25
ResNet-50
47.3
58.8
44.0
26.1
LWS
25
ResNet-550
47.7
57.1
45.2
29.3
MISLAS
7/2
ResNet-50
52.7
62.9
50.7
31.0
LA 40
ResNet-50
51.1
DisAllgn
67
ResNet-50
52.9
61.3
52.2
31.4
BCL 75
ResNet-550
56.0
PaCo
10
ResNet-50
57.0
NCL
31
ResNet-50
57.4
LiVT
63
ViT-B/16
60.9
73.6
56.4
41.0
Fine-tume Fine-tuning CLIP
Linear Prob (LA)
ViT-B/16
74.2
77.8
73.3
67.4
BALLAD 37
VIT-B/16
75.7
79.1
74.5
69.8
Decoder [58
ViT-B/16
73.2
Full-FT (LA)
VIT-B/16
73.9
79.8
71.9
63.9
cRT 25
ViT-B/16
72.6
81.1
70.6
56.1
LIFT 50
VIT-B/16
77.0
80.2
76.1
71.5
Model Soups 60
VIT-B/16
76.0
81.5
74.5
65.5
LT-Soups (Ours)
VIT-B/16
77.4
81.2
76.1
70.7
Table 17: Comparison of methods for training on NIH-CXR-LT.
Methods
Backbone
Overall
Many
Medium
Few
Training from ImageNet- IK pre-trained backbone
cRT (25
ResNet-50
38.0
43.3
37.4
30,0
LWS [25]
5esNet-50
28.0
45.7
23.0
08.3
CB LDAM-DRW 5
5esNet-50
37.7
47.6
35.6
25,0
CB Softmax
|II
ResNet-50
33.3
29.5
41.5
21.7
Fine-tuning CLIP
Linear Prob (LA)
ViT-B/16
17.5
13.3
21.1
16.7
BALLAD RM
ViT-B/16
34.5
36.7
38.9
20.8
Full-FT (LA)
ViT-B/16
38.0
43.8
41.5
20.0
cRT (25]
ViT-B/16
37.7
42.9
39.3
25.0
LIFT 50
ViT-B/16
38.5
43.3
40.4
25.5
Model Soups 60
VIT-B/16
38.0
45.6
40.2
20.0
LT-Soups (Ours)
ViT-B/16
39.3
42.4
40.7
30.8
28

---

Table Comparison methods for training on iNaturalist 2018.
Methods
Backbone
Overall
Many
Medium
Few
Training from scratch
cRT [25]
ResNet-50
65.2
69.0
66.0
63.2
LWS 25
ResNet-50
65.9
65.0
66.3
65.5
MiSLAS 72
ResNet-50
71.6
73.2
72.4
70.4
DiVE 18
ResNet-50
69.1
70.6
70.0
67.7
DisAlign 67
ResNet-50
69.5
69.1
69.9
69.4
ALA [71
ResNet-50
69.6
69.5
70.2
69.0
RIDE [57]
ResNet-50
71.5
72.4
73.1
70.4
RIDE+CR 38
ResNet-50
73.5
74.0
74.3
73.1
RIDE+OTmix 16
ResNet-50
73.7
74.1
75.2
72.8
BCL 75
ResNet-50
71.8
PaCo 10
ResNet-50
73.2
70.4
72.8
75,8
NCL
ResNet-50
74.2
72.0
74.9
73.8
GML 52
ResNet-50
74.5
LiVT
63
ViT-B/16
76.1
78.9
76.5
74.8
Fine-tuning CLIP
Linear Prob (LA)
VIT-B/16
60.4
48.9
60.0
63.9
Decoder 58
VYT-B/16
59.2
LPT 13
VIT-B/16
76.1
79.3
Full-FT (LA)
VIT-B/16
76.1
75.7
76.9
75.3
LIFT 50
VIT-B/16
79.1
72.4
79.0
81.1
Model Soups 600
VIT-B/16
76.4
77.1
76.8
75.6
LT-Soups (Ours)
VIT-B/16
78.2
76.7
78.5
78.2
29


Reviews:
[{"review_id": "UMgQtcDlv1", "reviewer": "Reviewer_snjd", "summary": "In this paper, the authors conduct experiments to show that previous methods for the long-tail problem cannot achieve better performance both on head and tail classes. Therefore, the authors propose to adapt the model soup for the problem by training different models on sub-datasets with different class imbalance ratios and averaging their weights to obtain better representation.", "questions": "Please refer to the weakness part.", "limitations": "Please refer to the weakness part.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The authors identify the problem of head-tail imbalance ratio which is neglected by previous works.\n2. The motivation of the paper is stated clearly by conducting preliminary experiments.\n\nWeakness:\n1. The writing is not proper enough. For example, throughout the paper, how the Logit Adjustment is performed is not mathematically formulated. Although it has been proposed in previous works, rephasing it in the current paper helps others understand it better. The caption of Figure 5 is neglected as well. The momentum coefficient $\\mu$ in line 210 is not defined either. If it is the same to $\\lambda$ above, then the last weight of the averaged model is almost the same as the first model so that the proposed methods is meaningless.\n2. It seems that the proposed method cannot solve the targeted problem. From the experiments, the proposed methods can improve the performance of model soup on tail classes, but the performance on head classes still degrades which is the problem that the authors stated to solve.\n3. The experimental setting is not comprehensive enough. For example, one of the setting of imbalance ratio or head-tail ratio in Table 1 and 2 should be kept when the other one is changing. Moreover, the results illustrated in Figure 2 is not convincing enough because they are averaged on head-tail ratio which will not be able to show the detailed results when the head-tail ratio is high or low.\n4. Can you show how the performance changes when the total number of models changes?", "was_revised": true, "final_justification": "The authors' response solve most of my problems. Considering the simplicity but effectiveness of the proposed method, I recommend acceptance.", "rebuttal": "We thank the reviewer for their feedback and address their concerns in detail below:\n\n---\n> ### **W2. It seems that the proposed method cannot solve the targeted problem. From the experiments, the proposed methods can improve the performance of model soup on tail classes, but the performance on head classes still degrades which is the problem that the authors stated to solve.**\n---\nIt is well established in the class imbalance literature [1,2,3,4] that, given an imbalanced dataset, the goal is to maximize balanced accuracy (see L122–124) on a uniformly distributed test set. This, in turn, requires achieving a well-balanced performance across both head and tail classes.\n\nWhile Model Soups favours head-class performance by fully fine-tuning on the original imbalanced distribution, this improvement comes at the cost of reduced accuracy on tail classes. On the other hand, PEFT limits weight updates to a small subset of parameters, which negatively impacts head-class performance. In contrast, LT-Soups fine-tunes and averages models trained on distributions with progressively increasing imbalance ratios. This strategy not only allows adaptation to head classes but also preserves the balanced representations needed for tail-class generalization. Across all six benchmarked datasets, LT-Soups consistently outperforms Model Soups in overall accuracy (by 1.3 to 2.3 percentage points) and shows substantial improvements in tail-class accuracy (by 2.6 to 10.9 points), with only a modest decrease in head-class performance (up to 3.2 points).\n\n*[1] Cao et al. Neurips 2019.*\n*[2] Ren et al. Neurips 2020.*\n*[3] Menon et al. ICLR 2020.*\n*[4] Wang et al. ICLR 2020.*\n\n---\n> ### **W3. The experimental setting is not comprehensive enough. For example, one of the setting of imbalance ratio or head-tail ratio in Table 1 and 2 should be kept when the other one is changing.**\n---\n\nPlease note that we followed the experimental protocol established in prior benchmarks [1,2]. Particularly, we conduct a comprehensive analysis across six datasets: four synthetic long-tailed distributions, CIFAR100-LT, ImageNet-LT, TinyImageNet-LT, and Places-LT, and two real-world long-tailed distributions, NIH-CXR-LT and iNaturalist2018. Each dataset features a distinct combination of imbalance ratio and head-to-tail ratio, providing a diverse representation of imbalance scenarios.\n\nFurthermore, in Section 3.2 of the paper, we systematically vary the imbalance ratio ($\\rho$) across three levels and the head-to-tail ratio ($\\eta$) across eleven values by synthetically modifying the CIFAR-100 distribution. Each configuration is evaluated across four baselines, including LT-Soups, resulting in a total of 396 training runs. While we expect the trends observed on CIFAR-100 to generalize to other datasets such as TinyImageNet-LT and those in Tables 1 and 2, we are happy to include additional results in the Appendix if the reviewer believes it would strengthen our findings.\n\n[1] Liu et al. CVPR 2019. [2] Shi et al. ICML 2024.\n\n---\n> ### **W4. Can you show how the performance changes when the total number of models changes?**\n---\nIncreasing the total number of models consistently improves overall performance, as it has already been presented in Figure 5b and L296 in the main paper and Table 7 in the Appendix. For the reviewer’s convenience, we summarize these results in the two tables below.\n\nTo recap, the total number of models used in LT-Soups is governed by two key hyperparameters: $N$, the number of subsamples with varying imbalance ratios, and $M$, the number of models trained per subsample. \n\nTable 1 explores the effect of increasing the total number of models by varying $N$, from the most balanced case ($N=1$) to the full dataset ($N=8$) while keeping the number of models per subsample fixed at $M=2$. Table 2 examines the impact of increasing $M$ while holding $N=8$ fixed. Overall, we observe consistent improvements in performance as the total number of models increases. To keep the experiments manageable, we fix $M = 2$ in the main benchmarks.\n\n\n*Table 1: Effect of increasing the total models on performance by varying the number of subsamples ($N$)*\n|                  | N = 1 | N = 2 | N = 3 | N = 4 | N = 5 | N = 6 | N = 7 | N = 8 |\n| ---------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| **Total Models** | 2     | 4     | 6     | 8     | 10    | 12    | 14    | 16    |\n| **All**          | 71.7  | 74.0  | 75.9  | 77.9  | 77.6  | 78.0  | 78.1  | 78.6  |\n| **Head**         | 74.6  | 76.6  | 78.5  | 81.2  | 82.0  | 83.0  | 84.5  | 85.0  |\n| **Tail**         | 70.1  | 72.5  | 74.5  | 76.1  | 75.2  | 75.4  | 74.7  | 75.2  |\n\n*Table 2:  Effect of increasing the total models on performance by varying the number of models trained per subsample ($M$)*\n|                  | M = 1 | M = 2 | M = 12 |\n| ---------------- | ----- | ----- | ------ |\n| **Total Models** | 8     | 16    | 96     |\n| **All**          | 78.2  | 78.6  | 78.8   |\n| **Head**         | 84.8  | 85.0  | 85.5   |\n| **Tail**         | 74.6  | 75.2  | 75.5   |\n\n---\n> ### **W1.1. The results illustrated in Figure 2 is not convincing enough because they are averaged on head-tail ratio which will not be able to show the detailed results when the head-tail ratio is high or low.**\n---\nWe would like to clarify that the analysis requested by the reviewer corresponds to Figure 1 of the main paper. This figure reports the performance of four methods, including LT-Soups, on synthetic CIFAR100 across three levels of imbalance ratio ($\\rho$) and eleven head-to-tail ratio ($\\eta$) values. Figure 2 is derived from the same data by marginalizing over both $\\rho$ and $\\eta$ to emphasize a central insight of our work: an effective LT method must perform well across the full spectrum of imbalance conditions.\n\n---\n> ### **W1.2. Throughout the paper, how the Logit Adjustment is performed is not mathematically formulated. Although it has been proposed in previous works, rephasing it in the current paper helps others understand it better.**\n---\nWe appreciate your suggestion and will revise the description of the Logit Adjustment loss function in Section **3.1** for greater clarity.\n\n---\n> ### **W1.3. The caption of Figure 5 is neglected as well.**\n---\nTo improve clarity, we split the original **Figure 5** into two separate figures, each with its own caption.\n\n---\n> ### **W1.4. The momentum coefficient $\\mu$ in line 210 is not defined either. If it is the same as $\\lambda$ above, then the last weight of the averaged model is almost the same as the first model, so that the proposed method is meaningless.**\n---\nWe use the following formulation for EMA:\n\n$\\theta_{ema} = (1-\\mu) \\cdot \\theta_{ema} + \\mu \\cdot \\theta_0,$ \n\nwhere $\\theta_0$ is the initialization weights and $\\theta_{ema}$ denotes the fine-tuning parameters. We fixed $\\mu=0.01$ throughout all the experiments. Therefore, the last weight of the averaged model is clearly not the same as the first model. We will clarify this in the revised manuscript."}, {"review_id": "CDZZoEy6JU", "reviewer": "Reviewer_Q5RH", "summary": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "questions": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "limitations": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "was_revised": true, "final_justification": "Please ignore my rating. I still don't understand why I have to go through this even though i notified early on that i would not able to perform my reviews this year.", "rebuttal": ""}, {"review_id": "5siC0kUHZJ", "reviewer": "Reviewer_9QKC", "summary": "This paper addresses the trade-off between head- and tail-class performance when fine-tuning large pre-trained models on long-tailed (LT) datasets. The authors begin by introducing a new metric, the head-tail ratio (η), which they use alongside the standard imbalance ratio (ρ) to more comprehensively characterize the structure of long-tailed distributions. Through experiments, they show that Parameter-Efficient Fine-Tuning (PEFT) methods excel in tail-heavy scenarios, while full fine-tuning is more advantageous when head classes are dominant.\n\nTo resolve this issue, the authors propose LT-Soups, a two-stage model merging framework. In the first stage, instead of training models on the entire imbalanced dataset, LT-Soups creates a series of data subsets with progressively increasing imbalance ratios. It fine-tunes a full model on each subset and then merges the weights of these \"specialist\" models via recursive weighted averaging to form a unified feature extractor. In the second stage, to recover head-class information potentially lost during subsampling, the framework freezes the merged backbone and retrains only the classifier layer on the full training set. Experimental results across several synthetic and real-world long-tailed datasets demonstrate that LT-Soups achieves a superior performance balance compared to PEFT and traditional model soups across diverse long-tailed regimes.", "questions": "None", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper's analysis of the head-tail performance trade-off is insightful. The introduction of the head-tail ratio (η) as a new analytical dimension provides a more fine-grained framework for understanding the complexities of long-tailed recognition.\n\n2. The paper's experimental section is robust. The authors validate their method on a wide range of standard benchmarks and use well-designed ablation studies to demonstrate the contribution of each component. The visualizations effectively support the core claim that different methods are optimal for different imbalance structures.\n\n**Weaknesses**\n1. While LT-Soups is an effective framework, its core ideas are more of an intelligent combination of existing techniques—Model Soups, data subsampling, and two-stage training—rather than a fundamentally new concept. The innovation lies in the specific recipe, which, while successful, may not meet the high bar for conceptual novelty at a top-tier venue.\n\n2. The framework requires training N×M models in the first stage. Although parallelizable and trained on smaller subsets, the total computational budget is significantly higher than for single-run methods like PEFT or Full-FT. A direct comparison of total GPU hours is missing, making it difficult to assess the practical cost-benefit trade-off.\n\n3. The recursive, ordered merging strategy is shown empirically to be superior to alternatives. However, the paper lacks a deep, intuitive explanation for why this curriculum-like fusion (from balanced to imbalanced) is the optimal approach. It remains unclear if this is a robust principle or an empirical finding specific to this setup.\n\n4. Lack of discussion on recent literature. For ensemble long-tail recognition: Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition, Mdcs: More diverse experts with consistency self-distillation for long-tailed recognition. For recent literature: LTRL: Boosting Long-tail Recognition via Reflective Learning, DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge, Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition.", "was_revised": true, "final_justification": "I thank the authors for their detailed and timely response. After reviewing their feedback, I believe that most of my concerns have been adequately addressed. I have therefore decided to raise my original rating, leaning toward acceptance.", "rebuttal": "We thank the reviewer for their thoughtful feedback and encouraging comments on our analysis, rigorous methodology, and well-designed ablation studies. We address their questions below:\n\n---\n> ### **W1. While LT-Soups is an effective framework, its core ideas are more of an intelligent combination of existing techniques.**\n---\nWe emphasize that our contribution goes beyond simply proposing a final model. Specifically:\n\n1. ***A new lens for analyzing long-tailed learning:*** We introduce the head-to-tail ratio ($\\eta$) as a complementary axis to the standard imbalance ratio ($\\rho$), offering a more comprehensive view of class imbalance. We argue that effective long-tailed methods must perform well across the full ($\\rho$, $\\eta$) spectrum—*a perspective that enhances understanding of real-world imbalance*. Through this lens, we examine state-of-the-art methods such as a very strong competitor, LIFT [1], which advocates parameter-efficient fine-tuning (PEFT) for long-tailed tasks, and Model Soups [2], which uniformly average fully fine-tuned models on the full imbalanced dataset. This analysis reveals performance trade-offs that are often have overlooked in conventional evaluations.\n2. ***LT-Soups (a principled adaptation of Model Soups):*** Building on these insights, we propose LT-Soups, which adapts the strengths of Model Soups to long-tailed settings. Model Soups boosts head-class performance but significantly degrades tail-class accuracy due to overexposure to head-heavy distributions. PEFT, on the other hand, limits parameter updates and struggles to adapt to head classes. LT-Soups addresses both issues by fully fine-tuning models on subsets with progressively increasing imbalance and combining them via recursive weight averaging. This balances head-class adaptation with tail-class generalization\n3. ***Comprehensive evaluation:*** Across six benchmark datasets, LT-Soups outperforms PEFT in overall accuracy (by 1.3 to 2.3 percentage points) on five of them, while achieving comparable tail-class performance. It also consistently surpasses Model Soups in overall accuracy (by 1.3 to 2.3 points) and yields substantial gains in tail-class accuracy (by 2.6 to 10.9 points), with only a modest drop in head-class performance (up to 3.2 points).\n\n*[1] Shi et al. ICML 2025.*\n*[2] Wortsman et al. ICML 2022.*\n\n---\n> ### **W2. A direct comparison of total GPU hours is missing, making it difficult to assess the practical cost-benefit trade-off.**\n---\nThe full computational analysis, including total wall-clock time for the parallel version of LT-Soups, is already provided in Appendix B. In response to the reviewers’ request, we also report the total GPU hours, assuming access to only a single GPU with modest memory.\n\nTo recap, LT-Soups consists of two stages: Stage 1 trains a series of specialist models on subsampled datasets with varying imbalance ratios. Stage 2 involves retraining only a linear classifier on the full dataset, which is computationally lightweight. Both stages use the same number of epochs.\n\nTables 1 and 2 report the Stage 1 fine-tuning time per subsample (each $\\rho$ corresponds to one subsample) and the total GPU hours when LT-Soups is trained on a *single GPU* for the ImageNet-LT and NIH-CXR-LT datasets. As we can see, the computational cost of LT-Soups depends primarily on the degree of imbalance in the dataset. For example, in ImageNet-LT (which comes with \\rho=256), following our subsampling strategy, the total absolute GPU time is ~8 hours. LT-Soups is costly in this case because with our largest \\rho=64, it includes 80% of the entire training data. In contrast, in NIH-CXR-LT ($\\rho = 6401$), the largest subset ($\\rho = 256$) includes only 24% of the data, ***resulting in substantial savings, up to 18x lower GPU hours compared to Model Soups of the same size.***\n\n*Table 1: Computational cost per ImageNet-LT subset for LT-Soups with $M=2$.*\n| $\\rho$        | 1       | 2       | 4       | 8       | 16      | 32      | 64      | **Total GPU hours** |\n| ------------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| **GPU Hours** | 0:12:19 | 0:18:44 | 0:32:00 | 0:54:00 | 1:26:32 | 2:02:00 | 2:31:09 | 7:56:50 |\n\n\n*Table 2: Computational cost per CXR-LT subset for LT-Soups with $M=2$.*\n| $\\rho$        | 1       | 2       | 4       | 8       | 16      | 32      | 64      | 128     | 256     | **Total GPU hours** |\n| ------------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| **GPU Hours** | 0:01:01 | 0:01:05 | 0:01:21 | 0:02:38 | 0:04:06 | 0:06:16 | 0:10:26 | 0:16:46 | 0:25:04 | 1:08:12 |\n\nWe conclude by referencing Table 9 in the Appendix, which compares the total computational cost of LT-Soups and Model Soups *(both with parallelization)* against other baselines on ImageNet-LT and NIH-CXR-LT. *Notably, although LT-Soups is a full-rank method, it converges within 10 epochs on CXR-LT, while LoRA—despite its parameter efficiency—requires 50 epochs, resulting in longer wall-clock time and higher total GPU hours*. Thus, in response to the reviewer’s concern, the claim that “the total computational budget is significantly higher than single-run methods” does not always hold; LT-Soups often reduces computational cost through subsampling, with the budget depending on the choice of $N$ (Tab. 8 Appendix).\n\n| **Method**       | **Wall-clock time (H\\:M\\:S)** | **Epochs** |\n| ---------------- | ----------------------------- | ---------- |\n| **ImageNet-LT**  |                               |            |\n| Full-FT          | 1:37:56                       | 10         |\n| Model Soups      | 1:37:56                       | 10         |\n| LoRA (rank=64)   | 1:25:33                       | 10         |\n| LT-Soups         | 1:45:38                       | 10         |\n| **CXR-LT**       |                               |            |\n| Full-FT          | 0:53:43                       | 10         |\n| Model Soups      | 0:53:43                       | 10         |\n| LoRA (rank=64)   | 2:14:32                       | 50         |\n| LT-Soups         | 0:32:17                       | 10         |\n\n---\n> ### **W3. The paper lacks a deep, intuitive explanation for why this curriculum-like fusion (from balanced to imbalanced) is the optimal approach.**\n---\nWe believe our recursive weight averaging (WA) is a principled approach, as it enables an effective trade-off between stronger adaptation—potentially biased toward head classes—and more balanced generalization. It can be interpreted as an exponential moving average (EMA) over fine-tuned models sorted by increasing imbalance severity, with a tunable parameter that adjusts the influence of more balanced (but smaller) versus less balanced (but larger) subsets. In contrast, uniform WA applies a simple arithmetic mean, giving equal weight to all models regardless of their imbalance level.\n\nIn all of our experiments in the paper, we use only two values for $\\lambda$: 0.3 and 0.7, corresponding to high and low adaptation needs, respectively. Intuitively, when the target dataset is close to the pre-training weights, the value of the $\\lambda$ becomes less important as even small datasets are enough for adaptations. However, when the shift becomes larger, subsets with more data (albeit biased towards head classes) become crucial.\n\nThe table below confirms our hypotheses. In particular, we compare recursive WA and uniform WA across two datasets with different similarities compared to CLIP-pretrained weights (according to the zero-shot performance). On ImageNet-LT, which is already well-aligned with CLIP-pretrained features, there is little to no difference between the two averaging schemes. However, for datasets that require significant adaptation [1], such as iNaturalist2018, recursive WA yields clear benefits by leveraging information from more data-rich subsets.\n\n\n*Table 1: Performance comparison under uniform and recursive WA across three datasets.*\n| Method      | **TinyImageNet-LT** |      |      |      | **iNat2018** |      |      |      |\n| ----------- | ------------------- | ---- | ---- | ---- | ------------ | ---- | ---- | ---- |\n|             | All                 | Many | Med. | Few  | All          | Many | Med. | Few  |\n| **Uniform** | 78.5                | 83.4 | 78.4 | 72.9 | 74.7         | 67.4 | 75.8 | 75.3 |\n| **Ours**    | 78.6                | 85.0 | 78.3 | 71.5 | 78.2         | 76.7 | 78.5 | 78.2 |\n\n---\n> ### **W4. Lack of discussion on recent literature.**\n---\nThank you for the suggestions. Given that our focus was on using pre-trained CLIP models for LT recognition, we have restricted the comparison to the recent literature that uses CLIP for LT recognition. In response to the reviewer’s request, we have expanded the related work section and now provide a brief comparison of LT-Soups with the methods mentioned.\n\nSADE [1], Mdcs [2], and DirMixE [5] are Mixture-of-Experts (MoE) methods that aggregate diverse experts trained with different logit adjustment (LA), targeting distributions like uniform, long-tail, and inverse long-tail. Unlike these methods, which require deploying all experts at inference, LT-Soups uses weight averaging to collapse models into a single, inference-efficient solution. Moreover, as noted in [6], fully fine-tuning foundation models like CLIP with LA loss alone is insufficient, as it can lead to inconsistent class-conditional distributions, particularly for tail classes.\n\nReflective Learning [3] promotes consistency across training iterations by minimizing KL divergence between predictions and soft labels induced from feature similarity. In this light, the EMA mechanism in LT-Soups can be viewed as a lightweight form of Reflective Learning or self-distillation [7].\n\n*[1] Zhang et al., NeurIPS 2022.*\n*[2] Zhao et al., ICCV 2023.*\n*[3] Zhao et al., ECCV 2024.*\n*[5] Yang et al., ICML 2024.*\n*[6] Shi et al., ICML 2024.*\n*[7] Allen-Zhu et al., ICLR 2023.*"}, {"review_id": "2wytpnfUbU", "reviewer": "Reviewer_tBqZ", "summary": "They authors propose LT-Soups, a two-stage procedure, where:\n\n1. Progressive subsampling + model-averaging. Multiple CLIP copies are fully fine-tuned (in parallel) on subsets whose imbalance ratios grow geometrically, then exponentially averaged to couple \"tail-friendly\" and \"head-friendly\" specialists\n\n2. Classifier re-tuning. Only the final linear head is re-trained on the full data with logit adjustment to regain head-class sharpness", "questions": "1. i'd like to see some small ablations if the authors can provide these :)\n- Performing PEFT, followed by performing the same final layer only tuning with logit adjustment that is performed in LT-Soups -- would this help in closing gap in head accuracy for PEFT ?\n- Performing Model Soups as done in Wortsman et al. [2022], followed by performing the same final layer only tuning\n\nAbove two ablations would tell if its the new model soup strategy which is driving the gains or the final logit adjustment.\n\n\n2. perhaps one more thing to test would be:\n- take a subsampled dataset that has the least imbalance, perform vanilla model soups (Wortsman et al. [2022]) on it or PEFT, followed by final layer only tuning -- this ablation would tell us that does one need multiple models at different levels of imbalance in the dataset, or does the least imbalanced dataset is enough\n\nlet me know if these make sense.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- it's a very simple approach\n- it's efficient since it can be parallelized\n- consistently outperforms PEFT and classical model-soup baselines on overall, head and tail accuracies\n- i really admire that the authors released code in the supplementary\n\n\nWeakness:\n- although the method works well, it's currently hard for me to pinpoint exactly where are the gains coming from -- look at questions section, some ablations will help disentangle the multiple components of the method", "was_revised": true, "final_justification": "i'd say the paper is borderline accept.\n\nReasons for not higher score:\n- the novely is a little limited, in the sense, the method seems like a mix-and-match of existing techniques -- this is not a bad thing per say. But mix-and-match in a very specific way adds to the complexity of the method. Overall method is still simple, but its not \"aesthetic\".\n\nReasons for not lower score:\n- authors provide extensive experiments\n- authors did perform the clarifying experiments during rebuttal which helped in understanding\n- authors released their code during submission", "rebuttal": "We thank the reviewer for their insightful feedback and for recognizing the simplicity and effectiveness of our work. Our responses are below:\n\n---\n> ### **Q1. Performing PEFT and Model soups, followed by performing the same final layer only tuning with logit adjustment that is performed in LT-Soups -- would this help in closing gap in head accuracy for PEFT/Model soups ?**\n---\nWe found that additional final-layer tuning with logit adjustment on PEFT and Model Soups has little to no effect. The table below summarizes the results on TinyImageNet-LT. We hypothesize that, unlike these baselines, LT-Soups does not fully exploit the entire training set, due to the downweighting effect introduced by weight averaging. Consequently, fine-tuning the final layer helps LT-Soups recover head-class sharpness and improves overall performance.\n\n*Table 1: Comparison of baselines with and without Classifier Re-training*\n| **Method**                           | **All** | **Head** | **Tail** |\n| ------------------------------------ | ------: | -------: | -------: |\n| PEFT                                 |    77.1 |     83.0 |     73.9 |\n| PEFT + Classifier re-training        |    77.0 |     83.0 |     73.8 |\n| Model Soups                          |    77.6 |     85.9 |     73.0 |\n| Model Soups + Classifier re-training |    77.6 |     85.5 |     73.4 |\n| LT-Soups Stage 1 (without Classifier re-training)  |    78.1 |     84.9 |     74.5 |\n| LT-Soups                             |    78.6 |     85.0 |     75.2 |\n\n---\n> ### **Q2. Take a subsampled dataset that has the least imbalance, perform vanilla model soups on it or PEFT, followed by final layer only tuning -- this ablation would tell us that does one needs multiple models at different levels of imbalance in the dataset, or does the least imbalanced dataset is enough**\n---\nTable 3 in the main paper has already addressed this question in the context of Model Soups. Per your request, we extend the same experiment to the PEFT baseline, with full results shown in the table below.\n\nSpecifically, we compare LT-Soups with Model Soups and PEFT baselines that follow the same two-stage framework as LT-Soups. The only difference lies in the first stage: PEFT trains a single model on a subset with a fixed imbalance ratio, while Model Soups averages 16 models, all trained on subsets sharing the same imbalance ratio.\n\nResults show that performance varies significantly depending on the imbalance ratio used. For instance, Model Soups with $\\rho = 8$ yields the highest tail accuracy (75.0), while using the full dataset ($\\rho = 100$) results in the highest head accuracy (85.9). A similar trade-off is observed with PEFT. However, under any single imbalance setting, both baselines fall short of LT-Soups, which achieves performance of 78.6 (overall), 85.0 (head), and 75.2 (tail). \n\nRather than optimizing for a single point on the imbalance spectrum, LT-Soups averages across subsets with varying imbalance levels. This enables it to integrate the strengths of both low- and high-$\\rho$ models, resulting in a more balanced head–tail trade-off overall.\n\n\n| Imbalance ratio ($\\rho$) | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 100 |\n| ------ | - | - | - | - | -- | -- | -- | --- |\n**Model Soups**\n| All | 71.7 | 75.9 | 76.0 | 77.2 | 77.2 | 77.3 | 77.9 | 77.6 |\n| Head | 74.6 | 78.6 | 78.7 | 81.0 | 82.8 | 84.7 | 85.5 | 85.5 |\n| Tail | 70.1 | 74.4 | 74.6 | 75.0 | 74.1 | 73.3 | 73.7 | 73.4\n**PEFT**\n| All | 73.9 | 74.3 | 74.8 | 76.2 | 76.4 | 77.1 | 77.0 | 77.0 |\n| Head | 75.9 | 75.8 | 77.0 | 78.2 | 80.4 | 81.8 | 81.9 | 83.0 |\n| Tail | 72.8 | 73.5 | 73.6 | 75.1 | 74.2 | 74.5 | 74.4 | 73.8 |"}, {"review_id": "pVZaYABLyD", "reviewer": "Reviewer_g8Rp", "summary": "The paper introduces LT-Soups, a two-stage model soups framework designed to address long-tailed (LT) class imbalance in machine learning. It focuses on the problem where head classes dominate, and tail classes are severely underrepresented, a scenario common in real-world datasets. Previous work on parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer show improved tail-class performance but often at the cost of head-class accuracy. The authors propose LT-Soups, which averts this trade-off by combining model soups and tailored fine-tuning techniques. The framework works in two stages:", "questions": "Real-World Generalization: How does LT-Soups perform when tested on highly skewed datasets or under conditions with noisy labels or missing data? Would the approach still provide strong results?\n\nStatistical Robustness: The paper lacks statistical significance measures such as error bars. Could you provide additional experiments or statistical tests to support the stability of your findings across multiple runs or data splits?\n\nAlternative Subsampling Strategies: Have you considered experimenting with more advanced or adaptive subsampling techniques, such as those based on active learning or dynamic class balancing during training? Could they offer better performance than the current progressive subsampling strategy?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nOriginality: The introduction of LT-Soups as a two-stage framework that balances performance across both tail and head classes is novel. It builds on the understanding that class imbalance is a complex, multi-dimensional problem and uses a dual-axis framework for imbalance characterization.\nRobustness: The method shows significant improvement across both synthetic and real-world long-tailed datasets, including scenarios with varying imbalance ratios and head-tail class distributions.\nPractical Application: The proposed method shows better adaptability and performance stability across diverse datasets, making it applicable to real-world tasks, especially those with imbalanced data distributions.\n\nWeaknesses:\nComplexity: While the approach is robust, it involves two stages of fine-tuning, which could increase computational costs and complexity compared to simpler methods like PEFT or traditional model soups.\nLimited Statistical Analysis: The paper lacks error bars or statistical significance tests in the experimental results, which could make it difficult to assess the variability and robustness of the reported results fully.\nAssumptions: The reliance on the head-tail ratio and imbalance ratio, while insightful, might oversimplify the complexity of real-world class distributions. A more flexible or generalized model could be explored to account for other potential distributional shifts.", "was_revised": true, "final_justification": "The authors’ response addressed my concerns, and I will maintain my positive rating in the final evaluation.", "rebuttal": "We thank the reviewer for their positive feedback on the originality, robustness, and practical applicability of our method. Below, we address their concerns in detail.\n\n---\n> ### **Statistical Robustness: The paper lacks statistical significance measures such as error bars. Could you provide additional experiments or statistical tests to support the stability of your findings across multiple runs or data splits?**\n---\nPlease note that we followed the experimental protocol established in prior benchmarks [1,2]. While it is not feasible to run statistical tests across all datasets within the rebuttal timeline, we report results on TinyImageNet-LT using three different random seeds in the table below. As shown, the performance variance is negligible, supporting the robustness of our method. If the reviewers believe it would add value, we would be happy to include statistical tests on the remaining datasets in the final version of the paper.\n\n| **Method**  | **All**     | **Head**    | **Tail**    |\n| ----------- | ----------- | ----------- | ----------- |\n| PEFT        | 77.0 ± 0.31 | 82.7 ± 0.44 | 73.8 ± 0.38 |\n| Model Soups | 77.5 ± 0.04 | 86.0 ± 0.04 | 72.9 ± 0.03 |\n| LT-Soups    | 78.5 ± 0.16 | 85.4 ± 0.4  | 74.6 ± 0.15 |\n\n*[1] Liu et al. CVPR 2019.*\n*[2] Shi et al. ICML 2024.*\n\n---\n> ### **Real-World Generalization: How does LT-Soups perform when tested on highly skewed datasets or under conditions with noisy labels or missing data? Would the approach still provide strong results?**\n---\nPlease note that two of the benchmark datasets—iNaturalist2018 and NIH-CXR-LT—are real-world datasets with naturally long-tailed distributions. They are characterized by markedly different values of imbalance ratio ($\\rho$) and head-to-tail ratio ($\\eta$): $\\rho=500$, $\\eta=0.11$ for iNaturalist2018 and $\\rho = 6491$, $\\eta = 5.66$ for NIH-CXR-LT. While incorporating additional distribution shifts, such as noisy labels or missing data, is a promising direction for future work, we follow prior work in the long-tailed learning literature [1,2] and focus solely on the imbalance aspect.\n\n*[1] Liu et al. CVPR 2019.*\n*[2] Shi et al. ICML 2024.*\n\n---\n> ### **Alternative Subsampling Strategies: Have you considered experimenting with more advanced or adaptive subsampling techniques, such as those based on active learning or dynamic class balancing during training? Could they offer better performance than the current progressive subsampling strategy?**\n---\nThank you for the suggestion. We agree that more advanced subsampling strategies, such as selecting samples based on information content (e.g., based on entropy), could potentially improve performance and represent a promising direction for future work. However, in this study, we prioritize simplicity to keep our core message focused and interpretable. Notably, even our basic subsampling strategy already outperforms existing baselines, demonstrating the effectiveness of our overall approach."}]

Author rebuttal:

Meta-review: Despite some concerns about conceptual novelty and computational efficiency, the paper makes a meaningful and well-validated contribution to long-tailed recognition with foundation models. The introduction of the head-tail ratio as an analytical tool and the empirical success of LT-Soups across diverse imbalance regimes provide strong reasons for acceptance. The authors’ detailed rebuttal and additional experiments effectively addressed reviewer concerns, and multiple reviewers raised their scores accordingly. The Author should address all reviewers' concerns in their final version.
Decision: Accept (poster)