Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Authors: Zhe Zhao
HaiBin Wen
Pengkun Wang
ShuangWang
Zhenkun Wang
Qingfu Zhang
Yang Wang
Venue: ICML 2025 poster (2025)
Keywords: Neural network pruning，Long-tail learning
Primary area: general_machine_learning->representation_learning

Abstract: Long-tailed distribution datasets are prevalent in many machine learning tasks, yet existing neural network models still face significant challenges when handling such data. This paper proposes a novel adaptive pruning strategy, LTAP (Long-Tailed Adaptive Pruner), aimed at balancing model efficiency and performance to better address the challenges posed by long-tailed data distributions. LTAP introduces multi-dimensional importance scoring criteria and designs a dynamic weight adjustment mechanism to adaptively determine the pruning priority of parameters for different classes. By focusing on protecting parameters critical for tail classes, LTAP significantly enhances computational efficiency while maintaining model performance. This method combines the strengths of long-tailed learning and neural network pruning, overcoming the limitations of existing approaches in handling imbalanced data. Extensive experiments demonstrate that LTAP outperforms existing methods on various long-tailed datasets, achieving a good balance between model compression rate, computational efficiency, and classification accuracy. This research provides new insights into solving model optimization problems in long-tailed learning and is significant for improving the performance of neural networks on imbalanced datasets. The code is available at https://github.com/DataLab-atom/LT-VOTE.

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed
Data
Zhe Zhao 12
HaiBin Wen
Pengkun Wang 13
Shuang Wang' Zhenkun Wang" Qingfu Zhang?
Yang Wang 135
Abstract
1. Introduction
Long-tailed learning aims to address the problem of highly imbalanced class distributions, where most classes (i.e., tail classes) have scarce samples, while few classes (i.e., head classes) have abundant samples (Liu et al., 2019; Wang et al., 2017; Tan et al., 2020; Li et al., 2020). This data distribution is prevalent real-world applications, such as e-commerce product classification, speech recognition, and natural lan- guage processing (Ouyang et al., 2016; Yang & Xu, 2020). Although deep learning models perform excellently on head classes, their performance on tail classes remains limited, mainly because models tend to overfit head classes and ne- glect feature learning for tail classes, leading to insufficient overall model generalization (Kang et al., 2019).
Long-tailed distribution datasets are prevalent in many machine learning tasks, yet existing neu- ral network models still face significant chal- lenges when handling such data. This paper pro- poses a novel adaptive pruning strategy. LTAP (Long-Tailed Adaptive Pruner), aimed balanc- ing model efficiency and performance to better address the challenges posed by long-tailed data distributions. LTAP introduces multi-dimension:. importance scoring criteria and designs a dynamic weight adjustment mechanism to adaptively de- termine the pruning priority of parameters for dif- ferent classes. By focusing on protecting param- eters critical for tail classes, LTAP significantly enhances computational efficiency while main- taining model performance. This method com- bines the strengths of long-tailed learning and neural network pruning, overcoming the limita- tions of existing approaches in handling imbal- anced data. Extensive experiments demonstrate that LTAP outperforms existing methods vari- ous long-tailed datasets, achieving good balance between model compression rate, computational efficiency, and classificattin accuracy. This re- search provides new insights into solving model optimization problems in long-tailes learning and is significant for improving the performance of neural networks imbalanced datasets. The code is available at https:/ github.com. DataLab-atom/ LT-VTTT
improve long-tailed learning performance, researchers have proposed various methods, including multi-expert sys- tems and modular designs (Wang et al., 2020; Liu et al., 2019). However, these traditional methods face numerous challenges in practical applications. For instance. multi- expert systems often require training and maintaining mul- tiple independent sub- model, resulting in enormous com- putational and storage resource consumption (Xiang et al., 2020). Modular designs rely on predefined module struc- tures, lacking dynamic adaptability and struggling to cope with constantly changing data distributions (Ren et al., 2020). Moreover, these methods often struggle to efficiently utilize parameters when dealing with tail classes, limiting model performance on scarce data (Zhang et al., 2021a).
As a model compression and optimization technique, prun- ing optimizes model structure and improves computational efficiency by removing redundant or unimportant neurons connections (Han et al., 2015; Liu et al., 2018; Zhu & Gupta, 2017). In recent years, pruning methods have shown signifi- cant advantages in improving model performance, reducing parameter counts, and accelerating inference (Frankle & Carbin, 2018: Blalock et al., 2020). However, ordinary pruning methods face special challenges when applied to long-tailed learning:
Eqqual contribution 'University of Science and Technology of China, Hefei, China 'City University of Hong Kong, Hong Kong "Suzhou Institute for Advanced Research, USTC, Suzhou, China "Southern University of Science and Technology, Shenzhen, China Annhi Provincial Key Laboratory of High Performance Computing, Hefei, China. Correspondence to: Pengkun Wang <pengkun @ ustc. aasc.edu.cn2>. Yang Wang <angyan ustc.edu.cn.>
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Pruming bias due to class imbalance: conventional pruning methods, blind to class-specific contributions,

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
risk exacerbating the very imbalance they aim ad- dress by inadvertently removing neurons crucial for tail class recognition (He et al., 2021a).
2. LTAP: Adaptive Pruner for Long-tailed Distribution
We propose LTAP, novel pruning framework that adap- tively protects crucial parameters while achieving efficient model compression. In this section, we first formalize the problem setup, then present our multi-criteria importance evaluation framework, followed by our dynamic weight adjustment mechanism and progressive pruning strategy.
Difficulty in dynamic adjustment: the static nature of traditional pruning methods conflicts with the dy- namic evolution of the association between parameters and data distribution during training, potentially lead- ing to suboptimal or even harmful pruning decisions (Molchanov et al., 2019).
2.1. Problem Formulation
Single evaluation criterion: the reliance on simplistic pruning criteria fails to capture the nuanced importance of neurons in the complex landscape of long-tail tail dis- tributions, potentially misleading the pruning process (Frankle et al., 2020).
Consider a deep neural network fo X - parameterized by 8, trained on a long-tailed dataset The number of samples per class in D follows power law distribution, leading to significant imbalance between head and tail classes. The network parameters are orga- nized into groups G = (91..... 9Mr,, correspond to a layer, filter, or other structural components.
{(z, y))
where each group may
To address these challenges, this paper proposes novel pruning strategy called Long-Tailed Adaptive Pruner (LTAP), specifically optimized for long-tailed learning envi- ronments. LTTPP is rooted in the understanding that effective long-tailed learning requires a fundamental rethinking of how we allocate and utilize model capacity (Kang et al., 2019). Our method makes innovative contributions in the following aspects:
Our objective is to identify a binary mask m E {0, 13110 that: I. Achieves a target sparsity ratio Tooaal 2. Maintains or improves model performance, especially tail classes 3. Preserves essential parameter interactions across different network components
This presents a constrained optimization problem:
Through the LT-Vote (Long- Tailed Voting) mechanism, we dynamically adjust the weights of different pruning criteria based on the classi- fication accuracy of different classes, enabling the prun- ing process to more specifically optimize the learning performance of tail classes, enhancing model robust- ness long-tail -ail distribution data.
New LT-Vote mechanism:
min L(Jo0(1-m)) D)
s.t. ||m||| nonal 10 Ae (e 2 Ter V
(I)
where A. represents the accuracy on class c, and To is class-specific performance threshold.
Multi-stage dynamic pruning: Our method divides the pruning process into multiple stages, gradually remov- ing redundant parameters, and dynamically adjusts the pruning strategy at each stage based on current model performance, ensuring continuous performance opti- mization during the pruning process.
2.2. Multi-criteria Importance Evaluation
Traditional pruning methods often rely on singular criteria such as weight magnitude or gradient- based importance. However, in long-tailed scenarios, different criteria may be more relevant for different classes or network regions. We propose a comprehensive importance scoring framework that integrates multiple complementary perspectives:
Efficient resource utilization: By reducing model pa- rameter count and computational requirements through pruning, we improve model operational efficiency in resource-constee constraied environments while maintain- ing or even improving classification accuracy on tail classes.
Definition 1 (Parameter Group Score). For each param- eter group E G. we define its importance score as a weighted combination of multiple criteria:
K
K
Sy
as
Sg.ks
E R
Σ
a aa=1
=1
(2)
Experimental results show that our proposed LTAP method significantly outperforms traditional pruning methods and other long-tailed learning methods on multiple long-tailed distribution datasets, validating its effectiveness in enhanc- ing tail class recognition ability, optimizing model structure, and improving computational efficiency.
The scoring criteria can be categorized into two main fami- lies:
1. Magnitude-based Criteria:
These capture the static
2

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
importance of parameters:
2.4. Progressive Multi-stage Pruning Strategy
To ensure stable model compression while protecting tail- class performance, LTAP employs progressive pruning strategy that distributes the overall pruning ratio "tocal across P stages:
gg.mag
Wgll2 E R4
(absolute magnitude)
Wgll2
E R+
(normalized magnitude)
gvagg-mag
ng
2. Gradient-based Criteria: These reflect the dynamic importance during training:
Tootl
7/p =
where Tp denotes the pruning ratio for each stage. The pruning process consists of four key steps:
Wg. Vw,4
(update alignment)
g,00s W V w, fl2
Sg tayloor--1
Vw, CI- |wyl
(first-order impact)
1. Compute importance scores Sn for each parameter group E G based on current model state
taylorr =
Vo, CI w
(second-order stability)
(4)
2. Select redundant parameter groups R {9 S. I Streabood, |R|/|G| 7p
Each criterion provides unique insights: magnitude-bart based criteria identify dominant parameters, while gradient-bast
Zero out weights of redundant groups: 0g + 0g o (1 my) for g E R
criteria capture training dynamics and optimization geome-
try. The cosine similarity term $g,00s specifically measures how well parameter updates align with the current optimiza- tion trajectory, helping identify parameters that consistently contribute to learning.
Update remaining parameters according to the pruned structure
The optimization process alternates between training and pruning:
2.3. Dynamic Weight Adjustment Mechanism
A key innovation in LTAP is its adaptive weighting scheme that dynamically adjusts the importance of different criteria based on class-specific performance. This ensures that pa- rameters crucial for tail classes are properly evaluated and protected.
0(++1) = = -/VoL((()))
(8)
where g(t) represents model parameters at step f, is the learning rate, and the loss function. During parameter updates, LTAP incorporates both gradient information and pruning masks:
Definition 2 (Class-Criteria Interaction). We model the interaction between classes and criteria through matrix multiplication:
1. = DN DERKXC NN C a softmax(JJ) ER
V SgS,
Σ Ogk V Vg5g.
(5)
(9)
0g + fg 11 S
0g 0g o (l)m))
mg)
Here, D represents a learnable criteria weight matrix that captures the effectiveness of each criterion for each class, and N₂ is the class distribution vector that encodes the long- tailed nature of the dataset.
where VgS, combines K different importance criteria weighted by ag, and mg E {0, 13%, is the pruning mask. The LT- Vote mechanism dynamically adjusts {ox] based on validation performance to protect tail-class pa- rameters.
The weights are updated through a performance-driven mechanism:
ft Dd +B ((11))
(6)
3. Theoretical Analysis: Tail Classes Benefit More from Overparameterization
This update rule has several important properties: 1. It strengthens criteria that lead to improved class performance maintains separate criterion preferences for differ- ent classes It naturally adapts to the learning dynamics throughout training
To verify the effectiveness and soundness of our method, we first establish a series of foundational definitions. Then, through lemmas and theorems, we systematically argue that tail classes in long-tailed distributions have higher require- ments for model overparameterization. Based on this, we
3

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Masking target
Long-taie Long-tailed Dataset
Model Parameters Weight Selection
0:25
Mask Generation
0.25
Updating
0.25
Pruning
0.25
Augmentation
Select
Parameters' Weight
Guided
Model
Normalization
Magnitude
magaitude
Evaluating
Cosine similarity
Processing
Taylor first order
fe 6
Taylor second order
Neural Networks
Model Parameters,
Criteria Weights
Accuracy,
Accuracy,
Result
Val Accuracy
Figure 1. Overview of the Long-Tailed Adaptive Pruner (LTAPP) methodology. The diagram illustrates the iterative process of model evaluation, parameter processing, and pruning, highlighting the integration of multiple importance criteria and the dynamic weight adjustment mechanism.
propose differentiated parameter allocation strategy and the tail-biased pruning proposition. Finally, we synthesize these theoretical results and prove the effectiveness of the tail-biased pruning strategy in learning from long-tailed distributions.
demand for model overparamettrization, which further supported by the imbalance in overparametteriztton demand (Corollary 2 in Appendix).
Based the differentiated overparameterization demand theorem, propose the parameter allocation strategy corol- lary, which guides how to reasonably allocate model param- eters in long-tailed learning.
To gain deeper understanding of the learning difficulty of different classes under loigetailed distributions, we intro- duce the sample complexity lemma (Lemma in Appendix), establishing the relationship between sample size, VC di- mension, and generalization error. Based on this lemma, we further define the class-specific VC dimension (Definition 1 in Appendix) and derive the learning difficulty for each class (Corollary in Appendix).
Theorem 2 (Parameter Allocation Strategy). In long-tailed learning, to optimize overall model performance, relatively more parameters should be allocated to tail classes. Specif- ically, for class c, the ideal parameter allocation ratio ar should satisfy Oae Ne No where N₁ is the number of samples in the head class (the class with the most samples). and N is the number of samples in class
Ni.
Theorem (Differentiated Overparameterization Demand). In a long-tailed setting, to achieve the same generaliza- tion performance, tail classes require a higher degree of overparameterication than head classes. Specifically, for tail classes, "t Q Nc ) and for head classes, Tee O()), where N1 is the number of samples in the head class (the class with the most samples), and N. is the num- ber of samples in class c.
Theorem 3 (Performance Gains from Parameter Allocation). Assume that model performance is logarithmically related to the number of effective parameters for each class, i.e., for class e, its performance perf, satisfies perfe x log Pa. where P. is the number of effective parameters for class c. Under the above parameter allocation strategy, compared to uniform allocation, the performance gain A is A Z where C. is the total number of Ω + L log classes.
The differentiated overparameterizztion demand theorem reveals that tail classes in long-tailed dataset have higher
N.

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
This theorem shows that by reasonably allocating param- eters, can significantly improve overall model perfor- mance on long-tailed datasets, especially improving the classification accuracy of tail classes.
50, 100). ImageNet-LT is long-tailed version of ImageNet, with 1,000 classes and natural long-tailed distribution. iNat- uralist 2018 is a large-scale real-world dataset with 8,142 species categories and inherent long-tailed distribution.
Based on the above theoretical foundations, we propose the tail-biased pruning proposition, which guides how to prune models in long-tailed learning.
Implementation Details. We use the knowledge generated from the long-tailed recognition task to guide the pruning of the backbone network. Specifically, for each parame- ter in the model, we calculate scores using 'magnitude', "avg wrgg-maanitdde',
Proposition 1 (Tail-biased Pruning). In long-tailed learn- ing, to reduce the number of parameters while main- taining overall model performance, a pruning strategy that favors retaining parameters for tail classes should be udopted. Specifically, the optimization objective is minm Ee
brosine_similartty'. 'taylor "taylor_frst..om www.st..odderrrr...re.rr.....e.e and 'taylor second order' during the gradient descent pro- cess. These scores are then weighted based on the cumula- tive change in accuracy for each class on the validation set. The weighted sum of the scores is used to determine whether to prune a parameter. We start the continuous pruning pro- cess after the 100th epoch, and the final model retains 30% of the original parameters. For the final evaluation phase, we use the same settings as DODA (Wang et al., 2024) for all baseline methods and our method. For the CIFAR-100- LT dataset, we follow the general experimental settings of (Cao et al., 2019) and use ResNet-32 (proposed by (He et al., 2016)) as the backbone network. The network is trained for 200 epochs using the GD optimizer with an initial learn- ing rate of 10=4, momentum of 0.9, and weight decay of 2x 10=4. For ImageNet-LT and iNaturalist 2018 datasets, we use ResNet-50 as the backbone network, train the network for 100 epochs with an initial learning rate of 0.1, and re- duce the learning rate by 0.1 at the 60th and 80th epochs. For all experiments, we set the value of the hyperparameter pau to 0.5.
(oo(fffoxx)))) + Al[m|lo. where m E (0.1) is a binary mask vector indicating whether pa- rameter is retained, We x Ne Nog.N. is the weight for class c, Le is the loss function for class c, foom denotes the masked model, X is a hyperparameter controlling the pruning strength, and ||mlll is the Lo norm of m.
N1.
This proposition, through a weighted loss function and pa- rameter sparsity, guides how to prioritize the retention of tail class parameters during pruning, ensuring that model param- eters are reduced while maintaining or improving overall performance on long-tailed datasets.
To ensure the effectiveness of the tail-biased pruning strat- egy in practical applications, we propose the following per- formance guarantee theorem.
Theorem 4 (Performance Guarantee of Tiil-biased Pruning). Assume that the initial model achieves a training error of c on each class. After applying the tail-biased pruning strategy, the expected generalization error [lEe for class
Baselines. For fair comparison, all methods are evalu- ated under the same experimental conditions. We use three strong long-tailed baselines, e.g., Balanced Softmax (BS) (Ren et al., 2020), LDAM-DRW (Cao et al., 2019), DBLP (Zhou et al., 2024) and two SOTA pruning method, ATO et al., 2024), RReg(Stewart et al., 2023). Our pro- posed method is denoted as 'BS LTAP', 'LDAM-DRW + LTAP", and 'DBLP LTAP". We report the classification accuracy of the head, medium, and tail classes, as well as the overall accuracy across all classes. Additionally, we compute the ratio of accuracy to FLOPs (f) as a key metric to evaluate both performance and computational efficiency.
loog (N where 8 is small (V
satisfies Ejēc] S SE+O constant (e.g., 0.05), representing the confidence level.
This theorem shows that despite the pruning process, the generalization error for tail classes can still be effectively controlled, and the strategy ensures that overall performance does not significantly degrade, especially in terms of the performance of tail classes. The relevant proofs can be found in Appendix A.
4. Experiments
4.2. Benchmark Results
In this section, we evaluate the performance of our proposed method on multiple long-tailed datasets. Furthermore, we assess the computational efficiency of each method by com- paring the ratio of floating-poit operations (FLOPs) and the ratio of accuracy improvement.
CIFAR-100-LT. Table I presents the classification results for different methods on the CIFAR- 1000L- 100-LT dataset under two imbalance ratios (IR = 50 and 100). LTAP consistently achieves higher C compared to other pruning methods in both imbalance ratio settings, demonstrating superior effi- ciency in terms of both accuracy and computational cost. For instance, in the [IR = 50] setting, LTAP achieves a tail accuracy of 34.1% 1% compared to 23.8% by RReg, while reducing FLOPs by 77.4% compared to the baseline BS.
4.1. Experimental Setup
Datasets. CIFAR- 100-LT is a long- tailed version of CIFAR- 100, containing 100 classes with two imbalance ratios (IR =
5

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Table 1. Accuracy (%) on CIFAR-I00-IT dataset (Imbalance (50, 100)). denotes the ratio FL.Pss between the target method and the baseline method. C' denotes the ratio of accuracy (acc) between the target method and the baseline method. The gray column represents the primary observed metrics, and the gray row indicates the baseline method for the current block.
100
Method
F(X)
Headt
Medium!
Tailf
Allf
C(X)1
F1
Head
Mediumt
Taily
All
018)
al.. 2020)
100.0
62.3
46.1
37.0
51.2
100.0
1.0
62.6
48.5
27.0
100.0
BS + ATTOWW et 2021)
81.7
39.6
30.6
21.8
32.9
6/.2
0.7
40.8
28.9
16.5
29.5
62.5
0.7
RRRggSttewat
2023)
35.1
35.6
40.2
78.5
2.2
36.3
79.0
RS + I.TAP
22.6
$7.6
43.4
34.1
47.8
93.3
4.1
55.8
44.7
23.2
42.4
89.8
3,9
LDAM- DRW (Cao et al., 2019)
100.0
61.5
43.0
26.4
19.1
100.0
1.0
65.1
48.1
20.1
45.8
100.0
1.0
LDAM-DRW AT 2024)
84.7
40.1
34.6
25.5
33.7
68.6
0.8
41.8
30.9
18.5
31.0
67.6
0.8
LDAM-DRW RRegSSt al, 2023)
35.1
52.9
39.8
23.7
42.4
86.3
2.4
54.3
37.8
16.7
37.6
82.0
2.3
LDAM-DRW LTAP
24.8
39.9
44.8
91.2
40.2
DBLP (Zhou et 2024)
100.0
61.2
46.5
32.3
50.2
100.0
1.0
61.4
46.9
23.6
45.3
100.0
1.0
DBLP ATD/Wu al. 2024)
81.7
50.7
37.0
26.2
38.5
76.6
0.9
40.8
32.6
DBLP RReg(Stewart al., 2023)
35.1
50.8
40.4
24.1
43.8
87.2
2.4
52.1
39.2
17.5
37.5
82.7
2.3
DBLP LTAP
24.0
56.1
43.5
31.5
46.7
93.0
3.9
54.7
43.3
25.8
42.0
92.7
3,9
Table Accuracy (%) on ImageNet-I.T and iNaturalist 2018, F denotes the ratio of FLOPPs between the target method and the baseline method. C' denotes the ratio of accuracy (acc) between the target method and the baseline method. The gray column represents the primary observed metrics, and the gray row indicates the bascline method for the current block.
ImageNet-LL
iNaturalist
Method
F(X)
Headt
Medium!
Tail
All"
C(X)
Ft
Head!
Mediumt
Tailp
Allf
C(%)
at... 2020)
100.0
60.9
51.0
1.0
67.4
100.0
BS + ATOOWW et 2021))
65.3
37.1
35.7
17.8
33.8
66.2
1.1
34,8
42.5
12.2
11.5
61.6
0.9
RRoggStewant 2023)
52.1
41.1
35.5
69.6
1.3
30.5
64.4
+ LTAP
30.6
58.5
45.0
30.1
48.1
81.9
2.6
59.2
60.7
60.7
60.5
89.8
LDAM- DRW (Cao al., 2019)
100.0
60.1
16.9
30.7
19.8
100.0
1.0
63.2
66.3
65.4
65.6
100.0
1.0
LDAM-DRW AT 2024)
65.3
36.9
34.6
17.5
33.1
66.4
1.1
36.5
30.9
30.7
31.3
47.7
0.7
LDAM-DRW RRegiStewnn al., 2023)
52,1
38.8
36.8
18.7
35.1
70.4
1.3
41.8
32.5
32.0
33.2
50.6
0.9
LDAM-DRW LTAP
30.8
41.9
45.4
91.1
59.9
91.7
DBLP (Zhou et al., 2024)
100.0
61.7
47.1
30.3
50.4
100.0
1.0
65.0
66.9
65.6
66.1
100.0
1.0
DRI ATD(Wa 2024)
65.3
17.5
33.7
66.8
1.1
41.0
30.6
47.6
DBLP RReg(Stewart. et al., 2023)
52.1
40.0
35.7
18.8
35.0
69.4
1.3
48.8
32.5
32.3
34.0
51.4
0.9
DBLP LTAP
30.0
58.5
45.2
23.0
47.3
93.8
3.1
58.7
59.3
59.8
59.4
90.1
3.0
The C ratio of 4.1 for LTAP is nearly double that of RReg FF (i.e.. 2.2). Similar trends are observed for LDAM-DRW and DBLP, where LTAP consistently improves tail class accuracy and achieves the highest FF ratios.
about 70% compared to the baselines, while achieving the highest & ratios. This reduction in computational cost, cou- pled with maintained or improved accuracy, demonstrates the practical utility of our method for essurcce-constrane environments where high accuracy required.
C
ImageNet-L. and iNaturalist 2018. Table 2 shows the re- sults on ImageNet-LT and iNaturalist 2018 datasets. These larger and more complex datasets further validate the effec- tiveness of I.TAP. On ImageNet-L.T, 'BS + I.TAP' achieves a tail accuracy of 30.1%, significantly outperforming 'BS + RReg' (i.e.. 18.2%), while reducing FLOPs by 69.4%. The J ratio of 2.6 for 'BS + LTAP' is double that of 'BS + RReg'. For iNaturalist 2018, LTAP shows consistent per- formance across all classes (i.e., head, medium, and tail classes), indicating its robustness in handling extreme class imbalance. Notably, LTAP maintains high accuracy across all class types while significantly reducing FLOPs. For ex- ample, on ImageNet-LT, 'DBLP- + ours' reduces FLOPs by 70% while achieving 93.8% of the baseline accuracy, resulting in & ratio of 3.1.
4.3. Further Analysis
In this section, we conduct a detailed analysis of the mech- anism of LTAP and discuss the following issues. More empirical results are reported in Appendix C.
Discussion I:
How are neurons masked under dif- ferent pruning strategies?
Figure 2 illustrates how neurons are masked under differ- ent pruming strategies. First, the visualization of w.o. vote shows that after removing the long-tailed feedback mech- anism, the flexibility of the pruning process decreases sig- nificantly, with pruning limited to specific rows of neurons. This rigid pruning strategy restricts the model's adaptability to varying data distributions, especially in handling long- tailed data, where it struggles to preserve neurons critical for tail classes. contrast, our proposed LTAP method, as
Across all datasets, our method achieves significant reduction in FLOPs while maintain- ing competitive or superior accuracy. For ImageNet-LT and iNaturalist 2018, our method consistently reduces FLOPs by
Efficiency Evaluation.
6

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailes Data
OUTs W.O. vote
Durs
ours y 0.7
Parems indes
Parami ind
0.2
-92
21000 -00
5005000355000 100
0000
-22
000
2
Paramsindex
Params Indix
Params index
Params Index
Figure 2. Visualization of neuron masking. Each small cell represents the sum of the parameters of a 3x3 convolutional kemnel, and each subfigure represents a cross-scction a layer of neurons. Each layer should contain 1 xx64 x4 x34x3xx x3 convolutional kernels, and we visualize the top-left 1 x 10x 10x3 part of each layer. The values and colors represent the sum of the parameters in the 3 convolutional kernels, and the blank areas indicate neurons that have been masked. The variable y represents the masking ratio, and w.o. vote denotes the removal of the long- taigta feedback mechanism LT- Vote.
shown in the y = 0.7 visualization, exhibits much greater flexibility.
that traditional pruning methods fail in long-tailed learning scenarios due their excessive focus on head classes.
Pruning is no longer confined to specific rows of neurons but instead dynamically adjusts based on the parameter val- ues of neurons across different layers and positions. This adaptive pruning strategy allows the model to better retain neurons that are critical for tail classes, improving classifica- tion accuracy on long-tail data, Additionally, as the y value increases, we observe that the pruning intensity increases, with more neurons being masked. However, the distribu- tion of pruning remains dynamic and flexible. This further demonstrates that the LTAP method can maintain both ef- ficiency and effectiveness under varying levels of pruning intensity. In summary, the LTAP method achieves more pre- cise neuron importance estimation through the long-tailed feedback mechanism, balancing computational efficiency and classification performance during the pruning process.
Accuracy on Tail Classes. Tail classes pose the biggest challenge in long-tailed learning. Traditional pruning meth- ods (e.g., ATO) perform disastrously on the tail classes, suffering a catastrophic drop in accuracy, which reflects their extreme inability to adapt to long-tailed classes. In comparison, our pruning method, even at a high pruning ra- tio (7 = 0.9), maintains strong performance, demonstrating its robust adaptability to tail classes.
FLOPs and Parameter Comparison. The last subplot shows the comparison of FL.OPs and parameter counts under different pruning strategies. Our method allows for varying degrees of pruning, and even at a pruning ratio of 7 = 0.9, it maintains high average performance and strong performance tail classes. At this point, compared to the baseline, it demonstrates significant advantages in both computational efficiency and performance.
Discussion 2: Dynamic changes in the performance of different pruning methods during training.
The comparisons above demonstrate that our method not only reduces parameter and computational costs while maintaining high performance, but also adapts effectively to different frequency classes. Notably, it shows significant advantage over traditional methods, particularly in handling tail classes
From Figure 3, it is evident that different pruning strategies exhibit significant performance differences on long-tailed data. We analyze from the perspective of different classes.
Accuracy on Many Classes. Even with a pruning ratio of y = 0.9. LTAP maintains a high accuracy. Meanwhile, the ATO shows slightly better performance this region, which indicates that traditional pruning exacerbates the imbalance in long-tailed distributions.
Discussion 3: Additional Ablation Studies on CE Baseline and Logit Adjustment
We conducted experiments with different loss functions (CE and Logit Adjustment) in table 4.3, pruning methods, and ablation studies of our tail-class protection mechanism (w.o.T). We have included comparisons with the Logit Ad- justment baseline for reference. As a method considering long-tailed distributions, Logit Adjustment demonstrates
Accuracy on Medium Classes. LTAP continues to maintain high accuracy at pruning ratios of 7 = 0.5 and = 0.9, following a similar trend as the head classes. In contrast, the accuracy of the ATO baseline significantly decreases, and it even surpassed by ITAP at y = 0.9. This suggests
7

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Many Accuracies
Medium Accuracies
Accuracies
FLOPS params
FLops(params)
8G(0.8M)
50
20
Accuracy
Accuracy
Accuracy
66(0.6M)
4G(0.4M)
A0
2G(0.2M)
50
100
150
200
50
100
150
200
0
50
100
150
200
Epoch
Epoch
Epoch
ours. 0=0.1
ours. 0.5
0.5
ours y= 0.9
base
ATO
Figure 3. Training dynamics comparison under different pruning strategies. The first three subplots show the accuracy changes over training epochs for the head, medium, tail classes, respectively. Different colored curves represent various pruning intensities, with the purple curve (ATO) representing the baseline for non-long- lon-lon-ttilldd tailed pruning methods. The last subplot shows the comparison of FLOPs and parameter counts across different pruning strategies ( 0.1, 0.5, and0.9) as well as the baseline (BS) and ATO methods.
Table 3. Experiments on CIFAR with IR=50
Table S. Performance Comparison across Different Imbalance Ra-
Mellad
RSU
Heal/S)
Maliann/3)
AlleS)
C/S/T
CF1
tios (IR)
CE
103.0
68.0
36.0
13.2
46.0
100.0
10
IR
6.83
0.7
Methed
PYS0
Head(9y7
Medium 937
Tall/S1
AlllWey
CXS6/1
CPT
Ours W.t.
CE
100,0
68.0
38.0
1322
46.0
100.0
1.0
CE Ours
25.3
64.8
317
1.2
41.1
09.3
3.8
ReGG
52.1
43.8
0,83
24.5
53.2
LA ATO
84.7
34.5
33.9
29.1
34.2
66.7
0.7
LA Re00
12:1
11.0
90.3
25.8
39.9
S6.
1.1
LA Ous w.o.T
22.8
53.5
42.7
30,0
44.8
X7.3
3.8
100,0
70.7
40.0
7.2
41.0
100.0
1.0
LA Ours
434
4.0
CE ReGG
52.1
47.7
13.6
06
21.9
53.4
1.0
Table 4. Experiments on CIFAR with IR=100
Method
RSU
Healrky
Mediumi/3
Tux
Alle%)
C/S)1
CF1
number of criteria. Even for datasets with long-tailed distri- butions, this remains much smaller than O(d), as our total complexity is O(d). These results strongly demonstrate the applicability of our method. Notably, approach main- tains high performance while significantly reducing model parameters, achieving superior C/F ratios across different imbalance settings.
CE
102.0
70.7
40.0
7.2
41.0
100.0
1.0
CE ATO
86.2
50.4
16.5
6.5
35.2
0.2
1.0
36
31.7
2.5
47.3
29.5
LA ATO
30.6
18.4
31.4
0.7
LA Re00
52:1
18.5
37.0
14.8
27.6
$7.6
1.1
LA Ous w.o.T
22.8
55.1
42.0
18.5
39.5
X2.4
3.6
LA 4 Oun
22.8
56.1
45.3
22.6
42.6
85.9
3.9
more balanced performance and higher tail-class accuracy. In this setting, our method still achieves the highest C/F ratio while maintaining superior tail-class performance compared to other pruning baselines.
5. Conclusion
We have presented ITAP,, dynamic pruning strategy de- signed to enhance model efficiency and performance long-tailed datasets. By dynamically adjusting pruning cri- teria based on class-specific performance, LTAP addresses the inherent pruning bias in conventional methods, partic- ularly for tail classes. Our theoretical analysis establishes that tail classes benefit more from model overparameteriza tion, which informs our tail-biased pruning approach. Ex- tensive experiments on benchmark -oilet datasets, in- cluding CIFAR- 100-LT ImageNet-L.T, and iNaturalist 2018, demonstrate that LTAP consistently improves classification accuracy, particularly for tail classes, while significantly reducing the model's 's computational overhead. By offering balanced trade-off between model compression and accu-
Discussion& 4: Analysis of Computational Overhead and Performance
In fact, our method introduces minimal additional overhead. The time complexity of pruning method is O(d), where d denotes the number of model parameters. This is signif- icantly lower than ATO's O(Dd), where D represents the size of the supernet used in that method. The additional complexity the dynamic scoring and balancing mecha- nism O(kk), where n the number of classes and the
8

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
racy, LTAP provides robust solution to the challenges of long-tailed learning and opens new possibilities for optimiz- ing neural networks in imbalanced and resource-constrained environments.
Advances neural information processing systems, 28, 2015.
He, C.. Wang. S., Zhang, D., and Zeng, M. Rethinking class-balanced methods for long-tailg visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7610-7619, 2021a.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
He, H., Bai, Y., Garcia, A., and Li, Adasyn: Adap- tive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelli- gence), pp. 1122-138..2 IEEE, 2008
Acknowledgements
The authors gratefully acknowledge the support from the National Natural Science Foundation of China (NSFC) un- der Grant Nos. 62402472, and 12227901. This work also supported by the Natural Science Foundation of Jiangsu Province (No. BK20240461), the Key Basic Research Foun- dation of Shenzhen (No. JCYJ202208 81 810000501 1), the Research Grants Council of the Hong Kong Special Admin- istrative Region (GRF Project No. CityU 11215723), the Project of Stable Support for Youth Team in Basic Research Field, CAS (No. YSBR-005), and the Academic Leaders Cultivation Program at USTC.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
He, Y.-Y., Wei, J., Zhu, X., Wang. X., Feng, Z., and Dong. J. Distilling virtual examples for long-tailed recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13173-13182, 2021b.
Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
References
Blalock, D., Ortiz, J. G., Frankle, J., and Guttag, J. What is the state of neural network pruning? arXiv preprint 2:Xiv:2003.030333 2020.
Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, P. Pruning filters for efficient convnets. In arXiv preprint 000i::/60..0080 2016
Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. Learning imbalanced datasets with label-distribution- aware margin loss. In Advances in neural information processing systems, volume 32, 2019.
Li, Y., Wang, T., Kang, B., Tang, S., Wang, C., Li, J., and Feng, J. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Advances in Neural Information Processing Systems, volume 33, pp. 1513- 2020.
Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. Smote: synthetic minority over-samplin technique. Journal of artificial intelligence research, 16:321-357, 2002.
Lin, J., Rao, Y., Lu, J., and Zhou, J. Dynamic neural net- works: survey. In IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2020.
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Class- balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
Liu, J. Ye, F., Zhang, X.,
Xu, H., and Cheng, Y. Dy- namic sparse training: Find efficient sparse network from scratch with trainable masked layers. In International Conference on Learning Representations, 2021a.
2019.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1rXiv:1803.0340 2018
Liu, Y., Zhang, Y., Wang, H., Fan, J., and Zhang, Z. Learn- ing to learn the future: Modeling concept drift in contin- ual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8415 8415-8423, 2021b.
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. arXiv preprint arXiv: /9::::922.000 19/2.05671, 2020.
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. Learning efficient convolutional networks through net- work slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736 7744,, 2017.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network.

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. Re- thinking the value of network pruning. arXiv preprint 2018
Wu, X., Gao, S., Zhang, Z.., Li, Z., Bao, R., Zhang, Y., Wang, X., and Huang. H. Auto-train-once: Controller network guided automatic network pruning from scratch. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2024.
Liu, Z., Miao, Z., Zhan, X., Wang. J., Gong, B., and Yu, S. X. Large-scale long-tailed recognition in an open world. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537- 2276, 2019.
Xiang, L.,, Ding, G., and Han, J. Learning from multiple experts: Self-paced knowledge distillation for aag-tailed classification. In European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Mohri, M. Foundations of machine learning, 2018.
Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J.
Yang. Y. and Xu, Z. Rethinking the value of labels for improving dass-imbalanced learning. In Advances in Neural Information Processing Systems, volume 33, pp. 19290-19301, 2020.
Pruning convolutional neural networks for resource effi- cient inference. In International Conference on Learning Representations, 2016.
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., and Kautz, J. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1264-11272, 2019.
Yin, X., Yu, X., Sohn, K., Liu, X., and Chandraker, M. Feature transfer learning for face recognition with under- represented data. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 2019.
Ouyang, W., Wang, X., Zhang. C., and Yang, X. Factors in finetuming deep model for object detection with long-tail distribution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 864-873, 2016.
Zhang, S., Li, Z., Yan, S., He, X., and Sun, J. Distribu- tion alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 2361-2370, 2021a.
Ren, J., Yu, Ma, X., Zhao, H., Yi, S., et al. Bal- anced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33: 4175 4186, 2020.
Zhang, Y., Kang, B., Hooi, B., Yan, S., and Feng, J. Deep loig-tailed learning: A survey, In arXiv preprint arXiv:2 10.04596, 2021b.
Zhou, Z., Liu, L.., Zhao, P., and Gong, W. Pareto deep long-tailed recognition: A conflict-averse solution. In The Twelfth International Conference on Learning Repre- sentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //tpp///////peeiie.... met/fooummicco 6o Pu10k115
Stewart, J., Michieli, U., and Ozay, M. Data-free model pruning at initialization via expanders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR) Workshops, June 2023.
Tan, J., Wang, C., Li. B., Li, Q., Ouyang, W., Yin, C., and
Yan, J. Equalization loss for long-tailed object recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11662-
Zhu, M. and Gupta, S. To prune, or not to prune: explor- ing the efficacy of pruning for model compression. In International Conference on Learning Representations, 2017.
11671, 2020.
Wang, B., Wang, P., Xu, W., Wang, X., Zhang, Y., Wang, K., and Wang, Y. Kill two birds with one stone: Rethinking data augmentation for deep long-tailed learning. In The Twelfth International Conference on Learning Represen- tations, 2024.
Wang, X., Long, Z., Wang, L., and Wang, L. Long-tailed recognition by routing diverse distribution-avv aware experts. In International Conference on Learning Representations, 2020.
Wang, Y.-X., Ramanan, D., and Hebert, M. Learning to model the tail. In Advances in Neural Information Pro- cessing Systems, pp. 7029-7039, 2017.
10

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Appendix
Definition 1 (Class-specific VC Dimension). For class in long-tailed dataset, the class-specift VC dimension dy is defined the VC dimension of the hypothesis space that can effectively separate that class from all other classes.
Balancing Model Efficiency and Performance:
Adaptive Pruner for Long-tailed Data
Corollary 1 (Class Learning Diffieulty). In a long-tailed dataset, the learning difficulty for class can be approx- imated as De z dVC,e where Ne is the number of samples N.
The content of the Appendix is summarized as follows:
1) in Sec. A, we provide detailed proofs and theoretical foundations for the results in the main paper.
in class
This corollary shows that for head classes, where No is large, the learning difficulty De is small. For tail classes, where Ne is small, even if dVC, remains the same or slightly smaller, the learning difficulty Dj increases signifi- cantly. This difference in learning difficulty directly affects the model's complexity (i.e., the number of parameters) re- quired for different classes, as stated in the differentiated overparameterization demand theorem (Theorem 1).
2) in Sec. B, we briefly present the state of the art in the field of long-tailed learning and neural network pruning.
3) in Sec. C, we illustrate more detailed empirical results and analyses of ITAP.
4) in Sec. D, we present a theoretical analysis of our proposed dynamic feedback pruning algorithm.
Corollary 2 (Imbalance in Overparameterizzaton Demand), In long-tailed dataset, the demand for overparameteri- zation is inversely proportional to the number of samples in each class. Specifically, for any two clusses i and j. if N, log N. N.. No
5) in Sec. E, we presents the detailed process of LTAP.
A. Supplementary theory
N₂ Nj. then It S0
In this supplementary section, we provide detailed proofs and theoretical foundations for the main results presented in the paper. We start by introducing the sample complex- ity lemma and related definitions, which form the basis for understanding the learning difficulty of different classes in long-tailed distributions. Then, we prove the main theorems and propositions presented in the paper, including the differ- entiated overparameterivation demand theorem, the parame- ter allocation strategy theorem, and the tail-biased pruning proposition. Finally, we provide the proof for the perfor- mance guarantee theorem, which ensures the effectiveness of the tail-biased pruning strategy in practical applications.
This corollary further supports the differentiated overparam- eterization demand theorem (Theorem 1), revealing that in long-tailed dataset, the number of samples in class decreases, the demand for overparameterization increases significantly, eggiffiantly.emphaszziingg the importance of providing more parameter protection for tail classes.
A.2. Proof for Theorem
Theorem 5 (Performance Guarantee for Tail-Biased Prun- ing). Assume that the initial model achieves training error of E for each class. After applying the tail-biased pruning strategy, the expected generalization error E Ei for class c satisfies
A.1. Appendix: Sample Complexity and Differentiated Overparameterization Demand
Lemma 1 (Sample Complexity Lemma). For class c in a binary classification problem, given hypothesis space He, target generalization error >> O, and confidence level 1-5 < 8 < 1), the minimum required sample size Ne satisfies N. 22 (AdVC,clog 12 + log ))), where dyc,e is the VC dimension of the hypothesis space H., associated with class
(1
log(No/0)
Eléc] S 5
(10)
N₂
where 8 is small constant (e.g.. 0.05) representing the confidence level.
Remark 1. (i) This bound shows that the required sample size is approximately linearly related to the VC dimension dyc.e and inversely proportional to the square of the target generalization error e. (üi) In practical applications, we usually focus on asymptotic behavior, which can be sim- (ii) Although this lemma is for binary classification, it can be extended to multiclass problems through the one-vs-all strategy.
Proof. Let fe denote the initial model, and feom denote the pruned model, where E 0, 1 is the binary mask vector obtained according to Proposition 1. Let D. represent the data distribution for class c, and D. represent the empirical distribution for class c. The expected generalization егтоr for class e can be expressed as:
plified to N. 2 Ω (dvgc)
E(éc] E(x, v) Del1(f0 m(x) v)).
(11)

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
According Theorem 1, for tail classes, the degree of over- parameterization "ye satisfies:
The above proof demonstrates that the tail-biased pruning strategy effectively controls the generalization error of tail classes while ensuring reduction in the total number of model parameters by prioritizing the retention of critical parameters for tail classes. The proof utilizes series of previous theoretical results, including the Differential Over- parameterization Demand Theorem (Theorem 1) and the Tail-Biased Pruning Proposition (Proposition 1), and ap- plies standard generalization error bounds this basis to ultimately obtain the bound expected generalization er- ror. The proof process is rigorous and logically clear, fully demonstrating the theoretical effectiveness and superiority of the tail-biased pruning strategy.
N1 No
Te 2 Ω
(12)
log No
Furthermore, according to Proposition 1, the tail-biased pruning strategy ensures that critical parameters for tail classes are preferentially retained. Therefore, for tail classes, the number of effective parameters Peoef in the pruned model foom satisfies:
Peeff ((min,e),
(13)
where e is the minimum number of effective parame- ters for class (Definition 1).
Theorem 6 (Performance Gain from Parameter Allocation). Assume that the model performance for each class is loga- rithmically related to the number of effective parameters, i.e., for class c, its performance perf. satisfies perfe x log Pa- where Pa is the number of effective parameters for class c. Under the parameter allocation strategy described in The- orem 2, compared to uniform allocation, the performance gain A satisfies
Combining equations (12) and (13), for tail classes, the effective degree of over-parameetrizzation fo in the pruned model f0 @ m satisfies:
Pe,eS
N1 N
Rcc
Ω
(14)
P Pini,
log No
A2 (2 1c=
(NI) N:
)
(18)
According to standard generalization error bounds (e.g., see Mohri (2018)), for class e, the generalization error éc of the pruned model f0 @ m satisfies the following probability inequality:
where is the total number of classes.
Proof. Let punif denote the number of effective parameters for class under uniform parameter allocation, and pelloc denote the number of effective parameters for class under the allocation strategy described in Theorem 2. According to the theorem assumption, the performance gain A. for class e can be expressed as:
)
P & (655600)
te (0g(1 Ne
2 - (15)
where 6, is the training error for class
Substituting equation (14) into equation (15), and using < (according to the theorem assumption), for tail classes, we have:
Pollon pansf
A. perf perfunis
log
(19)
(
)
log(No/0)
PP & se+0
21 21-6.
(16)
According Theorem 2, the parameter allocation strategy satisfies:
Ne
N1 N
palloc & N log
(20)
Finally, taking the expectation of equation (16), we obtain:
(V (v Mogg(N//0)
Under uniform allocation, the number of effective parame- ters for each class is independent of the sample size, so we
Eled| < Se+o
(17)
No
have:
punif 1.
(21)
This proves that for tail classes, the expected generalization error of the model from obtained by the sail-biased pruning strategy satisfies the bound in Theorem 5. For head classes, due to sufficient samples, the impact of pruning on general- ization performance is minimal, and is easy to verify that the theorem's bound also holds. Therefore, Theorem 5 is proved.
Substituting equations (20) and (21) into equation (19), we
get:
No
Ni No
))
N₂
N1
Ao x log
log
= =log
log log
Ni
N.
N.
(22)
12

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Since log log
Ni N.
is a higher-order infinitesimal, we have:
large pre-trained models. However, these methods often rely additional pre-trained models or complex training strategies, increasing computational complexity and model dependencies. Multi-expert systems (Wang et al., 2017; Xi- ang et al., 2020) and modular designs (Zhang et al., 2021b; Liu et al., 2021b) design specialized sub-models or modules for different data subsets. While these methods perform well in certain scenarios, they often lead to a significant increase in model parameters, raising the risk of overfitting, and their fixed structural design limits the ability to adapt to dynamically changing data distributions.
N1 N₂
A 2 ?
log
(23)
Taking the average of equation (23) over all classes, we obtain the total performance gain:
1 Σ c=1
N1
= c
Σ e=1
A
Ω
log
(24)
No
This proves the performance gain bound in Theorem 6.
Although the above methods have made some progress in addressing long-tailed problems, they still face challenges such as insufficciet flexibility, low computational efficiency, and difficulty in adapting to dynamic environments.
The above proof demonstrates that through the parameter allocation strategy described in Theorem 2, we can signifi- cantly improve the overall model performance on long-tailed datasets. Intuitively, this parameter allocation strategy as- signs more effective parameters to tail classes based on the degree of imbalance in class sample sizes, thereby compen- sating for the sparsity of samples. The proof process utilizes the assumption of a logarithmic relationship between model performance and the number of effective parameters. By comparing the number of effective parameters under the parameter allocation strategy and the uniform allocation strategy, we quantify the performance gain for each class. Furthermore, by taking the average over all classes, ob- tain a quantitative characterization of the total performance gain. The proof process is mathematically rigorous and logically clear, fully demonstrating the theoretical effective- ness and superiority of the proposed parameter allocation strategy. Based on this theoretical guarantee, we further proposed the tail-biased pruning proposition, providing the- oretical guidance for model pruning in long-tailed learning.
Neural Network Pruning. Neural network pruning, an important model compression and optimization technique, has received widespread attention in recent years. Existing pruming methods mainly include magnitude based pruning, importance- based pruning, structured pruning, and dynamic pruning.
Ingnitude-based pruning methods (Han al., 2015; Li et al., 2016) compress networks by removing connections or neurons with small weight magnitudes. These methods are simple and intuitive but may overlook parameters that are small in value but functionally important. Importance-bust based pruning methods (Molchanov et al., 2016; 2019) evaluate the importance of parameters by calculating their impact on model output, but typically rely on a single scoring criterion, making it difficult to comprehensively capture parameter importance in complex tasks. Structured pruning meth- ods (Li et al., 2016; Liu et al., 2017) aim to remove entire convolution kernels or neurons to achieve higher hardware acceleration effects. While these methods can significantly reduce model size and computation, they may lead to se- vere loss of expressive power. Recent dynamic pruning strategies (Lin et al., 2020; Liu et al., 202 allow dynamic adjustment of network structure during inference, provid- ing greater flexibility, but mainly focusing on improving computational efficiency.
B. Related Work
Long-tailed Learning. Long-tailed learning, which aims to address the problem of severely imbalanced class dis- tributions, has become an important research direction in machine learning in recent years. Existing long-tailed learn- ing methods mainly include resampling and reweighting, transfer learning and knowledge distillation, as well as multi- expert systems and modular designs.
A Although existing pruning methods have achieved signif- icant results in model compression and acceleration, they still have notable shortcomings in addressing long-tailed learning problems: (i) these methods typically assume uni- form data distributions, ignoring the special characteristics of long-tailed data. (ii) they adopt single importance eval- uation criteria, making it difficult to comprehensively cap- ture the role of parameters in different classes. (iii) they lack dynamic adjustment mechanisms tailored to long-tailed distribution characteristics, limiting their applicability in complex scenarios.
Resampling methods (Chawla et al., 2002; He et al., 2008) and reweighting techniques (Cui et al., 2019; Cao et al., 2019) balance data distribution and learning processes by adjusting sample sampling probabilities or loss weights. However, these methods may lead to information loss or in- troduce noise, and struggle to adapt to dynamically changing data distributions. Transfer learning (Yin et al., 2019; Liu et al., 2019) and knowledge distillation (Xiang et al., 2020; He et al., 2021b) techniques attempt to transfer knowledge from head classes to tail classes, or extract knowledge from
13

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Based on the above analysis, we believe it is necessary develop a pruning method specifically for long-tailed learn ing, which can both fully leverage the advantages of pruning techniques and effectively address the special challenges posed by long-tailed distributions. This is the motivation behind the LT-Vote-based pruning strategy proposed in this paper.
in refined manner to maintain performance across different classes.
D. Symbol Definitions and Assumptions
Categories and Sample Numbers The dataset contains C categories, where the sample count Ne for category follows a Pareto distribution with parameter 3:
C. Supplementary experiments
N₂ N. A == 1,2, .., C,
where N is scale parameter ensuring the total sample count meets the dataset size requirement.
Model and Parameter Groups
Deep neural network model is denoted as so X + RC defined by parameters O.
Figure 4. Pruning visualization of layers the front of the neural network, with other settings the same as in Fig. 2.
0g represents the g-th parameter group of the model,
G is the set of all parameter groups.
Classification Accuracy For category k, its classification accuracy is defined as:
(o) NR
Σ ( fo();); Vj kk) fayemk
where 1(-) is the indicator function.
Figure 5. Pruning visualization of layers near the middle of the neural network, with other settings the same as in Fig. 2.
Pruning Algorithm and Dynamic Feedback Mechanism
Standard pruning algorithm P with hyperparameter produces pruned parameters:
@ P(0:0).
Dynamic feedback pruning algorithm PLTAP adjusts importance criteria weights (!!) E RRK' based on clas- sification accuracy changes:
Figure Pruning visualization of layers near the end of the neural network, with other settings the same in Fig. 6.
8.4. (e( Ax (o(t-11)
From these three figures, the patterns exhibited within the same layer, across multiple layers, our method demonstrates significantly greater flexi- bility compared to traditional pruning methods. In contrast, traditional methods show low pruning efficiency in the lay- ers near the front, with almost no pruning, but neurons are pruned in large quantities after the middle layers, leaving fewer neurons than with ? 0.9. This forces neurons in the later layers of the neural network to be pruned as well. In comparison, our method prunes more evenly, achieving higher efficiency even in the early stages while still retaining a large number neurons the middle layers the neural network. This indicates that our pruning method adopts a more precise and flexible pruning strategy, pruning neurons
observe that, in addition to
and computes comprehensive importance score Sg for parameter groups:
K S, =
Sy.k-
k=1
Dynamic weight adjustment rule:
0, (t)
if k kmasked
(f+1)
0
O'
otherwise,
where Kmasked represents the masked category in round t, and K is the number of importance criteria.
14

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Parameter Group Set The of parameter groups strongly correlated with tail classes is defined as:
Since k: is tail class, Ni increased (or weights are redistributed) through dynamic feedback mechanism, resulting in significant increase in 8Sg.
1 is large, and 6A₃ is effectively
Grall {g E 9 g is strongly correlated with tail classes).
2. Pruning decision mechanism: Standard pruning algorithm P ignores changes in class accuracy. All param- eter groups' importance scores Sy are calculated with fixed weights, leading to relatively consistent pruning probabili- ties:
Theorem A (Tail Class Protection Effect of Dynamic
Feedback Pruning)
For a long-tailed distribution dataset D {(2 y1 H/) with C' categories, where sample count N& for category k follows Pareto distribution with parameter B. Let fo be deep neural network model defined by parameters 0, 02 be a parameter group, and 9 be the set of all parameter groups.
P ((5h))
S/t)) (6.)
PP) similar, independent of class.
Dynamic feedback pruning algorithm PLTAP dynami- cally adjusts importance scores through 85g. especially for tail classes, where Sg increases significantly:
Given dynamic feedback pruning algorithm PLTAP that adaptively adjusts importance criteria weights (9 based on classification accuracy changes 5 AA and computes com- prehensive importance score Sz- After I' rounds of prun- ing, the probability of tail-class-related parameter groups 0g. 9 E Grail being retained by PLTAA is significantly higher than by standard pruning algorithm P:
PP (S > ((O)) PLLAA) x (6Sg > A)
where A is the threshold change.
3. Comparing retention probabilities of both pruning algorithms:
Vg E Srail:
For g E Geam:
P ( | ((T) PLTAP (0()
(s > ((0) PPLAP)
> (5
PS/0 > 5(6) IP).
P ( + 0 (T)) = P(O 8))
This further implies:
Lemma a (Impact of Category Accuracy Changes on Importance Score)
PP ((()) + 0 (
> P ( (000)
Statement: In dynamic feedback pruning algorithm PIYTAP, for any parameter group g associated with category k. the change in importance score 85,- = 5 ((t-1) s(t) satisfies:
4. Cumulative effect after multiple iterations:
After T rounds of pruning:
(O(T) 40| 10LTAP)) 9
II
s(t s(t)) ((a))
|PITAP).
oS,- α N₂
8A
where 6A2. = (0(0) (o(c-1)).
PP ( + 0 p
P
s > s
(0,) IP)
1=1
Lemma b (Impact of Dynamic Feedback Pruning Algorithm on Retention Probability of Tail Class Parameter Groups)
Since for all t,
PP ($
5(0) s(() /PLAAP)
> P P(()))
s(6) IP).
Statement: In dynamic feedback pruning algorithm PLTAP, for parameter group g associated with category k, the in- crease in importance score 85, results in a significantly higher retention probability compared to standard pruning algorithm P.
we have:
T
T II
II
(5(0 > (0)) PLTAP)
>
P
s() > 5 IP).
t=1
(=1
Therefore:
Proof:
P ( + 0 0
> P
((T) ( + 0 0
1. Change in importance score:
According to Lemma 1, for E Gran:
Thus, dynamic feedback pruning algorithm PITAA sig- nificantly increases the retention probability of parameter groups bg associated with tail classes.
88, x N₂ -AA
15

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
Proof of Theorem A
therefore:
Objective: Prove that the dynamic feedback pruning algo- rithm РITTAP retains tail relatee parameter groups fg with significantly higher probability than standard pruning algorithm P.
IP ( #01 PLTAP)
> P
(
+ # #01P).
Thus, the dynamic feedback pruning algorithm PLTAP sig- nificantly increases the retention probability of parameter groups 0g associated with tail classes.
Proof:
Enhancement Importance Scores for Tail Class Parameter Groups by Dynamic Feedback:
E. Pseudocode
According to Lemma 1, for g E Stail, have:
8Sg x No -8Ak
Since is tail class with small Nr, therefore:
8.g is relatively large.
Consequently, the importance scores Sy of tail class pa- rameter groups receive significant enhancement after each pruning round.
Probability Calculation for Parameter Group Prun- ing:
Pruning decisions are based on Sg > s(t) (B,) ic:::
PP (s
s(0) (0.)
For E Gtait, due to significant increase in Sg. the probabil- ity of being pruned decreases substantially.
Comparison of Retention Probabilities between Two Pruning Algorithms: Standard Pruning Algorithm P:
P 1 + 0 P =
(5())
((())
(t) (0.) P)
Since does not consider class accuracy changes, all param- eter groups have similar retention prohabilities. Dynamic Feedback Pruning Algorithm РITAP:
s(t) > Stlt) (B.) PLTAP)
st0 > 5(1) IP)
>
(B.)
4. Cumulative Effect After Multiple Iterations:
After I' rounds of pruning, for g E Grall, we have:
T
( 1 =
II
P
s(t) g
> (( .PITAP):
1=1
T
P ( # P
P ((t))
(t)
IP)
=
>
(6,)
Since for all d:
P ( >
(0.) PITAP))
(t)
> P > 5(00) IP)
16

---

Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data
O: Input. Pretraining variable æ; learning rate termina- tion tolerance 2, preset pruning ratio "pp sample steps T, penalty X, and prunable variable partition G,class weight vector w.
0: Warm up and compute importance scores.
0: Initialize to store importance scores for each g G.
O: Initialize violating group V
+ {g: : E 9 with bottom-K importance scores).
0: Initialize historical set H f-
O: while | s 2 do
0: Initialize trial violating group set 1 +- 0.
O: Initialize 8 < 3, Xo + A, and x + æ.
0: for t 0........000
do Compute gradient of f over (t) ((x(1)) Compute trial ((t-1) (t))) git) f (ax(t)).
0:
0:
0:
Penalize variables in the violating set.
[ (1(+0)))
[b(++)]>> \e
0:
Compute the accuracy A of the (t+1)
each class.
O:
Update the importance criteria weight matrix.
D(++)) - (Att1 A D., w)
0:
Compute importance scores of G and collect
into S.
S- 9 < D(+11)
D(++1))
Update trial set D if new violating groups ap-
0:
pear.
D + (o: G with bottom-K K scores) /V
0:
Update penalty (0) and learning rate 3(0)
O: end for
0: Update violating set V < 1/H.
0: Update historical set + HUV.
0: end while
0: Set redundant GR upon importance score collection
S.
GR {g g with bottom-K scores in S}
0: Return. Identified redundant group set GR and impor- tant group set Gr G/GR. =0
17


Reviews:
[{"review_id": "qsRhLa9SvS", "reviewer": "Reviewer_xkjm", "summary": "This paper introduces ​LTAP (Long-Tailed Adaptive Pruner), a pruning strategy tailored for long-tailed data distributions. LTAP addresses the challenge of class imbalance by dynamically adjusting pruning priorities through a multi-criteria importance evaluation framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort. We are encouraged by your high appreciation of the novelty and contribution of our paper. We will continue to work on exploring how to better solve long-tailed problems in real-world scenarios. If you have any further questions, we will address them at any time. Thanks again!", "claims_and_evidence": "Formal proofs establish that tail classes inherently demand higher overparameterization, justifying LTAP’s tail-biased parameter retention strategy.", "methods_and_evaluation_criteria": "Extensive experiments on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 demonstrate LTAP’s superiority over baseline methods.", "theoretical_claims": "The paper provides theoretical analysis (e.g., Theorem 1–4) to justify why tail classes require higher parameter protection.", "experimental_designs_or_analyses": "LTAP achieves state-of-the-art efficiency-accuracy trade-offs across multiple architectures (ResNet-32/50) and datasets.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strength:\n\nLTAP is the first pruning framework explicitly designed for long-tailed data. The LT-Vote mechanism effectively mitigates pruning bias toward head classes by dynamically reweighting criteria (magnitude, gradient alignment, Taylor impact) based on per-class validation accuracy. This innovation directly addresses the core challenge of class imbalance in pruning.\n\nOverall, this work makes a contribution to long-tailed learning by bridging pruning and class imbalance mitigation. The LTAP framework is both theoretically grounded and empirically robust, offering a practical solution for efficient model deployment.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "NA.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YZ0yQKAm5b", "reviewer": "Reviewer_LjiE", "summary": "This paper introduces Long-Tailed Adaptive Pruner (LTAP), a model pruning framework designed for long-tailed class distributions. LTAP integrates multi-criteria importance scoring and a dynamic LT-Vote mechanism to prioritize preserving parameters crucial for tail classes. It employs multi-stage pruning, gradually refining the model while maintaining performance. Theoretical analysis supports the claim that tail classes require more capacity, and experiments on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 demonstrate that LTAP outperforms traditional pruning and long-tail learning methods, improving tail-class accuracy while reducing model size by 70%. LTAP provides a new approach to balancing efficiency and fairness in imbalanced learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your time, effort and recognition of our work. Your comments are very important for us to continue to improve this work. Based on your comments, we continue to make the following efforts**:\n\n- **About Pseudocode**\n\nThis is our negligence. We have completed and corrected the pseudocode in the original paper.\n\n- **About the broader baselines**\n\nThank you for your approval of the experimental section. Based on your comments, **we tried our best to supplement some baselines in the limited time, including Focal Loss, Logit Adjustment, and RIDE**. The following are the experimental results regarding supplemental baseline on the CIFAR-100-LT dataset. The experimental results show the superior performance of LTAP.\n\n|Method|F(%)↓|IR=10 Head↑|IR=10 Medium↑|IR=10 Tail↑|IR=10 All↑|IR=10 C(%)↑|IR=10 C/F↑|IR=50 Head↑|IR=50 Medium↑|IR=50 Tail↑|IR=50 All↑|IR=50 C(%)↑|IR=50 C/F↑|IR=100 Head↑|IR=100 Medium↑|IR=100 Tail↑|IR=100 All↑|IR=100 C(%)↑|IR=100 C/F↑|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Focal Loss|100.0|65.1|43.4|0.0|58.8|100.0|**1.0**|67.2|38.6|13.8|46.2|100.0|**1.0**|67.8|39.3|8.0|40.2|100.0|**1.0**|\n|Focal Loss + ours|22.0|57.1|32.3|0.0|49.7|84.5|**3.8**|61.9|29.6|5.4|38.7|83.7|**3.8**|63.4|35.9|4.8|36.4|90.5|**4.1**|\n|logit adjust|100.0|57.5|63.2|0.0|59.4|100.0|**1.0**|59.3|46.7|43.2|51.4|100.0|**1.0**|62.4|47.1|27.9|46.9|100.0|**1.0**|\n|logit adjust + ours|22.0|45.2|58.3|0.0|49.4|83.1|**3.7**|50.7|43.7|36.4|45.5|88.5|**4.0**|55.1|45.5|23.0|42.5|90.6|**4.1**|\n|RIDE|100.0|70.5|42.0|0.0|61.6|100.0|**1.0**|68.5|48.8|44.0|51.1|100.0|**1.0**|68.1|49.2|23.9|48.0|100.0|**1.0**|\n|RIDE + ours|22.0|62.8|32.5|0.0|53.4|86.6|**3.9**|62.8|40.1|32.0|43.9|85.9|**3.9**|62.0|46.8|18.2|45.3|90.6|**4.1**|\n\n\n- **About Figure 2**\n\nThanks for your suggestion, we have adjusted the font of Figure 2 in the original paper to improve readability.\n\n- **About Questions**\n\nIndeed, the comparison with smaller models is an interesting point, which is fundamental to pruning studies but has unique significance in long-tailed scenarios.\n\nFirst, as visualizations in Figures 2 to 6 demonstrate, **LTAP pruned parameters in a non-uniform, category-aware manner, which is difficult to achieve with manual architecture design**. By simultaneously considering multiple importance metrics and adjusting their weights based on accuracy feedback, LTAP may identify subtle parameter interactions that might have been missed in simple architectural reductions. This is fundamentally different from simply reducing the model size uniformly.\n\nIn addition, we try our best to organized a contrast experiment: the performance comparison between the manually designed small model and the original standard model, where the flops of the small model is 30% of the original standard model. If needed, you may kindly compare this with Table 1 in the original paper.\n\n|Method|F(%)↓|IR=10 Head↑|IR=10 Medium↑|IR=10 Tail↑|IR=10 All↑|IR=10 C(%)↑|IR=10 C/F↑|IR=50 Head↑|IR=50 Medium↑|IR=50 Tail↑|IR=50 All↑|IR=50 C(%)↑|IR=50 C/F↑|IR=100 Head↑|IR=100 Medium↑|IR=100 Tail↑|IR=100 All↑|IR=100 C(%)↑|IR=100 C/F↑|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|BS|100.0|64.9|55.5|0.0|61.5|100.0|1.0|62.3|46.1|37.0|51.2|100.0|1.0|62.6|48.5|27.0|47.2|100.0|1.0|\n|BS|30.0|54.8|45.1|0.0|52.1|84.7|2.8|53.3|40.8|30.0|44.1|86.1|2.8|55.1|41.6|21.0|40.4|85.5|2.8|\n|LDAM-DRW|100.0  |60.8  |43.3|0.0|55.4|100.0|1.0|64.5|43.0|26.4|49.1|100.0|1.0|65.1|48.1|20.1|45.8|100.0|1.0|\n|LDAM-DRW|30.0|51.0|30.1|0.0|45.0|81.2|2.7|54.8|34.1|20.4|40.2|81.8|2.7|56.0|36.0|14.6|36.9|80.3|2.6|\n|DBLP|100.0|65.3|43.4|0.0|58.7|100.0|1.0|61.2|46.5|32.3|50.2|100.0|1.0|61.4|46.9|23.6|45.3|100.0|1.0|\n|DBLP|30.0|57.9|30.0|0.0|49.5|84.3|2.8|62.8|29.3|5.5|38.9|77.4|2.5|63.7|32.1|3.3|34.6|76.3|2.5|", "claims_and_evidence": "The paper makes three major claims: (a) Tail classes require more model capacity than head classes, (b) Dynamically adjusting pruning criteria improves tail-class retention, and (c) LTAP achieves better accuracy-efficiency trade-offs than existing methods.\nOverall, the claims are well-supported by theoretical proofs and extensive experiments across multiple datasets. The results consistently show LTAP’s advantage in balancing efficiency and tail-class performance.", "methods_and_evaluation_criteria": "The methodology is well-designed, effectively addressing pruning bias in imbalanced data while maintaining efficiency. The evaluation is comprehensive, though hyperparameter robustness (e.g., pruning schedule, weight updates) remains an open question.", "theoretical_claims": "The paper presents a strong theoretical foundation supporting LTAP’s approach, with multiple theorems validating its core ideas. These claims are supported by rigorous proofs in the appendix, which follow established generalization theory and sample complexity principles. While some assumptions may not hold perfectly in practice, the results provide a strong theoretical justification for LTAP's adaptive pruning strategy. The presence of formal guarantees enhances the paper’s credibility, making LTAP a well-grounded contribution to long-tailed learning and pruning research.", "experimental_designs_or_analyses": "The experimental design is rigorous and well-structured, validating LTAP across CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, covering both controlled and real-world imbalance scenarios. The method consistently improves tail-class accuracy while reducing FLOPs, with results showing higher accuracy-per-FLOP (C/F) than competing methods. The evaluation metrics include head/medium/tail accuracy breakdowns, highlighting tail-class improvements.", "relation_to_broader_scientific_literature": "This paper bridges long-tailed learning and model pruning, two traditionally separate research areas, by introducing adaptive pruning tailored for class imbalance. Unlike prior long-tail learning methods that focus on reweighting, resampling, or architectural modifications (e.g., expert models, transfer learning, logit adjustments), LTAP optimizes model structure dynamically to preserve tail-class critical parameters, making it a novel contribution to long-tail research. Similarly, while pruning methods have primarily targeted efficiency and overall accuracy, LTAP introduces a distribution-aware pruning strategy that prioritizes fairness across head and tail classes.", "essential_references_not_discussed": "While the paper includes strong baselines such as LDAM-DRW and DBLP, there exist many other long-tailed learning approaches that could have further strengthened the evaluation. Methods like Focal Loss, BBN, and Logit Adjustment are widely used in long-tailed classification but were not explicitly included in the comparisons. Additionally, some recent state-of-the-art long-tailed learning strategies, such as MiSLAS and RIDE, which utilize representation learning and multi-expert models, could have been relevant baselines. While this omission does not undermine the paper’s key contributions, including a broader range of baselines would have further solidified LTAP’s effectiveness by demonstrating its adaptability across different long-tail learning paradigms.", "other_strengths_and_weaknesses": "Please refer to the above response.", "other_comments_or_suggestions": "The text in Figure 2 is too small, making it difficult to read.", "questions_for_authors": "How does an LTAP-pruned model (30% parameters retained) compare to a manually designed smaller model (with ~30% of the original parameters) trained with long-tail techniques? Does LTAP find a more effective sub-network than simply training a smaller model from scratch?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "66VezkZXfM", "reviewer": "Reviewer_T1Bx", "summary": "This paper introduces an adaptive pruning method called LTAP to address the challenge of handling long-tailed distribution data. The authors propose a multi-dimensional importance scoring criterion and design a dynamic weight adjustment mechanism to adaptively determine the pruning priority of parameters for different classes. Experimental results on various benchmark datasets, such as CIFAR-100-LT and ImageNet-LT, demonstrate improvements in both computational efficiency and classification accuracy for tail classes.\n\n**update after rebuttal**\n\nThank you for your rebuttal. I will keep my score unchanged and remain positive about this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your recognition of our work. We have carefully understood the relevant weaknesses you mentioned and have made the following efforts:**\n\n- **About presentation form**\n\nThank you for your valuable suggestions. We will supplement the theorems and textual explanations in Section III with some visual instructions to help the reader understand. We will illustrate (i) why pruning in the long-tail scenario requires special protection, and (ii) the core idea of distribution-aware pruning through a schematic illustration.\n\n- **About symbolic expression**\n\nThank you for your advice. We carefully proofread the notations used in this paper and improved some inappropriate descriptions.\n\n**Thank you again for your comments and we will continue to work on improving the quality of the manuscript！**", "claims_and_evidence": "Please refer to Strengths And Weaknesses.", "methods_and_evaluation_criteria": "Please refer to Strengths And Weaknesses.", "theoretical_claims": "Please refer to Strengths And Weaknesses.", "experimental_designs_or_analyses": "Please refer to Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "Please refer to Strengths And Weaknesses.", "essential_references_not_discussed": "Please refer to Strengths And Weaknesses.", "other_strengths_and_weaknesses": "**Strengths**\n1. The concept of adaptive pruning to balance model efficiency and performance on long-tailed data is intriguing and has the potential to attract interest from the research community.\n2. The author provides theoretical analysis to strengthen the persuasiveness of the proposed method.\n3. The overall organization of the paper is well-structured.\n\n**Weaknesses**\n1. Personally, I believe that instead of presenting numerous theorems and textual explanations in Section 3, incorporating visual elements would be more effective. This could help readers gain deeper insights into the methodology more easily.\n2. A minor issue: Some symbols used in the formulas are not well explained, sometimes I feel confused when reviewing this paper. I suggest that the author improve the writing and enhance the clarity of the paper’s content.", "other_comments_or_suggestions": "Please refer to Strengths And Weaknesses.", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uIcrBjoj18", "reviewer": "Reviewer_A5yk", "summary": "This paper introduces Long-Tailed Adaptive Pruner (LTAP), a novel pruning strategy designed to enhance neural network efficiency while preserving performance on long-tailed datasets. LTAP addresses this challenge by incorporating multi-dimensional importance scoring and a dynamic weight adjustment mechanism, ensuring that essential parameters for tail classes are retained. The method employs progressive multi-stage pruning, gradually removing redundant parameters. Extensive experiments on multiple benchmark datasets demonstrate LTAP's effectiveness.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your recognition of our work. We have carefully understood the relevant weaknesses you mentioned and have made the following efforts:**\n\n- **About broader references**\n\nWe have incorporated the studies you mentioned into our discussion and references as follows:  \n\nRecent advances in long-tailed learning address class imbalance through diverse strategies:  \n\n1. **Theoretical & Unified Frameworks**: [Wang et al., 2023] proposed a data-dependent contraction technique to unify re-weighting and logit adjustment.  \n2. **Hierarchical Label Variation**: [Yang et al., 2023] introduced **DirMixE**, leveraging Dirichlet meta-distributions to model global-local test-time variations, with variance regularization for stable generalization.  \n3. **Ensemble Calibration**: [Aimar et al., 2023] formulated **BalPoE**, a Fisher-consistent ensemble combining logit-adjusted experts calibrated via mixup.  \n4. **Weight Regularization**: [Alshammari et al., 2023] demonstrated that simple weight decay and MaxNorm constraints outperform complex rebalancing methods.  \n5. **Contrastive Rebalancing**: [Cui et al., 2023] proposed **PaCo**, integrating parametric class centers into contrastive learning to mitigate gradient bias toward head classes.  \n6. **Margin Adjustment Theory**: [Kini et al., 2023] analyzed **VS-loss**, combining additive/multiplicative logit adjustments to optimize margins in overparameterized regimes.  \n7. **Test-Agnostic Aggregation**: [Zhang et al., 2022] developed **SADE**, a self-supervised expert aggregation framework for unknown test distributions.  \n\nThese works collectively advance the field by bridging theory-practice gaps [1,6], enhancing model flexibility [2,7], and simplifying regularization [4], while addressing both label and group imbalances [3,5,6]. \n\n- **About the supplementary experiment on CIFAR-100-LT (IR=10)**\n\nAccording to your suggestion, we supplemented the comparison experiment with IR=10, and the results are shown in the table below.\n\n|Method|F(%)↓|Head↑|Medium↑|Tail↑|All↑|C(%)↑|C/F↑|\n|-|-|-|-|-|-|-|-|\n|BS|100.0|64.9|55.5|0.0|61.5|100.0|1.0|\n|BS + ATO|84.7|45.1|37.2|0.0|41.1|66.8|0.8|\n|BS + RReg|52.1|46.2|38.7|0.0|42.5|69.1|1.3|\n|BS + ours|22.6|56.4|47.3|0.0|53.8|87.4|3.8|\n|LDAM-DRW|100.0|60.8|43.3|0.0|55.4|100.0|1.0|\n|LDAM-DRW + ATO|84.7|41.5|26.2|0.0|37.3|67.3|0.8|\n|LDAM-DRW + RReg|52.1|39.8|25.1|0.0|35.2|63.5|1.2|\n|LDAM-DRW + ours|22.6|51.9|35.7|0.0|47.1|85.0|3.7|\n|DBLP|100.0|65.3|43.4|0.0|58.7|100.0|1.0|\n|DBLP + ATO|84.7|49.5|28.3|0.0|42.3|72.0|0.9|\n|DBLP + RReg|52.1|48.3|27.2|0.0|41.1|70.1|1.3|\n|DBLP + ours|22.6|57.3|30.0|0.0|49.2|83.8|3.7|\n\n\n- **About criteria weight matrix**\n\nTo facilitate your understanding, we have supplemented the images of the weight matrix $D$ in the last training process, where the horizontal axis represents the index of the class, different color blocks represent different pruning weight calculation methods, and the length of different color blocks represents the numerical value of different pruning weights in the weight matrix $D$.\n\n[https://anonymous.4open.science/r/AEFCDAISJ/D_matrix_pic.png](https://anonymous.4open.science/r/AEFCDAISJ/D_matrix_pic.png)\n\n- **About weight adjustment mechanism**\n\nThe observations you make are very sharp. In a single step of equation (6), for the improved category $c$, the weights of all relevant criteria indeed increase by the same value $\\beta$. We understand that this may raise questions about how the mechanism actually chooses the 'most effective criteria', and we provide the following clarifications in this regard:\n\n(i) Equation (6) is not designed to directly identify a single 'most efficient criterion', but rather to implement a more nuanced and multi-dimensional dynamic balancing mechanism.\n\n(ii) Among other things, the reasons for this design rather than directly identifying the 'best criteria' are:\n\n1. In practice, multiple criteria often work together to achieve the best results, rather than a single criterion being dominant.\n2. As reviewer LjiE said, the core goal of this paper lies in introducing distribution-aware capabilities for pruning strategies. The above standard is the carrier of this distribution-aware capability to effective pruning. In future work, we plan to explore mechanisms for more precise and interpretable distribution-aware parameter pruning. \n- **About novelty**\n\nAs stated in the previous point, our goal is to introduce distributional awareness to pruning, and to this end, LTAP redefines the pruning problem in the long-tailed scenario from 'uniform compression' to 'differentiated parameter assignment', a conceptual shift that is significantly innovative in its own right. **This method of directly correlating distribution properties with parameter importance opens a new path for the application of neural network compression on imbalanced data.**\n\n- **About notation and spelling errors**\n\nWe appreciate your comments and will fully revise them in a subsequent version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "- In Section 2.3, the authors state that the updating rule strengthens the criteria that lead to improved class performance. However, in Eq. (6), it appears that if the accuracy of class $c$ improves, the weights for all criteria of class $c$ are increased by the same value $\\beta$. This suggests that the method does not actually select the most effective criteria for class $c$. More explanation is needed to clarify this point.\n\n- The CIFAR-LT-100 dataset with an imbalance ratio of 10 is widely used. Including this dataset could be better but is not strictly necessary.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The authors do not present the final criteria weight matrix $D$ in the experiments. It would be helpful to include it to validate the effectiveness of the weight adjustment mechanism.", "relation_to_broader_scientific_literature": "The proposed method contributes to the broader scientific literature by tackling the challenges of model pruning in the presence of long-tailed data distributions.", "essential_references_not_discussed": "Several relevant works of long-tailed learning have not been cited or discussed, including:\n\n- Parametric Contrastive Learning, ICCV 2021\n- Label-Imbalanced and Group-Sensitive Classification under Overparameterization, NeurIPS 2021\n- Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition, NeurIPS 2022\n- Long-Tailed Recognition via Weight Balancing, CVPR 2022\n- A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning, NeurIPS 2023\n- Balanced Product of Calibrated Experts for Long-Tailed Recognition, CVPR 2023\n- Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition, ICML 2024", "other_strengths_and_weaknesses": "**Strengths**\n\n- Addressing model pruning in the presence of long-tailed data distributions is both meaningful and highly relevant to real-world scenarios.\n\n- The proposed method achieves a relatively significant performance improvement.\n\n**Weaknesses**\n\n- The novelty of the method is somewhat limited. It primarily focuses on combining different criteria with adaptive weights. However, as mentioned in the *Method and Evaluation Criteria* part, the weight adjustment mechanism does not seem effective in selecting the most effective criteria. This aspect may require further explanation.\n- Some notations are not properly defined. For example, the definition of $n_g$ in Eq. (3) and $\\beta$ in Eq. (6) is missing.", "other_comments_or_suggestions": "- There are some typos. In the Implementation Details section, there appears to be a formatting issue with the superscripts in the learning rate and weight decay values.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After review, the paper received four positive evaluations. Following the authors' rebuttal, two reviewers increased their ratings, while the other two maintained their original scores. All reviewers acknowledged the paper's contributions and expressed satisfaction with its technical merits.

The AC concurs with the reviewers' assessments and recommends acceptance.
Decision: Accept (poster)