Exploring Learning Complexity for Efficient Downstream Dataset Pruning
Authors: Wenyu Jiang
Zhenlong Liu
Zejian Xie
Songxin Zhang
Bingyi Jing
Hongxin Wei
Venue: ICLR 2025 Poster (2025)
Keywords: data efficiency
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel and efficient hardness score for downstream dataset pruning.

Abstract: The ever-increasing fine-tuning cost of large-scale pre-trained models gives rise to the importance of dataset pruning, which aims to reduce dataset size while maintaining task performance.
However, existing dataset pruning methods require training on the entire dataset, which is impractical for large-scale pre-trained models.
In this paper, we propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC), to identify informative images and instructions from the downstream dataset efficiently.
Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters.
Specifically, we define the Learning Complexity to quantify sample hardness and utilize a lightweight weights masking process for fast estimation, instead of the costly SGD optimization.
Based on DLC, we further design a flexible under-sampling strategy with randomness (dubbed FlexRand), replacing the top-K strategy, to alleviate the severe subset distribution shift.
Extensive experiments with downstream image and instructions dataset pruning benchmarks demonstrate the effectiveness and efficiency of the proposed approach.
In the images pruning benchmark, DLC significantly reduces the pruning time by 35$\times$ while establishing state-of-the-art performance with FlexRand.

Published as a conference paper at ICLR 2025
EXPLORING LEARNING COMPLEXITY FOR EFFICIENT DOWNSTREAM DATASET PRUNING
Wenyu J Zhenlong Liu', Zejian Xie', Songxin Zhang', Bingyi Jing', Hongxin Weilt Department of Statistics and Data Science, Southern University of Science and Technology State Key Laboratory for Novel Software Technology, Nanjing University
ABSTRACT
The ever-increasing fine-tuning cost of large-scale pre-trained models gives rise to the importance of dataset pruming, which aims to reduce dataset size while maintaining task performance. However, existing dataset pruning methods require training on the entire dataset, which is impractical for large-scale pre-trained mod- els. In this paper. we propose straightforward, novel. and training-frt free hardness score named Distorting based Learning Complexity (DLC), to identify informa- tive images and instructions from the downstream dataset efficiently. Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters. Specifically, we define the Learning Complexity to quan- tify sample hardness and utilize a lightweight weights masking process for fast estimation, instead of the costly SGD optimization. Based on DLC, we further design flexible under-sampling strategy with randomness (dubbed FlexRand), replacing the top-K strategy, to alleviate the severe subset distribution shift. Exten- sive experiments with downstream image and instructions dataset pruning bench- marks demonstrate the effectiveness and efficiency of the proposed approach. In the images pruning benchmark, DLC significantly reduces the pruning time by 35x while establishing state-of-the-art performance with FlexRand.
INTRODUCTION
The paradigm of pre-training and fine-tuning (PT-FT) (Kornblith et al., 2019) is increasingly popular with the rapid advancements in foundation models. Instead of training from scratch, the PT-FT paradigm first pre-trains large and general model on broad data, and then fine-tuning the pre- trained model for a variety of downstream tasks. This is particularly advantageous for employing deep models on edge devices (e.g., telephone) (Dhar et al., 2021), where data privacy is crucial and training from scratch is infeasible due to limited computation resources. Unfortunately, on-device fine-tuning costs can escalate sharply with the increasing scale of data, due to the enormous size of pre-trained models!. This highlights the importance of downstream dataset pruning, which aims to reduce the dataset size for fine-tuning while maintaining the performance on downstream tasks.
In the literature, existing dataset pruning methods generally designed for training from scratch. Those methods quantify the sample importance through a scoring function, which usually requires training the model on all the candidates. For example, EL2N (Paul et al., 2021) calculates the expected norm of loss gradients by training for multiple trials, sometimes even longer than the training time large-scale datasets (Qin al., 2023). When transferring to the PT-FT paradigm, the training or fine-tuning of the entire dataset can be prohibitively expensive and even infeasible as pre-trained models generally have huge parameters. Additionally, current methods typically do not take advantage of the pre-traines model, which is accessible prior to the fine-tuning and potentially impacts the selection of optimal data subset. These concerns motivate us to explore efficient dataset pruning methods that leverage pre-trained models without relying on backpropagation.
In this work, we propose a training-free hardness score, implementing the Learning Complexity (Definition 2) by lightweight pre-training parameters masking process. From the hardness per-
WWrr done SUSTech visiting scholar.
I 'Coresponding author (weihx@sustech edu. cn)
chttps://epoot . hatr/ddataaa
-model

---

Published as a conference paper at ICLR 2025
1.01 ZORN
1033
BOXX
2012
2033
Accuracy (%)
63,15
2614
$6.24
100
S7.41
$1.43
20
Hendey KC0 Porget
Brinesherrg Method
Grde ELOM Opn
Nendes 800 Prrget U
beenseMargi Method
C9
Grofd 1L26
Our
(a)
(b)
Figure 1: Performance comparison of different dataset pruning methods. (a): Time for downstream dataset pruning. The costs of existing training- baased methods are expensive, but we achieve 35x speed up. (b): Accuracy the downstream task. Our method outperforms the random baseline and achieves state-of-the-at performance. More results can be found in Section
spective, the above learning complexity aims to quantify sample importance via calculating the classification loss integral over model sequence with ever-increasing classification performance, which defined as the Learning Path (Definition I). The proposed scoring function, Distorting- based Learning Complexity (dubbed DLC), is efficient learning complexity implementation, by the observation that distorting (masking) the pre-training weights can construct a learning path with- out training fine-tunine Specifically, mask pre-training weights multiple times with random ratios, and the learning complexity can be efficiently approximated by the Monte Carlo method (Caflisch, 1998). As result, DLC can distinguish easy samples from hard ones. To further verify the effectiveness of DLC, we empirically demonstrate a high ranking correlation between DLC and the common optimization-bud based learning complexity. Moreover, we extend DLC to quantify the instruction hardness for efficiently fine-tuning the large language models (LLMs).
To identify informative samples based on DLC, design flexible under-sampling strategy with randomness, named FlexRand, replacing the common top-K strategy. For the multinomial distri- bution of sader-sampling, FlexRand aims to flexibly adjust sampling preference different data regimes. Moreover, the subset from FlexRand should not be significantly shifted away from the original data distribution. Specifically, we randomly select easy and hard intervals respectively, whose lengths are flexibly tuned with splitting hyperparam- eter y to accommodate different data regimes. In this way, we have the following two advantages: (a): FlexRand can adapt to different data regimes; (b): FlexRand avoids severe distribution shift.
same number samples from the
To verify the efficacy and efficiency of our method, we conduct extensive experiments on diverse setups, including different pre-training paradigms, model architectures, pruning ratios. Empirical results show that our method establishes soate-of-thh-all performance over existing methods for dataset pruning. For example, our approach improves the downstream accuracy averaged over various setups from 59.21% % to 60.45% a 1.24% improvement over the random method. Meanwhile, DLC significantly reduces the pruning time by 35x as shown Figure la. On the contrary, the compared methods achieve comparable or even worse performance than the random shown in Figure lb. For LLMs muningg our method consistently outperforms the random with various pre-tree pre-trained models and instruction fine-tuning datasets. In addition, our analysis indicates that DLC sensitive the pre-trained model, and the proposed sampling principle works well with different implementations of learning complexity.
datasets, and
PRELIMINARIES
2.1 BACKGROUND
Setup. In this paper, we consider the setting of supervised multi-class classification with pre- trained encoder. Let X' Rd denote the input space and y {1, ..., K) denote the corresponding label space. The downstream dataset D { 1 is drawn i. from the joint data distribution P Pxxy. We use h to denote the pre-trained encoder and g denote the prediction head. Given the
2

---

Published as a conference paper at ICLR 2025
downstream dataset, train classifier g h: which maps an input to the label space. An ideal classifier fo can be obtained by minimizing the following expected risk: R₂(f) = (e [=[ 0),
: X + RD with learnable parameters 0 IR,
In practice, we optimize the classifier by minimizing the following empirical risk:
RE (I.D)
N Σ [c[((2;; 0),
where L is the commonly used cross-entropy loss with the softmax activation function. Let 2 and denote the feature æ from the initial and fine-tuned h respectively.
Problem statement. The warateeraeteriized networks can be too heavy to optimize with limited computing resources. To accommodate the training budget v (0, 1) for downstream tasks, the dataset pruning aims to select a subset = {( vi) C D (M S N), which can be used to train classifier f by minimizing the empirical risk R2_ (f, D). The classifier from the ideal subset should have the minimal expected risk Rc(f). This can formulated as a bilevel optimization problem with cardinality constraint:
min Re(f) D
s.t.
ara R D)
Enumerating and evaluating all the (M) subsets is non-trivial. An alternative solution performs the dataset pruning by level-set estimation:
pruning,
if S(( y)) Ty preserving. if S((z, y)) 2 T
r((x, 9)
where S(xx, y)) denotes scoring function to quantify sample importance and Ty is threshold specified to accommodate the budget 17 N/K for category y. By convention, the common under- sampling strategy keeps the top-K samples sorted by the S((x, y)) in descending order. However, existing scoring functions typically rely on fine-tuning the pre-trained model on the entire down- stream dataset (Coleman et al., 2020), which is undesirable for efficient edge fine-tuning. To reduce downstream dataset pruning costs, we propose an efficient scoring function solely based on pre-trained model without fine-tuning in the following section.
3
METHOD
To design an efficient scoring function, we first define the Learning Complexity to quantify sample importance from the hardness perspective. Specifically, calculate the integral of classification loss for sample predicted by an improving model sequence, which is defined as:
T) can be defined as a learning path if there exists a positive constant r < <Re(foo) such that lim Rc(fow) r. t-000
Definition (Learning Path) A sequence of model parameters O (0(t)
Thus, the above Learning Complexity can be formally defined follows:
Definition (Learning Complexity) For any training sample (x, y) E D. given a learning path θ, the learning complexity is defined
/ c(f(x; dt
Slo((2,y))=
ST
Intuitively, samples with low learning complexity mean correct classification by a weak classifier in the front part of the learning path, compared the hard ones.
3

---

Published as a conference paper at ICLR 2025
Hard
Iarrd
MwN
Los 6. Opttainntin
0.6
0.6
11.2
0.2
Capacity
Epoch
(a)
(b)
(C)
Figure 2: Ranking correlation between the loss integral over the optimization and masking process. We fine-tune the pre-trained ResNet-18 on the five downstream datasets for 50 epochs and present the CXRB10 (Nodule) results here due to the space limit (see Figure 7 for full results). The easy (hard) sample corresponds the data point with a minimum (maximum) loss integral over the opti- mization. We utilize Max-Min normalization with the number of parameters, i.e., model capacity. We produce models with different capacities via masking (2%, .... 98%) weights of the pre-trained ResNet-18. Average category representation acts as the prototype for classification and loss integral. (b): Loss trends with the optimization time. (c): High ranking correlation coefficient with p 0.54.
scale the loss to the range of (0, 1). (a): Loss trends
3.1 DLC: DISTORTED-BASED LEARNING COMPLEXITY
In this part, we propose an efficient scoring function, Distorting- Learning Complexity (dubbed DLC) by implementing the above learning complexity with lightweight distorting process. Con- cretely, we distort the pre-trained model by masking the initial weights with different ratios, and the resulting models constitute an effective learning path. For clarity of expression, we refer to distorting as masking in the remaining part.
Efficiently constructing the learning path with masking DLC is motivated by the observation that increasing the number of parameters leads to improved generalization (Valle-Perez et al., 2019; Neyshabur al., 2018; 2017). Although the pre-trained models have been fine-tuned on the downstream datasets, the expected risk (test error) on the target task still decreases with the increas- ing model capacity as shown in Figure 2a. Formally, when the capacity of pre-trained model fa at step t + is larger than that at step t, we have R Rc(fa((1)) < RR((f() Therefore, the sequence of pre-trained models with different capacities is viable and can be constructed efficiently without fine-tuning. Specifically, we directly produce the classifiers with different capacities by masking the initial pre-training weights with ratios ranging from 12%, 4%...., .8.%%,%% 100%), ). Note that param- eters with small LI norm are masked first (Han et al., 2015). We provide concrete formulation of the pre-training weights masking in Appendix B. To calculate the loss integral, i.e., Learning Complexity, the average category feature acts as the class prototype, and we further adopt the Monte Carlo method to approximate the loss integral (Caflisch, 1998) for efficiency, instead of enumerating all models with different capacities:
Sr.c((z, ())
[T] Σ
IT ((((x;(()).))
where T' comprises random points É from T. In this way, DLC can distinguish easy samples from the hard ones as shown in Figure 3. To further quantitatively verify the effectiveness of DLC, we implement the common optimization-basee correlation between those two implementations.
learning complexity and demonstrate high ranking
Optimization trajectories viable but costly learning path We first show that the optimiza- tion trajectories constitute a viable learning path. At the training stage, stochastic gradient descent (SGD) and its variants (Sutskever al., 2013; eon Bottou, 1998) are usually used for optimizing the Reup (f, D) starting from the initial 6o. The optimization trajectories can be formulated with the following updating rule:
0 0, - -arl
1
Σ
(B, V[C(f(E;;)) 0.).
B.
4

---

Published as a conference paper at ICLR 2025
Handt
(Nodule)
Weeds (Partheniam)
(prinided)
PGVCAircraft (DHC-))
(Revalver)
Samples distinguished by DLC. In detail, we randomly select five image pairs with the highest and lowest DLC scores from all downstream Dataset (Category) sets. Qualitatively, easy samples with the lowest DLC contain a full and clear structure for classification.
Figure 3:
where B, is the data batch sampled from D and a₁ is the learning rate iteration (I is discrete and starts from 0). Despite not guaranteeing convergence to global optimum, SGD tends to find the local minimum i with flatness (Li et al., 2018; Keskar et al., 2017; Hardt et al., 2016) such that
lim Re(Sec)) = Re(sg) = Re(Seco) 1-00
To empirically verify the asymptotic convergence and decreasing generalization error, we visualize the test loss trend in Figure 2b. Obviously, fo(o) gradually converges to fa with better generalization performance, Following the Definition 1, the optimization trajectory of SGD during the fine-tuning viable learning path, and we calculate the integral of classification loss over the fine-tuning process as the optimization-ban based learning complexity.
We denote the above different implementations learning complexity as Sopt and SMask respec- tively. Despite implementing Sr.c in different forms, we empirically demonstrate the strong ranking correlation between the two as shown in Figure 2c, which indicates the effectiveness of DL.. Specif- ically, the average Spearman's s coefficient (7ar, 2005) between the SOpt and SMask over different downstream datasets as high as 0.51. As a result, DLC can efficiently quantify the sample hardness without dependency training fine-tuning.
An intuitive interpretation As shown in Figure 2b, we can observe that the / can classify the easy example with a smaller loss any iteration t. The same conclusion holds when the pre-trained model has fewer parameters as shown in Figure 2a. In other words, easy samples learned faster can also be learned with fewer parameters.
3.2 FLEXRAND: FLEXIBLE UNDER- SAMPLING WITH RANDOMNESS
From previous analysis, show that DLC discriminate the hard samples from the easy. To identify informative samples with DLC, we further design a flexible under-sampling strategy with randomness, named FlexRand, replacing the common top-K strategy. For the multinomial distribution of under-sampling, FlexRand aims to flexibly adjust sampling preference at different data regimes. On the other hand, the subset from FlexRand should not be significantly shifted away from the original data distribution. To achieve these objectives, we randomly select the same number of samples from the easy and hard intervals respectively, whose lengths are flexibly tuned with splitting hyperparameter to accommodate different data regimes. The sampling probability for (ar, y) can be formulated as follows:
M 2N
if SLo((E, v)) S., SLe((z, v)) S.
(x
21-
where S., is the 7-percentage of sorted samples scores, N is the dataset size and M is the subset size. In this way, we have the following two advantages:
FlexRand can adapt to different data regimes. a data-poor regime, easy samples are more informative, and the opposite holds when more samples can be preserved for fine-tuning (Sorscher et al., 2022). However, the top-K (Easy or Hard) strategy, denoted as T-E or T-H. fixes sampling preference across different data regimes and shows inferior downstream performance compared to the random as shown Figure 4b & 4d. On the contrary, FlexRand can flexibly adjust sampling preference with splitting hyperparameter y accommodate different data regimes.
5

---

Published as a conference paper at ICLR 2025
1.01
1240
0.00
25 Cadonnappig
PB
Calennaple
16
10
Caleaapinn
(a)
(b)
(c)
(d)
Figure 4: Harmful distribution shift from the top-K under-sampling strategies. We conduct downstream dataset pruning with four under-sampling strategies according to the optimization- based learning complexity and DLC. (a): Distribution shift comparison of four subsets for optimization-basee learning complexity. (b): Downstream performance comparison of four sub- sets for optimization-basee learning complexity. (c): Distribution shift comparison of four subsets for DLC. (d): Downstream performance comparison of four subsets for DLC.
FlexRand avoids severe distribution shift. Due the importance of flexible sampling prefer- ence, we change the fixed top-K K strategy to accommodate different data regimes. Despite improve- ment over the original, its performance still lags behind the random under-sampling as shown in Figure 4b & 4d. To analyze and investigate such performance degradation from the top-K (Flexi- ble) strategy (denoted as T-F), compare the distribution distance of different subsets to the full datasets with maximum mean discrepancy (MMD) (Gretton et al., 2012). As shown in Figure 4a & 4c, the top-K sampling incurs severe distribution shifts compared with the random, which dete- riorates downstream performance. On the contrary, FlexRand avoids such distribution shift by ran- domly selecting the same number of samples from the easy and hard intervals respectively. When mixing the top-K (Flexible) with the random strategy (denoted as M-F), corresponding to the ex- treme splitting 7 values, we can find that the distribution shift is still mild with better downstream performance as shown in Figure 4b & 4d.
3.3 EXTENSION TO INSTRUCTION DATASET PRUNING
Recently, large language models (LLMs) (Touvron et al., 2023a;b; Jiang et al., 2024) driven by the PT-FT paradigm have shown incredible capabilities across a wide range of language tasks. De- spite the availability of pre-trained weights, the ever-increasing number of parameters still incur heavy computation overhead for the fine-tuning, which involves instruction-tuning (Ouyang et al., 2022) and reinforcement learning with human feedback (RLHF) (Knox & Stone, 2011) to align the pre-trained LLMs with human preferences. On the other hand, recent works (Zhou et al., 2023a; Cao et al., 2023) indicate that almost all knowledge in LLMs is learned during pre-training. and only limited instruction tuning data is necessary to teach models to produce high-quality output. Therefore, we further extend the learning complexity prune instruction datasets for efficient large language models fine-tuning (Dubey et al., 2024; Team et al., 2024). Specifically, we construct the learning path by masking different numbers of weights in the pre-trained large language models (LLMs), which are sorted by L1 norm in ascending order. Due to the prediction difference in image classification and text generation, we replace the original loss function L( (a; O y) with:
Sr.c((1, 9)) = _ Σ L(S :..:: Vo x; 0()), 93)
where C' is the length of the output y.
4 EXPERIMENTS
In this section, we present the diverse downstream image and instruction datasets pruning bench- marks and empirically validate the effectiveness and efficiency of the proposed method. Moreover, we perform ablation studies to understand better how different components and hyperparameters influence the performance. The code is available in the supplementary material.

---

Published as a conference paper at ICLR 2025
Table 1: Average classification accuracy over 5 diverse downstream datasets and pruning ratios with varying model architectures. Bold numbers are the optimal results, and underline numbers are suboptimal results. Detailed results can be found in Appendix C.2.2.
Accuracy (%)
Time (6)
Method
RN18
RNS0
VIT-S
VIT-B
Average
S((a.y))
Total
Random
55.92 0.21
60.22
61.40 0.01
59.31 0.25
59.21
+
Herding
48.25 0.88
52.26
59.53
1.519
54.31
0.42
1035.0
1035.0 1:
kCG
54.31
58.33
6.15
59.2
0.14
1035.2 4.4
1035.2 +
Forgetting
55.46
59.77
0.19
4.24
0.11
1033.0
1033.0
Least Conf
0.27
60.38 0.00
0.15
59.05
5..06
1033.0
1033.0 :
Entropy
0.35
60.53
6..45
0.26
59.04
1033.0
1033.0 + 1.4
Margin
0.240
6,01
60.42
0,36
0.003
1033.0
1033.0 0.8
CD
0.144
58.30
6.25
57.36 0.08
$7.48
1035.3 + 6.2
1035.3 + 6.2
GraNd
0.0
$7.62
60.41
5.08
5..88
1033.0
1033.0 +
EL2N
52.15
$7.10
6.31
0,444
56.43 0.04
4051.0 8.7
4051.0 8.7
56.72
57.31 0.133
60.90 0.33
59.60 0.26
58.63 a 0.25
1033.0 0.7
1033.0 a
Ours
57.31 + 0.22
61.45 0.02
62.46 0.12
60.59 0.20
60.45 0.04
21.5 0.1
29.5 0.1
CNRIBTO
Deep Weeds
DTD
Accuriey CAp (56
Krenaayy Cap (90
Accureey Gap (SS)
Lamp
Earor
10%
36%
45% 18%
40%
4P%
19%
87%
95
18%
30%
4%%
66%
Praning
Pruning
FGVCAirerad
Sketch
Axeragy
Acarre Gap (%)
Entop thes
Accurac Gap (%0
Accurary
38%
30%
48%
50%
60%
75%
89%
98%
10% 30%
40% Ratio
50%
50%
Figure 5: Trends of average accuracy gap over the random method at different pruning ratios. De- tailed results can be found in Appendix C.2.2.
4.1 SETUP
4.1.1 DOWNSTREAM IMAGE DATASET PRUNING BENCHMARK
Pre-trained encoders For comprehensive evaluation, adopt pre-training encoders with dif- ferent depths, architectures, and pre-training paradigms. In detail, the pre-trained encoders consist of ResNet- 18, ResNet-50 (He et al., 2016), ViT-Small, and ViT-Base (Dosovitskiy, 2020). Those mod- els are pre-trained on the canonical ImageNet- K (Deng et al., 2009) dataset via fully and weakly supervised learning (Yalniz al., 2019) respectively.
Downstream datasets Generally, the downstream dataset is domain-speciffe and different from the pre-training dataset ImageNet-IK. Therefore, we choose diverse downstream datasets from domains (Islam et al., 2021) to construct the large-scale benchmark, including CXRB102. Deep- Weeds (Olsen et al., 2019), DTD (Cimpoi et al., 2014), FGVCAircraft (Maji et al., 2013), and Sketch (Bitz et al., 2012). For hyperparameter tuning, split 20% as the validation set.
4.1.2 DOWNSTREAM INSTRUCTION DATASET PRUNING BENCHMARK
Pre-trained models We utilize public pre-trained language models, including Mistral 7B (Jiang et al., 2023), Llama3 8B (Dubey et al., 2024), and Gemma2 9B (Team et al., 2024).
1CXBB10 is created by selecting 10 balanced classes from the ChestX-ray14 (Borghesi & Maroldi, 2020).

---

Published as a conference paper at ICLR 2025
Table Average classification accuracy (%) of Random / Ours methods on the MMLU benchmark. Different base models are fine-tuned with 50% instruction data from Alpaca Cleaned and Dolly & HH-RLHF, respectively. Bold numbers are the optimal results.
Alpucu Cleuned
Dully &
Base Model
Humunity
Social Science
STEM
Other
Humanity
Social Science
STEM
Other
Mistral 7B
52.44 54.75
71. 72.64
51.74 52.74
68.58 70.20
52.50/ 53.82
69.58 77.44
51.18/ 53.30
68.01 / 66.91
Ilamad
56.75
71.90 72.05
52,33/
69.78/
52.52/ 53.94
69.13/ 72.60
49.92/
69.65
Gemma2 9B
$6.37 58.79
73.29 / 75.25
54,14/56.
71.3/71.52
/ 56.08
71.43/ 73.32
50.23 / 52,99
69.51/70.44
Instruction fine-tuning datasets To align the based pre-trained models, adopt 2 different in- struction fine-tuning datasets: Alpaca Cleaned (Taori et al., 2023) and Dolly & HH-RLHF" More setup details are presented in Appendix C.I.
4.2
MAIN RESULTS
4.2.1
FINE-TUNING FOR IMAGE CLASSIFICATION
Our method achieves superior accuracy with much less time costs. To investigate the effec- tiveness of dataset pruning methods with different pre-trained models, we evaluate the downstream accuracy for four architectures and present the results in Table 1. We find that the proposed method establishes stale-of-the-at performance on all four architectures, highlighting its validity without dependency model capacities and structure. On the contrary, the downstream accuracy of the most competitive baseline Forgetting deteriorates when changing the pre-trained encoder from ViT ResNet. As a result, existing dataset pruning methods perform worse than the random strategy, while our method outperforms the random by 1.24% % on average.
In addition the accuracy evaluated on different models and downstream setups, we also record the time cost of each baseline, including hyperparameter tuning associated with the scoring estimation and under-sampling. In Table 1, time results averaged over downstream datasets are presented. Benefiting from the training- free scoring function and efficient hyperparameter tuning, the proposed method significantly reduces the pruning cost by 35x compared with the one-shot fine-tuning base- lines while achieving state-of-the-ar downstream performance. Notably the acceleration is more remarkable with over-increasing model capacities. Therefore, our method gives rise to the efficient fine-tuning of the over- wee-pparameterized pre-trained models.
Our method works well at various pruning ratios. For efficacy verification of our method on diverse downstream setups, compare the accuracy gap trends the random strategy for each fine-tuning dataset. As shown in Figure 5, the proposed method shows consistent improvement for each downstream dataset different data regimes, indicating the efficacy of the flexible under- sampling strategy with randomness. Overall, the advantage is coherent and more evident when fine- tuning with fewer samples. For example, our method outperforms the random baseline by 2.6% % when removing 90% samples from the Sketch dataset. Despite the randomness in the custom under- sampling, our method demonstrates stability with smaller performance variance. However, the com- pared methods show inferior performance with fluctuating trends, which reveals the drawbacks of the fixed sampling principle and the challenge of diverse downstream dataset pruning benchmarks.
4.2.2 FINE-TUNING LLMS FOR TEXT CLASSIFICATION
In Table 2, we present the text classification results of different dataset pruning methods on the MMLU benchmark. We can find that the proposed method consistently outperforms the random method with various pre-trained models and instruction fine-tuning datasets. Such superiority further demonstrates that learning complexity is universal and applicable in visual and language domains.
4.3 ABLATION STUDIES
To better understand why our method can obtain superior efficiency and efficacy on the downstream dataset pruning task, we perform extensive ablation studies the learning complexity metric, under-
Phttps://hugginge
cm /do1 11 / 1y_hhrlh
8

---

Published as a conference paper at ICLR 2025
0.4
671
0.40
6700
56.8
135
5U.5
57.
512
51.
37a
Makiz
Opilasin
Learning Patl Sizee
Accurncy (76)
Learning Complesity
(a)
(b)
(c)
CANL39
Deeg Woods
14.0
170
(d)
Figure 6: Results for different ablation studies. (a): Ablation on the learning path size. (b): Ablation on the masking strategies. (c): Ablation on the under- under-sampling sampling strategies. T-H, T-E, and T-F denote the top-K Hard, Easy, and Flexible under-sampling, respectively. (d): Ablation on the splitting 7- Blue, green, and orange dotted lines correspond to 10%, 20%, and 30% of the data, respectively.
sampling strategy, and corresponding hyperparameters. We also present a comparison of FlexRand and Dataset Quantization (Zhou al., 2023b) in Appendix C.2.4. As for the setups, we fine-tune the fully pre-trained ResNet-18 18 on 5 downstream datasets with 9 pruning ratios with default parameters.
Learning path size In Figure 6a, we investigate the effect of the learning path size on the rank- ing correlation and downstream accuracy. In particular, we calculate the ranking correlation by the class-averaged Spearman correlation coefficients between the optimization-base ity and masking-basse The results show the ranking correlation between the two scoring functions is significant and saturated at size 5. As a result, the proposed method can estimate the learning complexity efficiently and establish superior classification performance.
learning complex-
Weight masking principle To elucidate the choice of weight masking principle, we compare the L1-based model pruning method with the random with 10 different seeds. Class-averaged Spearman correlation coefficients and accuracy are presented in Figure 6b. We find that randomly masking the pre-training weights incurs unstable ranking and inferior performance. On the contrary, the L1 Ll-based weight masking is deterministic with consistency and superiority across different seeds. Therefore, we choose to mask the weights according to the LI norm.
Under-sampling strategy As shown in Figure 6c, we show downstream accuracy with different under-samp sampling strategies to verify the effectiveness FlexRand. FlexRand outperforms the random by 1.39% with DLC. However, the easy and hard strategies lag far behind the random baseline. Compared with the flexible strategy, the improvement demonstrates the efficacy the randomness, which alleviates the distribution shift. Besides, the optimization-bant based learning complexity with the FlexRand also established the ofate-of-the-ar performance. This indicates that the proposed sampling principle works well with different implementations of learning complexity.
Interval splitting To examine the effects of interval splitting, set the 7 by traversing {0.05, 0.10, 0.95), and present the results in Figure 6d. We can find that small splitting values (r < 0.5) generally deliver better performance in data-poor regimes. Therefore, random baseline. While y tuned by the linear classifier does not always achieve the optimal, significantly reduces the cost of hyperparameter tuning and establishes superior performance.
method outperforms the

---

Published as a conference paper at ICLR 2025
Table Average accuracy (%) over downstream datasets and varying pruning ratios with weakly pre-trained ResNet-18 18. Bold and underline numbers are the optimal and suboptimal results.
Method
10%-30%%
40% 40%%
70%--0055
Average
CXR
DW
DTD
FA
Sk
CXR
DW
DTD
FA
Sk
CXR
DW
DTD
FA
Sk
Random
27.27
87.42
54.67
32.15
$2.80
66.32
55.36
72.81
34.68
94.45
69.44
66.97
77.35
61.43
Herding
76.40
53.29
kCG
25 48
87:30
49.15
28:50
55.83
31.27
93.25
65.67
54.74
73.63
35.30
94.66
70.29
67.42
77.60
60.67
Forgetting
28.00
86.53
54.99
35.94
58.75
34.17
91.89
66.86
59.86
73.12
36. 18
94.33
70.29
68.39
76.92
62.45
Least Coaf
27.13
83.48
55.60
30.63
58.07
32.93
89.71
64.49
51.20
71.09
36.20
93.26
69.18
64.24
76.68
60.26
Entropy
35.72
64.27
76.80
Margin
28.12
83:20
55.80
30.52
58,43
33.92
89.85
64.21
50.59
71.35
36.52
93.41
68.63
64.30
76.61
60.36
CD
23.00
87.07
52,97
34,49
54.85
32.03
94,13
66.14
59.53
73.73
36.07
95,11
70.52
68.67
78.20
61.77
GraNd
21.77
86.40
53.43
35.08
54.24
30.70
94,04
66.02
60.38
73.38
36.03
94.85
70.35
68.91
78.00
61.57
EL2N
53.70
60.94
36.17
95.08
70.35
68.91
77.75
61.19
Ours
29.69
88.49
57,45
34,67
58.41
34,79
93.40
67.12
36.97
73.65
37.42
94.90
70.66
67.29
17.79
62.85
Table 4: Average classification accuracy (%) over downstream datasets and pruning ratios for different scores. Bold numbers are the optimal results, and underline numbers are suboptimal results.
(((),)))
ResNet-50
VIT-Small
ViT-Base
Average
CXR
BW
DTT
FA
Sk
CXR
DW
DTT
FA
Sk
CXR
DW
DTD
Sk
Random
31.73
91.99
67.44
06.75
63.63
33.13
92.60
66.56
48.02
66.27
31.88
91.85
64.81
41,91
61.10
60.38
Original
32,90
92.50
68.29
47.61
64.48
34.12
92.99
65.79
48.41
66.93
32.63
92.17
65.26
45.63
65.06
61,08
Transfer (RN18)
90.43
68.05
46.78
34,71
92.93
67.45
48.37
32.94
91.82
5 DISCUSSION
Is the proposed method affected by the quality of pretraining datasets? While our method provides outstanding performances models pre-trained by fully supervised learning, it is underex- plored whether the proposed method can work well on those models pre-trained on low-quality data, i.e., weakly supervised learning. To investigate the effectiveness of our method in this scenario, utilize the ResNet-18 18 model pre-trained on the dataset with missing labels (Yalniz et al., 2019). We present the classification accuracy of those pre-trained models by linear probing in Appendix C.2.5. As shown in Table 3, the proposed method keeps state-of-thearr performance and outperforms the random baseline by 1.42% % on average, not affected by the quality of the pre-training dataset. We provide detailed results in Appendix C.2.3.
Can we use small models to select samples for large models? For better efficiency in scoring estimation, we transfer the proposed scoring function from a pre-trained ResNet-18 to models with more parameters. The pruning time cost and accuracy averaged over 5 downstream datasets with 9 pruning ratios are presented Table 4. Notably, its improvement over the random by 0.32% % implies the effectiveness of our method. However, find that transferring does not achieve the original performance while consuming less time.
CONCLUSION
In this paper, we propose Distorted-based Learning Complexity (DLC), novel and straightforward hardness score without relying on training or fine-tuning. We construct the learning path by dis- torting (masking) the initial pre-training weights with different ratios and calculate the loss integral via the Monte Carlo method for efficiency. Additionally, we design a flexible under-sampling strat- egy with randomness, named FlexRand, which can adapt to different data regimes while avoiding severe distribution shift. We conduct extensive experiments on the downstream image and instruc- tion datasets pruning benchmarks, and the results show that our method establishes state- state-of-the-ar performance. Notably, our method significantly speeds up pruning by 35x in the visual benchmark.
The success of our method relies on the capability of pre-trained models. Thus, method may fail to improve the downstream performance if the pre-trained model cannot provide meaningful representations.
Linitations
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the Shenzhen Fundamental Research Program (Grant No.JCY120230807001 1809020) and the SUSTech-NUS Joint Research Program. Bingyi Jing is sup- ported in part by the National Natural Science Foundation of China under grant 12371290. We gratefully acknowledge the support of the Center for Computational Science and Engineering at the Southern University of Science and Technology, and the Collaborative Innovation Center Novel Software Technology and Industrialization at Nanjing University for our research.
REFERENCES
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. European Conference on Computer Vision, pp. 137-153, 2020.
Andrea Borghesi and Roberto Maroldi. Covid-19 outbreak in italy: experimental chest x-ray scoring system for quantifying and monitoring disease progression. La Radiologia Medica, 125(5):509- 513, 2020.
Russel E Caflishh. Monte carlo and quasi- quasi-monte monte carlo methods. Acta numerica, 7:1-49, 1998.
Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: high-quality instruction data selec- tion for large language models. arXiv preprint arXiv:2307.06294, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL https:/ openreview net www.ummddd-uuuoooo..
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: a large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255. leee, 2009.
Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A survey of on-device machine learning: an algorithms and learning theory perspective. ACM Transactions on Internet of Things, 2(3):1 49, 2021.
Alexey Dosovitskiy. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint www.v:20000019990000 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Mélanie Ducoffe and Frédéric Precioso. Adversarial active learning for deep networks: margin based approach. ArXiv, abs/ htbs/8002.0944/ 2018. URL https: / /aaippi....//../ htttps::/ap..aemaaees. sew.smiccctcichoo..............e org/Corrus ID: 3550773.
Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM Transactions Graphics (TOG), 310) 2012.
L con Bottou. Online learning and stochastic approximations. Online learning in neural networks, 17(9):142, 1998.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891 2020.
Amirata Ghorbani and James Zou. Data shapley: equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251, PMLR, 2019.

---

Published as a conference paper at ICLR 2025
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: comprehensive library for coreset selec- tion in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: stability of stochastic gradient descent. In International conference on machine learning, pp. 1225-1234. PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033000 2020.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 09685, 2021.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A broad study on the transferability of visual representations with contrastive learning. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 0845-8855, 2021.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Rishabh K. Iyer and Jeff A. Bilmes. Submodular optimization with submodular cover and sub- modular knapsack constraints. In Neural Information Processing Systems, 2013. URL https: //api. semant wwwchlar.......ooooooccoo. wacholaa..rrg/oorpuusse 133315673.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.040088 2024.
Vishal Kaushal, S. Kothawade, Ganesh Ramakrishnan, Jeff A. Bilmes, and Rishabh K. Iyer. Prism: a unified framework of parameterized submodular information measures for targeted data subset selection and summarization. ArXiv, abs/2103.00128, 2021. URL https://aapt semant comanticscholas
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: generalization gap and sharp minima. In International Conference Learning Representations, 2017. URL https: /pppnrevvew.. net/fformm ? i d= Ygg.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021.
W Bradley Knox and Peter Stone. Augmenting reinforeement learning with human feedback. In ICML 201 Workshop on New Developments in Imitation Learning (July 2011), volume 855, pp- 3, 2011.
12

---

Published as a conference paper at ICLR 2025
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1894. PMLR, 2017.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661- 2671, 2019.
Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh K. Iyer. Similar: submodular in- formation measures based active learning in realistic scenarios. In Neural Information Processing Systems, 2021. URL https:tpp..... appi semant cschol ar. www.o///oopssse : 2
Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation frame- work for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 8780-8802. PMLR, 2022.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Robert F Ling, Residuals and influence in regression, 1984.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arViv: 130.,5151, 2013.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 650-663, 2021.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. Advances in neural information processing systems, 30, 2017.
Behnam Neyshabur, Zhiyuan Li. Srinadh Bhojanapalli,, Yann LeCun, and Nathan Srebro. To- wards understanding the role of wer-paramettriztioon in generalization of neural networks. arXiv preprint 1 12066 2018
Alex Olsen, Dmitry A Konovalov, Bronson Philippa, Peter Ridd, Jake C Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, al. Deepweeds: a multiclass weed species image dataset for deep learning. Scientific reports, 91):2005, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: an imperative style, high- performance deep learning library. Advances in Neurul Information Processing Systems, 32, 2019.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng. Daquan Zhou and Yang You. Infobatch: lossless training speed up by unbiased dynamic data pruning. arXiv preprint arXiv:2303.044 04947, 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: a core-set approach. In International Conference Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2025
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536, 2022.
Iyy Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pp- 1139-1147. PMLR, 2013.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: an instruction-following llama model. https:/ hete:://gittttt wwwccoot//tteeeeeeeeeeeeee ttt//ttttsuuuuueeeeee labllllaaaaateeeee anforr__alaaca, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. International Conference on Learning Representations, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09888998., 2023b.
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes be- cause the parameter-function map is biased towards simple functions. In International Confer- ence on Learning Representations, 2019. URL https: /ttp///pnneeeeeeeeeeeeeeeeeeeeep net forum?id- rye4g3AqFm.
Kai Wei, Rishabh K. lyer, and Jeff A. Bilmes. Submodularity in data subset selection and ac- tive learning. International Conference on Machine Learning, 2015. URL nttpss://pp..
wcc.ooar..cccll...coommmom org/ CCrppsrr 9176553...
Max Welling. Herding dynamical weights to learn. In International Conference on Machine Learn- ing. pp. 1121-1128, 2009.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: a universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Reprexentations, 2023. URL https: opwnrevieeweeeeeeeeeeeeeeeeeeeee net/ forum? id=7D5EECbOoaf.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint (005:/99000000 2019.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: reducing training data by examining generalization influence. In International Conference on Learning Representations, 2022.
Shuo Yang, Zhe Cao, Sheng Guo, Ruiheng Zhang, Ping Luo, Shengping Zhang, and Liqiang Nic. Mind the boundary: coreset selection via reconstructing the decision boundary. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learn- ing, volume 235 Proceedings of Machine Learning Research, pp. $5948-55940 PMLR, 21-27 Jul 2024. URL https: https::/proceeeee/ee http///pooooecce/e//eee/ee///sse mll wwt.ggss.m..........mmmmmmmmmeee press v235/ yang 4b. html.
Jemold H Zar. Spearman rank correlation. Encyclopedia of Biostatistics, 7, 2005.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In International Conference on Learning Representations, 2023, URL https: //openreview. net. fort//orumm/m wtw.rmmmddoook...wco...o d=Q.kvi.6wCBvi.
14

---

Published as a conference paper at ICLR 2025
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat,, Ping Yu, Lili Yu, et al. Lima: less is more for alignment. arXiv preprint arXiv:2300..1226, 2023a.
Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17205-17216, October 2023b.
15

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
To accommodate the computation budget, dataset pruning reduces the number of training iterations by selecting the most informative subset. naive solution evaluates the downstream test perfor- mance drop caused by excluding each possible subset, which also underlies the influence function (Ling, 1984; Koh & Liang. 2017: Feldman & Zhang, 2020; Yang et al., 2022) and data Shapley values (Kwon& & Zou, 2022: Ghorbani & Zou, 2019). While intuitive, the computation cost is unac- ceptable because it requires training the model with an additional classification head for 2 times given dataset with size n. Therefore, the inefficiency has cast a key challenge to identify the critical data subset. Indeed, another line of work turns to replacing the costly counterfactual enu- meration with a level-set estimation. Specifically, the most informative samples identified based on customized scoring functions, which can be roughly divided into the following categories:
Geometric: Herding (Welling, 2009), k- E-CenterGreedy (Sener & Savarese, 2018), and Con- textual Diversity (Agarwal et al., 2020) define the scoring function as the sample similarity in the feature space and redundant points are removed for better diversity. SSP (Sorscher et al., 2022) adopts the k-means algorithm with features from the encoder to search the coreset. Moreover, CCS (Zheng et al., 2023) is an innovative data selection method, which maintains coverage in high-density areas of dataset.
Uncertainty: Least Confidence, Entropy and Margin (Coleman et al., 2020) select the most uncertain samples that may have greater impact on model optimization.
Forgetting (Toneva et al., 2018), GraNd and EL.NN (Paul al., 2021) mea- sure the sample informativeness according to the error or loss during the course of training, and the samples easy to learn are pruned.
Error Loss:
Decision boundary: Adversarial DeepFool (Ducoffe & Precioso, 2018) and Contrastive Active Learning (Margatina et al., 2021) preserve samples hard to separate based on the distance to the decision boundary. BoundarySet (Yang al., 2024) is a novel coreset construction method by selecting training samples to reconstruct the decision boundary of deep neural network learned on the full dataset.
Gradient matching: CRAIG (Mirzasoleiman et al., 2020) and GRAD-MATCH (Killam- setty et al., 2021) search for a subset with weighted gradients close to the full gradients.
Submodularity: FASS (Wei et al., 2015), PRISM (Kaushal et al., 2021), and SIMILAR (Kothawade et al., 2021) construct the coreset by maximizing the submodular functions (lyer & Bilmes, 2013), such as Graph Cut, Facility Location and Log Determinant (lyer et al., 2021), which naturally measures the diversity and information.
To improve the robustness of different dataset pruning methods, the concept of Moderate Coreset (Xia al., 2023) discussed. Specifically, given any score criterion of data selection, different scenarios prefer data points with scores in different intervals. However, the above-predefined scoring functions introduce additional pruning costs due to the parameter updating, which not negligible (Qin et al., 2023). To reduce downstream dataset pruning costs, we propose efficient scoring function named DLC solely based on the pre-trained model without fine-tuning.
B METHOD DETAILS
In this section, we provide a concrete formulation of the pre-training weights masking operation. Given the pre-training weights W E In and masking ratio r [0, 1),, the masking matrix M E f0, I is constructed by:
(0, if Wisl <T
M
11 if W Tr,
where t, is the (n * m * r) element in (Wi,.., WT Wnom) sorted by LI norm in ascending order. Finally, the masked pre-training weights W can be formulated as:
W W o M.
16

---

Published as a conference paper at ICLR 2025
C EXPERIMENTS DETAILS
C.I SETUPS
C.I.1 DOWNSTREAM IMAGE DATASET PRUNING BENCHMARK
Table 5: Downstream datasets for fine-tuning.
Domain
Downstream Dataset
Training Size
Test Size
Number of Classes
Total Size
Medical
CXRBI0
4000
1000
10
5000
Natural
DeepWeeds
6400
1600
8000
Texture
DTD
3760
1880
5640
Manufacture
FGVCAircraft
9000
2000
100
10000
Illustrative
Sketch
16000
4000
250
20000
Baselines. Aside from the random, we also compare the following methods: Herding (Welling, 2009), kCG (Sener & Savarese, 2018), CD (Agarwal et al., 2020), Forgetting (Toneva et al., 2018), Least Conf, Entropy. Margin (Coleman al., 2020), GraNd, EL2N (Paul et al., 2021), and SSP (Sorscher et al., 2022). If not specified, one-shot fine-tuning on the full downstream dataset is necessary to estimate the above scoring functions. For the masking- based learning complexity, the size of the learning path increases gradually, starting from the one with initial pre-training weights, until the ranking correlation with previous no longer changes. The hyperparameter y different data regimes is selected from the range {0.05, 0.10, 0.15, .... 0.95} by the validation accuracy.
Fine-tuning. We sequentially attach a linear layer on top of the pre-trained encoder for the down- stream image classification. Then, the above classifier is fully trained on the pruned dataset for 50 epochs using SGD with a momentum 0.9, a weight decay initial learning rate is 1c-3 and decays by factor of 10 the 25th and 37th epochs. For a fair com- parison, we preserve the model weights at the last epoch for classification evaluation. The above settings are the same for all models and datasets.
le-5, and a batch size of 128. The
Evaluation We prune the downstream datasets at 9 pruning ratios, ranging from 10% to 90%, for thorough verification and comparison. For example, keep 10% of each category in the original dataset when the pruning ratio is 10%. The efficacy and efficiency of fine-tuning dataset pruning is evaluated by measuring the following metrics: (1) downstream accuracy at different pruning ratios. (2) total time for scoring estimation, under-sampling, and associated hyperparameter tuning.
Implementation details. To ensure reliable reproduction, we have run the compared baselines using the DeepCore (Guo et al., 2022) library".. The code is based on PyTorch (Paszke et al., 2019).
C.1.2 DOWNSTREAM INSTRUCTION DATASET PRUNING BENCHMARK
Fine-tuning. We fine-tune the base model for epochs using SGD with batch size of 32, momentum of 0.9, learning rate of 7e-6 scheduled by cosine function, and weight decay of 0.01. Note that the learning rate increases linearly at the warmup stage (the first 100 steps). Due to the prohibitive large scale of base pre-trained models, LoRA (Hu et al., 2021), which only updates a small number of model parameters for efficiency.
the parameter-effficent fine-tuning method
Evaluation We prune 50% instructions from the original datasets. For instruction-tumed models, the MMLU with shots is the common benchmark (Hendrycks et al., 2020).
Implementation details. Regarding the pre-trained models and instruction datasets, we use the HuggingFace library Fine-tuning is based on the PEFT library", and evaluation is based on the NVIDIA L40.
LM-Eval library". The code is based on PyTorch and all the experiments
"We perform three full fine-tuning with different seeds estimate EL.NN accurately. 'https: //tptttttbtbbb/// com/Patrickktt/eeeeeeettt DeepCore 'nttps: httgs//gggggggggag.geeeeeeee Pnttps://gt thub. com/huggingg face///enaaceeeee "https: thub. htt.:/ooo////////ttte http///eetterrrree /1m-eva uat on harnass
17

---

Published as a conference paper at ICLR 2025
C.2 RESULTS
C.2.1
MORE RESULTS FOR DLC
11
Hard Easy
Hard Easy
Ww
W
Text
Toxt
Las 8 Optentratie
0.4
2.4
9.4
0.0
20%
50%
50%
100%
0.0
40
0.0
02
8.4
11.
1.8
1.0
Capacity
Epoch
Lan Masitng
(a)
(b)
My momm
Oyttnttatee
Teal
Tiat
0.2
0.2
Capacit
Epoch
(d)
(c)
1.0 Band
Hard
Cpttedralie
0.6
0%
30%
50%
80%
100
3.0
25
0.0
82
Caparity
E Poch
Las Masiing
(g)
(h)
3.2
Easy
3.5
Easy
Teat
Tist
0.4
58%
Capadit
Epoch
Masking
(k)
Haad
Thard
0.F
3.6
1%
50%
40%
100%
Caparity
Fpoch
Loa Maskinz
(m)
(n)
(0)
Figure 7: Ranking correlation between the loss integral over the optimization and masking process. The first fifth rows represent the results on CXRB10 (Nodule), Deep Weeds (Parthenium), DTD (Sprinkled), FGVCAircraft (DHC-6), and Sketch (Revolver) datasets respectively. (a,d.g...m:: Loss trends with the number of parameters. (b,e,h,k,n): Loss trends with the optimization time. (c.f,I,L,o): High ranking correlation coefficient with p {0.54, 0.46, 0.65. 0.68, 0.55).
18

---

Published as a conference paper at ICLR 2025
C.2.2 DETAILED MAIN RESULTS
Table 6: Classification accuracy (%) over 5 diverse downstream datasets and pruning different average ratios with the fully pre-trained ResNet-18.
Method
CXRB10
DeepWeedss
FGVCAireraft
Sketch
Averagg
20%
50SS
80%
50%
80%
20%
50%
20%
80%
20%
Kandom
24.90
31.13
33,00
83.83
91.71
93.42
51.83
63.44
67:25
20.87
42.75
55.44
41.19
63.88
70.82
55.77
Herding
24.73
31.20
33.87
45.50
76.65
91.17
31.38
54.43
66. 55
13.85
53.36
52.76
79.
$7.59
69:40
47.63
LCG
23.63
29.73
34.00
82.58
91.71
93.25
46.99
64.20
68. 19.19
15:45
37:72
53.80
35.00
63.96
71.13
54.06
CD
20.77
29.33
34.33
81.88
93.02
94.21
43.65
63.95
68.48
15.78
36.73
54.08
33.03
61.68
70.69
53.44
Coef
70
87.27
56 26.25
Entrogy
27.53
32.37
91.94
Margin
32.67
34.10
57.75
92.02
52.93
61.26
67.04
48.56
64.63
GraNd
19.50
30.75
34.27
80.48
92.67
94.29
40.05
62.66
68.3
15.05
35.52
54.36
31.14
59.97
70.993
$2.68
EL.NN
14.80
28.67
34.27
80.54
92.75
94.08
18.35
62.61
68 85
14.94
55.71
54.29
79.98
59.71
71.25
$2.05
Forgetiing
25.80
33.10
35.63
81.33
89.94
93.10
50.39
64.20
68 26
19.13
39.90
55.11
40.74
63.08
71.03
55.38
Ours
27.23
32.87
35.33
85.21
92.19
93.58
53.67
65.39
68
22 78
43.61
56.02
44.37
65.05
71.15
57.08
Table 7: Classification accuracy (%) over diverse downstream datasets and pruning different average ratios with the fully pre-traie pre-trained ResNet-50.
Method
CXRB10
DeepWeeds
DTD
FGVCAircraft
Sketch
Average
20%
50%6
30%
20%
30%
80%
201
30%
30%6
20%
30%
80%
20%
30%
S076
Random
26.30
31.67
36.57
86
85.50
93.75
94.85
39.95
70:43
73.44
24.78
51.16
63.81
45.66
69.01
75.26
60:19
Herding
32.47
36.00
83.81
70.62
17.90
35.52
73.78
30.70
95.58
19.07
39.88
31.27
95.67
20.37
39.63
75.24
Lease Coof
26.93
32.63
36,03
83.02
90.31
93.94
60.82
68.74
72.64
28.47
52:33
64.24
52.01
68.23
74.52
00.36
Fntropy
26.70
32.13
35.40
83.52
90.40
94.
60.96
68.97
72.57
28.78
$3.02
64.81
52.27
68.54
74.47
60.44
Marpin
25.60
33.37
35.87
83.23
90.10
94.25
60.14
68.71
72.13
28.26
51.73
64.42
51.13
68.63
74.53
60:14
GraNd
20.10
31,63
36.90
82.98
94.40
96.04
51.45
68.76
74.06
20.
45.09
63.59
38.89
65.98
74.93
$7.66
BL2N
32.40
36.00
74.36
19.00
63.38
36.75
57.21
Forgetting
27.13
32.70
37.00
92.00
94.98
57.98
74.18
23.48
46.78
59.84
Ours
34.63
37.10
87.77
94.06
69
61.58
70.69
74.73
72
$1.59
64.56
85
69.83
75.60
61.55
Table 8: Classification accuracy (%) over diverse downstream datasets and pruning different average ratios with the fully pre-trained ViT-Small.
Method
CXRB10
DeepWeeds
DTDD
FGVCAircraft
Sketch
Average
20%
50%
30%
20%
50%
80%
2015
50%
5056
20%
50%
80%
20%
50%
S096
Random
27.93
33.20
36.73
85.00
94.40
95.55
58.28
68.63
73.62
28.61
$2.00
63.68
$1.46
71.64
76.59
61.40
Herding
31.70
37.13
91.33
95.04
72.82
25.55
47.20
63.16
49.86
76.24
39.30
38.60
93.46
95.77
73.81
30.08
63.34
52.71
76.50
32.87
37.03
95.98
73.99
27.30
47.78
71.92
60:87
Least Conl
30.37
35:27
37.07
80.02
90.04
94.40
39.49
68.53
72.02
29.91
43.86
61.59
54.20
69.44
76.08
60:49
Entropy
30.57
35.17
37.40
80.50
90.69
94.60
28.83
68.87
72.00
30.70
49:38
61.54
54,27
69.86
75,93
00.67
Margin
30.93
34,57
38.00
80.46
90.65
94.56
59.11
68.17
72.46
29.30
43.65
61.26
54.24
70.13
75.95
60.59
GraNd
22.73
32.07
37.27
89 89.54
94.94
95.81
53.67
69)
74.31
28.20
$4.09
64.26
46.19
71.20
76.86
60:20
EL.N
21.47
32.57
37.13
85.60
95.
96.04
52.00
68.72
73.65
27.60
53.46
64.35
46. 18
71.39
77.06
60.36
Fongetingg
36.50
38.70
95.81
73.92
53.72
62.72
Table 9: Classification accuracy (%) over diverse downstream datasets and pruning different average ratios with the fully pre-trained ViT-Base.
Methed
CXRB10
DeepWeeds
DTD
PGVCAdreraft
Sketch
Average
20%
50%
50%
20%
50%
80%
20%
50%
50%
20%
50%
80%
20%
50%
80%
Random
26.20
32.07
34.30
$6.50
93.15
94.71
55.32
67.43
70.25
26.38
19.72
61.22
45.03
69.28
75.46
59.13
Herding
24.33
32.40
35.67
82:46
90.94
93.77
49.69
64.41
70.51
20.97
44.17
60.41
39.68
67.61
75.07
56.79
kCG
28.25
31.77
35.23
94.58
70.43
43.94
61.03
41.86
69.83
76.03
39.02
31.80
71.17
21.16
37.01
57.31
Couf
28.70
35.07
30.70
00:37
52.60
69.11
75.13
59:13
Entropy
29.23
33.87
35.93
78:23
88.
93.52
36.10
64.70
69
30.90
50:57
60.72
53.16
69.26
75.03
39.32
Margin
29.40
31.47
36.60
78.53
$8.96
93.60
56.51
65.12
69 89
29.05
50.11
60.58
53.17
68.97
75.46
50
GraNd
22.07
31.83
35.60
85.48
94.08
94.81
44.08
66.26
71.28
18.52
46.23
62.15
36,24
67.37
75.68
56.82
EL2N
19.03
29.87
34,23
84.77
94.60
95.02
41.79
65.80
70.35
20
45.53
61.86
33.47
60
76.13
56.00
Fargetting
33.77
34.77
71.01
50.25
60.70
19

---

Published as a conference paper at ICLR 2025
C.2.3
MORE RESULTS FOR WEAKLY PRE-TRAINED MODELS
Table 10: Classification accuracy (%) over 5 diverse downstream datasets and pruning different average ratios with the weakly pre-trained ResNet-18.
Method
CXRB10
DeepWeeds
FGVCAireraft
Sketch
Averagg
20%
50%
80%
20%
50%
20%
80%
20%
Kandom
26.17
32.43
35.07
57.04
53.15
94.52
54.86
66.19
69.29
32:57
55.89
66.65
56.01
72,90
77.13
61.31
Herding
23.60
30.57
35.37
51.60
78.96
90.75
31.99
56.13
66.33 06.81
24.78
50.96
66. 33
44.05
67.25
76.35
$3.03
LCG
24.27
30.33
35.57
87.42
93.81
94.75
49.10
65.18
70.
29.05
55.44
67.
14.98
73.11
77.22
60:52
CD
21.87
30.33
35.70
87.15
94.15
94.96
53.25
66.21
70.96
35.36
59.82
68.59
54.17
73.25
78.09
61.59
Coef
31.87
68.79
64.19
60.22
Entrogy
28.50
32.20
35.13
90.19
93.27
68.76
58.45
60.32
Margin
28.70
36.00
90.04
93.58
56.26
67.85
30.29
58.29
71,44
76.46
GraNd
21.33
30.80
35.33
86.00
94.06
94.77
54,22
66.54
70.11
36.14
00.41
08.75
54,16
73.03
77,80
61.56
EL.NN
18.63
10.37
35.40
87.61
94.5
95.00
53.51
66.58
70.21
35.56
61.23
60.07
53.07
72.81
77.33
61 (1.40
Forgetingg
29.10
34.60
35.63
86.85
92.31
94.54
54.02
67.18
70.62
36.56
60.58
68.05
58.85
72.95
76.73
62.57
Ours
29.90
34.33
36.83
88.50
93.46
95.00
$7.48
66.67
70.37
33.38
57.63
67.32
58.07
73.14
77.37
62.63
Table 11: Classification accuracy (%) over 5 diverse downstream datasets and pruning different average ratios with the weakly pre-trained ResNet-50.
Method
CXRB10
DeepWeeds
DTD
FGVCAIrcraft
Sketch
Average
20%
50%6
30%
20%
30%
80%
201
30%
30%6
20%
30%
80%
20%
30%
S076
Random
27.63
34.37
37.23
90.31
95.33
96. 64
63.21
72.75
75.53
42.55
66.65
76.64
66.80
76.62
82.13
67.08
Herding
31.33
37.03
96.85
74.08
34.58
54.66
80.72
58.
33.17
96.17
97.02
66.24
32.50
97.04
60.94
72.66
76.50
46.56
78.06
66.09
79.13
67.24
Lease Coof
28.47
35:70
$7.80
87.73
92.92
96.25
60.83
70.53
75:00
41.63
63:31
74.36
67.45
27.27
81
06.06
Fatropy
27.97
34.47
36.87
87.19
93.48
96.06
61.06
70.46
75. 14.14
41.82
63.56
75.01
67.52
77.35
81.52
65
Marpin
29.23
34,33
37.00
87.40
93 48
96.15
61.15
70.25
75.
41.34
63.25
74.80
67.40
77,43
81.67
66.00
GraNd
24.30
32,63
37.03
91.19
96
96.88
60.87
73.51
76.47
47.36
70.53
77.75
65.63
79.15
K2.15
67. 46
BL2N
24.37
33.10
37.00
96.56
60.34
64.62
82.27
67.41
Forgetting
35.43
36.00
96.63
62.22
73.26
76.21
46.72
77.46
67.10
78.88
67.77
Ours
30:40
35.73
37.87
91.00
95.00
96.88
64.57
73.55
76.37
43.66
77.23
67.85
79.53
87.36
68.
Table 12: Classification accuracy (%) over diverse downstream datasets and pruning different average ratios with the weakly pre-trained ViT-Small.
Method
CXRB10
DeepWeeds
DTTD
FGVCAircraft
Sketch
Averuge
20%
50%
30%
20%
50%
80%
201
50%
5056
20%
50%
80%
20%
50%
S056
Random
28.50
33.80
37.00
91.31
95.25
96.73
60:71
73.63
76.67
51.87
56.81
68.46
54.56
74.08
79.14
63. 63.94
Herding
33.77
36.30
94.21
96.60
76.47
27.14
67.56
50.98
79.07
61.26
33.77
36.63
96.98
77.65 07.80
33.60
57.92
67.70
54.56
79.33
64.13
33.00
37.43
91.17
96.67
73.81
77.70
33.64
68.76
49.98
Least Conl
27.77
33.30
36.30
86.21
92 02.29
96.08
61.79
71.70
76.06
31.76
52.78
66 68.18
56.90
72.39
78.55
62.67
Entropy
29.80
34.97
35,00
85.10
92.67
95.98
62.04
71.56
75.05
30.78
53.20
68.98
57.19
72.04
78.45
62.76
Margin
28.70
33:93
37.00
85.96
92.65
95.92
62.30
72.23
75.85
31.95
53.44
63.97
56.95
72.43
78.27
62.90
GraNd
20:17
31.67
36.23
91 91.44
96.54
97. 10
54.15
73.74
77.59
34.85
59.70
(9. 28
18.18
74,66
79.99
63 63.09
EL.N
19.97
32.73
36.00
90.90
97.25
97.4
52.04
73.25
77.43
33.30
83
69.
58
74,83
79.83
62.88
Forgeting
35.90
37.53
97.38
77.50
33.72
36.12
Table 13: Classification accuracy (%) over 5 diverse downstream datasets and pruning different average ratios with the weakly pre-trained ViT-Base.
Methed
CXRB10
DeepWeeds
DTD
PGVCAdreraft
Sketch
Average
20%
50%
50%
20%
50%
80%
20%
50%
50%
20%
50%
80%
20%
50%
80%
Random
20.00
31.07
32.37
93.92
97.52
97.81
$8.19
74.06
78.53
37.57
62.21
72.33
55.26
78.52
82.33
64.83
Herding
17.03
25.50
29.33
84-48
96.30
97.48
55.30
71.45
78.30
32.06
60:41
71.09
50.78
77.32
81,93
61.93
KCG
28.25
34.73
97.08
97.88
61.33
72.93
78.26
37.66
63.36
72.46
62.05
82.16
65.34
27.07
30.90
98.04
78.75
73.79
56.26
62.17
Couf
29.57
32.67
97.00
71.74
76.70
31.50
70.73
62.95
78.08
63.36
Entropy
18.10
30.83
33.90
84-65
94.02
97.
61.90
71,86
76.90
35.79
59.62
71.19
62.38
77.68
K1.76
63.85
Margin
20.07
70 00
29.13
84:04
93.06
96.83
63.49
70.60
75. 14
36.
58.78
70.70
62. 4%
76.87
83.02
63.
GraNd
16.07
27.97
34.57
93.77
97.31
96.21
51.06
73.50
76.57
38.08
61.83
72.85
56.11
78.87
60.95
63.05
EL2N
15.00
26.60
33.13
94.23
97.58
93.02
56.95
72.06
76
30.5 54
65.51
73,59
57.55
79.07
K2.84
64.70
Fargetting
25.83
78.39
59.76
63.25
31.57
33.93
79 36
72.59
66.67
20

---

Published as a conference paper at ICLR 2025
C.2.4 COMPARISON OF DIFFERENT UNDER-SAMPLING STRATEGIES
Table 14: Average classification accuracy (%)
9 pruning ratios with the fully pre-trained
ResNet- 18 from different under-sampling strategies. Bold numbers are the optimal results.
Under-sampling Strategy
CXRB10
DeepWeeds
DTD
FGVCAircraft
Sketch
Average
Random
29.88
89.65
61.01
39.69
58.63
55.77
DQ
30.07
89.49
61.17
40,34
58.82
55.98
FlexRand
31.81
90.33
62.42
40.64
60.19
57.08
In this section, we further compare different under-sampling strategies to verify the superiority of FlexRand. Specifically, the strategy in Dataset Quantization (Zhou al., 2023b) dividing datasets into different bins and randomly sampling from each bin, alleviates distribution shift and similar to ours. Differently, FlexRand divides the full dataset into two bins with different sizes: easy and hard bins. In this way, samples in different bins have different probabilities of being selected, which enables adaptation in different data regimes. Moreover,
empirically compare the performance of FlexRand and Dataset Quantization (DQ) using the same DLC score. The experiments are con- ducted with the fully pre-trained ResNet-18 18 model and we keep the same fine-tuning setting in the manuscript. As shown in Table 14, FlexRand outperforms the DQ. verifying its superiority.
C.2.5 PERFORMANCE OF PRE-TRAINED MODELS BY LINEAR PROBING
Table 15: Accuracy (%) of pre-trained models by linear probing on the full downstream dataset.
Weakly Supervised
Fully Supervised
Dataset
RN18
RN50
VIT-Small
VIT-Base
RN18
RN50
VIT-Small
VIT-Base
CXRB10
26.10
30.70
38.70
35.10
25.40
27.10
35.80
36.70
DeepWeeds
80.88
87.38
97.06
98.13
77.38
80.50
95.94
94.31
DTD
67.87
74.95
77.61
78.51
61.70
66.12
75.69
71.76
FGVCAireraft
35.55
46.38
69.10
71.92
27.96
30.06
64.21
61.12
Sketch
58.00
63.05
79.70
81.95
50.28
51.43
76.50
74.93
As shown in Table 15, we present the classification accuracy of those pre-trained models by linear probing by linear probing the full downstream dataset.
21


Reviews:
[{"review_id": "rbDga81QBf", "reviewer": "Reviewer_N2di", "summary": "The paper introduces Distorting-based Learning Complexity (DLC), a novel training-free hardness score for efficient downstream dataset pruning. DLC quantifies sample hardness by masking pre-trained weights and approximating loss integration via the Monte Carlo method. The authors also propose FlexRand, a flexible under-sampling strategy to adapt to different data regimes and avoid distribution shift.", "questions": "see the Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.) The significance lies in its potential to reduce the computational burden of fine-tuning large pre-trained models while maintaining performance.\n2.) The paper is well-structured, and easy to follow.\n3.) The introduction of FlexRand adds another layer of adaptability to the pruning process, making it more robust across different data regimes, a valuable contribution to data pruning strategies.", "weaknesses": "1.) The paper suggests that DLC is not sensitive to the quality of pre-trained models, but this claim could be further experimented on different size level pre-trained models.\n2.) The method requires storing multiple masked models, which could be a limitation in environments with constrained memory resources, potentially affecting the practicality of the approach.\n3.) The paper could benefit from a more detailed discussion on scenarios where DLC might underperform or fail, providing a more comprehensive understanding of its limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RJnia92Q7P", "reviewer": "Reviewer_FdCJ", "summary": "A novel training-free hardness score, Distorting-based Learning Complexity, is proposed to identify informative images and instructions from downstream dataset. Also, a flexible under-sampling method with randomness named FlexRand is proposed to alleviate the severe subset distribution shift. Extensive experiments demonstrate the effectiveness and efficiency of the proposed approach.", "questions": "When referring masking the pre-training weights, what specific operation is performed on the network parameters?\nWhat's the meaning of dotted line in blue(10%), green(20%) and orange(30%) in Figure(d)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed scoring function, Distorting based Learning Complexity, is an efficient training-free score for dataset pruning. A under-sampling strategy with randomness, FlexRand, is designed to adapt to different data regimes and avoid distribution shift. Extensive experiments demonstrate the effectiveness of the proposed approach. DLC significantly reduces the pruning time by 35× in images pruning benchmark.", "weaknesses": "Some typo: Line 021, \"a flexible under-sampling with randomness\" -> \"a flexible under-sampling strategy with randomness\"\nIn Figure 4(a), the MMD value of Random is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j4PehtEBZT", "reviewer": "Reviewer_aZeo", "summary": "This work describes a novel dataset pruning method without the need of pre-training on the target dataset. Given models pre-trained on large scale datasets, this work proposes a Distorting-based Learning Complexity score to identify informative images and instructions. Sample hardness is estimated by randomly masked neural networks, representing networks with different capabilities. Then samples are randomly sampled from the easy and hard groups, respectively. The proposed method achieves effective dataset pruning with 35x less pruning time.", "questions": "1. How is the loss integration implemented? In the integration figures, the upper bound of loss is 1.0. Is it normalized to the range of (0, 1)?\n2. How is the masking applied to the neural network? \n3. How is the splitting hyper-parameter $\\gamma$ determined in the actual use? If multiple values need to be tested, the tuning time should also be counted towards the pruning time in Figure 1.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The design of using random masks to produce classifiers with different capabilities is interesting and practical. With the averaged feature serving as the prediction head, there is no more need to fine-tune the classifier on downstream tasks. \n2. Detailed experiments are conducted to illustrate the effetiveness of the proposed method. The method can be applied to both image and instruction datasets, both demonstrating performance improvement. \n3. The writing is generally fluent and easy to follow.", "weaknesses": "1. The authors claim that easy samples are more likely to be correctly classified by a weak classifier in the front part of the learning path. However, the overall Learning Complexity score is acquired by averaging classification loss of multiple randomly sampled networks. The definition of learning path seems not to be utilized in the method design. \n2. Can the utilization also be applied to some of previous methods? For example, the Herding method uses parameter influence as scores for each sample. Here the fine-tuned model can also be substituted by a pre-trained model with averaged features as the prediction head. Although the direct employment of pre-trained models is practical, it is not a unique design. And it will be interesting to see if applying the strategy to previous methods also leads to performance improvement. \n3. The strategy of dividing datasets into different groups and randomly sampling from each group is similar to the idea in Dataset Quantization [1]. Dataset Quantization first iteratively separate the data into multiple bins with coreset selection methods. Normally the early groups tend to cluster around the distribution center, while the later groups show more diversity. By sampling from each bin, the overall distribution will be kept similar to the original one. This paper has a similar claim that FlexRand avoids severe distribution shift. Please discuss the difference of the proposed strategy from Dataset Quantization and the advantages of it. \n4. Section 5 discusses the quality of pre-trained models. The authors claim that the method is not sensitive to the quality of pre-trained models. But weakly supervised models are not always worse than fully supervised models. Please also show the original performance comparison between these two groups of models. \n5. Minor:\n    - The use of pretrain and pre-train need to be unified in the paper. \n    - The sample number is represented both by N (line 105) and |D| (line 129). Please unify the use. \n\n[1] Zhou, Daquan, et al. \"Dataset quantization.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)