Efficient Core-set Selection for Deep Learning Through Squared Loss Minimization
Authors: Jianting Chen
Venue: ICML 2025 poster (2025)
Keywords: Core-set selection
Dataset pruning
Loss reduction attribution
Primary area: general_machine_learning->online_learning_active_learning_and_bandits
TL;DR: A straightforward and effective core-set selection method identifying valuable samples early in the training process.

Abstract: Core-set selection (CS) for deep learning has become crucial for enhancing training efficiency and understanding datasets by identifying the most informative subsets. However, most existing methods rely on heuristics or complex optimization, struggling to balance efficiency and effectiveness. To address this, we propose a novel CS objective that adaptively balances losses between core-set and non-core-set samples by minimizing the sum of squared losses across all samples. Building on this objective, we introduce the
Maximum Reduction as Maximum Contribution criterion (MRMC), which identifies samples with the maximal reduction in loss as those making the maximal contribution to overall convergence. Additionally, a balance constraint is incorporated to ensure an even distribution of contributions from the core-set. Experimental results demonstrate that MRMC improves training efficiency significantly while preserving model performance with minimal cost.

Efficient Core-set Selection for Deep Learning Through Squared Loss Minimization
Jianting Chen
Abstract
rapidly growing data scales and constrained resources.
Core-set selection (CS) for deep learning has be- come crucial for enhancing training efficiency and understanding datasets by identifying the most in- formative subsets. However, most existing meth- ods rely on heuristics or complex optimization, struggling to balance efficiency and effectiveness. To address this, we propose a novel CS objec- tive that adaptively balances losses between core- set and non-core-set samples by minimizing the sum of squared loss across all samples. Build- ing this objective, we introduce the Maximum Reduction as Maximum Contribution criterion (MRMC), which identifies samples with the max- imal reduction in loss as those making the max- imal contribution to overall convergence. Addi- tionally, balance constraint is incorporated ensure even distribution of contributions from the core-set. Experimental results demonstrate that MRMC improves training efficiency signifi- cantly while preserving model performance with minimal computational cost.
CS is often regarded a static data pruning technique, in contrast to dynamic data pruning (DDP) (Raju et al.. 2021; Nguyen et al., 2023; Qin et al., 2024). Both aim to enhance training efficiency but differ significantly: CS retains only selected data, while DDP requires access the full dataset. Moreover, CS focuses data value, whereas DDP emphasizes the model and training process. While both are valuable, this work mainly focuses on CS.
Mainstream CS methods can be categorized into four types: !I Geometry-hased methods analyze data distribution in the feature space and select representative samples by clus- tering algorithms, that cover the data distribution, ensur- ing diversity and consistency (Sorscher et al., 2022; Xia et al., 2023). 2 Uncertainty-based methods prioritize sam- ples that provide the highest information gain based on the model's prediction uncertainty. Heuristic metrics include forgetting events (Toneva et al., 201 9), gradient norms (Paul et al., 2021), loss variance (He et al., 2024), and decision boundary errors (Yang al., 2024). 3 Optimization-aann methods frame CS mathematical optimization problem, aiming to maximize the core-set's ability to substitute the original dataset from the perspective of model parameter op- timization (Killamsetty et al., 202 lb;a). 4 Hybrid methods combine geometric properties and uncertainty metrics to balance diversity and information gain (Zheng et al., 2023).
1. Introduction
Data-driven deep learning heavily depends on large-scale scale datasets, but the growing size of models has led to expo- nentially increasing computational costs. This issue is par- ticularly critical in essource-constrainn efficient learning an area of rising interest (Shen et al., 2023). Core-set selection (CS), one of the tasks in efficient learning, focuses on selecting small, yet representative subset (core- set) from the original dataset to reduce sample size while preserving model performance (Katharopoulos & Fleuret, 2018; Coleman et al., 2020). By optimizing data utilization, CS plays a crucial role in alleviating the tension between
With the rise of large-scale models like large language mod- els and multimodal systems, CS has gained new importance. Recent studies focus on tailoring CS techniques to the spe- cific characteristics of data and models, optimizing both training and fine-tuning processes (Xia et al., 2024a; Wang et al., 2024b; Evans et al., 2024).
scenarios, making
Despite significant progress in CS, several challenges re- main to be addressed. First, balaneing diversity and un- certainty remains challenging. Diverse samples reflect the global distribution, while uncertain samples emphasize lo- cal information gain. Diversity can be understood as the uncertainty of the overall distribution given known local information. Therefore, both factors are essential for a core- set, but their conflicting objectives complicate achieving an effective balance. Second, there is gap between heuristic and principled methods. Heuristic methods are simple but
'Department of Computer Science and Technology, Tongji University Shanghai, China. Correspondence to: Jianting Chen bnnenjt@tongji.cdu...
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
lack theoretical rigor, while principled methods are resource- intensive and often fall short in practical applications.
2. Related Work
Geometry-based methods leverage the spatial structure of sample features to select core-sets that preserve data diversity and distribution consistency. Theoretically, diver- sity can enhance model robustness by increasing intra-class feature variance (Yu et al., 2020; Chan et al., 2022). Rep- resentative methods include clustering algorithms based on Euclidean distance, such as K-center (Sener & Savarese, 2018) and K-means (Sorscher et al., 2022), which improve feature diversity by removing redundant samples. Addition- ally, Moderate-DS (Xia et al., 2023) prioritizes samples near the median of intra-class centers maintain distribution consistency. FastCore (Chai et al., 2023) ensures that the core-set retains gradient information by maximizing the dis- tances between a sample and others within the same cluster. However, these methods heavily rely on the reliability of fea- ture representations. Low-quality or insufficiently trained encoders are likely to undermine their effectiveness.
To address these challenges, we propose novel CS objec- tive that replaces complex bi-level optimization strategies in principled methods with a simpler goal: to minimize the sum of squared loss of all samples, i.e. min ((.)). The L2 norm amplifies the penalty on high-loss samples, preventing overfitting to the core-set and promoting generalization to the non-core-set, achieving an adaptive balance between the two. Assuming consistent initial loss values, the objective can be reformulated into two components: maximizing loss reduction and balancing loss reduction.
Building on this objective, we introduce the Maximum Reduction as Maximum Contribution criterion (MRMC), which leverages the concept of loss reduction attribution to evaluate mutual sample contributions to loss reduction. MRMC addresses the two components as follows: ! Max- imizing loss reduction prioritizes samples with significant contributions, identified by their loss reduction during the initial training process. Samples with high contribution are those that substantially reduce the overall dataset loss. 2 Balancing loss reduction employs a proxy-based reg- ularization to adjust selection weights, ensuring balanced contributions and promoting diversity. The MRMC criterion is simple to implement and is theoretically inspired.
Incertainty-based methods assume that samples with higher uncertainty contain more information and thus have greater learning value. Classical uncertainty measures, such as Least Confidence (Wang & Shang, 2014) and Entropy (Coleman et al., 2020), are widely utilized in various ap- plications. In CS methods, forgetting events (Toneva et al., 2019) capture samples that are repeatedly forgotten by the model. EL.2N and GraDN (Paul et al., 2021) quantify the uncertainty of the sample through prediction errors and gra- dient norms. Additionally, the distance between sample and the decision boundary is another critical measure of uncertainty (Liu et al., 2019). (Yang et al., 2024) proposes selecting samples near the decision boundary to preserve the original decision boundary. Although uncertainty- based methods are simple and efficient, their heuristic nature of- ten lacks a rigorous theoretical justification. Furthermore, noisy samples may also exhibit high uncertainty, potentially interfering with the performance.
We evaluate MRMC on image recognition tasks, demon- strating strong performance in low-resource settings and consistently outperforming other methods. Parameter sen- sitivity analysis further confirms the effectiveness of the criterion and regularization technique.
The contributions of our work are as follows:
1. We propose a novel CS objective based on minimizing the sum of squared loss, which balances convergence between the core-set and non-core-sed set samples while heing intuitive and easy solve.
Optimization- methods focuses on model optimiza- tion, aiming to ensure that the contribution of the core-set to parameters is consistent with that of the original dataset. These methods typically rely sample gradient informa- tion. CRAIG (Mirzasoleiman et al., 2020) and GradMatch (Killamsetty et al., 2021a) select samples by minimizing the gradient difference between the core-set and the orig- inal dataset. ADACORE (Pooladzandi et al., 2022) em- ploys exponentially weighted Hessian approximations to estimate loss curvature and construct the core-set. RCS (Xia et al., 2024b) employs lexicographic optimization to balance model performance while reducing data size. An- other widely used tool is Data Shapley (Ghorbani & Zou, 2019), which quantifies each sample's contribution to the model and selects the core-set accordingly (Wang et al., 2024a). Although optimization methods are theoret-
2. We introduce the concept of loss reduction attribution to quantify sample interactions, bridging the gap be- tween the objective and the solution algorithm.
3. We propose the MRMC criterion to identify high- contribution samples and introduce a regularization to ensure balanced contributions. Both approaches are easy to implement.
4. Experiments on image recognition tasks demonstrate its superior performance across various core-set sizes and validate the reliability of its key modules and pa-
rameters.
2

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
ically well-founded, their high computational complexity poses significant challenges in practice.
as a bi-level optimization problem:
Σ EED
C(z,, O
min
Hybrid methods aim to integrate the strengths of various approaches. For example, (Sorscher et al., 2022) incorpo- rates geometric distances and uncertainty metrics to improve the selection process. D2Pruning (Maharana et al., 2024) employs graph- based message-passinn passsig mechanism, where sample scores are iteratively refined based on the difficulty scores and feature distances of neighboring nodes. Molleg (Chen et al., 2024) adopts a dual-model training strategy to select both representative and challenging samples. By combining diverse characteristics, hybrid methods not only enhance the core-set quality but also offer a fresh perspec- tive for developing unified theories and frameworks.
(I)
0 = arg min 0"
Σ C(zi,00), AEC
where o is obtained by solving the inner optimization prob- lem. Although bi-level optimization captures the objective of CS, it is computationally expensive because each outer- loop optimization requires solving the inner problem, which may be high-dimensional and non-convex. This contradicts the motivation of CS, which is to improve efficiency.
To address these challenges, we propose simple and ef- ficient first-order optimization objective. Specifically, the objective is reformulated minimizing the squared loss over the entire dataset:
3. Preliminary
Consider a dataset D - |_ = (( ui)) consist- ing of n samples, where each sample 2; is composed of an input 2, and its corresponding label y;- We aim to train a deep learning model parameterized by by minimizing a loss function L. The training process employs mini-batch stochastic gradient descent (SGD) to update the parameters over K steps. Let 0* denote the parameter values after the t-th update, and let the loss for sample 2; at these parameter values be 1 [(2,, 8*).
min Σ (C(2, min Σ (1 FIED
(2)
2EDD
The squared loss reduces overall loss while suppressing extreme values, balancing losses across samples, and im- proving generalization.
The final loss 1/5 can be expressed as the initial loss minus the reduction in loss during training: 1 = L A4. As- suming that the initial loss is constant across all samples ( V2, E D, 1 = 10), the objective can be rewritten as:
The task of CS, which is the focus of this study, involves selecting a subset C' C D from the dataset according to a selection ratio w 1, such that |C|- [wn]. This subset C serves the core-set, replacing the original dataset for model training. The objective is to reduce the data size and improve training efficiency, while preserving model quality as much as possible.
min Σ (2 min Σ A1,)
(3)
AEDD
&DD
Expanding and ignoring constants gives:
min Σ AA. + Σ (AL)
AA
(4)
270
4. Proposed Method
XED
2EDD
This section begins with the motivation for minimizing the sum of the squared loss of all samples, followed by the introduction of the concept of loss reduction attribution. Finally, it presents the MRMC criterion and the proxy-based regularization balancing technique.
This objective function consists of two components:
The optimization term: Σ. Al, which maxi- mizes the total reduction in loss. This aligns with the fundamental goal of training deep learning models.
The regularization term: 1
2/00 EA,)((A)))) which controls the balance of loss reduction. This prevents overfitting to the core-set samples, serving a role simi- lar to regularization.
4.1. Motivation and Objective
The training objective of deep learning models is typically to fit a given dataset D by minimizing a loss function, which can be formalized as: min Σ..eD 1,K where 15 denotes the loss value of the i-th sample after K iterations. CS, only the core-set C D participates in training. However, the model must also generalize to the non-core-set samples 2 E D C). Consequently, the objective is often defined
Notably, a similar optimization problem is also discussed in (Jain et al., 2023).
4.2. Loss Reduction Attribution
The objective function in Equation (4) uses the loss reduc- tion Al, as the variables. The reduction in loss is achieved
3

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
through updates to the model parameters using SGD. Dur- ing the i-th parameter update, given mini-batch B' and learning rate 77, the parameter update is computed as:
Here, dj,s represents the contribution of the training sample 2j to the loss reduction of 2i, thereby capturing the sample relationships during the learning process.
8¹ = 0 -1 Σ Vo ZGEB*
(5)
When training a model with the core-set C, the loss re- duction attribution can expressed as Al; = Σxyec 9,,5- Assuming the attribution values {az.i) are stable (i.e., they do not vary significantly across different core-sets), and in- corporating the objective function from Equation (4), the optimal core-set C is defined as:
where Voh(zz, 0 represents the gradient of the loss for sample 2, under the parameter ft-1
Based on first-order Taylor expansion approximations and the assumption of local smoothness, the change in the loss value for sample 2, between consecutive updates, denoted as Ad,, can be approximated as:
C argmin [Qup((C) + YPreg (C)), (10)
CCS|JC|-
where the optimization term chope (C) and the regularization term Oreg (C)) are given by:
A = L(22, 01-1 ((2, R VoL(zi, 0- (00-))(001111 - 0²)
Papt (C) =- Σ
Σ
(I))
(6)
R.ED 2,EC
Σ 0F V 0-11.
7
2
EB
Oreg (C) (C) Σ
Σ
(12)
For simplicity, let us denote the gradient inner product as
BED
5jEC
G(z4, 2g, 1- = (z 10 Vo[(2,, 01-).
The hyperparameter y controls the trade-off between the two terms. Specifically, the optimization term dupi (C)) focuses selecting samples that contribute most to the overall dataset's loss reduction. The regularization term &reg (C) ensures that contributions are distributed more evenly across all samples in the dataset, rather than being dominated by a few samples. To meet these two requirements, we will design algorithms to address them separately.
On a larger scale, when the model trained on the dataset D over K iterations, the total loss reduction of sample zi can be expressed as:
K Σ
K
A
A
1
0--1) Glan, 2)
t=1
t=1 EB
K
G(zi, 0² M) 1²))
= DED
4.3. MRMC Criterion
To solve the optimization term (C), we leverage the approximate symmetry of the attribution values. For any pair of samples (z,, 2j), we assume |aj, مو.ن Xij- where Xiς denotes permissible symmetry error. Referring to Equation (9), the gradient inner product used to compute the attribution values satisfies the commutative property, i.e., G( Zj. (((-1)) = G(2j, 2; 0(t-1)). However, the symmetry may be affected by the indicator function I(z, e B'). Specifically, if 2, E Ba and 2j E B, during an epoch, their gradient inner products are calculated using different parameters 8a-1 and gh-1_ Such differences may cause deviations in the gradient inner products. Nevertheless, under small learning rates in SGD, parameter updates are minimal, allowing us to approximate attribution values as symmetric.
where T(2; E B+) is an indicator function that equals if 2y is included in mini-batch Bs, and otherwise. From the above equations, it can be observed that the loss reduction Al, of sample 2; is determined by the cumulative contribu- tions of the gradient inner product G(2,, 2j, 0r11) from all training samples. This observation inspires the definition of a concept: Loss Reduction Attribution.
Definition 4.1 (Loss Reduction Attribution). Assume a loss function C 0), optimized over dataset D through SGD updates. The loss reduction attribution expresses the total loss reduction Al, of the sample 2; E D as the sum of attribution values (contributions) of all training samples 2j D. plus negligible residual
AG
Σass 3ED
azs te,
(8)
Based on this assumption of approximate symmetry, the optimization term can be reformulated as:
where the attribution value ajs is defined as:
dope (C) = Σ
Σ
0ss = Σ Σ
ajı
4ED 2,EC
2,EC
K
(13)
03.5 Σ
G( Zj, 0 ((-1))[[23 E B¹).
(9)
Σ
Σ
(ai. + >i.i),
2,ED

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
and by ignoring the symmetry error (i.e., Ar,5 = -), and referring to Equation (8), we derive:
pairwise interaction of attribution values between core-set samples across the entire dataset D. Our focus is on the sec- ond term, which captures the interrelations of attributions across samples.
Pupa (C) - ΣA1 A/
(14)
3YEC
Observing the attribution value based on gradient inner prod- ucts in Equation (9), we find that a sufficient condition for the second term to zero is:
This linear additive relationship suggests that minimizing (C) can be achieved by selecting samples with the largest Ar as these samples tend to contribute most signif- icantly other samples during training. We thus propose the Maximum Reduction Maximum Contribution cri- terion (MRMC), expressed as: {MRMC (2₃) -Aj-
Theorem 4.2. If the gradients of two samples are orthogo- nal, their attribution interaction is zero.
Proof. The interaction between attribution values can be simplified by ignoring the indicator function term:
To compute Álj. we argue that only a small number of training epochs on all samples is sufficient. As training pro- gresses, a sample's 's self-connribution continuously accumu- lates and becomes significantly larger than its contribution to other samples, i.e., Oj.j > ai.j: Vi + j. However, CS places greater emphasis on sample's 's contribution other samples. Therefore, only minimal training is required to measure sample contributions, aligning with the motivation of CS to improve efficiency.
03 x G( Ry (), 2k: 0).
(18)
Here, the gradient inner products G(zi, 2z 0) and G 2g. 0) quantify the similarity of 2, and 2 to 23 via their gradients. If the gradients of 2j and 25 satisfy orthogo- nality, then
V2 D, G(z, 24, 0)C 2k, 0) 0.
(19)
The gradients of the core-set samples are expected to be orthogonal. This implies that the core-set samples must satisfy diversity constraints.
Specifically, we train the model on the original dataset D for R epochs to compute (MRMM(²²), during which each sample 2j is associated with loss sequence I.g 15"),190 ,,.),,,,40) 1(5)) Due to the stochasticity of SGD, we avoid directly computing the difference between initial and final loss values. Instead, we adopt a simple mathemati- cal model, assuming the loss values decrease following a negative exponential trend:
To achieve this, we propose a proxy-based regularization technique to enhance the diversity of the core-set. Specifi- cally, we decompose the selection of core-set samples into two-stage optimization process. In the first stage, we com- pute {MRMM(22) as described in the previous section and select the top p|C'l (p < 1) samples with the highest scores to form an initial subset C². Since these samples have high contributions, they are very likely to remain part of the core- set, even after considering the regularization term. In the second stage, we aim enhance diversity by optimizing the selection of the remaining (1- p)]C| samples such that their gradient directions differ from those of the initial subset C".
5 m q
(15)
By fitting 4 and wy each loss sequence Lj, the MRMC criterion is computed as:
PMRMC (22) = ر1 = 93 w ))
(16)
Samples with higher MRMC criteria make greater contribu- tions to the dataset and should be prioritized for selection.
We train a small proxy model (using only the output layer) on the features For and labels Yo, of samples in C". The model 0' is trained for a few epochs until convergence. For any remaining sample 2; E D',C", the loss value is [(2,, ('). If the proxy model poorly fits 2;, it suggests that 2, has gradient differences from the initial subset C". Thus, the regularization score for the remaining samples is defined as:
4.4. Proxy-based Regularization
Minimizing the regularization term involves solving a quadratic optimization problem, which is challenging to address directly. To simplify, we further expand the regular- ization term as:
Prey (2:) exp ( -(=z, 0*))
(20)
2
The exponential operation normalizes the scores, thereby en- hancing regularization performance in practice. By combin- ing the MRMC criterion with the regularization constraint, we define the final scoring function as:
Orey (C) = Σ
Σ
=C 1
Σ
DED
EC
5C 2DD
(17)
Σ
Σ
Σ
i I(k: # f).
E
EVEC 2
<(2,) = (2))MC(2)) - 7 PPeg (Pgg(()) (2,),
(21)
The first term represents the L2-norm of all attribution val- ues within the core-set. The second term reflects the total
where the (1 - p)[C| samples with the highest scores are selected for inclusion in the core-set.
5

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
4.5. Selection Algorithm
Algorithm 1 Core-set Selection Algorithm
Input: Model L(-, 0), dataset D, core-set size |C], hyper-
Given a model and loss function ((: as well as the entire dataset D, the detailed algorithm (see Algorithm 1) proceeds as follows:
parameters {R. p: p}
Output: Core-set C
for to R do
1. Initial training and observation: First, the model is trained on the full dataset D for R epochs to observe the loss sequences across samples and obtain their feature represen- tations. This step is standard for most CS algorithms. The advantage of our algorithm is that R is very small, requiring minimal resources.
Update 8 with mini-batch SGD on D Collect the loss values (15)) | E D} end for Compute (sMMMC((E)) 2 E D} using Eqs. (15)(16) if p then C <- Topk(D, IC', (MMMc(())))
2. MRMC criterion computation: Using Equations (15) and (16), the MRMC criterion for each sample is calculated based on the observed loss sequence.
else if < then
C' < Topk plC|, (M)MM(Z;))
Train a lightweight proxy model O' using C'
Compute (xx(z) = E D' C²) using Eqs. (20)-211)
3. CS without regularization: If regularization not applied, the top |C| samples with the highest MRMC values selected to form the core-set. The process ends here.
C+ C" Topk(D \ C", (1 p)([),
end if
4. CS with regularization: If regularization is applied, an initial subset O' is first constructed by selecting the top p|C| samples with the highest MRMC values. A lightweight proxy model 0' then trained on the samples in C". For the remaining samples D \ C², the top (1 - P)|CI samples are selected based on their combined scores, computed using Equations (20) and (21). These selected samples are then added to the core-set.
we use ResNet-18 (11.2M parameters) and train with a batch size of 128 for 200 epochs. For ImageNet-1k, Ik, we use ResNet-34 (21.3M parameters) with batch size of 256, training for 100 and 60 epochs, respec- tively. The optimizer is SGD across all datasets, with an initial learning rate of 0. momentum 0.9, and weight decay of 5 x 10 (CIFAR) or 1 x 10-4 (others). A cosine an- nealing schedule reduces the learning rate to a minimum of x 10 10-4, and data augmentation includes random horizon- tal flipping and cropping. The number of training epochs is fixed for all core-set sizes to ensure smaller core-sets reduce training time, aligning with the goal of efficiency.
and
Efficiency Analysis. Our method is characterized by high efficiency. It requires only a few epochs of model training, significantly reducing the cost of preparation. The selec- tion algorithm operates with near O(N) complexity and relies solely on each sample's loss change, avoiding expen- sive pairwise distance calculations. Additionally, the proxy model for computing is simple linear layer, which ensures extremely high training efficiency.
Competitors. We compare the proposed method with the following classic or recently proposed CS algorithms': Ran- dom selection; FL2N & GraNd (Paul et al., 2021) mea- sures and selects samples with high uncertainty; Glister (Killamsetty et al., 2021b) formulates a bilevel optimiza- tion problem to maximize the log-likelihood on the entire dataset. CCS (Zheng et al., 2023) improves the coverage of core-set data by applying stratified sampling based on EL2N scores; Moderate prioritizes samples that are at intermedi- ate distances from class centers; Dyn-Une (He et al., 2024) selects samples with high uncertainty based on the standard deviation of loss values during the training process; Bound- CryCCS (Yang et al., 2024) identifies the distance of sam- ples to the decision boundary using adversarial learning and combines CCS to improve sample coverage; D2pruning (Maharana et al., 2024) computes sample difficulty based on EL2N scores and improves core-set diversity using a graph- propagation method.
5. Experiments
This section includes four groups of experiments: ! Com- paring MRMC with strong baselines; (2) Verifying the ef- fectiveness of balancing the losses; 3 validating the as- sumptions behind loss reduction attribution; ④ Sensitivity analysis of key parameters.
5.1. Experimental Setup
Datasets. We use four widely adopted datasets to evaluate the effectiveness of the proposed method, namely CIFAR- 10, C00, Tiny-ImageNee, and ImageNet-1k. The scale and complexity of these datasets increase progres- sively, with the number of classes ranging from 10 1000. These datasets enables the evaluation of CS algorithms der different task complexities.
Evaluation protocols. We evaluate both the quality and efficiency of CS algorithm. Core-set quality is assessed
Models and training. For CIFAR-10 10 and CIFAR-100,
LLabel balance constraint was not enforced in reproduction.
6

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
Table 1. Comparison of model accuracy (%) under two protocols for different core-set selection algorithms.
Dataset
CIFAR-10 10 (95.44)
CIFAR-100 (79.37)
Iin--ImaeNeet (62,40)
Selection ratio w
70%
50%
30%
Avg
70%
50%
30%
Avg
70%
50%
30%
Avg
Random
94.45
93.60
91.06
93.04
76.09
72.83
67.01
71.98
58.14
52.43
45.25
51.94
Full training
Glister
94.80
94,20
90.15
93.05
76.80
73.21
66.90
72.30
57.20
55.05
48.23
53.49
Moderate
94.25
92.43
89.22
91.97
76.38
73.22
67.14
72.25
57.54
52.94
44.32
51.60
protocol
Dyn-Une
91.50
89.59
85.82
88.97
73.91
70.09
64.06
69.35
52.98
44.43
35.57
44.33
Boundary
94.72
93.18
91.21
93.04
75.95
73.22
65.48
71.55
57.88
53.12
44.65
51.88
D2pruning
92.76
91.31
88.97
91.01
77.79
74.10
69.65
73.85
60.05
57.37
49.04
55.49
EL2N
95.40
95.23
91.94
94.19
77.61
67.60
32.55
59.25
56.08
38.96
13.79
36.28
GraNd
95.01
94.59
88,91
92.83
77.02
65.58
49.23
63,94
55.98
39.67
20,41
38.68
Early
Glister
94.05
92.59
89.10
91.91
76.39
72.49
66.05
71.64
56.50
53.98
47.68
52.72
training
CCS
95.40
95.03
92.65
94.36
75.31
71.15
64.63
70.36
56.51
51.46
43.72
50.56
protocol
Moderate
94.20
92.94
91.51
92.88
76.27
72.97
66.58
71.94
57.32
$2.01
44.37
51.23
R 20
Dyn-Une
92.72
87.86
79.47
86.68
75.29
67.05
47.78
63.37
56.66
39.97
12.57
36.40
Boundary
94.50
93.68
92,08
93.42
75,94
72.16
66.10
71.40
57.76
53.01
45.01
51.93
D2pruning
92.22
90.89
87.95
90.35
75.97
73.56
69.21
72.91
59.42
56.88
50.15
55.48
MRMC
94.94
94.61
92.27
93.94
77.76
74.43
67.28
73.16
59.89
56.89
49,99
55,59
MRMC-R
95.46
95.24
93.13
94.61
77.17
74.46
68.66
73.43
60.42
57.03
50.88
56.11
The best results are highlighted in bold, and the second-best results are indicated with underline.
by the accuracy of the model on the test set after training on the selected core-set. Following standard practices, we prepare core-sets of varying sizes and compare the model's performance trained on core-sets selected by different algo- rithms. Higher test accuracy indicates better core-set quality. For selection efficiency, the main computational cost arises from training the model prior to selection to obtain sample uncertainties or feature representations. Methods such as MRMC and EL2N suggest this cost can be reduced by train- ing for only few epochs. Hence, we adopt two protocols: (1) Full training protocol: CS is performed after fully train- ing the model (all epochs). (2) Early training protocol:
D2pruning performs differently from the original results (Maharana al., 2024), likely due differences training iterations. Under the early training protocol, Moderate and Boundary outperform, indicating that full training model may lead to overfitting on simpler tasks.
Table 2. Comparison results on ImageNet- IK.
Selection ratio
50%
30%
10%
Avg
Random
68.51
64.57
48.26
60.45
D2pruning 68.71
65.26
52.14
62,04
MRMC
69.81
66.47
50.48
62.25
MRMC-R
69.85
66.05
49.66
61.85
CS is performed after training the model for only R = 20 epochs. Core-set quality is evaluated under both protocols for all algorithms except EL2N, CCS, and MRMC. Most experiments are conducted three times with different seeds.
On CIFAR-100, D2pruning achieves the best results, fol- lowed by MRMC-R. However, under the early training pro- tocol, most methods degrade, while MRMC-R becomes the best. Even the basic MRMC without regularziation also outperforms others. Notably, when compared to EL2N and Dyn-Unc, both of which are similarly simple algorithms that observe and leverage outputs from the model during training, our MRMC criterion proves to be more effective.
Hyperparameters. For MRMC, the number of initial train- ing epochs is set to R = 20. For the regularized version (MRMC-R), the regularization parameters are set to p = 0.3 and 7 = 2. The proxy model is trained with SGD (learning rate 0.01, batch size 512) for 10 epochs. For competitor algorithms, we follow the publicly available hyperparameter settings during reproduction.
The results on Tiny-ImageNet are similar to those on CIFAR- 100. Both MRMC and D2pruning demonstrate superior performance. This suggests that for more complex tasks, is critical for CS to balance diversity and uncertainty. Furthermore, employing the MRMC criterion alone can achieve comparable outcomes, as it inherently adapts to balance diversity and uncertainty.
5.2. Comparison with Baselines
We compare MRMC with other CS algorithms across the CIFAR- 10/100 and Tiny-imagenet datasets. Table presents the model performance under different core-set sizes.
On CIFAR-10, MRMC-R achieves the best performance across all core-set sizes, with CCS as the second-best.
The results on ImageNet- IK under the early training proto- col are shown in Table 2. With a fixed training of 60 epochs, MRMC outperforms Random and D2pruning at selection
7

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
Random
EL2N
CCS
Moderate
Dyn-Unc
Boundary
D2pruning
MRMC
MRMC-R
Nop Nep-com-st Loss
2.5
93
0.75
90
Tol Accurasy
0.9
8
94
0.5
0.6
3
0.5
0.5
0.3
(a) CIFAR-10 =70%
(b) CIFAR-10 =50%
(c) CIFAR- 1EARR-004 =30%
Non-coesst Lose
Tcll Ausuney
76
S
1.0
14
Non-cond
2.6
0.5
2.8
S:
0.0
1.0
(d) CIFAR-100 =70%
(e) CIFAR- 100 & =50%
(l) CIFAR-100 =30%
4,0
8 &
Non corestt Los
3.0
56
Tox Actunay
6.4
& 88
2.0
3.6
1.0
0.0
60
2.0
60
(g) Tiny-ImageNet =70%
(h) Tiny-ImageNet w =50%
(i) Tiny-ImageNet:a w =30%
Figure Relationships hetween non-core-set loss (lines) and test accuracy (bars) across different algorithms in the early training protocol.
ratios of 50% and 30%. The suboptimal performance of MRMC-R likely due to hyperparameter settings.
enhanced methods (e.g., D2Pruning, MRMC) retain higher non-core-set loss while achieving superior accuracy. This highlights the importance of introducing diverse samples into the core-set for complex tasks.
For the time cost of sample selection on CIFAR-10/100 and Meayeem methods such as MRMC(-R), EL2N, GraNd, CCS, Moderate, and Dyn-Une operate a second- level scale with near-linear complexity. In contrast, Glister and D2Pruning require tens of seconds, while Boundary incurs a significantly higher cost, taking hundreds of sec- onds due to adversarial training. On ImageNet- IK, MRMC- completes sample selection in approximately minute, whereas D2Pruning takes around minutes.
5.4. A Assumption Validation
We hypothesize that loss reduction attribution possesses stability and symmetry. To validate this, we conduct case study. Specifically, we randomly sample 50 instances from CIFAR-100 and compute their attribution scores over 10 training epochs under four settings: using the full dataset and core-sets of 30%, 50%, and 70%. This results in four 50 x 50 attribution matrices, as shown Figure 2.
5.3. Effectiveness of Balancing Losses
The proposed MRMC and regularization techniques aim to reduce and balance the losses of core-set and non-core-set samples, allowing the model better fit all data. Given that core-set losses are generally minimized, our focus shifts to the losses of non-core-set samples. Figure shows the relationship between non-core-set loss and test accuracy.
For stability, we normalize the attribution scores and calcu- late the average relative error against the full-data attribu- tion matrix. The relative errors for the 30%, 50%, and 70% core-sets are 27%, 26%, and 22%, respectively, indicating improved stability with larger core-sets.
For symmetry, visualizations of the attribution matrices consistently exhibit strong diagonal symmetry across all set- tings. The relative symmetry errors are 16%, 19%, 22%, and 24% for the 30%, 50%, 70% core-sets, and full dataset, re- ppectively-showiee that symmetry decreases the amount of training samples increases.
For MRMC and MRMC-R (last two columns), regulariza- tion reduces non-core-set loss and improves model perfor- mance most cases. With small core-sets (w =30%), non-core-set loss directly correspond to model performance: lower loss yield better accuracy. However, on CIFAR-100 and ImageNet with large core-sets, uncertainty-based methods (e.g., EL2N, Dyn-Unc) reduce non-core-set loss but degrade overall performance. In contrast, diversity-
8

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
due to better sensitivity to gradient differences. However, for the 70% core-set CIFAR-100, P 0.8 performs better, suggesting that larger initial subsets benefit from regularization to capture uncertain samples.
Effect of y. We compare Y = 1, and 5 with = 20 and p = 0.5. The tradeoff parameter y balances the MRMC criterion and regularization. Results show that the optimal ? varies with core-set size and task complexity.
(a) Full dataset
(b) Selection ratio 70%
6. Conclusion
We proposes a simple yet effective criterion (MRMC) for CS, which identifies samples contributing most to loss re- duction, thereby enhancing the model to fit the known data. Compared other algorithms, MRMC requires only lim- ited initial training, offering advantages of simplicity and low computational cost. Additionally, we introduce proxy- based regularization to enhance the diversity of core-sets. This approach adaptively balances the losses of core-set and non-core-set samples, improving the model performance.
(e) Selection ratio 50%
(d) Selection ratio 30%
Figure 2. Visualization of loss reduction attribution values for the
50 samples under training with core-sets of different sizes.
Limitations and future work. We make several assump- tions, such as the stability and symmetry of loss attribution values. Further optimization is possible if loss attribution can be efficiently computed and modeled. Furthermore, the regularization technique relies on multiple hyperparameters, which complicates practical implementation. Future work should focus on reducing this dependency for applicability.
Table 3. Results of parameter sensitivity validation experiments.
Dataset
CIFAR-100
Tiny-Im
Selection ratio
70%
30%
70%
30%
R = 30
77.83
66.38
60.25
49.66
R 20
77.76
67.28
59.89
49.99
R
76.52
67.82
58.74
48.73
5
75.51
66.33
57.67
46.76
Acknowledgments
P 0.8
77.81
66.98
58.87
47.2
P = 0.5
77.21
68.26
58,94
49.83
This work completed during the visit to The Chinese University of Hong Kong. We sincerely thanks Professor Irwin King for his guidance. Gratitude is also extended to the co-advisor at Tongji University, Professor Yang Xiang, for his financial support.
= 0.3
77.17
68.62
60.42
50.88
7 = 5
77.73
63.61
60.24
48.79
2
77.21
68.26
58.94
49.83
7 77.08
67.51
59.86
50.26
Impact Statement
5.5. Parameter Sensitivity Analysis
We analyze the sensitivity of three key parameters on CIFAR-100 and Imag- datasets, using selection ratios of 70% and 30%. The parameters include the number of initial training epochs R, the scale of the regularization subset size p, and the trade-off parameter y. The results are shown in Table 3.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
5, 10, and 30 with p = 1 (no regularization), most results show that larger R improves core-set quality. However, for the 30% core-set CIFAR- 100, R = 10 performs best, while for the 30% core-set Tiny- NetgeNe, R = 20 performs best. This suggests that optimal Rdepends on core-set size and task complexity.
Effect of R. Testing
Chai, C., Wang, J., Tang, N., Yuan, Y., Liu, J., Deng, Y., and Wang, G. Efficient coreset selection with cluster-based methods. In Singh, A. K., Sun, Y., Akoglu, L., Gunopulos, D., Yan, X., Kumar, R., Ozcan, F,, and Ye, J. (eds.), Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6- 10, 2023, pp. 167-178. ACM, 2023. doi: 10.1 0w1.:3500000035999990. URL https:
Effect of p. For p = 0.3. 0.5, and 0.8 with R = 20 and y smaller p generally produces better performance, likely
9

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
3599326.
Katharopoulos, A. and Fleuret, F. Not all samples are cre- ated equal: Deep learning with importance sampling. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learn- ing, ICML. 2018, Stockholmsmüsssnn, Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2530-2539. PMLR, 2018. URL http::/ppooeeedigg mer.eiinnss..i...........mm. presss/ v80 wwD/katharoppuuloslaa..ba
Chan, K. H. R., Yu, Y. You, C., Qi, H., Wright, J., and Ma, Y. Redunet: A white-box deep network from the principle of maximizing rate reduction. J. Mach. Learn. Res., 23:114:1-114::03, 2022. URL https: / /mmzz. html.
http:/apers//22////
Chen, D., Li, Z., Ni, Y., Zhang, G., Wang, D., Liu, Q. Wu, S.. Yu, J. X., and Wang. L. Beyond efficiency: Molecular data pruning for enhanced generalization. CoRR, (bs/2409.0108) 2024. doi: 10.48550/ARXIV. 2409.01081. URL http:::tt///t......t http::://doi.or//1//// 48550/ 2409v.2499, 01081.
Killamsetty, K., S. D., Ramakrishnan, G., De, A., and Iyer, R. Grad-match: Gradient matching based data subset selection for efficient deep model train- ing. In Meila, M. and Zhang, T. (eds.), Proceed- ings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Mu- chine Learning Research, pp. 5464-5474, PMLR, 18- 24 Jul 2021a. URL https:::p///cc.cc htthttps:: press/v133 vvc//vv333 19/ http://w19/kkilllee htt.kkklaammttttylaa.... html.
Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Se- lection via proxy: Efficient data selection for deep learn- ing. In Sth International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: //openreview.cooeeoo//e/e net/ hwt.://eet//orrrr///eee forum? ii-..j2b.............
ngs .mlr.
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. K. GLISTER: generalization based data sub- set selection for efficient and robust learning. In Thirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Conference on Innovative Applica- tions of Artificial Intelligence, IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intel- ligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 81 11108811 8118 18. AAAI Press, 2021b. doi: 10.1609/AAAI. V35I9. 16988. URL ht ht : www.doo..oooooooooooooooooooo... 1609/ aaai. Shai.v35.0 v3519. 11.55.9... .. 6988.
Evans, T., Parthasarathy, N., Merzic, H., and Hénaff, J.
Data curation via joint example selection further ac- celerates multimodal learning. CoRR, (2d/2406.177111,
2024. doi: 10.48550/ARX11.2406.1777 //doi org/ 110 http:11..885555
URL https: 48540/ htt5555////iiii. http:/awww.......... 2406..7711...
Ghorbani, A. and Zou, J. Y. Data shapley: Equitable valua- tion of data for machine learning. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2242- 2251. PMLR, 2019. URL http: /ttppeeeeceee.... mlr. wwwppppessspaeeeee v http//ghobbaaannnie
Liu, K., Liu, W., Cheng, J., and Lu, X. UHRP: uncertainty- based pruning method for anonymized data linear regres- sion. In Li, G., Yang, J., Gama, J., Natwichai, J., and Tong, Y. (eds.), Database Systems for Advanced Applica- tions 24th International Conference, DASTAA 2019, Chi- ang Mai, Thailand, April 22-25, 2019, Proceedings, Part III, and DASTAA 2019 International Workshops: BDMS, BDQM, and GDMA, Chiang Mai, Thailand, April 22- 25, 2019, Proceedings, volume 11448 of Lecture Notes in Computer Science, pp. 19-33. Springer, 2019. doi: 2. URL https: doi. //di. 03 030 18590-9
html.
He, M., Yang, S., Huang, T., and Zhao, B. Large-scale dataset pruming with dynamic uncertainty. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2024 Workshops, Seattle, WA, USA, June 17-18, 2024. pp. 7713-7722. IEEE, 2024. doi: URL https://
10. 22g/10.1007//
10. AA11199CVPRW3882220400000 doi. 111111109//// http:////vwww333333 2024. 00767.
Maharana, A., Yadav, P., and Bansal, M. D2 pruning: Mes- sage passing for balancing diversity & difficulty in data pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. https: / //openreview net/ forum? btt.ruummp..tthtttooc.....
Jain, E., Nandy, T., Aggarwal, G., Tendulkar, A., Iyer, R. K., and De, A. Efficient data subset selection to generalize training across models: Transductive and inductive net- works. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Mirzasoleiman, B., Bilmes, J. A., and Leskovec, J. Core- sets for data-efficient training of machine learning mod- els. In Proceedings of the 37th International Confer- ence on Machine Learning, ICML. 2020, 13-18 July
10

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 1950-6960) PMLR, 2020. URL http:tt/prccceeeinneee mlr wwm.r.poe press/v119/ mi mirzasoleiman20..coo
2023. doi: 10448550ARRXX122334000558 URL https: / //doi.org/10. 48150/
03589.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. Beyond neural scaling laws: beating power law scaling via data pruning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Nguyen, T., Gerofi, B., Martinez-Noriega, E. J., Trahay, F., and Wahib, M. KAKURENBO: adaptively hiding sam- ples in deep neural network training. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.). Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.neL, 2019. URL htwps:htt////tt //pp:///////e.ie..../ net wtrt/forum?00 -id--idddBBBla 3m00mm.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on data diet: Finding important examples early in training. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Infor- mation Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pp. 20596-20607, 2021.
Wang, D. and Shang. Y. A new active labeling method for deep learning. In 2014 International Joint Conference on Neural Networks, IJCNN 2014, Beijing, China, July 6- 6-11, 2014, pp. 112-119. IEEE, 2014. doi: 10.1109/ JICNN.2014.6889455 URL https: / /////ddd /doi org/10. 1109/I 2014. 6889457.
Pooladzandi, O., Davini, D., and Mirzasoleiman, B. Adap- tive second order coresets for data-efficient machine learn- ing. In Chaudhuri. K., Jegelka, S., Song, L.,, Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Confer- ence on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedingy of Machine Learning Research, pp. 17848-17869. PMLR, 2022. URL http:: httpp:::p:ppcceeepepiett www.ong.........mmo.o........ee. mlr. press// v1 #162/pooladzandi22.. html.
Wang. J. T., Yang, T., Zou, J., Kwon, Y., and Jia, R. Re- thinking data shapley for data selection tasks: Misleads and merits. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.ned 2024a. URL https: / //openreviee ew. net/ forum? d=mKYBMf11
Qin, Z., Wang, K., Zheng, Z., Gu, J.. Peng, X., Xu, Z.. Zhou, D., Shang, L., Sun, B., Xie, X., and You, Y. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: // //openreview net. foret//ooummmm wdtcmmiskcccccceecoocmm
Wang, Y., Chen, Y., Yan, W., Fang, A., Zhou, W., Jamieson, K., and Du, S. S. Cliploss and norm-based data se- lection methods for multimodal contrastive learning. CORR, abs/2405. 195477, 2024b. doi: 10.48550/ARXIIV. 2405 URL https: /tt////ooioo/////////oo/ 488500 a arXiv 2xiv.2405.02 19547.
Raju, R. S., Daruwalla, K., and Lipasti, M. H. Accelerating deep learning with dynamic data pruning, 2021. URL https: /twiv..ivvvvv/////v////////aaavv org/ abs//b////// /21111111111 12621.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, LESS: selecting influential data for targeted instruc- tion tuning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URL https: / //openreview.c www.eeww.eeeeeeeeeee net/ wttt://frrmmmiico http://rrrmmm.p-cp////
Sener, O. and Savarese, S. Active learning for convolu- tional neural networks: A core-set approach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: http://errvvvieeeeee ew. net/ forum? Id=HlaIuk-RM.
Xia, X., Liu, J., Yu, J., Shen, X., Han, B., and Liu, T. Mod- erate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL https: http:///ppeeeieeeeeee net/ pdf?id=7D5ERCBB
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., and Tao, D. On efficient training of large-scale deep learning models: A literature review. CoRR, abs/2304.03589,

---

Efficient Core-sct Selection for Deep Learning Through Squared Loss Minimization
Xia, X., Liu, J., Zhang, S., Wu, Q. Wei, H., and Liu, T. Refined coreset selection: Towards minimal coreset size under model performance constraints. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL https: /tt.eccaee//ppeeeeeeeee.....eeee forum?
id=yb5xV8LEDq.
Yang, S., Cao, Z., Guo, S., Zhang, R., Luo, P., Zhang, S., and Nie, L. Mind the boundary: Coreset selection via reconstructing the decision boundary. Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httpp:://pppeeeeeeeeeeeee ew www.eeerrreee..eeeforrmee wwt//foounm
id-hWng00XeE4.
Yu, Y., Chan, K. H. R., You, C., Song. C., and Ma, Y. Learn- ing diverse and discriminative representations via the prin- ciple of maximal coding rate reduction. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, vir- tual, 2020.
Zheng, H., Liu, R., Lai, F., and Prakash, A. Coverage-centrric coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL httptttt/pppeeeeeeeeeeeeee///eee net/ DddFidd=OwkVLL6WB1.
12


Reviews:
[{"review_id": "M56VKJttDQ", "reviewer": "Reviewer_hRSX", "summary": "The authors propose a new objective to select subsamples for training deep learning models. They call it MRMC (Maximum Reduction as Maximum Contribution) which essentially says that a data point that leads to more reduction in the squared error loss during the first few epochs of training relative to other points, is a more important point and should be a part of the subsamples used for further training. They also introduce a regularization term in their objective to take care of overfitting. They show the effectiveness of their sampling method compared to a variety of methods in their experiments on standard image datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and fair assessment. We appreciate the opportunity to  address your concerns.\n### **Response #1: Lack of Theoretical Guarantees**\nWe acknowledge that our method does not provide formal theoretical guarantees. This work aims to bridge the gap between purely heuristic methods and theoretically grounded but computationally expensive approaches. Our method (closer to a heuristic method) is motivated by a clear theoretical intuition and achieves higher performance than other heuristic algorithms. We will provide more analysis and supporting empirical evidence to justify this intuition and our contributions in the revised manuscript. \n1. **Complexity analysis** of MRMC and **runtime comparisons** (see Responses #1 and #4 to Reviewer *opAA*) to demonstrate its practical efficiency.\n\n2. A **case study validating the assumption** that attribution scores are approximately symmetric and stable, which is fundamental to our method (see Response #2 to Reviewer *JTrd* ).\n\nWhile a full theoretical guarantee is absent, we believe this combination of intuition, analysis, and empirical validation provides a meaningful step forward in the design of practical and reliable core-set selection algorithms.\n\n### **Response #2. Comparison with Optimization-based Methods**\nThank you for highlighting the importance of comparison with optimization-based methods. We addressed this concern by attempting to reproduce GradMatch[1] and Glister[2] as optimization-based baselines. Unfortunately, we found that GradMatch (with GPU) was too computationally demanding—core-set selection time exceeded even full model training time in our setting. Other optimization-based methods also face this issue. Therefore, we selected Glister as a more practical baseline and adapted it by using only output-layer gradients to reduce its computational cost (selection time reduced to 27 seconds on CIFAR-100), but still much slower compared to MRMC-R (3s) and MRMC (0.2s).\n\nWe have now added more comparisons (i.e.  Glister, GraNd) in the revised manuscript. The results show that MRMC achieves better core-set quality while being significantly more efficient. (C10 and C100 correspond to CIFAR-10 and CIFAR-100, respectively, and TI refers to Tiny-ImageNet. The numbers in parentheses indicate the core set size.)\n\n\n|         Method         | C10(70%) | C10 (50%) | C10 (30%) | C100 (70%) | C100 (50%) | C100(30%) | TI(70%) | TI(50%) | TI( 30%) |\n|:----------------------:|:-----------:|:-----------:|:-----------:|:------------:|:------------:|:------------:|:-----------------:|:-----------------:|:-----------------:|\n| MRMC                   |   94.94     |   94.61     |   92.27     |    77.76     |    74.43     |    67.28     |       59.89       |       56.89       |       49.99       |\n| MRMC-R                 |   95.46     |   95.24     |   93.13     |    77.17     |    74.46     |    68.66     |       60.42       |       57.03       |       50.88       |\n| Glister                |   94.80     |   94.20     |   90.15     |    76.80     |    73.21     |    66.90     |       57.20       |       55.05       |       48.23       |\n\n### **Summary**\nWe hope that the added analyses and new experimental comparisons comprehensively address your concerns and help position MRMC as an effective and scalable solution to the core-set selection problem. Thank you again for your thoughtful feedback.\n\n[1] GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training,ICML,2021\n\n[2] GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning,AAAI,2021", "claims_and_evidence": "The paper is written clearly and is easy to follow. The claims are convincing enough.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are not many theoretical claims in the pair. However, the theoretical intuition behind the main algorithm is clearly explained.", "experimental_designs_or_analyses": "Seems good but some more experiments may be added. See weaknesses.", "relation_to_broader_scientific_literature": "With the huge deep learning architectures being used in today's age, subsampling algorithms are very important. There are either algorithm with theoretical guarantees but practically tough to implement or some well performing heuristics without any guarantees. This paper tries to bridge the gap by providing a new subset selection objective which is theoretically motivated and grounded and at the same time good in practice.", "essential_references_not_discussed": "Not to the best of my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n1) Subset selection is becoming a very important problem in the era of huge models and this paper would be of interest to a broad community.\n2) The algorithm is intuitive and simple.\n3) Experiments are presented on good number of datasets and show very good.\n\nWeaknesses:\n1)Although the theoretical intuition behind the idea and framework is clear, the method does not really give theoretical guarantees. \n2) In the experiments section, the authors have not made comparisons with what they call the optimization-based methods like GradMatch. I think these comparisons are also required as they are essentially the methods with some theoretical guarantees. Even though I admire the detailed experiments authors have done, I believe this kind of study needs to be even more comprehensive.", "other_comments_or_suggestions": "See Weaknesses", "questions_for_authors": "None. Please address the weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "1XSrqdnKJP", "reviewer": "Reviewer_JTrd", "summary": "This paper presents a novel coreset selection method, which minimizes the squared loss to balance contributions between coreset and non-coreset samples. The method follows a two step process: first select samples with the highest MRMC value, then train a proxy model to select samples to increase diversity. Extensive experiments on benchmarks demonstrate that the proposed method significantly accelerates training and reduces computational costs while improving model performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your suggestions are very helpful for improving the quality of this work. Below we address each of your concerns:\n### **Response #1: Missing Related Work**\nThank you for pointing this out. In the revised manuscript, we will add a discussion on “Refined Coreset Selection” (ICML 2024), which formulates the problem as a bi-level optimization and employs Lexicographic Optimization for solution. This work belongs to the class of optimization-based methods and provides an important reference.\n“Mind the Boundary” (ICML 2024) is already discussed in our original manuscript and also included as a baseline named \"Boundary\" in our main experiments (see Section 5.1 Competitors). We will clarify this more explicitly in the revised version.\n### **Response #2: Empirical Validation for Assumptions**\n We appreciate the reviewer’s observation. It was indeed an oversight on our part not to empirically validate the assumptions. To address this, we conducted a case study to assess the **symmetry and stability** of the attribution scores. Specifically, we randomly sampled 50 instances from CIFAR-100 and computed their attribution scores throughout the 10 training epochs under four different settings: using the full dataset and coresets of 30%, 50%, and 70%. This resulted in four 50*50 attribution matrices.  **Symmetry:** Visualizations of the attribution matrices consistently exhibit strong diagonal symmetry across all settings. The relative symmetry errors are 16%, 19%, 22%, and 24% for the 30%, 50%, 70% coresets, and full dataset, respectively—indicating that symmetry degrades as the amount of training data increases. **Stability:** We further normalized the attribution scores and computed the average relative error against the full-data attribution matrix. The relative errors for the 30%, 50%, and 70% core-sets are 27%, 26%, and 22%, respectively, suggesting improved stability with larger core-sets. (Due to the review policy, we are unable to include external links to visualizations.) \n### **Response #3: Comparison with GraNd**\nGraNd, proposed in [1], is a well-known and efficient method, and EL2N is its approximation. As expected, the two methods yield similar experimental results. Following your suggestion, we implemented GraNd and will update the results in the revised manuscript. Similar to EL2N, GraNd’s performance drops significantly as the core-set size decreases. In contrast, our MRMC-R method consistently outperforms GraNd at 50% and 30% coreset sizes, especially on more challenging tasks. This demonstrates the strength of MRMC in scenarios where effective selection is needed.\n### **Response #4: Proxy Model Details**\nThe proxy model used in our method is the final linear classification layer (i.e., copy.deepcopy(model.classifier)). Analyzing gradients from the output layer is a common and efficient practice in coreset literature. As described in line 350, the proxy model is trained using SGD with a learning rate of 0.01, batch size of 512, for 10 epochs. Since it contains only one linear layer, training is very fast—even on large datasets like ImageNet-1K, it completes in under one minute.\n### **Response #5: Training Epochs R  > 20**\nWe originally evaluated R = 5, 10, and 20 in the sensitivity analysis, but did not include larger values. Thank you for pointing this out. To address this, we  conducted an experiment with R = 30. The results show that, except for the 70% and 50% core sets of CIFAR-100, performance generally degrades. This indicates that an excessively large R is inappropriate. This suggests that larger R may lead to overfitting in early training and reduce the effectiveness of the attribution signal.\n\n| Method    | C10(70%) | C10(50%) | C10(50%) | C100(70%) | C100(50%) | C100(30%) | TI(70%) | TI(50%) | TI(30%) |\n|:----------------:|:--------:|:--------:|:--------:|:---------:|:---------:|:---------:|:-------:|:-------:|:-------:|\n| MRMC-R (R=20)    | 95.46    | 95.24    | 93.13     | 77.17     | 74.46     | 68.66   | 60.42   | 57.03   | 50.88 |\n| MRMC-R (R=30)    | 95.35    | 95.16    | 93.00     | 77.83     | 74.99     | 66.38   | 60.25   | 56.32   | 49.66 |\n| GraNd            | 95.01    | 94.59    | 88.91     | 77.02     | 66.58     | 49.23   | 55.98   | 39.67   |  20.41 |\n| EL2N             | 95.40    | 95.23    | 91.94     | 77.61     | 67.60     | 32.55   | 56.08   | 38.96   |  13.79  |\n\n\n### **Summary**\nWe hope that the added analyses and new experimental comparisons comprehensively address your concerns.\n\n[1] Deep Learning on a Data Diet: Finding Important Examples Early in Training. NeurIPS 2021", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper introduces a novel coreset selection criterion that reframes sample importance via squared loss minimization. It extends prior gradient-based approaches by introducing the Maximum Reduction as Maximum Contribution (MRMC) metric to quantify each sample’s impact on training.", "essential_references_not_discussed": "[1] Refined coreset selection: Towards minimal coreset size under model performance constraints. ICML 2024\n\n[2] Mind the Boundary: Coreset Selection via Reconstructing the Decision Boundary. ICML 2024", "other_strengths_and_weaknesses": "Strengths:\n1. The method is built on theoretical foundation, providing rigorous justification for the proposed approach.\n2. The novel MRMC criterion is both technically sound and intuitive, which offers a novel perspective on evaluating sample importance.\n3. The experimental analysis is thorough and demonstrates promising performance gains across multiple datasets.\n\nWeaknesses:\n1. Several works in coreset selection is missing, as mentioned in section above.\n2. The assumption of symmetry in attribution values based on mini-batch SGD can be fragile, especially since it relies on a small learning rate, while training in the experiment begins with a high learning rate 0.1\n3. GraNd score proposed in [1] should be included for comparison.\n\n[1] Deep Learning on a Data Diet: Finding Important Examples Early in Training. NeurIPS 2021", "other_comments_or_suggestions": "1. The paper should empirically validate key assumptions made in the theoretical analysis. For instance, it is essential to verify whether the attribution values are indeed approximately symmetric as assumed in Section 4.3, and if they remain stable throughout training (as mentioned in line 207).\n2. The Taylor expansion used in Eq. (6) assumes that the high order terms are negligible. Empirical study is needed to assess the impact of the residual terms on approximating $\\Delta l_i^t$.", "questions_for_authors": "Please address the Comments section.\n1. Following weakness 3, how does GraNd perform on the benchmarks?\n2. In section 4.3, you argue that a small number of training epochs on full samples are sufficient. How does the MRMC's performance change when the number of training epochs $R$ exceeds 20?\n3. Can you provide the architecture of the proxy model used for regularization and the hyperparameter for training the proxy model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5XGFPl81F", "reviewer": "Reviewer_uBL3", "summary": "The authors:\n\n*propose a new core-set selection approach that seeks to balance losses between chosen and unchosen samples by minimizing the overall sum of squared loss.\n\n* Introduce the Maximum Reduction as Maximum Contribution (MRMC) criterion, which pinpoints those data points that most substantially reduce the loss—indicating they have the greatest influence on achieving model convergence.\n\n* To maintain a fair representation, they impose a balance constraint so that the contributions of the core-set remain evenly distributed. \n\n\nI like the idea of the paper and the writing style. My main concerns are related to the experiments and mainly competing methods,", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the feedback and positive comments on our idea and writing. Below we address the main concerns:\n### **Response #1: Theoretical Guarantees in Prior Work**\nWe realize that our original phrasing may have led to confusion. Our intention was to emphasize the practical gap between efficient heuristics and theoretically grounded methods, rather than to dismiss existing work with guarantees. We will update the manuscript to better reflect this nuance.\n### **Response #2: Theoretical Claims**\nFollowing your suggestion, we reviewed the relevant literature and will modify Equation (1) to use D instead of D\\C. As for Equation (6), we use a first-order Taylor expansion to approximate the change in loss, a standard approach in prior work. We avoid second-order terms to maintain algorithmic simplicity. We will add clarifications in the revised manuscript to justify this approximation.\n### **Response #3: No Theoretical Guarantees**\nThank you for pointing this out. We acknowledge  that our method does not provide formal guarantees on generalization or approximation error. Rather, it is a heuristic framework inspired by theoretical insights. We will revise the language to clearly state that our method is “theoretically inspired” rather than “theoretically guaranteed.”\n### **Response #4: Comparison to SOTA Baselines**\nBeyond accuracy, a key contribution of our method is its selection efficiency. While achieving accuracy comparable to SOTA methods, our approach incurs significantly lower computational cost. We acknowledge the lack of runtime and complexity analysis in the original submission and will include it in the revision (see Responses #1 and #4 to Reviewer opAA).\n\nWe also note that coreset evaluation protocols vary widely across prior work, making fair comparisons difficult. Key differences arise in two stages:\n\n- **Preparation before selection**: Some methods (e.g., *D2Pruning*, *Boundary-CCS*) require a fully trained model (e.g., 200 epochs) prior to selection, which is computationally expensive. In contrast, our method selects using a partially trained model (e.g., 20 epochs), offering substantial efficiency gains.\n\n- **Training after selection**:\n  - **Warm start**: Training proceeds without reinitializing the model after selection. This typically improves performance. We provide results showing our method further benefits from this tip.\n  - **Fixed iterations**: Keeping the number of SGD steps constant across core-set sizes nullifies the efficiency benefit of using smaller subsets.\n  - **Fixed epochs (ours)**: Using the same number of training epochs across different core-set sizes reduces the overall training cost as the size of the coreset decreases.\n\nOur experiments adopt the fixed-epoch, no-warm-start setting and include reproductions of the baseline methods for relatively fair comparison. Even under these constraints, our method—trained with only a few epochs—achieves comparable results to *D2Pruning*, which relies on full model training. Furthermore, our selection algorithm has near-linear time complexity (≈O(N)), making it scalable to large datasets such as ImageNet-1K.\n\nRegarding Tukan et al. (ICML 2023), as cited by the reviewer, their focus is on extremely small coreset sizes under a highly specific setup (learning rate 0.01, batch size 20, 300 epochs). This setting differs significantly from our experimental protocol and most existing data pruning benchmarks, making direct comparison less appropriate.\n\n| Method    | C10(70%) | C10(50%) | C10(50%) | C100(70%) | C100(50%) | C100(30%) | TI(70%) | TI(50%) | TI(30%) |\n|:----------------:|:--------:|:--------:|:--------:|:---------:|:---------:|:---------:|:-------:|:-------:|:-------:|\n| MRMC-R   | 95.46    | 95.24    | 93.13     | 77.17     | 74.46     | 68.66   | 60.42   | 57.03   | 50.88 |\n| MRMC-R-warm   | 95.65    | 95.25    | 93.88     | 77.83     | 75.00     | 70.05   | 61.44   | 58.46   | 55.01 |\n\n\n### **Response #5: Empirical Validation of Key Assumptions**\nWe appreciate the suggestion. We conducted a case study to empirically validate these assumptions (see Response #2 to Reviewer JTrd). For symmetry, the average relative error between attribution scores a_ij and a_ji is consistently below 24%, with clear diagonal symmetry observed. For stability, attribution scores remain stable across different core-set sizes, with relative errors below 30%.\nInitial observations suggest that optimizer settings and core-set size influence these properties.\n\n### **Summary**\n We acknowledge the limitations of our current work and will revise the manuscript to clarify assumptions, experimental comparisons, and better position our contributions. Thank you again for the valuable feedback.", "claims_and_evidence": "1. The authors claim that prior work neglected the theoretical guarantees. However, theoretical methods for practical subset selection have been proposed already, both for data (https://proceedings.mlr.press/v202/tukan23a.html, https://ieeexplore.ieee.org/abstract/document/9941065) and model (https://proceedings.neurips.cc/paper_files/paper/2022/hash/f7fc38fdd95fd146a471791b93ff9f12-Abstract-Conference.html) compression.", "methods_and_evaluation_criteria": "The authors evaluate using the accuracy metric on CIFAR10, 100, and Tiny ImageNet.\nThey also see how the loss on the corset is translated to accuracy in the test set.\n\nThey also analyze accuracy as a function of the three key parameters-> the number of initial training epochs, the scale of the regularization\nsubset size, and the trade-off between the losses parameter.", "theoretical_claims": "1. Equation 1, why D\\C and not the full data D?\n2. I don't see how the second derivation works in (6), please provide the details carefully for review/\n\n-----------------------\nIn general, the paper is not a theoretical one, while I agree it has a good theoretical *motivation*, it does not have theoretical guarantees. Thus I would not use the statements \"balance efficiency and theoretical guarantees\", \"Both approaches are theoretically robust and easy to implement.\", and many others -- I don't see theoretical guarantees on the training, approximation error, generalizability or else.", "experimental_designs_or_analyses": "Experiments are not good enough as the authors does bt compete with SOTA baselines which achieve a much better accuracy. See cifar10/100 results in the paper \"https://proceedings.mlr.press/v202/tukan23a.html,\" and all the other competing methods there.", "relation_to_broader_scientific_literature": "x", "essential_references_not_discussed": "x", "other_strengths_and_weaknesses": "Strengths:\n\n1. good idea with some theoretical motivation -- the idea is that a good coreset should satisfy two things: diversity, and effect on the loss. So the authors compute the subset that contributes the most to reducing the loss, and add a regularization to constraint diversity of the coreset. \n\n2. Writing is good.\n\nWeaknesses:\n\nMainly experiments, prior work has some better results.", "other_comments_or_suggestions": "x", "questions_for_authors": "You assume that “attribution values” (the pairwise gradient inner products) are stable and roughly symmetric. Have you measured how large the asymmetry (i.e., ∣aj,i−ai,j∣∣aj,i​−ai,j​∣) is in practice? Do you see any specific scenarios or datasets where this assumption might break?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lx0Ef6wSWR", "reviewer": "Reviewer_opAA", "summary": "This paper proposes a two-phase core-set method for selecting a small but representative subset of training data. The first phase selects samples with the highest contributions, while the second phase employs a lightweight proxy model to evaluate the differences between the remaining samples and already selected samples, further selecting samples that can increase overall diversity.\n\n# **update after rebuttal**\nI would like to thank the authors for their sincere efforts to address my concerns. I have increase my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your insights have greatly helped us identify key areas for improvement. We address the main concerns below:\n### **Response #1: Insufficient Novelty in Contributions**\nRather than introducing a new theoretical guarantee framework, our contribution focuses on the design of a simple, efficient, and practical selection strategy. MRMC can be computed with minimal overhead by measuring sample loss reduction. Although computing the OPT value requires training a proxy model, it is a single linear layer, making training extremely fast. The overall complexity is near-linear (≈O(N)).\n\nAs the title suggests, “Efficient Core-set Selection” emphasizes not just coreset quality but also selection cost. E.g., MRMC uses only 20 epochs for preparation, whereas methods like D2Pruning require a fully trained (200-epoch) model on CIFAR. Moreover, D2Pruning involves costly feature distance computations and information propagation. On ImageNet-1K, D2Pruning takes 350 seconds to build a 30% core-set, while MRMC-R takes 50s and MRMC under 10s. \n### **Response #2: Ablation Experiments**\nTable 1 already compares MRMC, which only maximizes loss reduction, with MRMC-R, which incorporates balanced loss regularization— the two core techniques in our framework. The regularized version shows clear advantages on more challenging tasks and under smaller coreset sizes. Section 5.3 specifically analyzes the effectiveness of the balancing term. An ablation study would be unnecessary. \n### **Response #3: Extreme Pruning**\nWe acknowledge this as a limitation of our work. As a heuristic method, we focus on selection efficiency, which comes at the cost of reduced performance under extreme pruning. While small subsets are highly valuable in  interpretability and incremental learning, our method is primarily optimized for moderate ratios, where it achieves comparable performance with low cost.\n### **Response #4: Lack of Runtime**\nWe appreciate your comment. Although efficiency is central to our work, we omitted a runtime analysis — this was an oversight. We will include wall-clock time  in the revision to better support our efficiency claims.\nAs shown in the table, many strong SOTA methods suffer from high computational overhead, which limits their practicality on large-scale datasets. In contrast, MRMC and MRMC-R achieve a favorable balance between coreset quality and selection efficiency.\nWall-clock time (seconds) for constructing a 50% coreset on CIFAR-10, CIFAR-100, and TinyImageNet, and a 30% coreset on ImageNet-1K:\n| Method     | CIFAR-10 | CIFAR-100 | TinyImageNet | ImageNet-1K |\n|-----|-----:|------:|--------:|-------:|\n| EL2N       |      0.1 |       0.1 |          0.2 |           — |\n| GraNd      |      0.1 |       0.1 |          0.2 |           — |\n| Glister    |     20.4 |      27.4 |         75.1 |           — |\n| CCS        |      0.3 |       0.3 |          0.9 |           — |\n| Moderate   |      0.8 |       0.8 |          1.5 |           — |\n| Dyn-Unc    |      0.2 |       0.3 |          0.7 |           — |\n| Boundary   |    102.1 |     127.1 |        290.7 |           — |\n| D2Pruning  |     10.9 |      11.3 |         25.1 |        352.9 |\n| MRMC       |      0.2 |       0.2 |          0.4 |          9.3 |\n| MRMC-R     |      1.0 |       1.3 |          2.9 |         50.7 |\n### **Response #5:  Theoretical Assumptions**\nOur method is heuristic and not theoretically guaranteed. Nevertheless, we aim to ground our design in theoretically inspired approximations. Thus, we validate the assumptions empirically (see Response #2 to Reviewer JTrd). Our method aims to strike a balance between complex, inefficient theoretical methods and simple but less effective heuristics. \n### **Response #6:  Comparison to DDP**\nWe appreciate your suggestion and will include a discussion of Dynamic Data Pruning (DDP) methods in the Introduction. We agree that DDP methods (e.g., Kakurenbo, InfoBatch) have demonstrated strong performance in accelerating convergence.\nHowever, DDP and Static Data Pruning (SDP) differ fundamentally in both motivation and usage:\n- **SDP** selects a fixed subset  in the early phase of training and discards the remaining data. This enables savings not only in computation, but also in memory and storage, making SDP more suitable for on-device learning or communication bottlenecks.\n- **DDP** assumes access to the full dataset throughout training and aims to reduce computational cost via dynamic sample scheduling. DDP can be seen as an alternative to random data scheduling in SGD, and is more closely related to curriculum learning or advanced sampling strategies.\n\nMoreover, prior SDP works typically do not compare directly with DDPs, and we followed the same convention. Nonetheless, we agree that DDP is an important direction and will acknowledge it in the revision.\n\n### **Summary**\nWe hope that these responses address your concerns and help position MRMC as an effective  solution to the core-set selection problem.", "claims_and_evidence": "In my view, balancing diversity and uncertainty in core-set selection is indeed an important issue. However, the relative importance of diversity and uncertainty may vary in different scenarios. From my perspective, which admittedly may be incorrect, for moderate levels of pruning, more attention should on hard-to-learn samples. This ensures that all rare patterns, which can only be learned through memorization effects, are included in the training set. In this case, the optimization objective of the core set is to reduce redundant samples while retaining the information represented by all samples in the original dataset. On the other hand, in extreme pruning scenarios where very few samples are retained, the focus should be on easy-to-learn samples that represent simple and common patterns. This helps preserve the basic pattern information present in the original dataset in extremal setting.\n\nBased on this perspective, maximizing loss reduction represents easy-to-learn samples, while balancing loss reduction attempts to incorporate hard-to-learn samples. Therefore, the authors' claims themselves are not problematic. However, the evidence provided by the authors is quite lacking:\n\n1. The authors should provide very detailed ablation experiments for maximizing loss reduction and balancing loss reduction to evaluate their respective contributions. Instead, the authors devote a large portion of pages 3-6 to the formalized expression of the proposed method, which is neither critical nor important. The authors should use some of this valuable space (or include it in the appendix) to clearly experimentally verify the contributions of the two-stage method.\n\n2. The authors should provide experiments with extreme removal rates similar to the baseline methods, such as the 90% pruning ratio used in D2pruning, to test the performance of their method in extreme removal settings (it's worth noting that this is precisely the setting where the authors' method has the most theoretical advantage).\n\n3. The paper lacks a detailed evaluation of wall-time runtime. Even if the authors' method does not have an advantage in this aspect, they still need to conduct experiments to report the results.", "methods_and_evaluation_criteria": "See Claims And Evidence.", "theoretical_claims": "In the derivations presented on pages 3 to 6 of the paper, the authors primarily propose a conceptual framework to formalize their proposed method. I do not suggest  this constitutes theoretical contribution. While I think it is perfectly reasonable and unproblematic for the paper to lack theoretical contributions (definitely not affect my score), the weakness of the authors' experimental evidence leads me to expect some theoretical contributions from this paper. Moreover, as mentioned in the limitations section of the paper, the theoretical derivations in this work are based on too many assumptions.", "experimental_designs_or_analyses": "See Claims And Evidence.", "relation_to_broader_scientific_literature": "The authors proposed two-phases method position in two main approaches in **Dataset pruning**:\n\n- **Static Selection**:\nPerformed before or in the early stages of training, aiming to identify a representative subset of the training data. Key research includes Core Set and Data Pruning studies by:\n\nHuggins et al. (2016) Coresets for scalable...\n\nPaul et al. (2021) Finding important examples early in training\n\nKrishnateja et al. (2021) ``Glister''\n\nXia et al. (2022, 2024) on moderate and refined core set approaches\n\n- **Dynamic Selection**:\nAlso known as Dynamic Data Pruning. Involves continuous sample selection throughout the training process. Notable works include:\n\nRaju et al. (2021) Accelerating deep learning with dynamic data pruning\n\nTruong et al. (2023) ``Kakurenbo''\n\nQin et al. (2024) InfoBatch", "essential_references_not_discussed": "I notice a minor oversight in the paper regarding the discussion and comparative experiments with Dynamic Data Pruning (DDP). Based on empirical evidence, DDP based methods demonstrate superior performance at moderate pruning rates, while static pruning methods regain their advantage under extreme pruning conditions. I strongly recommend that the authors incorporate discussions of the following papers and include the latest state-of-the-art DDP method in their baseline comparisons in Table 1:\n\n[1] Accelerating deep learning with dynamic data pruning, arXiv 2021.\n\n[2] KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training, NeurIPS 2023.\n\n[3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n\n[4] Instance-dependent Early Stopping, ICLR 2025.", "other_strengths_and_weaknesses": "This article is well-organized and the author attempts to address important challenge in core-set.", "other_comments_or_suggestions": "I suggest this paper currently falls slightly below ICML acceptance standards.\nI should note that the 5-point scale used here (unlike the traditional 10-point system common in ML conferences) makes the evaluation confusion. I am assigning a Weak Reject, though my precise assessment would be a borderline reject. I hope the authors understand this current status more clearly.\n\nIn my view, the paper suffer from insufficient novelty in its contributions and inadequate experimental evaluation. I would direct the authors' attention to D2pruning, which shares similar motivations in addressing both DIVERSITY & DIFFICULTY, but provides substantially much more comprehensive experimental validation. To strengthen this work, the authors need to provide extensive additional experiments (I guess other reviewers will suggest specific evaluations as well) to demonstrate their method's contributions beyond D2pruning.\n\nA minor suggestion: The term ''Selection ratio'' should be replaced with ''Pruning rate'' (30% Pruning rate = 70% Selection ratio) to maintain consistency with the established terminology in previous literature.", "questions_for_authors": "Not applicable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the theoretically motivated and computationally efficient algorithm, and the solid experimental evaluation. Although the problem setting has been studied many times, this work is a solid contribution. Thus, I recommend acceptance.
Decision: Accept (poster)