Efficient Representativeness-Aware Coreset Selection
Authors: Zihao Cheng
Binrui Wu
Zhiwei Li
Yuesen Liao
Su Zhao
Shuai Chen
Yuan Gao
WEIZHONG ZHANG
Venue: NeurIPS 2025 poster ()
Keywords: Coreset Selection
Deep Neural Networks
Efficient Representativeness-Aware
Primary area: general_machine_learning

Abstract: Dynamic coreset selection is a promising approach for improving the training efficiency of deep neural networks by periodically selecting a small subset of the most representative or informative samples, thereby avoiding the need to train on the entire dataset. However, it remains inherently challenging due not only to the complex interdependencies among samples and the evolving nature of model training, but also to a critical *coreset representativeness degradation issue* identified and  explored in-depth in this paper, that is, the representativeness or information content of the coreset degrades over time as training progresses. Therefore, we argue that, in addition to designing accurate selection rules, it is equally important to endow the algorithms with the ability to assess the quality of the current coreset. Such awareness enables timely re-selection, mitigating the risk of overfitting to stale subsets—a limitation often overlooked by existing methods. To this end, this paper proposes an **E**fficient **R**epresentativeness-**A**ware **C**oreset **S**election method for deep neural networks, a lightweight framework that enables dynamic tracking and maintenance of coreset quality during training. While the ideal criterion—gradient discrepancy between the coreset and the full dataset—is computationally prohibitive, we introduce a scalable surrogate based on the signal-to-noise ratio (SNR) of gradients within the coreset, which is the main technical contribution of this paper and is also supported by our theoretical analysis. Intuitively, a decline in SNR indicates overfitting to the subset and declining representativeness. Leveraging this observation, our method triggers coreset updates without requiring costly Hessian or full-batch gradient computations, maintaining minimal computational overhead. Experiments on multiple datasets confirm the effectiveness of our approach. Notably, compared with existing gradient-based dynamic coreset selection baselines, our method achieves up to a 5.4\% improvement in test accuracy across multiple datasets.

Efficient Representattveness-AAarr Coreset Selection
Zihao ; Binrui Wu'; Zhiwei Li', Yuesen Liao', Su Zhao², Shuai Chen", Yuan Gao², Weizhong Zhang 1.4f Fudan University, Meituan Inc, Wuhhn University Shanghai Key Laboratory of Intelligent Information Processing (zhcheng25, brwu23, zwli23, ysliao24] @m. fudan. edu, cn (zhaosu04, chenshuai31) tuan. com, ethan y w.w.gggaaaa....mm com weizhongzhang edu. cn
Abstract
Dynamic coreset selection promising approach for improving the training efficiency of deep neural networks by periodically selecting small subset of the most representative or informative samples, thereby avoiding the need train the entire dataset. However, it remains inherently challenging due not only to the complex interdependencies among samples and the evolving nature of model training, but also to critical coreset representativennss degradation issue identified and explored in-depth in this paper, that is, the representativeness information content of the coreset degrades over time training progresses. Therefore, we argue that, in addition to designing accurate selection rules, it is equally important to endow the algorithms with the ability to assess the quality of the current coreset. Such awareness enables timely re-selection, mitigating the risk of overfitting stale subsets-a limitation often overlooked by existing methods. To this end, this paper proposes an Efficient Representativeness: ware Coreset Selection (ERACS) method for deep neural networks, a lightweight framework that enables dynamic tracking and maintenance of coreset quality during training. While the ideal rriterio--gradienn discrepancy between the coreset and the full dataset- -is computationally prohibitive, we introduce a scalable surrogate based on the signal-to-noise ratio (SNR) of gradients within the coreset, which the main technical contribution of this paper and is also supported by our theoretical analysis. Intuitively, decline SNR indicates overfitting to the subset and declining rep- resentativeness. Leveraging this observation, our method triggers coreset updates without requiring costly Hessian or full-batch gradient computations, maintaining minimal computational overhead. Experiments on multiple datasets confirm the effectiveness our approach. Notably, compared with existing gradient-based dy- namic coreset selection baselines, our method achieves up to 5.4% improvement in test accuracy across multiple datasets.
1 Introduction
The success modern deep neural networks largely relies on large-scale datasets, which are typically collected from the internet. These datasets often contain significant amount of redundancy, noise, and even mislabeled samples [24]. To address this issue, coreset selection has been extensively studied, originally proposed in the context of classical machine learning models, e.g., SVM [23]. logistic regression [22 and Gaussian mixture model [9). The goal is to find a small subset of weighted training samples, referred to coreset, which ensures that training on this coreset yields
"Equal Contribution.
t Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

performance comparable to that on the full dataset. Their key idea is based on function approximation, i... the training loss defined on the coreset should be close to the one defined on the whole training data. These early methods are predominantly static, meaning that the coreset is selected before training and reused throughout the training process. Lots of promising results have been reported in the literature [23] 12
19 g) Nevertheless, coreset selection is inherently discrete and high- dimensional optimization problem and remains highly challenging to large-scaled applications. s Static coreset selection strategies often underperform in deep neural networks [3]. The main reason is that compared to machine learning models, deep neural networks exhibit highly non-stationary training dynamics and more intricate inter-sample correlations, making it difficult ensure that importance estimations made before training remain valid throughout the entire training process. Therefore, it is extremely challenging to select the coreset accurately in the early stage of training or even before training. As result, a new line of research, i.e., dynamic coresel selection [3, has emerged. These methods periodically update the coreset during training to adapt to the current state of the model. Representative approaches include GradMatch [1]. which selects samples by matching gradients between the subset and the full dataset, and GLISTER [15]. which guides selection by optimizing a proxy validation loss via influence functions.
Gradien: Dur
Gradiam Dur
Epoch
Epoch
Epoch
Epoch
(a) Grad dif((CIFAR-IO) (b) Grad dIFF(CIFRR-)0O)
(c) SNR(CIFAR-10)
(d) SNR(CIFAR-100)
Figure 1: Gradient discrepancy and SNR across training epochs. Subplots (a) and (b) show the gradient difference between the coreset and the full dataset on CIFAR- 10 and 1 Subplots (c) and (d) report the SNR, defined as the ratio of the mean to the standard deviation of coreset gradient norms. As training progresses. the gradient discrepancy between the coreset and the full dataset increases, indicating a decline in coreset representativeness. Meanwhile, the SNR decreases, suggesting that the model is overfitting the current coreset.
Motivations. Beyond the intricate training dynamics of deep neural networks and the complex inter- sample correlations discussed earlier, this paper identifies another crucial but often overlooked issue:, ie., the degradation of coreset representativeness over time, which also adds significant challenge to coreset selection. We refer this observation the coreset representativeness degradation issue, which directly motivates the development of the effective coreset selection methods proposed in this work. Specifically, while the initially selected subset may align well with the full gradient direction, it gradually becomes outdated as training progresses. Using GradMatch [4 as an example, the experimental results presented in Figure (a) and (b) demonstrate that the gradient discrepancy between the coreset and the full dataset increases rapidly after each round of coreset selection. Based on these phenomenon, we argue that, in addition designing more precise selection criteria, is equally crucial to equip coreset algorithms with serlf-awaarnesss-hee ability to dynamically assess and maintain subset quality throughout training. Notably, few recent attempts have been made to partially address this issue. For example, CREST [25] introduces curvature-awaid perspective by locally approximating the loss as a quadratic function and leveraging Hessian- peeeian-prconittioneee gradients to determine whether the current coreset still accurately estimates the full-data gradient. This approach helps determine whether reselection necessary. This method demonstrates improved performance over other dynamic strategies, but the high computational cost of computing the Hessian or its inverse significantly undermines one of the core advantages of coreset methods, i.c., training efficiency.
In this paper, we propose an Efficient Representativeness-A Aware Coreset Selection method (ERACS) for deep neural networks, a lightweight framework that enables dynamic tracking and maintenance of coreset quality during training. It important to note that, since the gradient discrepancy between the coreset and the full dataset is computationally prohibitive, this ideal criterion cannot be directly adopted to develop an efficient coreset selection method. Our theoretical analysis in Section 4.2 exhibits that the Signal-to-Noise Ratio (SNR), defined as the ratio between the norm of the mean coreset gradient and its standard deviation, of the coreset gradients is closely related with the gradient discrepancy. As shown in Figure SNR consistently aligns with the gradient discrepancy
2

---

and significantly decreases after each round of coreset selection, which may be attributed to the model overfitting the outdated and representativeness-liiiee subset. Since SNR of gradients within the coreset can be computed efficiently, we therefore develop a scalable surrogate based on it in our approach, which is the main technical contribution of this paper. Leveraging this surrogate, our method triggers coreset updates without requiring costly Hessian or full-batch gradient computations, maintaining minimal computational overhead. Experiments on multiple datasets confirm the effectiveness of our approach. Across multiple datasets, our approach delivers test accuracy gains of up to 5.4% over state-of-the-an gradient-bant bbased dynamic coreset selection methods.
Our main contributions are summarized as follows:
We identify the phenomenon coreset representativeness degradation in dynamic coreset selection methods, particularly in late-stage training.
We analyse the theoretical relationships between SNR the coreset gradients and gradient discrepancy, which plays a fundamental role in developing our coreset selection methods.
We propose a lightweight dynamic coreset selection method, which enables dynamic tracking and maintenance of coreset quality during training.
We validate our method on multiple benchmark datasets, achieving higher test accuracy with negligible additional training cost.
2
Related Work
We first review various coreset strategies tailored to deep models. We then highlight gradient matching techniques, which select training samples that directly approximate the full-data gradients from an optimization perspective and have shown strong empirical performance across benchmarks.
2.1 Coreset Selection in Deep Learning
has been extensively studied since the era of traditional machine learning. However, classical coreset selection methods [IO 5 4 designed for conventional machine learning tasks (e.g., geometry-baat
Coreset selection 14
21 10
approaches relying on Gaussian mixture assumptions 8 I])
exhibit significant limitations when applied to deep learning, primarily due to their high compu- tational complcxity and fixed data representations. With the rapid advancement of deep learning, recent research [14 6 3 2 coreset selection has shifted toward novel methodologies tailored to the characteristics of deep neural networks. These approaches can be broadly categorized into: (1) Geometry- based methods (e.g., k-Center Greedy (22). which construct coreset by optimizing the spatial distribution of samples; (2) Prediction uncertainty- baased methods (e.g., entropy sampling (7],
which prioritize samples that are difficult for the model to classify; (3) Gradient matching methods (e.g., Craig 20), GradMatch [14), which enhance training efficiency by minimizing the gradient discrepancy between the subset and full dataset; (4) Bilevel optimization approaches [26 3 which formulate the selection process as an optimization problem based on validation performance; and (5) Submodular function-basse based techniques (e.g., Similar [6). Prism [7].. which leverage mathematical properties to ensure sample diversity. Although these methods demonstrate promising results in specific scenarios, empirical studies [ reveal that random sampling remains strong and compet- itive baseline, underscoring the need for further exploration to bridge the gap between theoretical guarantees and practical effectiveness in deep learning- baaseng-bbased core-set selection.
2.2
Grradient-Matching--
Methods
This paper primarily investigates
-att--mmttttttsmseeee coreset selection methods 14 20 15
25
These methods explicitly target the approximation of the full-dataset loss gradient, selecting training subsets from an optimization perspective. A representative approach is CRAIG 20 selects samples that are optimally aligned with the full gradient direction by formulating a submodular facility location problem solvable via greedy algorithms, yet suffers from computationally expensive gradient estimation that hinders large-scale applicability. GLISTER [15] formulates coreset selection as a bilevel optimization problem aiming to maximize validation performance, offering model adaptability despite its dependence on clean validation data and learning objective smoothness assumptions. Most relevant to this work is GradMatch [4] directly selects training samples to
3

---

ensure subset gradient approximation of full-data gradients, supporting online/offline modes with proven convergence guarantees under convexity and empirical superiority over traditional coreset methods across benchmarks, particularly for high-efficiency training or data-constrained scenarios. CREST [25] achieves convergence for non-convex models by approximating the loss with quadratic functions and extracting sub-region coresets, using iterative mini-batch selection to minimize gradient variance while pruning learned samples.
Preliminaries
In what follows, first present the core optimization objective of GradMatch [14), which seeks weighted subset that closely approximates the full gradient. Due to the combinatorial nature of this objective, then introduce efficient greedy solver based Orthogonal Matching Pursuit (OMP) that incrementally constructs a representative coreset.
3.1 Basics on GradMatch
GradMatch [4] is gradient-based data subset selection framework designed to accelerate deep neural network training by approximating the full gradient using a small, weighted subset of training samples. Let (Ø) denote the empirical loss over the full dataset, and VoL(0) its corresponding full-batch gradient. At each training step t, the goal is to approximate this gradient using a weighted combination per-sample gradients over subset X D with associated non non-negative weights w'.
To quantify the quality of approximation, GradMatch
14 defines the gradient matching error:
Err(w', X L, L,) =
Σ w V.V.L'(0. -VL NEX*
where 0, is the model parameter at step t. L (0:) is the loss on sample from the training set, and VaL' (0z) denotes the gradient of the individual loss.
To regularize the optimization and prevent overfitting to few incorporates an & regularization term and defines the regularized objective:
points, GradMatch
Frrx (w, X, I,, 0,) = Er w X, L, 0e) + /||
(2)
where À is a hyperparameter controlling the strength of the regularization.
The core optimization problem is thus formulated as:
w = arg min Errx(w, N.W 0c),
(3)
w. W
where | denotes the number of samples in X. budget on the maximum subset size. This formulation guides the design of subset selection algorithms within GradMatch (14)
3.2 Orthogonal Matching Pursuit for Subset Selection
exactly is intractable. To tackle this, GradMatch [4] adopts the Orthogonal Matching Pursuit (OMP) algorithm, greedy iterative approach that approximates the optimal subset.
Solving the combinatorial optimization problem Equation 3
OMP incrementally builds the subset X' by selecting at each step the data point most correlated with the current residual. The residual is defined as the gradient of the regularized error with respect the weights, evaluated at = 0:
Vw Err (X' w, L, 0:)l
This residual captures the contribution of each candidate point to the reduction error. The sample with the largest absolute component in is added to the subset.
OMP proceeds as follows:
Initialize the subset X' = @ and compute the initial residual at 0.

---

While X S k and Ex(2) 2 e, repeat:
Select e arg max, Iryl and update X {e}.
Re-optimize weights: w + arg minw Errx(w, X, 0,). Recompute the residual with the updated w.
Return the selected subset X' and weights w.
This procedure avoids exhaustive enumeration and efficiently constructs a representative subset that minimizes the gradient mismatch with regularization.
4
Method
This section is divided into three parts. Section 4.1 introduces our motivation experiment, which reveals the progressive divergence between coreset and full-dataset gradients during training. Section analyses an upper bound on the gradient approximation error via concentration inequality, which connects the gradient discrepancy with SNR and naturally motivates the use the SNR as tractable surrogate for representativeness. In Section 4.3 we propose ERACS, which is lightweight coreset selection framework that enables dynamic tracking and maintenance of coreset quality during
4.2
training.
4.1 Motivating Experiment: Coreset Representativeness Degradation Issue
Theorem in GradMatch indicates that if the full gradient is accurately approximated by the one defined on the coreset, the model learned with coreset selection should be close to the one trained on the full datasets. However, our preliminary reproduction of GradMatch shows that performance improvement over random selection is less substantial than theoretically expected. This inconsistency indicates potential progressive divergence between the gradients as training proceeds-thar that is, the representativenes: of the coreset may degrade rapidly over time. To investigate this discrepancy, we conducted full GradMatch training run and recorded the gradients of the full dataset and the selected coreset at every epoch.
coreset and full dataset
Specifically, we implement GradMatch with classification tasks on CIFARIO and CIFAR100 using ResNet18 8 11. At each epoch, we compute the gradient over the full dataset, denoted giall, and the cos geore) as the gradient
gradient over the selected coreset, denoted as Jcore We then used difference metric to quantify the representativeness of the coreset. As shown in Figure (a) and (b), the gradient difference remained consistently small in the early stages of training, indicating that the coreset effectively captured the full-dataset gradient direction.
However, in the later stages of training, we observed that immediately after selected, the gradient difference would rapidly increase, suggesting sharp drop in representativeness. We refer to this phenomenon coreset representativeness degradation issue. This phenomenon highlights a key limitation of existing static or infrequent-update coresct strategies and motivates our development of representativeness:
coreset was
dynamic coreset selection framework.
4.2 Connections Between Gradient Diserepancy and SNR
While the gradient discrepancy introduced in Section 4.1 serves as an accurate measure of coreset rep- resentativeness, it requires computing the full dataset gradient at every epoch, resulting in significant computational overhead. This indicates that this gradient discrepancy cannot be directly adopted to develop efficient coreset selection method. This issue motivates develop scalable surrogate for gradient discrepancy, which can approximate representativeness without requiring full gradient access.
Theoretical Analysis. We investigate the theoretical connection between gradient discrepany and SNR. Let μn denote the full-dataset gradient at iteration f, and fúe denote the gradient computed a coreset S, of size m. The SNR is defined as:
[ 112
SNR
5

---

where on denotes the standard deviation of sample gradients within the coreset. The following theorem provides a probabilistic bound on gradient approximation error, which characterizes the connection between gradient discrepancy and SNR.
Theorem (Gradient Approximation Bound via SNR). [informal] If the gradients is Sub- Gaussaassi:n:
Then with probability at least 1 8:
114 Pulle A
(2log(1/0) /m SNRcore,t
The formal version of and its complete proof can be found in the supplementary materials.
This result indicates that larger SNR leads to tighter approximation of the full-dataset gradient, validating the use of SNR as proxy for representativeness.
Empirical Observation. Figure 1 (c) and (d) present the dynamics of SNR of the experiments in Section[4.1] We can observe that SNR consistently aligns with the gradient discrepancy. Moreover, during the early stages of training, SNR remains high, reflecting high-quality coreset. However, later stages, SNR decreases sharply after each update, which may be attributed to the model overfitting to the outdated and representaaiveness-iime limited subset. The reason could be that when the model overfits the coreset, the gradient fú, vanishes while the variance o, stay constant.
Statistical Illustaation:
Linear Regression Example. To further understand the behavior of SNR and its connection to coreset overfitting, we consider simple linear regression setup. Suppose each input x; and noise term are drawn independently from Gaussian distributions:
VA A x; + E _N(0,
2; (12,,
where A is the optimal model trained on the coreset, i.c., the optimal solution of problem (5) Assume A is the intermidate model obtained during the training process on a coreset {(T;, y/ :=1 by minimizing the squared loss:
I(A)=
U1
Ax²)²
(5)
2m
The gradient of the loss with respect the model parameter A for each sample is:
g -, Ar;) 22(E; dz;), where ô A - A"
We now derive the expectation and variance of the per-sample gradient:
Eig] 8 E[
(6)
Var (g:) E[g²] (Big:))2
(7)
[[=,({s - 8z;) 6² (1{2)))
(8)
= E[u/e]] 28E[x 8 8² (E(x?)))
(9)
= B[² + 52 -B|=))((B|=))""
(10)
where we used the independence of x; and Es, and the fact that E[e,] and Bi= o2.
Let be the kurtosis of x;, i.e., N = (ا samples as:
B's we can have the SNR of the average gradient over m
[[|gs]]
SNR
(11)
Var /Va(g:)/m
+ (K 1
B.
As the model better fits the coreset (i.c., |8| + 0), the signal E[g:]] vanishes, but the variance term remains bounded due to the noise o². As result, SNR tends to zero. This illustrates that a low SNR an indicator of coreset overfitting and supports its use as a dynamic signal for triggering subset updates during training.
Time Efficiency in Computing SNR. shown in Table I computing the full-batch gradient on CIFAR-10 10 takes approximately 14.39 seconds, while computing the coreset SNR only takes about 1.45 seconds- an order of magnitude faster. Therefore, SNR can serve as lightweight, scalable surrogate metric evaluate the representativennss of the coresct.

---

Table Computation time (in seconds) for Gradient Diff vs. coreset SNR on CIFAR-IO.
Metric
Gradient Diff'
Coreset SNR
Time (s)
14.39
1.45
4.3 Represettaiveness-Aaarr Coreset Selection
now present our representativeness-avae dynamic selection framework that adaptively refreshes the coreset based its measured SNR. Unlike prior works that update subsets at fixed intervals regardless of quality, ERACS periodically checks whether the gradient SNR falls below preset threshold Tenr If so, subset is selected using an external method such as OMP; otherwise, training proceeds with the current coreset. Algorithm I gives the detailed steps of our method ERAC.
coreset selection method ERAC based on SNR, which
Algorithm Efficient Representaaiiveness-Awi Coreset Selection (ERACS)
Require: Training set UI; initial subset 20();; learning rate total epochs T;; coreset size k;; check interval C: threshold Tsnri batchsize B.
1: for = 1 to 7 do
2: if t. mod C then
Compute SNR, SNR(A((-1), 0),1) 4: if SNR, < Tsnr then
3:
5:
x²(1) w t OMP(L, 0,-,,,,
6:
else
7:
x2 (t(-1)
8:
end
9:
else
10:
X²(1) + x(1-1)
end if
12: 0g 6.+Ban BatchsGDD((()) w², a, B, Epochs = 1)
end for
14:
return 0r
Here, OMP(-) is greedy algorithm solve the coreset selection problem BatchSGD trains the model with a stochastic optimization algorithm using the weighted coreset for some epoches. Benefited from the computation efficiency of SNR, our method can track the representativeness of the coreset during training without requiring costly Hessian or full-batch gradient computations, maintaining minimal computational overhead.
Experiments
In this section, first provide detailed overview of our experimental setup, including the datasets, baseline methods, and training configurations. We then conduct a comprehensive comparison between proposed ERACS method and state-of-the-rs coreset selection algorithms, namely CRAIG, GLISTER, GradMatch, and CREST, as well as Random baseline. In addition to evaluating model performance, we compare training speedups study to analyze the sensitivity of BRACS to key hyperparameters.
methods. Finally, perform ablation
5.1 Experimental Setup
Datasets and baselines We evaluate our proposed method, Efficient Representaiiveess-AAare Coreset Selection (ERACS), on three standard datasets: CIFAR-10, CIFAR-100, and ImageNet. All experiments are conducted using ResNet-18 as backbone architecture. compare ERACS against six baselines: training on the full dataset (Full), uniform Random selection, and four representative coreset methods: CRAIG R GradMatch 14,, Glister [5] and CREST 25
Evaluation is based on test accuracy and training efficiency.

---

Training details. For training, we use stochastic gradient descent (SGD) with momentum 0.9, weight decay 5 10 and cosine-annealed learning rate starting from 0.1. Models are trained for 300 epochs on CIFAR datasets and 350 epochs on ImageNet. We fix the coreset budget at 10% across all settings. To decide when update the coreset, FRACS monitors the SNR of the gradient norms every C' epoch and triggers reselection only when the SNR exceeds threshold Torr- This mechanism ensures minimal overhead while preserving representativeness. All experiments were run on Nvidia 4090 GPUs.
5.2 Main Results
ERACS achieves consistently higher accuracy across datasets. Table reports the accuracy under 5%, 10% and 20% data budget. ERACS outperforms all other coreset selection baselines across all benchmarks. GradMatch by 5.4% and CRAIG by over 14%. On ImageNet, ERACS attains 63.91 % with 10% budget, surpassing both GradMatch and CREST by 4.75% and 2.27%, respectively.
CIFAR-100 with 10% budget, ERACS achieves 69.67%, improving over
Table 2: Test accuracy (%) under different data budgets using ResNet- 18. The bolded values indicate the best-performing method. Random, Craig, Glister, and GradMatch select coresets every 20 epochs. CREST selects mini-batches and decides when to update them based on its quadratic loss approximation. Our method selects coresets based the SNR indicator.
Dataset
Budget
Random
Craig
Glister
GradMatch
Crest
ERACS
Full
5%
84.23
82.74
84.5
85.4
85.83
87.27
CIFAR-10
10%
88.23
87.35
88.49
89.41
89.87
91.37
94.94
20%
90.20
89.07
89.86
91.38
91.69
92.75
38.89
37.04
29.55
40.53
45.81
46.86
CIFAR-100
10%
63.17
55.07
43.94
64.26
68.10
69.67
75.21
20%
62.18
60.66
52.75
64.26
68.92
71.03
5%
42.45
44.28
43.76
45.15
46.72
48.96
ImageNet
10%
57.41
55.47
53.09
59.16
61.64
63.91
70.50
20%
62.47
59.49
55.94
63.6387
65.97
68.69
ERACS maintains tighter gradient alignment during training. To better understand ERACS's behavior, we analyze the gradient discrepancy between the selected subset and the full dataset. Tigure[sshww that baseline methods like GradMatch suffer from large gradient gaps in later training stages, indicating overfitting on stale coresets. In contrast, FRACS keeps the gradient difference consistently low, thanks to timely reselection triggered by SNR signals.
Tost Accuracy (%)
Sone 190
Gradicnt Dit
GestMOnc
Gradicn DaT
GradMash
UndMarch
GracMacch
Epoch
Epoch
Epoch
Egoch
(a) CIFAR-10
(b) CIFAR-100
(c) CIFAR-10
(d) CIFAR-100
Figure 2: Comparison between GradMatch and our SNR-based method. Left: Test accuracy. Right:
Gradient difference between the selected subset and the full dataset. ERACS achieves better accuracy and stronger gradient alignment.
ERACS offers strong speed-accuracy trade-offs. Figure compares the training speedup and relative accuracy of different methods. ERACS matches the speed of GradMatch and Crest, while consistently achieving higher relative accuracy. Notably, on 1IFAR-100, ERACS maintains over 0.9 relative accuracy and achieves 7.4x speedup compared to full-data training.
These results confirm that FRACS delivers favorable balance between accuracy and efficiency, and outperforms prior methods under constrained training budgets.

---

CIFAR-10
(b) CIFAR-100
(c) ImageNet
Speedup and
10
1.0
1.0
Relative Accuracy
0.8
0.8
Speedup
0.6
0.6
-0.0
0.4
0.2
0.2
0.2
CMP
Glied
Cost
yidh
0695
yo!
Co9
GlAR
Figure 3: Speedup and relative accuracy comparison across subset selection methods. ERACS achieves higher accuracy than GradMatch while maintaining comparable speed.
5.3
Ablation Study
To understand the sensitivity of ERACS to its hyperparameters, perform ablation studies on the SNR threshold Tear and the checking interval C.
Effect of SNR Threshold Tsar We analyze Tsar E (0.1, 0.2 0.3, 0.4, to study its impact model performance and efficiency. Lower thresholds (e.g., 0.1) increase update frequency and improve gradient alignment but raise computational costs. Higher thresholds (e.g., 0.5) reduce overhead but may degrade accuracy. Our experiments show Tanr 0.3 achieves the best trade-off, balancing accuracy and efficiency.
Effect of Checking Interval C. We evaluate C E 21,2,3,4, 4, 10) to determine optimal SNR monitoring frequency. Smaller C (e.g.. 1) improves responsiveness but incurs high overhead, while larger C' (e.g., 10) sacrifices adaptability for efficiency.
Table 3: Effect of varying SNR threshold Tsar and checking interval C on CIFAR-10. Each cell shows accuracy, number of updates, and mean grad diff. Bold indicates best trade-off.
0.1
0.2
0.3
0.4
0.5
JUpd.
[Upd.
[rrad.
1Acc.
_Upd.
1Grad.
JUpd.
UUpd.
91.51
0.060
91.42
0.171
91.28
0.078
98.84
42
010085
90.32
40
0.0M0
2
91.45
56
0.063
91.34
48
0.074
91.21
40
0.081
90.70
34
0.089
90.10
32
0.095
91.00
5
90.92
36
0.076
90.81
28
0.084
90.45
22
0.099
89
18
0.114
89.60
16
0.122
10
90. 10
30
0,085
89.85
24
0.093
89.42
0.106
88.97
18
0.115
88.57
16
0.119
Our ablation study reveals that ERACS is robust to moderate variations in both the SNR threshold Tsur and the checking interval C. While smaller values of Taa and C' generally lead to higher accuracy and better gradient alignment, they incur more frequent updates and computational overhead. Notably, we identify Tanr 0.3 and C as the optimal setting, achieving a strong balance between performance and efficiency 110--991.33 91.37% accuracy, only 26 coreset updates, and stable gradient behavior. These results validate the effectiveness SNR as lightweight, principled signal for dynamic coresct re-selection during training.
6 Conclusion
This paper proposes ERACS, a lightweight and dynamic coreset selection framework to address the coreset representativeness degradation issue during deep model training. We introduce the SNR of coreset gradients as scalable proxy to monitor representativeness. When SNR drops below a threshold, ERACS triggers reselection using the GradMatch objective, preventing overfitting stale samples and preserving gradient alignment. Experiments across multiple datasets show that ERACS simple, fast, and effective, improving both training efficiency and final performance.
Limitations. While FRACS avoids full- full-gradient or Hessian computations, periodic SNR evaluations may still be costly at extreme scales. Additionally, SNR may not always reflect representativeenss

---

under highly non- non-Gause Gaussian gradients or non non-classification tasks. Future directions include adaptive thresholding, alternative proxies, and extensions broader learning settings.
7 Acknowledgements
This work was supported by the National Nature Science Foundation of China (62472097, 124B 1040) and AI for Science Foundation of Fudan University (FudamX24A1028), The computations in this research were performed on the CFFF platform of Fudan University.
References
[1] Olivier Bachem, Mario Lucic, and Andreas Krause. Coresets for nonparametric estimation- the case of dp-means. In International Conference on Machine Learning, pages 209-217. PMLR, 2015.
[2] Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio. Semantic redundancies image- classification datasets: The 10% you don't need. arXiv preprint arXiv: 1409, 2019.
[3] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33:14879- 14890, 2020.
[4] Trevor Campbell and Tamara Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Machine Learning, pages 698-706. PMLR, 2018.
[5] Kenneth L Clarkson. Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Transactions on Algorithms (TALG), 6(4):1-30, 2010.
[6] Michael B Cohen, Cameron Musco, and Christopher Musco. Input sparsity time low-rank approximation via ridge leverage score sampling. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithmss, pages 1758-1777. SIAM, 2017.
[7] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang. Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 1906. 10006.118200 1829, 2019.
[8] Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. Advances in neural information processing systems, 24, 2011.
[9] Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pages 181-195. Springer, 2022.
[10] Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the siitt annual ACM symposium on Theory of computing, pages 291-300, 2004.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[12] Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in Neural Information Processing Systems, 29:4080-444 4088, 2016.
[13] Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1289-1299. IEEE, 2019.
[14] Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464-5474. PMLR, 2021.
10

---

[15] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8110-8118, 2021.
[16] Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submod- ular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34: 18685-1888777, 2021.
[17] Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: unified framework of parameterized submodular information measures for targeted data subset selection and summarization. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI, 2022.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[19] Mario Lucic, Matthew Faulkner, Andreas Krause, and Dan Feldman. Training gaussian mixture models at scale via coresets. The Journal of Machine Learning Research, 18(1):5885-5999 2017.
[20] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950-6960. PMLR, 2020.
[21] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596-20607, 2021.
[22] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. arXiv preprint /rXiv:/708.0000
2017.
[23] Ivor W Tsang, James T Kwok, Pak-Ming Cheung, and Nello Cristianini. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research, 6(4), 2005.
[24] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333. 2024.
[25] Yu Yang, Hao Kang, and Baharan Mirzasoleiman. Towards sustainable learning: Coresets for data-efficient deep learning. In International Conference on Machine Learning, pages 19314-393380 PMLR, 2023.
[26] Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Prob- abilistic bilevel coreset selection. In International conference on machine learning, pages 27287-27302. PMLR, 2022.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contribution this paper lies identifying the gradient mis- match problem that occurs in the later stages of classical gradient-based core-set selection algorithms, proposing lightweight signal-to-noise ratio (SNR) metric indicate the ap- plicability of the current core-set, and providing theoretical analysis to demonstrate the rationality of the SNR metric.
Guidelines: The main contributions of this paper the efficient representativeness. coreset selection method, which is elaborated detail in the method section. Additionally, extensive experimental validation has been conducted, effectively reflecting the contributions of this paper.
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
12

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification:
We provide full derivation of our theoretical results in the appendix.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of all experimental configurations, and believe the results can be reproduced regardless whether the code and data are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
13

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, we will make experimental code available in the supplementary materials. The data used our experiments is open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//giide//Ddeeebbiiiss oow.ssobissiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //ipss.c/ pctpss//ccclllcggudeeeess. guides wwtp/oeeeiiiiiiinnnoooooo Pollccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The experimental configuration has been thoroughly described throughout section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
The statistical metrics we used are all officially defined and well-established.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
14

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The implementation details and training configurations are elaborated in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This work has been conducted in full accordance with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
15

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study does not carry these risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used public data and models under the license and terms, which were also properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
16

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papepswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We did not release any new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our experiments didn't 't include the crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiments did not include the crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
17

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used as part of the core methodology in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
18

---

Supplemental Material: Efficient Representativenes--Aaaa
Coreset Selection
A Proof of Theorem 1
Note: Due to the non-convexity of deep learning models, the complexity of gradient distributions, and the dynamic nature of training, our theoretical analysis here limited to simplified conditions that help understand the behavior of the proposed algorithm under idealized settings. These results are intended offer theoretical insight rather than serve as the main contribution of this work. The practical effectiveness of the method is ultimately demonstrated through empirical results.
Theorem 1 (Relative Gradient Approximation Concentration Bound). Consider a supervised learning task with parameter vector We E Fd at training step t. Let:
D {(x,, k)) denote the full training dataset
S D coreset with cardinality |S| = m
g: = Vi(uw; vi, Vi) E TRd represent per-sample gradients
Mt 1/2 E 9i be the full-data mean gradient (signal)
At = 1. EjES 9j denote the coreset gradient mean
of = s )l the spectral norm of coreset gradient covariance
SNR, 1//,, define the signal-to to-noise ratio
Under the following assumptions:
(Al) Independent Sampling: Coreset indices are sampled i.i.d. from 'D (or equivalently, N >> m
allowing approximation by i.i.d. sampling)
(A2) Sub-Gaussian Gradients: Centered gradients satisfy the Sab--Gussian condition
(u. Nj. - -1,)²
Yu Yu E Rdd ||u|| = 1] exp
<
o
Then for any confidence parameter å E (0, 1), the relative gradient approximation error
ea(S) Titell
|ime jeell
satisfies the probabilistic guarantee:
( 5d
Cologg1/8) SNR,
(e(S) <
log
=
2 1
(12)
SNR,
5
mm
where C > is a constant.
Proof. Let 9A VL(Wzzzj. wy) denote the per-sample gradient, and define the full-data mean gradient as:
N
9i.
N
and the empirical mean of the coreset gradients as:
Au Σ 91
1
19

---

We aim to bound the relative gradient approximation error:
1
6, (S)
I Pell
|
Step 1: Centered gradient representation.
Let Zij 9j Me be the centered gradients. Then:
M4
Zz-
j=1
so that:
1
(S)
Zz
p
Step 2: One-dimensional projection concentration.
By assumption (A2), for all unit vectors gd-1;
(u, 2,)
E
exp
2.
This implies that each projection (u, 2,) is sub-Gaussian random variable with parameter at most oz. Define the empirical projection mean:
Y1 U1
X
Zz
m
j=1
By standard sub-Gaussian concentration, for fixed Sd-1 and any >> 0:
mt²2
P (IX,,) 2 t) < 2exp
207
Step 3: Extend to vector norm via g-net.
We have:
23
sup wEB4-1
[
j=1
Let N1/2 be 1 1/2-net of gd-1 with cardinality at most 5d. Then for any vector Rd
||||| 2 max (u, v).
wEN1/2
Applying the union bound over the finite net:
ml2
P (mN N1/2: |xal
2. exp
20
Since
1 Σ Z3
$ 2 max
Xxul
nEN1/2
we obtain:
m
mt²
Σ
4
2 2t
2 5d exp
m
20
1=1
Step 4: Solve for in terms of confidence level d. Let the hand side be equal to
mt²
2 5d exp
20
20

---

Solving yields:
2 log
2.54
t == 8eg
8
Hence with probability at least
5d
li Mell S 2011
log 8
Step 5: Normalize by | and define SNR. Recall:
((S)= = 11 feell
SNR, = 112+1
||
5:
Thus with probability least 1 8:
2 5d
(x(S - SNR,
log
8
Optional simplification: For fixed d, the logarithmic term can be absorbed by constant into å:
CVlog(1/0)
c(S) S
V
B
Boader Impact
Coreset selection algorithms aim significantly reduce the training data volume and computational overhead while preserving model performance, making them a key strategy for improving training efficiency and reducing energy consumption in deep learning. By identifying small yet highly representative subset from the original training data, such methods can effectively shorten training time and lower hardware resource usage, thereby reducing carbon emissions and mitigating the environmental impact Al development. In light the growing global attention to sustainable AI, advancing efficient and eco-friendly training strategies holds strong practical and long-term value. Moreover, the reduced resource demand improves accessibility for researchers and developers with limited computational capabilities, promoting greater inclusivity and equity in AT research and innovation.
This study proposes a representativeness-eee aware coreset selection method based on gradient signal- to-noise ratio (SNR), which significantly improves final model performance while maintaining the method
computational efficiency of coreset selection. Compared to conventional approaches, possesses the ability to aware the representativeness selected samples during training, effectively addressing the issue of coreset degradation in later stages. As result, it achieves favorable balance between training efficiency and model performance. The method is theoretically grounded, simple to implement, and easy to integrate into existing deep learning pipelines. It is well-suited for both academic research and industrial deployment, particularly in edge scenarios and developer communities with constrained computational resources.
21


Reviews:
[{"review_id": "6od0E2jpag", "reviewer": "Reviewer_5gdu", "summary": "The paper is about improving dynamic coreset selection algorithms. \nCoreset algorithms are divided into static, that create a single coreset for the whole training process, and dynamic algorithms that adjust the coreset as the training progresses. Static methods are commonly used for classical, easy to analyze models, and dynamic algorithms are often used for deep neural networks that do not allow capturing a single subset suitable for all possible variations of their weights.\n\nAn interesting observation was made, dynamic coresets not only degrade in quality between coreset readjustment steps, but also with each subsequent readjustment. \n\nThis resulted in development of a coreset quality measure, that is evaluated independently of main coreset and model optimization to prevent overfitting to the coreset. A surrogate measure that is easier to compute was developed. A coreset meta-algorithm was presented, that reruns dynamic coreset selection at specific points in time, when representativeness diminishes below a threshold.\n\nExperimental results are presented, confirming superiority of the proposed algorithm, compared to all considered dynamic coreset construction methods.", "questions": "- SNR is calculated using coreset data only according to the formulas in Section 4.2. At the same time it is noted, that calculating representativeness metric on full dataset is prohibitively expensive. Would not it be possible to combine two ways, enjoying better generalization of the metric to full data and efficiency -- for example by a simple baseline of calculating Gradient Diff not against full data, but against a random sample of the full data of the same size as coreset? Would be nice to compare proposed method to similar simple probabilistic baselines both theoretically and practically.\n- Overfitting to coreset is discussed in the paper. I want to get more information about severity of the problem: even if we overfit a single coreset, gradient on it tends to zero, while gradient on full set increases. This should lead to selecting another representative coreset during the next coreset recalculation? Do we argue that such cycle (overfitting - fix on new coreset - overfitting again) will endlessly repeat or even diverge when the proposed method is not used? Would be nice to analyze this dynamic behaviour.", "limitations": "Limitations are well described.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Most importantly, the paper advances the state of the art in dynamic coresets as presented in the experimental results. At the same time, the method is thoroughly theoretically justified, as presented in Theorem 1.\n- A need for better awareness of the coreset representativeness is confirmed by a thorough literature review of baseline methods (such as GradMatch) and empirical results (such as DeepCore)\n- Representativeness itself is thoroughly analyzed; a representativeness metric (SNR) is presented and its theoretical properties are discussed.\n\nWeaknesses:\n\n- No statistical significance data (error bars, std. deviations, etc.) is presented in the Main results, Tagle 2. It is specifically concerning, because results are very close to each other. I expect that even with error bars the results will appear significant, due to the fact that in all settings ERACS outperforms other methods, however it is an important piece of information to add (NeurIPS checklist item 7 mentions this as well)\n\n- Minor suggestion related to comparison to baselines:\n- - While the properties of SNR were thoroughly studied and the choice thoroughly justified in Theorem 1, would be nice to extend /  explore modifications to SNR, and also compare to other baseline \"representativeness scores\" (see also the Questions section). \n- - While Ablation study in section 5.3 performs modifications to Algorithm 1, the SNR formula itself remains unchanged. Also comparison in Table 2 is done only against other coreset methods, would be nice to compare (if there exist any) to other \"coreset recalculation triggers\" such as other simpler representativeness metrics.\n\nUpdate after Author - Reviewer discussion:\n- We further confirm that paper advances state of the art, looking at the experimental results presented and the theoretical clarifications made. I am raising my score 5 to 6 to account for this strongly positive fact.", "was_revised": true, "final_justification": "The main positive point is strong advancement of state of the art both experimentally and theoretically:\n- Novel deep learning coreset reselection strategy is presented\n- It outperforms all currently available methods\n\nThis in my opinion justifies the highest score.", "rebuttal": "### W1: Lack of statistical significance data\nThank you for your suggestion. In the early stages of our experiments, we also conducted multiple runs and observed relatively small standard deviations. Based on this, we fixed the random seed in later experiments and reported the results accordingly.\n\nTo further address your concern, we have re-run all experiments in Table 2 five times and computed the mean and standard deviation of test accuracy for each method. The updated results are shown below:\n\n|Dataset|Budget|Random|Craig|Glister|GradMatch|Crest|ERACS|Full|\n|-|-|-|-|-|-|-|-|-|\n|CIFAR-10|5%|84.23±1.27|82.71±2.14|84.57±0.27|85.40±0.28|85.81±0.21|**87.28±0.18**|94.94|\n||10%|88.21±1.06|87.38±1.74|88.52±0.26|89.42±0.24|89.89±0.19|**91.37±0.17**||\n||20%|90.24±0.83|89.06±1.15|89.69±0.18|91.38±0.14|91.70±0.13|**92.75±0.15**||\n|CIFAR-100|5%|38.89±0.58|37.14±2.19|29.66±3.16|40.50±0.85|45.79±0.57|**46.88±0.22**|75.21|\n||10%|63.21±0.36|55.07±1.81|44.34±2.86|64.26±0.61|68.10±0.38|**69.62±0.24**||\n||20%|62.18±0.34|60.62±1.46|52.79±1.48|64.71±0.59|68.91±0.28|**71.03±0.17**||\n|ImageNet|5%|42.35±0.29|44.38±1.23|43.76±1.35|45.15±0.91|46.72±0.87|**48.96±0.63**|70.50|\n||10%|57.41±0.35|55.46±0.96|53.10±1.19|59.26±0.65|61.67±0.52|**63.93±0.46**||\n||20%|62.57±0.26|59.49±0.84|55.94±1.04|63.61±0.54|65.91±0.41|**68.63±0.37**||\n\n*Table: Updated accuracy results with standard deviation (mean ± std over 5 runs) across datasets and budgets.*\n\nAs shown in the updated table, our proposed method consistently demonstrates superior performance and stability across various datasets and budget settings, validating the statistical significance and robustness of the results.\n\n### W2 and Q1: Extension and Comparison of Coreset Representativeness Evaluation Metrics\n \nWe designed two experiments to evaluate and compare the effectiveness of different coreset representativeness metrics:\n\n**(1) Comparison of alternative metrics for triggering coreset reselection**  \nWe evaluated four metrics for coreset reselection on CIFAR-100 using a 10% coreset budget and report the resulting test accuracy, all applied on top of the GradMatch coreset selection algorithm:\n\n|Trigger Metric|Accuracy (mean ± std)|\n|-|-|\n|Geometric Dispersion Score|67.62±0.51|\n|Uncertainty Score|67.94±0.46|\n|GradientDiff (Coreset vs. 10% Full)|68.21±0.42|\n|**ERACS (ours)**|**69.62±0.24**|\n\n*Table: Comparison of coreset reselection triggers based on final test accuracy.*\n\nThe **Geometric Dispersion Score** [r1] computes the average minimum distance from unselected points to the coreset in feature space. A high GDS indicates poor coverage and degraded representativeness. We trigger coreset reselection when GDS exceeds a predefined threshold.\n\nThe **Uncertainty Score** [r2] monitors the overall uncertainty of the coreset based on entropy. A lower average entropy indicates that the model has already absorbed most of the information from the coreset, making it less informative. When the Uncertainty Score drops below a predefined threshold, we trigger coreset reselection to restore representativeness and maintain training effectiveness.\n\nThe **GradientDiff (Coreset vs. 10% Full)** measures the L2-distance between the average gradients of the coreset and a randomly sampled 10% subset of the full dataset. A large gradient difference indicates degraded representativeness and prompts coreset reselection.\n\nWhile each of these methods helps detect representativeness degradation, our proposed **ERACS** method consistently achieves higher accuracy by more effectively monitoring overfitting, thereby providing a more reliable signal of coreset quality.\n\n**(2) Comparison between SNR and a baseline method (GradientDiff (Coreset vs. 10% Full)) when integrated with different coreset selection algorithms**  \n\nWe conducted experiments on the CIFAR-10 dataset with a coreset budget of 10% to evaluate the generalization and adaptability of SNR as a trigger for coreset representativeness degradation across multiple coreset selection methods, including GLISTER, GradMatch, InfoBatch, and DDSA. GradientDiff serves as a simple probabilistic baseline, estimating representativeness by comparing the gradient shift between the coreset and a random subset of the full data.\n\nGLISTER[r3] and GradMatch[r4] are two dynamic coreset selection methods discussed in the main paper, and we additionally compare our approach with the latest methods, InfoBatch and DDSA.\n\n- **InfoBatch (ICLR 2024)** [r5]: Proposes a *lossless training acceleration* approach through unbiased dynamic pruning. It conducts soft pruning based on per-sample loss and rescales gradients to maintain unbiased estimation of full-dataset gradients.\n- **DDSA (ICML 2025)** [r6]: Introduces a dynamic coreset selection framework that combines *importance-based selection* and *selective augmentation*. Samples are chosen every epoch based on a combination of high loss and low confidence, and high-confidence samples are further augmented to boost generalization.\n\n|Coreset Method|Trigger Metric|Accuracy (mean±std)|\n|-|-|-|\n|GLISTER|Original|88.52±0.26|\n|GLISTER|GradientDiff (vs.10% Full)|88.94±0.36|\n|GLISTER|**ERACS(Ours)**|**91.12±0.23**|\n|GradMatch|Original|89.42±0.24|\n|GradMatch|GradientDiff (vs.10% Full)|90.51±0.38|\n|GradMatch|**ERACS(Ours)**|**91.37±0.34**|\n|Infobatch|Original|90.29±0.19|\n|Infobatch|GradientDiff (vs.10% Full)|91.37±0.14|\n|Infobatch|**ERACS(Ours)**|**92.18±0.27**|\n|DDSA|Original|90.52±0.22|\n|DDSA|GradientDiff (vs.10% Full)|91.88±0.13|\n|DDSA|**ERACS(Ours)**|**92.35±0.19**|\n\n*Table: Comparison of coreset trigger metrics based on final test accuracy.(cifar10, 10% dataset, the original method selects the core set once every 20 epochs)*\n\nAs shown, while GradientDiff (vs. 10% Full) provides modest improvements over fixed-interval reselection, its gains are consistently smaller than those achieved with our proposed SNR metric. In all four coreset methods evaluated, the SNR-based trigger (ERACS) more effectively detects representativeness degradation and enables timely reselection, resulting in more stable accuracy improvements. This demonstrates the generality and robustness of our approach.\n\n### Q2: Analyze of Overfitting Dynamic Behaviour\n \nWe share the same viewpoint as you in identifying the dynamic of overfitting to a fixed coreset when representativeness is not monitored. In our experiments, we observe that when coreset selection is performed at fixed intervals (e.g., as in GradMatch), the model tends to overfit the current coreset. Due to the use of relatively small learning rates typical in deep learning training, the gradients on the coreset do not vanish entirely. When we extend the application window of a selected coreset, the coreset loss continues to decrease to a small value, while the test loss first decreases and then increases, signaling overfitting. Upon reselecting the coreset, the coreset loss resets to a higher value and resumes decreasing, while the test loss also starts to decrease again. This pattern suggests that the newly selected coreset restores representativeness and improves generalization performance.\n\n|Epoch Range|Training Loss|Test Loss|Interpretation|\n|-|-|-|-|\n|x|1.76|1.68|Fresh coreset selected|\n|(x,x+Δ)|↓|↓|Training on the **coreset**, both training and test loss decrease|\n|x+Δ|1.38|1.42|Coreset becomes **overfitted**, test loss starts to increase|\n|(x+Δ,x+T)|↓|↑|Training loss continues to decrease, but test loss increases|\n|x+T|1.15|1.49|A **new coreset** is selected, test loss drops again|\n\n*Table: Illustration of overfitting dynamics under fixed coreset selection schedule.*\n\nFor example, after the y-th coreset selection at epoch x, test loss decreases initially but eventually rises, and after the (y+1)-th selection at epoch x+T, it drops again. This evidences a repeating cycle of **overfitting to the coreset – performance degradation – recovery via new coreset selection** when no representativeness-aware mechanism is applied.\n\nHowever, we emphasize that precisely verifying repeated absolute overfitting—i.e., when the gradient converges to zero—is challenging in deep learning applications. This difficulty arises because, in deep neural network training, the learning rate needs to be gradually decreased to ensure training stability. As a result, it is impossible to fully overfit the coreset an infinite number of times. \n\nWhat our proposed method adds is the ability to **detect the onset of overfitting early**—e.g., before test loss starts to rise—by tracking the signal-to-noise ratio (SNR) of gradients. This allows for timely replacement of stale coresets and leads to more stable and efficient convergence.\n\nWe agree that analyzing this dynamic in more depth is valuable, and we will include a visualization and tabular analysis in the revised version to support this explanation.\n\n[r1] Sener, Ozan, and Silvio Savarese. \"Active learning for convolutional neural networks: A core-set approach.\" arXiv preprint arXiv:1708.00489 (2017).\n\n[r2] Coleman, Cody, et al. \"Selection via proxy: Efficient data selection for deep learning.\" arXiv preprint arXiv:1906.11829 (2019).\n\n[r3] Killamsetty, Krishnateja, et al. \"Glister: Generalization based data subset selection for efficient and robust learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 9. 2021.\n\n[r4] Killamsetty, Krishnateja, et al. \"Grad-match: Gradient matching based data subset selection for efficient deep model training.\" International Conference on Machine Learning. PMLR, 2021.\n\n[r5] Qin, Ziheng, et al. \"InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning.\" The Twelfth International Conference on Learning Representations.\n\n[r6] Yang, Suorong, et al. \"When Dynamic Data Selection Meets Data Augmentation.\" arXiv preprint arXiv:2505.03809 (2025)."}, {"review_id": "dhJQbpvYjP", "reviewer": "Reviewer_vscD", "summary": "This paper proposes a dynamic coreset selection method for improving training efficiency. This paper first reveals that coreset representativeness degrades in existing dynamic coreset selection methods. Based on this finding, they theoretically analyze that Signal-to-Noise Ratio (SNR) is a promising approach to track and maintain the coreset quality during training. Extensive experiments demonstrate the effectiveness of the proposed method on multiple benchmark datasets.", "questions": "1. The compared methods are from 2023 and earlier, which limits the convincingness of the results. Could the authors provide more results with the state-of-the-art methods?\n2. I am curious whether coreset sampling should consider storage requirements like data distillation methods. If so, dynamic coreset selection might need to account for storage limitations and ensure fair comparisons with static sampling methods.\n3. For the first contribution—\"we identify the phenomenon of coreset representativeness degradation in dynamic coreset selection methods, particularly in late-stage training\"—the claim would be more convincing with experimental results across diverse model architectures, different optimization methods, and varied training scenarios.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-structured and easy to follow.\n2. Experimental results demonstrate its effectiveness.\n\nWeaknesses:\n1. The observed representativeness degradation may not be surprising, as the existing paper [1] has shown that coreset selection tends to favor high-density, easy samples. It is intuitive that representation degrades during training—in early stages, models learn easy samples with small gradient differences between the original and selected datasets, while in later stages, the model learns harder samples, naturally improving gradient differences.\n2. The compared methods are from 2023 and earlier, which limits the convincingness of the results.\n3. I am curious whether coreset sampling should consider storage requirements like data distillation methods. If so, dynamic coreset selection might need to account for storage limitations and ensure fair comparisons with static sampling methods.\n4. For the first contribution—\"we identify the phenomenon of coreset representativeness degradation in dynamic coreset selection methods, particularly in late-stage training\"—the claim would be more convincing with experimental results across diverse model architectures, different optimization methods, and varied training scenarios.\n\n[1] Zhou, D., Wang, K., Gu, J., Peng, X., Lian, D., Zhang, Y., ... & Feng, J. (2023). Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 17205-17216).", "was_revised": true, "final_justification": "The response has solved my concerns, so I raised my score to 4.", "rebuttal": "### W1: Further Discussion on Representative Degradation\n \nThank you for the insightful comment. While we agree that coreset selection methods may sometimes favor easy or high-density samples, we believe the representativeness degradation observed during training has deeper causes that deserve further discussion.\n\nFirst, taking GradMatch as an example, when an $\\ell_2$-norm regularization term is included (as is common in practice), the selection is no longer biased solely toward easy samples. In fact, a subset of harder examples---often with large gradient---are preferentially included due to the $\\ell_2$ regularizer. \n\nSecond, we posit that the degradation in representativeness is not primarily due to the selection of easy samples, but rather due to the intrinsic limitations of deep neural network training in high dimension nonlinear space. In almost all deep learning settings, the number of samples is typically much smaller than the dimensionality of the data distribution, and the samples always lie in a highly complex nonlinear space. Thus, even a well-chosen coreset cannot maintain long-term representativeness throughout training, since it cannot fully determine the classification surface, which will be reflected in over-fitting. Consider  a 2D example with linearly separable samples. In such a case, we can select a few (e.g., 10-20) points to represent the whole dataset, the representativeness may remain robust over time since the classifier determined by the coreset can be very close to the one trained on the full dataset. However, such stable coverage is infeasible in deep networks with millions of parameters and complex loss landscapes. Therefore, we argue that representativeness degradation is not only about \"sample difficulty\" and more importantly about the intrinsic limitations of deep neural network training in high dimension nonlinear space.\n\nWe hope this perspective helps clarify that representativeness degradation is a complex phenomenon that cannot be attributed solely to the selection of easy samples.\n\n### W2 and Q1: Comparison with the results of papers after 2023\n \nThank you for pointing this out. We appreciate your suggestion to include comparisons with more recent coreset selection methods. In response, we conducted a broader literature survey and identified two state-of-the-art methods published after 2023:\n\n- **InfoBatch (ICLR 2024)** [r1]: Proposes a *lossless training acceleration* approach through unbiased dynamic pruning. It conducts soft pruning based on per-sample loss and rescales gradients to maintain unbiased estimation of full-dataset gradients.\n- **DDSA (ICML 2025)** [r2]: Introduces a dynamic coreset selection framework that combines *importance-based selection* and *selective augmentation*. Samples are chosen every epoch based on a combination of high loss and low confidence, and high-confidence samples are further augmented to boost generalization.\n\nWe implemented both methods and evaluated them on the CIFAR-10 dataset with a fixed coreset budget of 10% of the training data. Specifically, we applied both methods with periodic coreset reselection every 20 epochs and compared them against their respective variants that use our proposed SNR-based trigger instead.\n﻿\nThe results are summarized below:\n\n|Method|Test Accuracy (mean ± std)|\n|-|-|\n|InfoBatch (fixed interval)|90.29±0.19|\n|**SNR + InfoBatch**|**92.18±0.20**|\n|DDSA (fixed interval)|90.52±0.22|\n|**SNR + DDSA**|**92.35±0.18**|\n\n*Table: Comparison with recent coreset methods using SNR-based reselection trigger (CIFAR-10)*\n\nThese results demonstrate that even for **non-gradient-based coreset methods**, our proposed SNR-based coreset representativeness detection framework can be *seamlessly integrated* and consistently improves model performance.\n\nWe will include these results in the revised manuscript if accepted.\n\n### W3 and Q2: Comparison of Distillation Methods under Fair Storage Restrictions\n \nThank you for raising this important and practical concern. In deep learning, **GPU memory (rather than disk storage)** is often the primary constraint during training. Since mini-batch training is standard, dynamic coreset selection does **not significantly increase memory usage per iteration** as the full dataset can be stored in the disk.\n\nMoreover, static coreset selection methods often require a **larger coreset size** to achieve comparable performance, which may result in higher memory usage during every training step. To examine this trade-off, we conducted the following experiments that take both storage and accuracy into account.\n\nWe implemented an additional experiment focusing on the **overall storage usage** throughout training. Specifically, in our ERACS configuration using the best trade-off threshold (as reported in the main paper), the full dataset is only accessed during **26 coreset reselection steps** out of **300 total epochs**. For the remaining **274 epochs**, only a 10% coreset is used for training. This means that the average effective dataset size used over time is approximately **17.8%** of the full dataset.\n\nTo ensure a fair comparison, we evaluated the static coreset selection method **CRAIG** that uses a static 17.8% subset of the dataset, and trains on it for the full 300 epochs. The results, shown below, demonstrate that ERACS still achieves higher accuracy under equivalent storage conditions.\n\n|Method|Avg. Storage Budget (%)|Accuracy (%)|\n|-|-|-|\n|CRAIG (static subset)|17.8|88.13±2.32|\n|ERACS (ours)|17.8|**91.37±0.17**|\n\n*Table: Comparison under equalized average storage budget (17.8% of full dataset).*\n\nThe result suggests that dynamic coreset selection methods like ERACS provide a compelling middle ground: they offer significant storage savings compared to full-data training, while outperforming static coresets in accuracy. While static approaches are more efficient in terms of raw storage, **the dynamic strategy captures training dynamics more effectively**—a promising direction worthy of continued exploration in coreset research.\n\n### W4 and Q3: Results Across Diverse Model Architectures, Different Optimization Methods, and Varied Training Scenarios.\n\nThank you for the suggestion. Since in the rebuttal we are not allowed to give the visualization of intermediate results as Figure 1 in the main text, we present the final results across diverse **optimization methods**, **model architectures**, and **training scenarios**. We will include detailed plots and analysis in the revised version. \n\nWe first evaluated ERACS under different **optimizers** on CIFAR-10 with a coreset budget of 10% using ResNet18. As shown in below, our method consistently outperforms the baseline (GradMatch) across **SGD**, **Adam**, and **RMSProp**, confirming that the observed representativeness degradation and our solution are not tied to a specific optimizer.\n\n|Optimizer|GradMatch Accuracy (%)|ERACS Accuracy (%)|\n|-|-|-|\n|SGD|85.4|**91.3**|\n|Adam|84.9|**91.2**|\n|RMSProp|84.7|**90.5**|\n\n*Table: Accuracy comparison of ERACS and baseline under different optimizers (ResNet18 on CIFAR-10).*\n\nTo demonstrate the **generality** of ERACS beyond image classification, we extended our experiments to both **text** and **regression** tasks.\n\nIn the **vision domain**, we repeated our experiments using **VGG16** on CIFAR-10. In the **text domain**, we applied ERACS to fine-tuning **RoBERTa** on the **SNLI** dataset. As shown in the table below, ERACS consistently improves accuracy across all tested architectures and modalities, and exhibits similar overfitting dynamics during late-stage training—further supporting our hypothesis on representativeness degradation.\n\n| Model Architecture         | GradMatch Accuracy (%) | ERACS Accuracy (%) |\n|----------------------------|------------------------|---------------------|\n| ResNet18 (CIFAR-10)        | 85.40                  | **91.37**           |\n| VGG16 (CIFAR-10)           | 84.17                  | **90.21**           |\n| RoBERTa (SNLI - fine-tune) | 91.92                  | **92.33**           |\n\n*Table: Accuracy comparison of ERACS and GradMatch under different model architectures.*\n\nTo further assess applicability to **regression problems**, we evaluated ERACS on the **California Housing** dataset using three model architectures: MLP, ResNet, and FT-Transformer. We report root mean square error (RMSE), where lower values are better. ERACS achieves consistently lower RMSE than both the baseline and GradMatch, demonstrating its effectiveness in regression settings as well.\n\n#### Regression Task: California Housing (RMSE ↓)\n\n| Method           | MLP     | ResNet  | FT-Transformer |\n|------------------|---------|---------|----------------|\n| Baseline         | 0.518   | 0.537   | 0.486          |\n| GradMatch        | 0.542   | 0.554   | 0.502          |\n| **ERACS (Ours)** | **0.529** | **0.547** | **0.495**        |\n\n*Table: RMSE comparison on regression task. Lower is better.*\n\n[r1] Qin, Ziheng, et al. \"InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning.\" The Twelfth International Conference on Learning Representations.\n\n[r2] Yang, Suorong, et al. \"When Dynamic Data Selection Meets Data Augmentation.\" arXiv preprint arXiv:2505.03809 (2025)."}, {"review_id": "yygZZON93O", "reviewer": "Reviewer_xjQ2", "summary": "The paper considers the dynamic coreset selection problem, that is, selecting a subset of the training dataset that represents the whole training dataset *during training*. As the training epochs increase, such methods tend to lose the performance of the coreset (parameters get apart and the prediction accuracy gets lower than that trained by whole dataset). Authors state that it is because the selected coreset loses \"representativeness\" of the whole dataset, and proposes measuring SNR (Signal-to-Noise Ratio) of the gradients of the losses among samples as the criterion of the representativeness; we re-select the coreset if SNR becomes lower than a predetermined threshold.", "questions": "### Key questions\n\n- As far as I read the paper of GradMatch, it re-selects the coreset in a regular interval, and I felt that it is the only difference of the proposed method from GradMatch. Is it true?\n  - Since the observation of the proposed method in the coreset representativeness is important, I think that the limited differences from GradMatch do not lose the importance of the proposed method, but at least the difference of the proposed method from GradMatch should be clarified.\n  - Related to this issue, In the caption of Figure 1, it is needed to be explained that GradMatch re-selects the coreset in a regular interval, to explain the reasons of the periodic fall of \"Grad diff\" and the periodic rise of \"SNR\".\n- The paper discussed the effect of using SNR by an example of linear regression, but is it possible to build a little bit more general discussion than linear regression?\n- This paper uses SNR as a criterion of representativeness, that is, focuses on the gradient information and found that SNR is useful. However, I felt that other types of information described in Section 2.1 (geometry, prediction uncertainty, etc.) may be available to evaluate the representativeness. How the authors think of this?\n\n### Major comments\n\n- Section 4.2, Theorem 1: As far as comparing with the complete result in Appendix A, it looks that the right side of the inequality in this informal result $\\frac{\\sqrt{2 \\log(1/\\delta)}}{\\sqrt{m}\\cdot\\mathit{SNR}\\_{\\mathit{core},t}}$ should be $\\frac{C \\sqrt{2 \\log(1/\\delta)}}{\\sqrt{m}\\cdot\\mathit{SNR}\\_{\\mathit{core},t}}$ instead ($C$ should be included).\n- Section 4.2, paragraph \"Statistical Illustration: Linear Regression Example\": Although the analysis showed that SNR becomes low as the model parameter (in this case $A$) approaches the optimum (in this case $A^*$), but it does not discuss the effect of the coreset selection. Can we get insight of what coreset is good from this discussion or not?\n- Section 4.2, paragraph \"Time Efficiency in Computing SNR\": It states that SNR is costless to compute than the difference of the gradient, but as far as reading the definition of $\\hat{\\sigma}_t$ in Appendix A, it looks costly to compute since the covariance matrix and the spectral norm are computed. (Authors also presented that it is costly for extreme scales in Section 6.) How large is the theoretical computational cost?\n- Section 4.3, Algorithm 1: Is the coreset size *during the algorithm* is always $k$? More specifically,\n  - Do we select the initial subset ${\\cal X}^{(0)}$ as a random subset of ${\\cal U}$ of size $k$?\n  - With OMP (line 5), do we newly select a coreset ${\\cal X}^{(t)}$ of size $k$ from *whole* dataset ${\\cal U}$?\n\n### Minor comments\n\n- Abstract, lines 16-18: It states that \"While the ideal criterion - gradient discrepancy between the coreset and the full dataset - is computationally prohibitive, we introduce a scalable surrogate based on the signal-to-noise ratio (SNR) of gradients within the coreset\", but as far as reading the paper, the main motivation to use SNR of gradients seems to be not the cost but the control of the overfitting. Are the importances of these two aspects are almost the same? (If so, please consider writing both.)\n- Section 1, line 45: It is confusing to write \"As a result, a new line of research, i.e., dynamic coreset selection [3], has emerged\"; it looks that the reference [3] established the concept of the dynamic coreset selection (but it seems not true). In order to avoid confusions, a possible way is to write \"As a result, a new line of research has emerged, which we call the dynamic coreset selection. These methods periodically update the coreset during training to adapt to the current state of the model [3].\"\n- Section 2: It seems better to group methods that can be used for dynamic coreset selection, so that the proposed method is highlighted.\n- Section 4.2, paragraph \"Theoretical Analysis\": $\\hat{\\sigma}_t$ is explained as \"the standard deviation of sample gradients within the coreset\", but the description is too vague to represent it. Please consider adding a reference to Appendix A that gives its formal definition. Also, it is better to show that $\\hat{\\sigma}_t$ is in reality defined via the covariance matrix, to highlight that it incorporates pairwise information of the samples.\n- Section 6, paragraph \"Limitations\": It states that \"Additionally, SNR may not always reflect representativeness under highly non-Gaussian gradients or non-classification tasks\", but I could not understand the reason of the latter. Why SNR may not work well for non-classification tasks?\n- Section 6, paragraph \"Limitations\": It states that \"Future directions include adaptive thresholding, alternative proxies, and extensions to broader learning settings\", but what are \"adaptive thresholding\" and \"alternative proxies\" in this problem?\n- Appendix A, Step 3: The property of 1/2-net is not so obvious. Are there appropriate references? Also, a brief explanation of 1/2-net is desired.\n\n### Misspellings\n\n- Section 1, line 60: \"serlf-awareness\" --> \"self-awareness\"", "limitations": "The paper states the limitation that the proposed criteria (SNR) is still costly if computed for a a large number of times (although costless than computing full gradient), and that its performance may fall for highly non-Gaussian gradients or non-classification tasks.  \nI did not aware of no other specific limitations, but strongly aware of the cost of computing SNR (above) since it is not well discussed in the paper (see \"Major comments\" above).", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper noticed the fact that, when we iteratively select the coreset, existing methods may select coresets apart from representing the whole dataset.\n- As a criterion to evaluate whether the coreset represents the whole dataset well, the paper focused on the standard deviation (more precisely, the covariance between samples) of the gradient in the training.\n  - I felt it is reasonable since it incorporates not only samplewise but also pairwise information of samples.\n  - Also, it is good that it is a simple criteria.\n\nWeaknesses\n\n- I expect more detailed discussions on \"representativeness\"; why SNR can evaluate the representativeness, and the possibility of other criteria to evaluate the representativeness.\n- Cost evaluations and computation procedures are not well described.", "was_revised": true, "final_justification": "After reviews and authors' responses, I expect that the paper will be good to be published. So I keep my score as 4.", "rebuttal": "### W1: More Discussion on Representativeness, SNR Justification and other criteria.\n\n**Intuitively**, coreset sizes are typically small relative to the full dataset and the model capacity. As training progresses, the model can easily overfit to the small coreset, leading to a situation where gradients from the coreset lose their informativeness. In this regime, the gradient updates become increasingly aligned with noise rather than meaningful signal. This phenomenon is naturally captured by the **signal-to-noise ratio (SNR)** of coreset gradients—when representativeness deteriorates, the SNR drops, signaling that the coreset no longer provides sufficient training signal.\n\n**Theoretically**, we provide analysis showing that SNR provides an upper bound on expected gradient discrepancy. Specifically, low SNR correlates with small informative variation in gradients, indicating that the coreset may no longer reflect the structure of the full data distribution.\n\n**Empirically**, we tested several alternative criteria for triggering coreset reselection, Please refer to **Reviewer 5gdu – W2 and Q1: Answer (1)** for detailed experiment results and analysis.\n\n### W2: Details of Clarity in Cost Evaluation and Computation\n\n**Hardware.** All experiments on CIFAR-10 and CIFAR-100 were conducted using a single NVIDIA RTX 4090 GPU. For large-scale experiments on ImageNet, we utilized a server equipped with 8 NVIDIA RTX 4090 GPUs.\n\n**Model Architectures.** Our primary architecture for image classification tasks was ResNet18.\n\n**Optimization Setup.** We used SGD as the default optimizer with momentum = 0.9, Nesterov enabled, learning rate = 0.025, and weight decay = 5e-4. For learning rate scheduling, we adopted CosineAnnealingLR with $T_{\\text{max}}=300$ epochs.\n\n**Loss Function.** We used standard cross-entropy loss without sigmoid activation.\n\n**Training Procedure.** Each experiment was run for 300 epochs with a batch size of 128.\n\nWe will include these details in the revised version to improve clarity and reproducibility.\n### Q1: Difference of the Proposed Method from GradMatch\n\nWhile we use GradMatch as the primary base method in our experiments, this is due to its **popularity and strong empirical performance**, not because our method is tied to it.\n\nThe core contribution of our work is the **SNR-based representativeness detection framework**, which is **model-agnostic** and can be **integrated with any dynamic coreset selection algorithm** as a triggering mechanism. ERACS is not a modification of any specific coreset selection method, but a general plug-in for adaptively monitoring and improving coreset quality over time. We will revise the manuscript to make the distinction between GradMatch and ERACS clearer.\n\nTo demonstrate this generality, we applied the ERACS trigger to multiple coreset selection methods—including **GradMatch**, **GLISTER**, **InfoBatch**, and **DDSA**. Please refer to **Reviewer 5gdu – W2 and Q1: Answer (2)** for detailed experiment results and analysis.\n\nRegarding the reviewer’s suggestion on **Figure 1**, We will make changes in the revised version.\n\n### Q2: Can the Illustration Extend Beyond Linear Regression?\n\n##### Notation\n\nLet\n\n- $\\mathcal{S} = \\{z_i = (x_i, y_i)\\}_{i=1}^{m}$ be a fixed subset.  \n- $\\mathcal{L}(\\theta; z)$ be the per-sample loss with gradient  \n  $g(\\theta; z) = \\nabla_\\theta \\mathcal{L}(\\theta; z) \\in \\mathbb{R}^{d}$.  \n- The empirical risk  \n  $F(\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} \\mathcal{L}(\\theta; z_i)$  \n  with minimiser $\\theta^\\star = \\arg\\min_\\theta F(\\theta)$ and deviation  \n  $\\delta = \\theta - \\theta^\\star$.  \n- The averaged single-sample gradient and its signal-to-noise ratio:\n\n  $G_m(\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} g(\\theta; z_i)$\n\n  $\\text{SNR}(\\theta) = \\frac{ \\| \\mathbb{E}[g(\\theta; Z)] \\|_2 }{ \\sqrt{ \\frac{1}{m} \\cdot \\text{Tr} \\left( \\text{Cov}[g(\\theta; Z)] \\right) } }$\n\n\n##### Assumptions\n\n1. **$L$-smoothness (or $B$-Lipschitz gradient)** in a neighbourhood of $\\theta^\\star$:  \n   $\\left\\| \\nabla F(\\theta) - \\nabla F(\\theta^\\star) \\right\\|_2 \\le L \\left\\| \\theta - \\theta^\\star \\right\\|_2$  \n   For non-smooth losses (e.g. ReLU, hinge) replace $\\nabla$ with a sub-gradient and $L$ with a Lipschitz constant $B$.\n\n2. **Non-vanishing gradient variance**: there exists $\\sigma_g^2 > 0$ such that  \n   $\\mathrm{Tr} \\left( \\mathrm{Cov}_{\\mathcal{S}}[g(\\theta; Z)] \\right) \\ge \\sigma_g^2$  \n   for every $\\theta$ in the same neighbourhood (guaranteed by input/label noise, model misspecification, etc.).\n\n##### Proposition\n\nUnder assumptions (1)–(2),  \n$\\mathrm{SNR}(\\theta) \\le \\sqrt{m} \\cdot \\dfrac{L}{\\sigma_g} \\cdot \\left\\| \\delta \\right\\|_2$,  \nso as $\\left\\| \\delta \\right\\|_2 \\to 0$, we have $\\mathrm{SNR}(\\theta) \\to 0$.\n\n**Proof sketch.**\n\n- Because $\\nabla F(\\theta^\\star) = 0$, smoothness gives  \n  $\\left\\| \\mathbb{E}[g(\\theta; Z)] \\right\\|_2 = \\left\\| \\nabla F(\\theta) \\right\\|_2 \\le L \\cdot \\left\\| \\delta \\right\\|_2$\n\nFor any differentiable or sub‑differentiable model with standard loss functions, the above derivation remains applicable and theoretically sound.\n\n### Q3: Are There Other Effective Representativeness Signals?\n\nWe conducted a controlled experiment to compare our SNR-based method (**ERACS**) with three alternative approaches based on different information modalities discussed in Section 2.1, including geometry and uncertainty.\nwe tested several alternative criteria for triggering coreset reselection, Please refer to **Reviewer 5gdu – W2 and Q1: Answer (1)** for detailed experiment results and analysis.\n\n### Major1；Minor2、3、4；Misspellings1: \nThank you for your suggestion. We will correct them accordingly in the revised version.\n\n### Major2: Linear Regression Example Does Not Inform Coreset Quality\n\nOur goal in this section is to illustrate how SNR evolves as optimization progresses — specifically, how representativeness degrades as overfitting to a fixed coreset occurs. The example is not intended to identify which coreset is “good,” but rather to support the motivation for **monitoring** representativeness over time. The actual selection of a good coreset remains the responsibility of the base selection algorithm (e.g., GradMatch), while **our framework focuses on determining whether a previously selected coreset remains valid**.\n\n### Major3: Clarify Practical Computation Cost of SNR\n \nThank you for the thoughtful question. We would like to clarify that the definition of $\\hat{\\sigma}_t$ in Appendix A—based on the spectral norm of the full empirical covariance matrix—is used **solely for theoretical analysis**, where a rigorous formulation is needed to establish our guarantees.\n\n**In practice**, we adopt a **diagonal approximation** of the covariance to avoid the prohibitive $O(d^2)$ memory and $O(d^3)$ computation cost. This approximation reduces the complexity to $O(d)$, making the SNR trigger highly efficient in real training scenarios. Such approximation is widely adopted in deep learning, such as the well-known techinique Batch Normalization, see the first paragraph of Section 3 in [r1] for details. \n\n### Major4: Clarify Coreset Size and Selection Process in Algorithm 1\n \nYes, you are correct.\n\n### Minor1: Dual Motivation of SNR—Cost vs. Overfitting Control\n \nThank you for the observation. Our primary motivation for using SNR is to monitor and prevent overfitting to a stale coreset. However, in practice, computing full-data gradient discrepancy is too expensive. Thus, proposing a lightweight alternative like SNR is also an important part of our contribution. We will revise the abstract to reflect both aspects.\n\n### Minor5: Why Might SNR Fail in Non-Classification Tasks?\n\nWe mentioned that SNR may be less effective in non-classification tasks primarily because we thin such tasks (e.g., regression) may require more precise gradient information—both in direction and magnitude. Since SNR only captures the ratio of mean to variance, it may not fully reflect the nuanced gradient behavior needed to evaluate representativeness in these settings.\n\nBut preliminary results on regression tasks (please refer to **Reviewer pdTr – Q2**) suggest that SNR can still effectively guide coreset reselection and lead to noticeable performance gains. We plan to include these findings in future versions of the paper.\n\n### Minor6: Clarify Future Directions—Adaptive Thresholding and Proxies\n \nWe clarify the intended meanings as follows:\n\n- **Adaptive thresholding** refers to allowing the model to automatically learn appropriate threshold values, instead of relying on manually specified ones.\n- **Alternative proxies** refers to other potential metrics that could be explored for evaluating the representativeness of the coreset.\n\n### Minor7: Explain or Reference 1/2-net in Appendix A\n \nThis result is based on Corollary 4.2.11 from Vershynin's *High-Dimensional Probability*, Chapter 4, Section 4.2, \"Nets, Covering and Packing\".\n\n**Corollary 4.2.11** (Covering numbers of the Euclidean ball)  \n*For any* $\\varepsilon > 0$, *the covering number of the unit Euclidean ball* $B_2^n$ *satisfies:*\n\n$(1/\\varepsilon)^n \\le \\mathcal{N}(B_2^n, \\varepsilon) \\le (2/\\varepsilon + 1)^n$\n\n*The same upper bound also holds for the unit sphere* $S^{n-1}$.\n\nIn particular, setting $\\varepsilon = 1/2$ yields a $1/2$-net on the unit sphere with cardinality at most $5^n$, meaning that every unit vector lies within Euclidean distance at most 0.5 of some point in the net.\n\n[r1] Ioffe, Sergey, and Christian Szegedy. \"Batch normalization: Accelerating deep network training by reducing internal covariate shift.\" ICML, 2015."}, {"review_id": "4v1zudxBPU", "reviewer": "Reviewer_pdTr", "summary": "The paper addresses the practical challenge of coreset representativeness degradation in dynamic coreset selection methods during deep learning training. The authors propose ERACS, a framework that uses a lightweight Signal-to-Noise Ratio metric to monitor coreset quality and trigger re-selection adaptively, rather than on a fixed schedule. Empirical results across CIFAR-10, CIFAR-100, and ImageNet show that ERACS outperforms state-of-the-art coreset methods such as GradMatch, Glister, and CREST in both accuracy and efficiency, with notable improvements in convergence speed and gradient approximation accuracy.", "questions": "How does ERACS behave when gradients are inherently noisy (e.g., due to high label noise or early-stage training instability)?\n\nCan ERACS be extended to regression, language models, etc?\n\nThe contribution about the discovery of coreset representativeness degradation in dynamic coreset selection methods should be rephrased to acknowledge prior work and emphasize quantitative replication and efficient handling, not original discovery.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) Tackles an important problem that affects coreset effectiveness in real-world training pipelines.\n2) Introduces a simple and efficient SNR-based metric to adaptively determine when a coreset has become stale, avoiding expensive full gradient or curvature computations.\n3) Provides a formal bound on the relationship between gradient discrepancy and SNR under sub-Gaussian assumptions, helping justify the proposed approximation.\n4) Extensive empirical evaluation\n\nWeaknesses: \n1) The paper claims as its first contribution that it “identifies the phenomenon of coreset representativeness degradation in dynamic coreset selection methods.” This is factually incorrect. The degradation issue in dynamic coreset selection was already identified and thoroughly discussed in CREST (arXiv:2306.01244). CREST explicitly shows that coreset gradient fidelity deteriorates even after periodic reselection, and proposes a curvature-aware solution. ERACS builds on this observation but does not originate it.\n2) Incremental novelty: the core idea — using SNR of coreset gradients as a reselection trigger — is conceptually incremental. The method improves over GradMatch primarily by monitoring coreset quality adaptively rather than modifying the selection algorithm itself.", "was_revised": true, "final_justification": "The clarification and the changes that the authors will add to the final version convinced me to change my ranking to 5", "rebuttal": "### W1: Prior Work Already Identified Representativeness Degradation\n \nThank you for this important clarification. We acknowledge that the CREST paper (arXiv:2306.01244) has already identified the issue of coreset representativeness degradation, and explicitly demonstrated that coreset gradient fidelity can deteriorate even under periodic reselection.\n\nWhile our paper initially framed this as a novel observation, we agree that CREST precedes our work in formally highlighting this problem. We will revise the statement of our first contribution in the final version to accurately reflect this prior work.\n\nThat said, our approach differs in both motivation and methodology. CREST uses **second-order curvature information** (i.e., Hessian-preconditioned gradients) as a supervision signal to guide coreset reselection. In contrast, we propose a **lightweight signal-to-noise ratio (SNR)**-based criterion as a proxy for monitoring coreset representativeness.\n\nMore importantly, our method is not merely an alternative trigger: we reinterpret representativeness degradation from the perspective of **overfitting to a stale coreset**, and show empirically and theoretically that SNR strongly correlates with this overfitting behavior. This leads us to design a simple, scalable, and efficient metric for dynamic coreset monitoring that avoids the computational overhead of second-order methods.\n\nThus, while we share similar concerns with CREST, our contribution lies in introducing the SNR-based perspective, providing a theoretical grounding, and offering an efficient implementation that performs well across tasks.\n\nWe will explicitly acknowledge the connection to CREST in the revised manuscript and clarify that our main contribution lies in the proposal and validation of the **SNR-based representativeness criterion**.\n\n### W2: Incremental Nature of Using SNR as a Trigger\n\nThank you for the feedback. While we use GradMatch as the primary base method in our experiments, this is due to its **popularity and strong empirical performance**, not because our method is tied to it.\n\nOur core contribution is the **SNR-based representativeness detection framework**, which is **model-agnostic** and can be **integrated with any dynamic coreset selection algorithm** as a triggering mechanism. ERACS is not a modification of any specific selection algorithm but a general plug-in for monitoring and improving coreset quality over time.\n\nTo demonstrate this generality, we applied our ERACS triggering mechanism to multiple coreset methods—including **GradMatch**, **GLISTER**, **InfoBatch**, and **DDSA**—without changing their selection logic. As shown in Table below, ERACS consistently improves accuracy across all baselines. These results clearly indicate that our method is not a narrow enhancement to GradMatch, but a **flexible and broadly applicable scheduling strategy**.\n\n|Coreset Method|Original Accuracy (%)|ERACS Trigger (%)|\n|-|-|-|\n|GLISTER|88.52±0.26|**91.12±0.73**|\n|GradMatch|89.42±0.24|**91.37±0.17**|\n|InfoBatch|91.37±0.13|**92.18±0.26**|\n|DDSA|91.88±0.14|**92.35±0.19**|\n\n*Table: Accuracy improvements by adding ERACS triggering to different coreset selection methods.*\n\nAs shown above, integrating our SNR-based ERACS trigger into various existing coreset methods leads to consistent accuracy improvements—despite not altering their selection mechanisms. This highlights the generality and robustness of our framework in improving training effectiveness by adaptively detecting representativeness degradation.\n\nWe will clarify this point further in the revised manuscript and include these cross-method results to reinforce that ERACS offers a general, efficient, and scalable solution for adaptive coreset scheduling.\n\n### Q1: Behavior of ERACS Under Noisy Gradients\n\nThank you for raising this important point. To evaluate the robustness of ERACS under noisy training conditions, we conducted additional experiments on CIFAR-10 by injecting **synthetic  label noise** into the training set.\n\nSpecifically, we adopted a label flipping strategy to inject noise: for 10% of the training samples, we randomly replaced their ground-truth labels with incorrect ones, drawn uniformly from the remaining 9 classes. This approach simulates real-world scenarios where label corruption occurs in a class-agnostic fashion and induces instability in gradient directions during training.\n\n|Method|Test Accuracy (%)|\n|-|-|\n|Random|72.3|\n|GradMatch|75.6|\n|GLISTER|74.8|\n|CRAIG|71.1|\n|**ERACS (Ours)**|**80.9**|\n\n*Table: Test accuracy (%) under 10% label noise on CIFAR-10. ERACS achieves the best robustness among all baselines.*\n\nWe then compared ERACS with existing coreset methods under this noisy setting. The results show that **ERACS remains stable and consistently outperforms other baselines**, even when gradients are corrupted by label noise. This suggests that our SNR-based criterion is not overly sensitive to noisy setting.\n\n### Q2: Applicability Beyond Classification\n\nThank you for this forward-looking question. Although our current manuscript focuses primarily on image classification, the **ERACS framework is inherently general** and can be extended to other learning paradigms such as **regression** and **language modeling**.\n\nTo validate the generality of ERACS, we conducted experiments on:\n\n- **Regression tasks** using the **California Housing** dataset.\n- **Language modeling** tasks using **RoBERTa fine-tuned on SNLI**.\n\nIn both cases, we compared ERACS against the baseline training as well as GradMatch-based coreset selection. Below, we summarize the results:\n\n#### Regression Task: California Housing (RMSE ↓)\n\n| Method           | MLP   | ResNet | FT-Transformer |\n|------------------|-------|--------|----------------|\n| Baseline         | 0.518 | 0.537  | 0.486          |\n| GradMatch        | 0.542 | 0.554  | 0.502          |\n| **ERACS (Ours)** | **0.529** | **0.547**  | **0.495**          |\n\nERACS consistently reduces RMSE compared to both the baseline and GradMatch, demonstrating its effectiveness on regression tasks.\n\n#### Language Task: SNLI with RoBERTa (Accuracy ↑)\n\n| Method           | Accuracy (%) |\n|------------------|--------------|\n| Baseline         | 91.92        |\n| GradMatch        | 92.08        |\n| **ERACS (Ours)** | **92.33**    |\n\nERACS improves classification accuracy on natural language tasks, showing its utility beyond vision.\n\nThese results suggest that our **SNR-based representativeness criterion** is broadly effective across model types (e.g., ResNet, FT-Transformer, RoBERTa) and task types (classification, regression, language).\n\nWe will explicitly incorporate these results and discussions into the revised manuscript, and highlight them in the **Limitations and Broader Impact** sections.\n\n### Q3 Clarification Request: Rephrase Discovery Claim as Quantitative Replication\n\nThank you for the valuable comment. We agree that the phenomenon of coreset representativeness degradation in dynamic coreset selection has been previously mentioned in related works. We will cite and properly acknowledge these prior contributions, such as CREST (arXiv:2306.01244), in the revised manuscript.\n\nOur contribution does not lie in claiming the original discovery of this phenomenon, but rather in offering a new **interpretation from the perspective of overfitting to a stale coreset**. Building on this interpretation, we propose a **lightweight, scalable, and model-agnostic** SNR-based criterion to monitor coreset representativeness and adaptively trigger reselection. Our method is compatible with a wide range of dynamic coreset selection algorithms and offers an efficient solution to this long-standing issue.\n\nIn the revised manuscript, we will rephrase the contribution accordingly to reflect these clarifications: emphasizing our novel interpretation and practical solution, rather than claiming the phenomenon itself as new."}]

Author rebuttal:

Meta-review: The paper provides a novel coreset selection mechanism, where it specifically focus coreset representativeness degradation issue. They connected this issue with SNR and provided a statistical illustration. The paper provides a reasonable set of experiments to make the case. Reviewers are generally positive about the paper. Reviewers had raised several concerns re. experiments, e.g., older baselines, generalization to other domains (like text). Authors successfully addressed their concerns.  I recommend acceptance.

One existing concern about the paper, which I would like to highlight is as follows. I think the paper should compare their method against the baselines based on end-to-end running time rather than coreset size. This is because, if coreset selection is costly that must be penalized accordingly.
Decision: Accept (poster)