Coresets from Trajectories: Selecting Data via Correlation of Loss Differences
Authors: Manish Nagaraj
Deepak Ravikumar
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: Deep learning models achieve state-of-the-art performance across domains but face scalability challenges in real-time or resource-constrained scenarios. To address this, we propose Correlation of Loss Differences ($\mathtt{CLD}$), a simple and scalable metric for coreset selection that identifies the most impactful training samples by measuring their alignment with the loss trajectories of a held-out validation set.  $\mathtt{CLD}$ is highly efficient, requiring only per-sample loss values computed at training checkpoints, and avoiding the costly gradient and curvature computations used in many existing subset selection methods. We develop a general theoretical framework that establishes convergence guarantees for $\mathtt{CLD}$-based coresets, demonstrating that the convergence error is upper-bounded by the alignment of the selected samples and the representativeness of the validation set. On CIFAR-100 and ImageNet-1k, $\mathtt{CLD}$-based coresets typically outperform or closely match state-of-the-art methods across subset sizes, and remain within 1\% of more computationally expensive baselines even when not leading. $\mathtt{CLD}$ transfers effectively across architectures (ResNet, VGG, DenseNet), enabling proxy-to-target selection with $<1\%$ degradation. Moreover, $\mathtt{CLD}$ is stable when using only early checkpoints, incurring negligible accuracy loss. Finally, $\mathtt{CLD}$ exhibits inherent bias reduction via per-class validation alignment, obviating the need for additional stratified sampling. Together, these properties make $\mathtt{CLD}$ a principled, efficient, stable, and transferable tool for scalable dataset optimization.

Published in Transactions on Machine Learning Research (11/2025)
Coresets from Trajectories: Selecting Data via Correlation of Loss Differences
Manish Nagaraj Electrical and Computer Engineering Purdue Universily
mmagara@purdue.comm
Deepak Ravikumar
trwnikumrupuuudue...cm edu
Electrical and Computer Engineering
Purdue University
Kaushik Roy
kaustik-puurdue edu
Electrical and Computer Engineering Purdue University
Reviewed on OpenReview: /ttp::://pperevviii net/ forum? QYQpbZTWJ9
Abstract
Deep learning models achieve srate-of-the-ars performance across domains but face scala- bility challenges in real-time or resource-constrainee scenarios. To address this, we propose Correlation of Loss Differences (CLD), a simple and scalable metric for coreset selection that identifies the most impactful training samples by measuring their alignment with the loss trajectories of a held-out validation set. CLD is highly efficient, requiring only per- sample loss values computed at training chockpoints, and avoiding the costly gradient and curvature compulations used in many existing subset selection methods. We develop gen- eral theoretical framework that establishes convergence guarantees for CLD-based coresets, demonstrating that the convergence error is upper-bounded by the alignment of the selocted samples and the representativeness of the validation set. On CIFAR- 100 and ImageNet- -kk CLD-based coresets typically outperform or closely match state-of-the-art methods across subset sizes, and remain within 1% of more computationally expensive baselines even when not leading. CLD transfers effectively across architectures (ResNet, VGG, DenseNet), en- abling proxy-to-taree target selection with 1% degradation. Moreover, CLD is stable when using only carly checkpoints, incurring negligible accuracy loss. Finally, CLD exhibits inherent bias reduction via per-class validation alignment, obviating the need for additional stratified sam- pling. Together, these properties make CLD a principled, efficient, stable, and transferable tool for scalable dataset optimization.
1 Introduction
Deep learning models rely on large and diverse datasets to achieve state-of-the-art performance across a wide range of tasks. However, training on such datasets increasingly constrained by compute and memory budgets, especially in real-time
mesource-limited settings. This raises fundamental question: Which subsets of data most effectively support generalization? A natural answer is offered by coresels, compact, reprosentative subsets of training data that retain full-dataset performance when used for training.
Coresets support a range of applications inchuding active learning Coleman et al. 2020 neural architecture search (Na et al.) 2021 Shim et al.] 2021), dataset distillation (Czzeaavetee et al. 2022), and continual al. 2020 However, most existing approaches are cither basod
lcarning Aljjnnii et al. 2019 Borsos
The code available on GitHub.

---

Published in Transactions on Machine Learning Research (11/2025)
heuristic criteria unrelated to generalization 2017). or expensive second-order or bilevel aptimization (Kllaasetty et al. 2021a Pruthi et al. 2020
Toneva et al. 2018
Belouadah et al. 2020;
Rebuffi et al.
Garg
& Roy 2023 Killamsetty et al. 2021b Xia et al. 2024b Borsos et al. 2020 limiting scalability.
We propose a simple, scalable, and theoretically grounded alternative to coreset generation using a metric we define as the Correlation of Loss Differences (CLD). This metric quantifies how closely the loss trajectory of training sample aligns with the average validation loss trajectory during training (Figure | left). Since the validation set reflects the test distribution, high positive CLD samples are likely to contribute positively to generalization. Selecting such samples yields compact coresets that preserve, or even improve, test accuracy by filtering out ambiguous or harmful examples (Figure 1 right).
Beyond simplicity, CLD provides strong theoretical guarantees. We prove that training on high-CLD samples achieves convergence in population risk with an error bound that closely matches full-data training, where the excess error is explicitly governed by the sample alignment parameter K and the validation representativeness 8 (see Theorem D. Our theory reveals that high CLD is not just sulficient but also necessary to minimize the convergence error- bound under coreset training.
We validate these findings across CIFAR-100 and ImageNet-Ik,, where CLD-sclected coresets typically out- perform or closely match state-of-the-an methods across a wide range of coreset sizes, and remain within 1% of more computationally expensive baselines even when not leading. Unlike these methods, CLD requires only per-sample loss values, allowing it to avoid the costly gradients, Jacobians, or feature embeddings used by many influence- and similarity-based selectors. Concretely, the only computational overhead beyond a standard training run on the full dataset of size N is single forward pass at each checkpoint, over a small, held-out validation set of size Q. Since the validation set is typically much smaller than the training set (Q & N), this weight approach yields significant gains in both computational and storage efficiency, which we quantify in our full cost analysis summarized in Table !
An additional strength of CLD is its robustness; the metric remains stable when computed using sparsely sampled training checkpoints and is consistent across random seeds, making il practical for large-scale or budgeted deployments. Furthermore, CLD coresets transfer effectively across architectures. This is one of the key advantages of CLD. Corescts computed using small proxy models (c.g., ResNet-18) generalize to larger models (c.g., ResNet-50, DenseNet) with performance drops consistently under 1%. Hence we compute coresets using smaller proxy model (e.g., ResNet-18) and reuse it to train larger target models (e.g., Reaeeee-b//VGGG/Deeeeeeee with minimal loss in accuracy, while substantially reducing selection cost.
Summary of contributions:
1. Correlation of Loss Differences for Coreset Selection. We introduce CLD, a simple and scalable metric for corcset construction based on the correlation between training sample's 's loss differonces and the average validation loss trajectory, serving as a proxy for generalization, in Section 4
2. Thcoretical Guarantees. We develop a general convergence framework showing that training on high-CLD samples yields population risk close to full-data training, with the suboptimality explicitly governed by sample alignment and validation representativeness, in Section 5
3. Experimental Validation. We show that on CIFAR-100 and ImageNet- 1k, CLD-sclected coresets typically outperform or closely match stale-of-the-ar methods across wide range of subset. sizes, and remain within 1% of more expensive baselines when not leading, in Section 6
4. Efficiency, Transferability, and Stability. CLD avoids gradient and curvature computations, incurs minimal compute and storage cost, transfers across architectures via proxy models, and remains stable under checkpoint subsampling and random seeds, making it highly practical for large-scale settings. We discuss these in Section 7 and Section
2 Related Literature
The need for scalable coreset methods has led to a variety of approaches, which can be broadly grouped into score-based, opfimizalionn-uused, and training properly- based methods.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Positive CLD
60
Lote at Ejnch
53.81
50
Testset accuracy from
10% correstsss (%)
40
35,30
Training Epochs
30
25.13
ImageNet- Ik Class 292: Tiger
Negative CLD
Toie Suuple Low
Avera Vodadaiio
1.3
Positive CLD
Negative CLD
Zero
CLD
Training Epochs
Figure 1: Correlation of Loss Differences (CLD) at a glance. Left: ImageNet-1k "Tiger" samples illustrating varying CLD scores. High-CLD samples (top row) closely track the validation loss trajectory, indi- cating informative and representative data. Low/negat /ngaative-CLD samples (bottom row) significantly deviate, typically corresponding
atypical, ambiguous, or mislabeled examples. Right: Performance comparison of coresets formed by selecting equal-sized subsets of the highest 10% positive, lowest 10% negative, and 10% zero-valued CLD samples of ImageNet-Iie 1k on ResNet-18. Corescts with high-positive CLD samples achieve superior accuracy over various seeds.
Score-based methods select training samples according to predefined metrics, often in the feature space based on model prediction confidence. Some methods score a sample by its distance to class center (Rebuffi
et al.
2017 Castro et. al. 2018 Belouadah et al. 2020 to the class feature median (e.g.. Moderate (Xia
ct al.
2022 ), or to other samples (Sener & Savarese 2018 Cal (Margatina et al. 2021 identifies contrastive
examples via the KL divergence between predictive distributions, while Herding (Chen et al.] 2010 selects
representative samples using kernel-based approach in Hilbert space. Other methods rely on prediction probabilities from (possibly proxy) model. For example, Forgetting (Toneva et al.,
2018) counts how often
sample is misclassified after previously being correct, GraNd and EL2N (Paul al. 2021 rank examples by their early-training loss gradient norm or l2 error norm, respectively, and AUM (Pleiss et al. 2020 scores the area under the maryin across training to flag potential label issues. While these approaches are simple and avoid expensive gradient or Hcssian computation, they often inherit biases from their heuristic scoring rules and offer no convergence guarantees. To mitigate bias, methods such as CCS (Zheng el al., 2022 employ stratified sampling for diversity, while D2-Pruning (Maharana et al. 2024) combines difficulty (prediction variance) and diversity (feature density) in a graph- based framework. Nonetheless, such strategies often make restrictive assumptions to avoid noisy samples (c.g., CCS discards up to 30% of data) and still lack theoretical generalization guarantees.
Optimization-baase methods formulate coreset selection as an explicit optimization problem, often with provable convergence guarantecs. CRAIG (Mirzasoleiman et al.) 2020) and GradMatch Killamsetty et al. 2021a) select samples that align with the full-data gradient direction, while Glister (Klllansetty et al, 2021b) maximizes held-out validation liceliicl Bilevel optimization (Borsos et al. (0200 has been used to leverage influence functions (Koh & Liang) (001)) for selecting samples with maximal generalization benefit, and GraphCut (Iyer et al. 2021) uses submodular information measures the objective. Recently, BoundarySet-CCS (Yang et al. 2024) minimized decision boundary reconstruction error while ensuring class diversity. While theoretically grounded, these approaches often require repeated optimization loops, making them computationally expensive for large datasets.
Training property-based methods exploit the dynamics of training to assess sample importance. SloCurv (Garg & Roy 2023) uses second-order loss curvature statistics to identify samples with better generalization potential, and TracIn (Pruthi et al. 2020 tracks gradient alignment with a validation set..
3

---

Published in Transactions on Machine Learning Research (11/2025)
TDDS (Zhang et al. 2024) extends this idea by projecting each sample's gradient onto the accumulated gradient to quantify its true contribution, and by monitoring this projection across multiple iterations to account for fluctuations in importance over time. Although effective, such methods depend on costly first- or sccond-order statistics, limiting scalability.
Sealable training property-boase baseed methods reduce this overhead by measuring per-sample dynamics using only forward-pass signals. Dyn-Unc (He et al, 2024) summarizes the variability of the true-class probability over sliding windows and prunes by thresholding aggregated uncertainty after training. DUAL (Cho et al., 2025) combines this uncertainty score with difficulty term and uses pruming ratioo adaptive Beta sampling schedule to reweight selection at high pruning ratios. In contrast, our method CLD ranks examples by how well their loss-difference trajeclories align with the class-wise validation loss trajectory, providing an explicit generalization-alignaenn
criterion. All three approaches admit identical minimal logging, one scalar per example per checkpoint (probability for cyn-Unc/DUALL loss for CLD), so compute and storage are directly comparable. Unlike uncertainty-only methods, however, CLD's alignment objective supports a convergence guarantee (Section 5), and it avoids ralio-specific sampling schedules and their hyperparameters; robustness at extreme cases (e.g,, coresets of size 5%) under identical logging (Section (6). In short, CLD combines the practicality of score-based metrics (low compute/ compute/storage via scalar logging) with validation- aligned training-dymamics signal that admits convergence guarantee.
report
3 Preliminaries and Problem Setup
We consider the supervised learning problem of learning a mapping from the input space to the output space, X + Y, where X C IRd and C R. The training dataset. S consists of N samples drawn from an unknown distribution D X x Y, with each sample denoted as Σ = (T,, 9:)- Thus, S = ( Additionally, we assume access to a query (held-out validation) set ~ D2 . {Q) containing Q samples, which represents the distribution D.
A learning algorithm A (c.g., SGD) is used to train model with parameters A Rp the training set S over T' iterations. We denote the model parameters at iteration / of training on S as Og,, where 0g corresponds to the random initialization prior to the first update. The performance of the model at step on sample zm is evaluated using a loss function E(fg, Em.) R x Rd + R, which quantifies the prediction error on ž. at that point in training.
The goal of training is to minimize the population risk RD(0),
arg min (D) arg min |8(0, 2)].
(1)
0
8
E~D
However, since D unknown, instead minimize the empirical risk R(,0,))
arg min R)( arg min
Σ
e(0, Zm)
(2)
N
0
The gradient of the loss with respect to the parameters 0 at step & for sample ž is denoted as Vol(Os, zt). The average gradient over the validation set (Gv) is,
Q
Gv(('))
Σ
VoE 93)
(3)
Q 1
4 Correlation of Loss Differences (CLD)
We now define the core quantity used in our method, the correlation of per-sample loss trajectories with the validation set.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Loss Trajectories For every sample ž we record the per-iteration change in loss during the model training run, and collect these T increments in loss-difference trajectory
((z) (c(0g, ž) e(0g, 2), ..... E(0g, e(05--1,)) ,2) IR..
(4)
The validation-average trajectory is defined similarly as,
1 1
Te (05, k)) - E E k,)
[e (0g - - E ER.
(5)
Definition 1 (Corrclation of Loss Differences (CLD)). The CLD score of training sample Zm E S is the correlation between the sample's loss trajectory A. and the average loss trajectory of the validation set V:
CLD(Zm) p ( (mm:B'v)
(6)
where p is the correlation metric. In experiments, employ Pearson correlation Pearson 1895 due to its scale invariance and computational simplicity. Intuitively, the CLD metric quantifies how well a training sample's loss dynamics align with the aggregate loss trajectory of the validation set, which serves as a proxy for generalization behavior. Samples with higher CLD values are deemed more influential and can thus be prioritized for coreset construction. We investigate this hypothesis both theoretically and empirically in the following scctions.
While Definition | defines CLD using global validation trajectory, our practical implementation class- specific averages to enstre semantic alignment; see Section 4.1
4.1 Coreset Selection Procedure
We first train source model 0s on the full dataset S, recording per-epoch losses for all training and validation samples. This logging piggybacks on the standard training loop; no additional passes over the training set are required. Consequently, all training samples are scored al the chosen checkpoints. We then compute Am and the class-specific average validation trajectory Av, for each class c.
Biy.c
1
[800] (3) ((02.
Σ TEV.
[((0], (1) 2(05- . ERT.
(7)
VV
[Vol
L.V.
where ====0 : c].
In accordance with standard practice for coresct selection, we score samples within each class independently. For training sample Em E S with label ym c, its CLD score is the Pearson correlation between its trajectory and the corresponding class-specific validation trajectory:
CLD pl (Zm)) (....) V Fm
(9)
After computing all scores, we select the top-ke training samples in each class to form a class-balanced coreset
C C.. C. Top ke (f SS: ym = c), CLD),
(10)
with total size fixed in advance as k= Ee ke- This per-class selection strategy ensures both label balance and stability of dynamics within semantic categories, improving the robustness and interpretability of the resulting coreset.
A key advantage is architectural flexibility, CLD scores can be computed using proxy model and transferred to larger deeper architectures. The full coreset selection procedure is summarized in Appendix A
5

---

Published in Transactions on Machine Learning Research (11/2025)
5 Theoretical Analysis of CLD-Coresets
We now provide theoretical justification for selecting high-CLD samples, showing that such coresets yield convergence guarantees close to full-data training under the following assumptions.
Assumption 1 (L-smoothness). For every fized sample z, defone s(0) = e(0, 2). Then I is L-smooth in 0, i.e.
I. ly x112 Vx, y E Rº.
f(y) S f(x) (Vf(x), y- x) +
(11)
Consequently, both the population risk RD(.) and the empirical risk R(.)) are also L-smooth.
Assumption (Bounded Gradient Norm). There erists > 0 such that, for all and every training sample 7m ES E and validation sample Øj V, IVol(O, 2m) S B, and [Vol(0, (1)ll S B. Consequently, for any inder set C {I,...,N}
Vol(O, Zm)
< BB
[[v(()) SB,
(12)
MEC
2
Q (e)(0, is the validation-averag gradient. Q j=1
where Gv(0)=
Assumption 3 (Validation Representativeness). With probability at least 8', the volidation gradient Gv(-) at every iterate encountered during training satisfies
([v - VoRD (0)l22 S 6, where 6= OBB///)
(13)
These assumptions mirror those commonly adopted in analyses of training dynamics (Pruthi et al. 2020
Dyas et. al. 2022 Veen et al.) 2020 Clemmensen & Kjærsgaard] 2022 Zhang et al. 2022); we merely state Assumption 3 explicitly for transparency, even though typically invoked implicitly and is widely regarded as reasonable.
Remark 1 (Per-Class Validation Trajectorics). In our implementation, we compute CLD using class-specific validation trajectories Š'yc rather than single global trajectory. This refinement aligns with standard coreset practices that enforce class balance, which reduces the variance of the corvelation estimates by matching each training sample with the validation subset most relevant to its semantic label. The theoretical guarantees stated here continue to hold, as long as the per-closs validation subsets satisfy the representativeness condition in Assumption 3 when interpreted conditolnii
Theorem (Convergence with CLD-Coresets). Consider a gradient descent algorithm trained over iter- alions on a truining dalasel S with held-out validalion set V. Given Assumplions 7 to 3 let the learning rate satisfy 17 S L/L. Let fg denote the porameters at iteration when training on a coreset C.
Then, training the coreset C, consisting of somples with high CLD scores:
C
{ Zm SSS CLD(Em) -c
O, +
(14)
guarantees that
(D())) Rint 1)
+ LnB² + (BVZk+
min 0st<T
<
(15)
where Rint := infs (0), and i. 2 is an aligument-gan term that quantifies the mismatch between the average coreset gradient and the validation prozy gradient Gv(f) along training (see Appendir B for the formal definition). Intuitively, decreases as the selected samples' CLD scores increase and as the coreset grows, and 0.

---

Published in Transactions on Machine Learning Research (11/2025)
Proof Sketch. The proof is based the observation that the change in population risk across training steps can be approximated by the inner product between the gradient of the risk and the update direction. A high CLD score implies a strong correlation between a sample's loss-change trajectory and that of the validation set, which in turn suggests consistent alignment between the sample's gradient and the validation gradient.
Due to the L-smoothness of the loss function, this alignment persists even when training on the coreset, allowing us to bound the cosine similarity between the average coreset gradient and the true risk gradient. This leads to controlled approximation error in the optimization update. The total error term (Bv2k + +5)2 is governed by three factors: the CLD scores of selected samples, the deviation between the coreset and full- data parameter trajectories, and the quality of the validation set as a proxy for the true distribution. Full details and supporting lemmas are provided in Appendix B
Interpreting the Theory Under the stated assumptions, training on the full dataset S yields the con- vergence bound
| RD (D)())))) 2 (+g) 05KET
Runf + LTBB
min
(16)
n/T
Theorem [ shows that training on a high-CLD coresct achieves a similar bound, up to an additive deviation
term (Bv2k 8)2. This deviation captures the aligmment of the coreset with the validation dynamics (k))
and the representativeness of the validation set (0), with 6 = O(B/vO//)) decreasing in the validation size Q.
and k decreasing the CLD selection is tightened (smaller E) or as the coreset size k increases (see Remark 3
and Remark in Appendix B
The alignment term & reflects both the informativeness of selected samples and the size of the coresct. Higher CLD scores indicate stronger agreement with validation loss trajectories and thus tighter gradient alignment, reducing x. Additionally, larger coresets more faithfully approximate full-data training dynamics, also lowering k. We note that making k extremely small can increase trajectory deviation ||&e]|, which is reflected inside (Remark 4 When the coreset size is fixed, the theorem implies that selecting higher- CLD samples improves convergence by minimizing this deviation. Thus, CLD-based selection emerges as a principled and necessary criterion for preserving the optimization behavior of full-data training.
Corollary 1 (Necessity of High CLD for Good Coresets). Under the hypotheses of Theorem 4 achieving convergence rates comparable to full-data training necessarily requires that the selected samples exhibit near- mazimal CLD scores and that the validation set provides a reliable prozy for the true risk gradient. Fulfilling these necessary conditions ensures the optimization dynamics induced by the coreset remain well-aligned with those of full-dato training.
6 Experimental Evaluation
We evaluate CLD empirically, focusing on its effectiveness and transferability.
Experimental Setup We benchmark on CIFAR-100 (Krizhevsky et al. 2009] and ImageNet- 1k Rus- sakovsky et al. 2015). CIFAR-100 has 50,000 training and 10,000 test images across 100 classes; TmageNet- 1k has ~ 1.28M training images and a 50,000-image validation set across 1,000 classes. For each random sced, we form a classwise held-out validation split from the training data (10% for CIFAR-100; 1% for ImageNet-1kk), ensuring equal per-class representation; a different split. generated per seed, and the re- sulting train/validation partitions are reused across all basclines for fairness. Unless otherwise specified, ResNet-18 (He et al.) (2016) is the default architecture for CLD scoring and for training on selected coresets. Coresets are constructed per seed in a samples under CLD. Subset sizes range from 000% on CIFAR-100 and 0.1%-100% on ImageNet-1k. We report the mean and standard deviation over 5 independent seeds.
manner by selecting, within each class, the top-ranked
Baselines. We compare against representative state-of-the-art methods from three families: score- score-based (Forgetting (Toneva et al. 2018), EL2N (Paul et al., (201)), and CCS (Zheng et al. (022) using AUM (Plciss) et al. 2020)), optimization-han based (Glister (Killamsetty et al. (0221)), D'--runiig (Maharana et al. 2024). ).
and training-proan pooiinngg-oopettt--e based (TDDS (Zhang et. al. 2024), SloCurv (Garg & Roy 2023) DUAL (Cho et. al.
7

---

Published in Transactions on Machine Learning Research (11/2025)
Randon CCS (AUM)
EL2N
Glister
TDDS
SloCurv
Forgetting
Drrniing
DUAL
Mean Top-1 accuracy seeds (%)
40
40
0,250 85 1% 2%
65%
75
10% 20% 50%
0.15% 0.5%
1%
55%
10%
30%
50%
Conosct of full datasot)
Coreset full dataset)
(a) CIFAR-100
(b) ImageNet-Ik
Figure 2: Test accuracy (mean over five seeds) for representative coreset selection methods on CIFAR-100 and ImageNet- with ResNet-18. CLD consistently matches or outperforms baselines across dataset sizes. (X-azis uses non-uriform coreset-size grid.) Color map: blues (score-based), oranges (optimization-baser greens property-baselee black (CLD). For reference, the full-data mean top-1 accuracy over five seeds is 70.95 on CIFAR-100 and 69.91 on ImageNet-IKe Ik. Complete numerical results are available in Appendix C
RosNet-33
RoaNet-50
DenseNet- 12"
VGG-19 (with Hatchnorm)
Moan Top 1 occurucy
ove: 5 soods (58)
Transter
Tronster
Tronstor
Tranter
15
Oraele
Oraole
Orarle
Oracle
100
200
Coresct: size (% of Fall dataset)
Coreset sine 1% of full dataset]
Cornsct size (% of full datasett;
Coreact size (% of full datuset)
Figure 3: Transferability of CLD-bosed coresets across architectures ImageNet-Ik: Each subplot reports test accuracy (mean over five runs) for largel models trained coresets of varying sizes. Transfer coresets (dashed black, diamonds) are selected using ResNet-18; 18; Oracle coresets (solid green, circles) are computed by the target itself. Transferred corcsets are within 1% of oracle coresets across all targets and sizes.
2025 plus Random. We use implementations from the DeepCore (Guo et al. 2022 library when available, and herwwie rely on official GitHuu repositories. All methods are run under a consistent training setup (40 pretraining epochs where required), without any additional fine-tuning or regularization. To ensure fairness, baselines, including ours, select and train coresets using the same backbone (ResNet-18).
Transfembillty protocol. On ImageNet-1k Transfer corescts using ResNet-18 and apply them to ResNet-34, ResNet-50, VGG-19, and 1enseNet-121. We compare this lo Oracle setting where each target model computes its own CLD scores and coresets from its dynamics.
additionally test cross-architecture transfer. We compute
Results and Observations Figure summarizes performance on CIFAR-100 and ImageNet-1k compared to other methods. Across both datasets, CLD consistently matches or outperforms the strongest baselines from each family. The most competitive alternatives are D2PPrruning and DUAL, though both degrade very small coreset sizes. On CIFAR-100, Glister, DUAL, CCs (AUM) can slightly edge out CLD at larger subsets (by < 1%), whereas CLD consistently leads ImageNet-1k. At large subset sizes, CLD converges full-data
8

---

Published in Transactions on Machine Learning Research (11/2025)
performance with negligible deviation from the strongest bascline. Full numerical tables (and additional methods beyond those plotted) are deferred to Appendix C to avoid clutter. A complementary analysis of the subset fraction required to match full-data accuracy is presented in Appendix D.2
For cross-architecture transfer on ImageNet- -ee Figure B shows that Transfer coresets selected with ResNet- closely track Oracle coresets computed by the target model itself. The gap remains below 1% across ResNet-34, ResNet-50, VGG-19, and DenseNet-121 and across coreset sizes, including transfers across archi- tecture families (ResNet DenseNet/VGGG,
Takeaways CLD achieves near-optimal accuracy across subset sizes while incurring the lowest compute and storage overhead among strong baselines, and its coresets transfer effectively from lightweight proxies to larger targets. Together, these properties make CLD a scalable, reliable choice for coreset selection in both single-architecture and cross-architecture regimes.
7 Computational and Storage Efficiency
Beyond accuracy, practical coreset method should keep both compute and storage costs low. CLD does by relying only per-sample loss scalors that standard training already produces; per-sample gradients, Hessians, or pairwise similarities are required. Practically, this entails scoring the full training set at a small number of checkpoints, but the scores are exactly the per-sample losses computed during training, i.c., no extra inference sweeps. The only extra work is a forward-only sweep over a small held-out query set each proxy epoch (Q < N) to record query losses. In contrast, gradient/adversaan /adversaria methods incur extra backward passes, while similarity cetaartty/eearree neerrst-neighhor methods require feature extraction and large feature caches. We quantify the end-to-end compute cost (sclection plus training the selected coresct) and the storage overhead in detail in Appendix D and summarize the symbolic complexity below.
Notation and setup. We measure compute in floating-point operations (FIOPs) and report storage overheads:
Data and epochs. N training samples, Q query samples; T epochs for the large model, T Tproxy for the prozy; Tearly (early scoring), Terocy (early proxy epochs in DUAL).
Model cost convention. Large model forward cost flarge: proxy forward cost f with < flarge One backward x 2 forwards + one training step & 3 forwards per example.
problem. k coreset size; d input dimension; c classes; R repeats (restarts/prolf ppobes); reselection interval.
CRAIG embeddings. F pemultimate-featun dimension; Def == F+c is the embedding size used by
CRAIG.
Method-specific.. J window length (DDD-USC/DUALLTDDDS) H message-pasinng rounds (DD--runing): ANN degree; U unlabeled-pool size (Cal); Yane anchor spacing and A=Tproxy/ /ann anchors (CRAIG); stochastic-rreddy tolerance; À trade-off in GraphCut.
Results and observations (compute). As summarized in Table methods that score during early training of the large model (e.g., Forgetting, EL2N, GraNd) require one or more full sweeps over all examples with the large network for Tearly epochs (and sometimes repeats), so their selection cost in- cludes terms like 3NTearyy R faarer making them compute-ineffiiieu even if the coreset used later is small. Optimization eedimization-vit---eeelleeee wrtth-reeelectinn methods (e.g., Glister) add frequent subset updates every Y epochs, driving O + N log(l/e)) fiarge T/7) on top of 3kT fiarge- Feature/gras /graph based selectors (Herding, Moderate, D D2-Pruning, Cal) pay 3N plus at least one Nf encoding pass (sometimes graph/kNN work). By contrast, CLD uses only proxy training and cheap per-epoch query forwards:
ComputeaL = pN + QT + 3KT flarger
with no gradient/Hessiiee Hessian sweeps, no adversarial steps, and no pairwise similarities.
Results and observations (storage). To make storage comparisons transparent, Table reports selection-slage storage overhead only, Le., method-specifie extras beyond storing the large model's weights.

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: End-to-end compute and storage overhead for CLD and baselines. Storage column lists method- specific extras during selection only. Notation Section
Computational cost (selection + coreset training)
Storage overhead
Method
(method-speciifcc extras)
Herding
3NT f + NJ + O(Ndk) + 3kT 3 flarge
( [features] (+ optional
O(k2) Gram)
Forgetting
3NT early Starge 3KT
((N) pper-sample counter)
AUM
p + 3kT Slarge
((N)
[running sums]
Cal
3KT procyf f + ff + ((Und) + 3KT flarge
(UUd) |proxy features
GraNd
cay R Riiarge + cN Rhharge + 3kTmmthfarg
((N) (scores/log)
EL2N
Tea Rflarge
Sttuage
O(N) [scores]
Moderate
3NT p + NS + O(Nd+ WlogNN + Jaage
( [prxxy features]
Di-Pruning
3NT proxy f + Nf + O(Nnd) + O(HNK) + 3kT flarge
( + O(Nk) [features +
KNN graph]
CRAIG
3NT J +
DeR) + 35T flarge
O (N((+cc) [per-anchor em- beddings]
Glister
S +
((Q) [validation cache]
GraphCut
3NT proxy f + N + O(N2k) + 3KT ftarge
((N [pairwise similarities]
SloCurv
3NT prosy. J + 3N(R+1) 3KT ftage
( + O [running stats
+ probe dirs]
TDDS
3NT proxy/ S + 3NT P 3kT Starge
O(NJ) [windowed logs)
Dyn-Unc
3NT proxy f + 3KT flarge
O(NJ) [windowed logs)
DUAL
3NT J + 3kT fiarge
O(NJ) [windowed logs]
CLD (Ours)
3NT prosy QT f 3kTffagg
0 (N Q Trrocy)
[loss logs)
Early-training methods (Forgetting, EL2N, GraNd, AUM) need only O(N) scalars; mandowed-umcortinngg
methods (Dyn-Unc, DUAL, TDDS) add O(NJ) logs. CRAIG stores (N(++c)) embeddings, similarity /feature methods cache O(Nd) (plus O(Nx) graphs), and GraphCut is O(N2). CLD uses only scalar loss logs (
A visual summary. (x-axis, log scale), with bubble size proportional to the selection-stage storage overhead. Points in the upper-lest. with small bubbles are closest to the "Pareto-elficient" frontier, combining high accuracy with low compute and storage cost. To provide a concrete, quantitative context for these trade-offs, the plot is generated using an illustrative setup: selecting 10% coresets of ImageNet-Ik with a ResNet-18 proxy, then training a ResNet-50 on the chosen coreset (see Appendix for details). In this setting, methods that score during early training of the large model (e.g., GraNd, EL2N) and those with frequent reselection (e.g,, Glister) appear far the right due to large compute costs, while ciiturre/imilarttyy based selectors (Herding, Moderate, Cal, -P2-Prunigg) have large bubbles from O(Nd) feature caches. CLD lies near the efficient frontier: its selection cost is proxy-only plus lightweight query forwards, and its storage is just scalar loss logs. We also show CLDgo (scores derived using loss values from all proxy epochs) and CLDas (first 45 epochs only); the latter cuts selection compute nearly in half while preserving accuracy (discussed in Section S This mirrors observations for DUAL, which also achieves minimal accuracy drop by using only carly proxy epochs, underscoring that temporal truncation can further improve efficiency without sacrificing performance. For clarity, the figure uses shades of blue for score-based methods, shades of orange for optimization-basee methods, shades of green for prainnng-rooeert--baseee methods, and dark red for CLD.
the trade-off between accuracy (y-axis) and end-to-end compute
10

---

Published in Transactions on Machine Learning Research (11/2025)
15
CL g0
ResNet-15 (proxy) ResNet--0
10% coreset
D Pruninng
DUAL
CraphCut
Cl Clister
Mean Top-1 Accuracy over 5 seeds
Forgetting
Slocurv
CRAIC
Moderate
Graldd
45
Cal
TDDS
Storage Overhead
155
MB
30
10 TB
Dyn-Unc Unc
20
Herding
20
ET.2N
10a
104
Compute (PFLOPs, log scale)
Figure 4: Efficiency summary: Accuracy vs. Compute (x-axis, log scale); bubble size is proportional to the selection-stage storage overhead. The plot uses illustrative setup for concreteness: selecting 10% coresets of ImageNet-Ik with ResNet-18 proxy and training ResNet-50 the coreset (see Appendix E). Both CLDso (all proxy epochs) and CLD45 (first 45 proxy epochs) are shown; the latter achieves similar accuracy at roughly half the selection compute. A similar trend is observed for DUAL when restricted to early proxy epochs (which we discuss in Section highlighting that early-epoch scoring can improve efficiency without harming performance.
Moan Top-1 ncoursey ovei 5 sears (%)
Top-1 SErIeS DANE seNTs 3
Full (90 epocias
spodlo
Pind (30 epachss
Domrustaple x2 [45 spocia))
Dommseaoll x3 epocias)
003 (CLD)
0.2% 0.00 0,501 1855 315
256
100
5.5
12
755
850
5%
10%
30%
00%
Coreeet size (% of full dataseet)
(36
dataset)
(a) Test accuracy when CLD is computed from early sub- (b) Ablation bios reduction. CCS-style stratified sampled checkpoints. Stability holds even at coarse resolution. sampling on top of CLD consistently reduces accuracy (X-aris uses non-uniform coreset-sise grid.)
coreset sizes CIFAR-100.
Figure 5: Stability and bias. (a) CLD is stable under reduced temporal resolution; (b) external stratified sampling is unnecessary, and often harmful, for CLD.
8
Discussion
We discuss practical considerations and empirical findings that further illustrate the applicability of CLD.
Stability under temporal subsampling. We first assess robustness to reduced temporal resolution on ImageNet-1k with ResNet-18. CLD is computed either from the first 30 or 45 checkpoints (out of 90) or from trajectories subsampled at 2x or 3x lower frequency, while training still runs for all 90 epochs. As shown Figure 5a] using only early checkpoints (30/45) yields accuracy nearly identical to the full-trajectory setting,
11

---

Published in Transactions on Machine Learning Research (11/2025)
10090
sareds (%))
ORIGINAL POOL
EQUAL-BIN
1
19930
LOWEST men
PROPORTIONAL
IICHEST-BeR
Sumple roonr
10000
0030
12
Randon
LOWEST-Bed
6 12
4000
Moan Top
HIGHEST--BOR
EQUAL FQUAL-BIN
2030
PROPORTIONAL
10,,0g
( 6.4]
1235
2075
nan bin
Coreset size (% train split)
(a) mem Scores of Validation Sets Tested. Counts
(b) Effect of Validation Set Composition CIFAR-
per bin for each heuristic (RANDOM, LOwEST-zem, HIGHEST-ment EQUAL-BIN, PROPORTIONAL); the pool's Validation sets are built by the five mem- mem-basee cas-based heuristics.
100, ResNet-18; mean over five seeds (shaded bands: std).
original distribution is included for reference.
No label cleaning was applied.
Figure 6: Validation proxy study on CIFAR-100: (a) distributional differences induced by mem-based heuris- tics; (b) downstream impact CLD coreset performance.
indicating that the informative signal is captured early in training. 2x subsampling preserves accuracy, and even 3x subsampling incurs only minor degradation due to reduced temporal resolution.
Bias reduction and stratified sampling. Several methods incorporate bias-reduction mechanisms, such as CCS (Zheng et al. 2022). which stratifies selection across score percentiles promote diversity. In contrast, CLD leverages per-class validation trajectories, yielding a generalizationn-waarr signal that naturally balances classes and downweights noisy or redundant samples. As shown in Figure 5b applying CCS-style stratified sampling on top of CLD scores consistently reduces accuracy across coresct sizes on CIFAR-100. This contrasts with metrics like AUM, where CCS can be beneficial; for CLD, percentile quotas perturb its validation-aligned ranking and reintroduce less informative points.
Validation proxy: composition and size matter. Our theoretical guarantecs for CLD require that the validation set be a faithful proxy for the test distribution (Assumption 3 Section 5). To understand this, ask: How sensitive is CLD to the composition of the validation set, and does bias in this validation set affect downstrewm coreset qualily? To probe this, we split CIFAR-100's 50k training set into a classwise 25k/25k train/port /pool partition. From the pool, we constructed 5000 example validation sets using five heuristics based publicly available memorization scores (Feldman & Zhang 2020), denoted mem. Intuitively, low-mem points correspond to canonical, stercotypical examples (often helpful for transfer and exploited by methods such as SloCurv), while high-mem points surface atypical or mislabeled examples. These heuristics, therefore, let us bias the validation set toward "typical" or "atypical" regions of the data. We trained ResNet-18 proxies on the 25k train split, computed CLD with respect to each heuristic-based validation set, built class-balancee coresets, and fine-tuned ResNet-18 across coreset sizes (five seeds; see Figure 6b). As shown in Figure 6a the non-random heuristics yield validation sets with markedly different. mem profiles. Two clear patterns emerge. First, validation sets biased toward HIGHEST-men examples consistently degrade performance across coresct sizes: overrepresenting atypical or mislabeled samples lowers accuracy and hinders learning. Second, validation sets built from LowesT-mem examples generally support stronger generalization, but retaining some high-mem points is beneficial for capturing long-tail behavior likely present in the test distribution. In our runs, PROPORTIONAL sampling, drawing examples according to the pool's original mem distribution, was the most reliable overall, and would likely improve further if mislabeled points were filtered or downweighted (which we did not do in this study). These findings underscore that CLD's effectiveness depends critically on the quality of the validation signal: coresets cannot exceed the fidelity of the validation dynamics they are aligned with. This motivates future work on principled procedures to build clean, represent ative validation sets (e.g., robust to label noise) and to select validation sizes that balance reliability with efficiency.
Overall, CLD already achieves implicit bias reduction via validation alignment: external stratified sampling is unnecessary and often counterproductive. We provide further results, including seed-wise stability (Ap-
12

---

Published in Transactions on Machine Learning Research (11/2025)
pendix D.1 and connections to influence functions and training data attribution methods (Appendix F in the appendix.
Loss Differences as a Gradient-Free Proxy for Influence. The CLD metric is built on correlating per-example loss differences, a deliberate choice over simpler signals like raw losses or gradient norms. This choice is theoretically motivated: as we formalize in Lemma [ a first-order expansion shows that the loss difference =)( approximates the gradient inner product (Vol(6r=1; :2), 50--1). This insight positions CLD as an efficient. proxy for the alignment dynamics tracked by influence methods that compute gradient similaritics, such as TracIn (Pruthi et al. 2020). The primary advantage of this proxy approach is computational efficiency. Methods like TracIn are powerful but expensive, requiring the storage and processing of -igeenimnni per-sample gradients. CLD captures the same underlying aligument dynamics while avoiding this overhead entirely. Simpler signals, such as raw losses or gradient norms, are even cheaper but conceptually flawed, as they are confounded by signal drift or discard crucial directional information. Empirically, CLD's performance comparable to TracIn on key attribution metrics, including the Lineur Datamodeling Score (a measure of how well scores correlate with true sample influence) and prediction brittlencss (a mcasure of how critical top-ranked samples are to model prodictions). See Appendix F for the full analysis and broader discussion of how CLD relates to influence methods.
Scope and Cavcats. Because CLD requires only per-cxample training losses and a small validation proxy, its core criterion can be adapted to other supervised set settings (e.g. con- trastive learning with a validation loss, or object detection by aggregating per-instance losses to a per-image score). In this work, we intentionally limit our scope to supervised image classification for controlled com- parisons and do not claim empirical validation outside this domain. Applying CLD modern language model fine-tuning, however, is non-trivial due to challenges like the "squeezing effect." (Ren & Sutherland 2025). As fine-tuning progresses, the model concentrates probability mass onto the exact training sequences, which can paradoxically lower the assigned probabilities of similar but non-identical validation samples. This causes divergence between training and validation loss trajectories, meaning that training samples promoting generalization are not guaranteed to have a high CLD score. Corroborating this finding, Xia et al. (2024a) observe that, unlike in vision, minimizing validation loss does not reliably improve model performance in instruction tuning. Adapting a CLD-style selector for LLMs will therefore require developing approprae generalization proxies, which is important direction for future work.
Beyond Supervised Vision:
9 Limitations
While CLD is scalable and effective, it does have limitations. First, requires access training loss tra- jectories across multiple checkpoints, which may not be feasible in settings where models are deployed black boxes when fine-tuning from pretrained checkpoints without full retraining. Second, although CLD requires a hold-out validation set, this reduces the number of samples for training.
10 Conclusion
We introduced Correlation of Loss Differences (CLD), a simple, scalable metric for identifying data that aligns with generalization. By relying only on per-sample losses across training checkpoints, without gradients, pairwise similarities, or second-order information, CLD enables principled coreset selection with low compute and storage cost. Across CIFAR-100 and ImageNet-lk, CLD matches or outperforms state-of-the-art methods over a wide range of subset sizes and allains full-data accuracy with substantially smaller subsets. CLD- selected coresets also transfer across architectures (ResNet, VGG, DenseNet) with < 1% degradation, remain stable when using only carly checkpoints, and inherently reduce bias via per-class validation alignment, obviating additional stratified sampling. Our theory further shows that the convergence gap under coreset training is controlled by sample-validation alignment and the representativenes of the validation set. Taken together, these properties make CLD a practical tool for large-scale, budgeted training and a principled foundation for future work on robust validation design and budget-aware data selection.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgements
This work supported in part by the Center for the Co-Design of Cognitive Systems (CoCoSys), DARPA-sponsored JUMP 2.0 center, the Semiconductor Research Corporation (SRC), the National Science Foundation, and Collins Aerospace. We are also thankful to Eistathia Soufleri, Akshita Gupta, Utkarsh Saxena, Amitangshu Mukherjee, and Sakshi Choudhary for their helpful discussions and feedback.
References
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in newral information processing systems (NeurIPS), 32, 2019.
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Baker Grosse. Training data attribution via approximate unrolling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
S Basu, P Pope, and S Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations (ICLR), 2021.
Eden Belouadah, Adrian Popescu, and Ioannis Kanellos. Initial classifier weights replay for memoryless class incremental learning. In BMVC 2020- The 31st British Machine Vision Virtual Conference (BMVC), pp.
1 1-13, 2020.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Corescts via bilevel optimization for continual learning and streaming. Advonces in neural information processing systems (NeurIPS), 33:14879 14890, 2020.
Léon Bottou. Largo-scale machine learning with stochastic gradient descent. In Proceedings of COMP- STAT'2010: 19th International Conference on Compulational Statislics, Paris France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177-186. Springer, 2010.
Francisco Castro, Manuel J Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End- to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233 248, 2018.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu, Dataset distilla- tion by matching training trajectories. Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4750-4759, 2022.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sizth Conference Uncertainty in Artificial Intelligence, pp. 109-116, 2010.
Yeseul Cho, Baekrok Shin, Changmin Kang, and Chulhee Yun. Lightweight dataset pruning without full training via example difficulty and prediction uncertainty. In Forty-secone International Conference on Machine Learning (ICML), 2025.
Line H. Clemmensen and Rune D. Kjærsgaard. Data representativity for machine learning and ai systems. arXiv preprint ar Xic:22203.0 04706, Mar 2022. URL https://arxiv: wwwriii.org//bb.. httgg/bb//200... 04706
C Coleman, Ych, Mussmann, Mirzasolciman, P Bailis, Liang, J Leskovec, and M Zaharia. Sc- lection via proxy: Efficient. data selection for deep learning. In International Conference on Learning Representations (ICLR), 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems (NeurIPS), 33:2881 2891, 2020.
Isha Garg and Kaushik Roy. Samples with low loss curvature improve data, efficiency. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 20290 20300, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coresct selection in deep learning. In International Conference on Dalabase and Expert Systems Applications, pp. 181-195. Springer, 2022.
Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems (NewrIPS), 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 770-778,
2016.
Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruming with dynamic uncertainty. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion (CVPR), pp.
7713-7722, 2024.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected con- volutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pp. 4700 4708, 2017.
Andrew Πyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial infor- mation measures with applications in machine learning. In Algorithmic Learning Theory, pp. 722-754 PMLR, 2021.
K Karthikeyan and Anders Søgaard. Revisiting methods for finding influential examples. abs/2111.04683, 2021.
CoRR,
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning (TCML), pp. 5464-54744 PMTR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Tyer. Glister: Gen- cralization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118 8118, 2021b.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML), pp. 1885 1894. PMLR, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). URL htlp://www. cs. toronto. edu/kriz/eiji htmll 2009.
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. In The Twelfth International Conference on Learning Representations
(ICLR), 2024.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empiricol Methods in Natural Language Processing (EMNLP), pp. 650-666 330,663, 2021.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning (ICML), pp. 6950-6960. PMLR, 2020.
Byunggook Na, Jisoo Mok, Hycokjun Choc, and Sungroh Yoon. Accelcrating neural architecture scarch via proxy data. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAT-21. International Joint Conferences Artificial Intelligence Organization, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Sung Min Park, Kristian Georgiev, Andrew Ilyss, Guillaume Leclere, and Aleksander Madry. Trak: attribut- ing model behavior at scale. In Proceedings of the 40th International Conference Machine Learning (ICML), pp. 27074 27113, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advonces in neural information processing systems (NewrIPS), 34: 20596-20607, 2021.
Karl Pearson. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240 242, 1895.
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems (NeurIPS), 33: 17044-17056, 2020.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Aduances in Neural Information Processing Systems (NeurIPS), 33:19920- 19930, 2020.
Sylvestro-Alvise Rebuffi, Alexander Kolesníkov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Compuler Vision and Pattern Recognition (CVPR), pp. 2001-20 2010, 2017.
Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirleenth International Conference on Learning Representations (ICLR), 2025.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Internaiional journal of compuler vision, 115:211 252, 2015.
Proceedings of the AAAI Conference Arlificial Intelligence, volume 36, pp. 8179 8186, 2022. Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Leorning Representations (ICLR), 2018.
Jae-hun Shim, Kyeongbo Kong, and Suk-Ju Kang. Core-set sampling for efficient neural architect search. arXiv preprint arXio:2107.06500 06669, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xrxi::1409. 1556, 2014.
Charles Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-100 101, 1904. doi: 10.2307 00-2307/141211 1412159. URL httpp:://ww.jsttttt sestorr.og//taabee/1122s
Tlya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning (ICML), pp. 1139 1147.
PMLR, 2013.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations (ICLR), 2018.
Duco Veen, Jan C. Grutters, Matthias F. M. F. Veta, Max A. Viergever, and Josien W. Pluim. The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity. PLOS ONE, 15(8):c0237009, 2020. doi: 10. /
Mike Wojnowicz, Ben Cruz, Xuan Zhao, Brian Wallace, Matt Wolff, Jay Luan, and Caleb Crable. "influence sketching": Finding influential samples in largo-scale regressions. In 2016 IEEE International Conference on Big Data (Big Data), pp. 8601-3612. IEFF, 2016.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for largeted instruction luning. In International Conference on Machine Learning (TCML), pp. 54104 54132. PMLR, 2024a.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-cfficient deep learning. In The Eleventh International Conference Learning Represenlations (ICLR), 2022.
Xiaobo Xia, Jiale Liu, Shaokun Zhang, Qingyun Wu, Hongxin Wei, and Tongliang Liu. Refined coresct selection: Towards minimal coreset. size under model performance constraints. In Forty-first International Conference Machine Learning (ICML), 2024b.
Shuo Yang, Zhe Cao, Sheng Guo, Ruiheng Zhang, Ping Luo, Shengping Zhang, and Liqiang Nie. Mind the boundary: Corcsct sclection via roconstructing the dccision boundary. In Forty-first International Conference on Machine Learning (ICML), 2024.
Wei Yang. Pytorch-classifccation https: http:/ggituub..o..... httppp/...../.aaaao///ooooooaoaa 2017.
Chih-Kuan Ych, Joon Kim, Ian En-Hsu Yen, and Pradcop K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing syslems (NeurIPS), 31, 2018.
Mengying Zhang, Mary L. Li, Tesi Liu, Yong Yu, and Zhi-Hua Zhou. Deep active learning by leveraging training dynamics. In Advances Neurol Information Processing Systems 35 (NeurIPS 2022), pp. 14737- 14750, 2022.
Xin Zhang, Jiawei Du, Yunsong Li, Weiying Xie, and Joey Tianyi Zhou. Spanning training progress: Tempo- ral dual-depth scoring (tdds) for enhanced dataset pruning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 86223-26232. 2024.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. The Eleventh International Conference on Learning Representations (ICLR), 2022.
17

---

Published in Transactions on Machine Learning Research (11/2025)
A CLD- LLD-CCresee Selection Algorithm
For completeness, we provide the full pseudocode for the coreset selection procedure described in Section 4.1 This algorithm computes per-class CLD scores by correlating each training sample's loss trajectory with the corresponding class-specific validation trajectory, and selects a fixed number top-scoring samples per class form a class- balanced coreset.
Algorithm 1: Coreset Selection Using CLD (Per-Class)
L Input: Training set $ = [Em m=11 Validation set V {á)) Class budgets (Ke) with E ker
Epochs T. Initial params (g, Loss é, Optimizer A, Hyperparameters
Output: Coreset C of size k
for to do
0/. A for m < 1 to N do
// update model
3
L Store ((0g) Zm) // record train loss
for j to Q do
L Store ((0g. 4)) record val loss
for m + 1 to N do (( + [00 =m) E(05-1 {m)] 1=1
for j to Q do
((7) [1(0) (,)
for
to C do
12
S. + f E S fmm c)
13
V. (1 E V U) ct
S'v. + 1 [V.l Ev. A(T);;
14
foreach em E S. do
L (L)) & ((mm), A'v..)
18
C. + top-ke elements of S. by CLD(
C C.; return
19
Implementation note. Loss values in Steps 3 6 are logged as part of the normal training loop; no additional passes are introduced.
B Detailed Theoretical Framework
In this appendix, we provide the complete theoretical framework supporting the results stated in Section We first outline the dotailed lemmas establishing gradient alignment and approximation properties CLD- selected coresets. We then conclude with a full proof of the convergence guarantee presented Theorem
B.1 Roadmap and Notation
Proof Outline. Our main convergence guarantee, Theorem is built upon three supporting lemmas:
Lemma which establishes that high CLD score implies strong alignment between a sample's gradient and the validation gradient.
Lemma 2 which shows that this alignment is preserved during coreset training, provided the coreset and full-data parameter trajectories remain close.
Lemma which leverages this stable alignment to bound the approximation error between the average coreset. gradient and the true population risk gradient.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Notation Summary. We use the following symbols throughout our analysis:
L The smoothness constant of the loss function.
B: An upper bound the per-sample gradient norm.
k: The size of the coreset.
Q: The size of the validation set.
A parameter controlling the strictness of CLD-basod sample selection.
K: The alignment-gap term, quantifying the gradient mismatch from coreset selection.
The validation representativeness error, which decays as O(vQ).
7: The learning rate (step size) of the optimizer.
T: The total number of training iterations.
0g 0g: Model parameters al iteralion / when training on the full datasel and the coreset, respectively.
B.2 Supporting Lemmas for Theorem
Lemma 1 (High CLD Implies Gradient Aligmment). Consider a training sample 7m E S with CLD (m)) = of (A) 2 - for some small 0. Let O's be the parometers obtained by running algorithm A S foor iterations. Let 80t-1 = 0' 0-5 be the parometer update at step t.
Suppose the learning algorithm is run for a sufficiently large namber of iterations T. Assume the sequence of parameler updates {50'-1)1- is sufficienlly varied. This means the updates are not persistently orthogonal to any fized non-zero vector direction in the relevent parometer subspace.
Then, for most training steps t where Gv(Ag) + O, the sample gradient Vol(os, Em) and the validation grudient (0g))) are well-aligned:
CO6 (V) 2 4
(17)
where e4 + 0 0.
Proof Outline. Use first-order loss changes to relate per-cxample loss differences and validation loss dif- ferences at consecutive steps. High correlation of differences implies an (approximately) positive lin- car link between their inner products with update directions, forcing the sample gradient to be a posi- tive scalar multiple of the validation gradient under sufficiently varied updates. Contimuity then yields cos(<(VE(,, Zm), Gy)) 2
with +0 as c
Proof. We first analyze the idcalized case where the corrclation is perfect (e 0) and the underlying ap- proximations hold exactly, and then argue by continuity.
Assune the first-order Taylor expansions are exact for the loss changes:
(Ā(Zm)), E(0s,, Zm) - 1005-1,mm) =m)) ~ (Vol(0' =m)),00 Itt
(18)
Q (AV): (e(o's ( -(0'='1))) (Gv 8,00--1 yt-
(19)
Assume perfect correlation () 1. This implies an exact positive linear relationship I; cyt+K* K' for all where c = and K' = H-cğ. Substituting the definitions of and y::
(Vol(6')) Zm); 2m)),8(-11 = (Gv(85 50,00-11
(C +)+1))111 K'.
(20)
Rearranging yields:
(Vol(0'-1 ;m,) 1,, K"
(21)
19

---

Published in Transactions on Machine Learning Research (11/2025)
Since the mean of the loss trajectory will be smaller compared to the variance of the terms (losses eventually reduce to 0), it is reasonable to assume K' Ri O. Thus, for = 1, I::
(Vol(0' ;zm) c.Gv((vV)))) 600000 0.
(22)
Let i i= V - cG The vector - is exactly orthogonal to the update direction 60'-1 at each step
Now, invoke the assumption that the sequence of updates { sufficiently varied. This means the updates are not persistently orthogonal to any fixed non-zero direction W1-1 If 1 were non-zero, the variation in updates would eventually yield 50 such that (17,-1 506-1) o 0. Since the inner product is exactly zero for all t in our idcalized case, the only possibility consistent with the sulficient variation assumption is that W must be the zero vector. Thus:
- Vol(o' Em) cGv =
(23)
This signifies that the sample gradient is exactly a positive scalar multiple (c> 0) of the validation gradient:
Vollog _mm = (o
(24)
Consequently, the vectors are perfectly collinear and point in the same direction (assuming Gv(A + 0).
The angle 7/-1 between them is exactly 0. Therefore, this idealized case:
cos(7--1) = cos (< (Vol(o' Z... Gv( =1
(25)
This derivation holds under the ideal conditions (e = 0, exact Taylor approx., K' = 0). Since the involved operations are continuous, when the conditions are only approximately met (i.e., p 2 with c - 0, Taylor approx. is good, K' is small), the resulting cosine similarity will be close to 1.
We express this conclusion as
cos(1)11) 2 G-11
(26)
where the error c/-1 - 0 as - 0. Assuming this alignment holds for most steps t (implying alignment at step relies on propertics at t 1). the lemma statement follows.
Remark 2 (On Update Sequence Variation). The assumption regarding the update sequence (50--1} is that il exhibits enough varialion over the trajectory lo ensure that no fixed non-zero vector can remain orthogonal to all updates. This property is weaker than requiring the updates to span the entire parameter space, but it is sufficient for the argument. It essentially prevents the gradient difference vector from hiding in a direction that the optimization process never explores. Stochastic optimization methods accumulating updates over many ileralions (large T) are often expected to salisfy this sufficient varialion condition.
Lemma 2 (Stability of Gradient Alignment). Suppose the conditions in Theorem hold: specifically, L- smoothness and bounded gradients Volo, 2)122 < B for all z). Consider coresel C constructed by selecting samples with high CLD scores:
C (Zm E E CLD(Z.)) 2 1] with
- +.
(27)
Assume that during braining, the difference between the porameler Inajeclories salisfies 110- - 00'00'l1 at step t.
115e112
Then, for each sample žm E C, the cosine similarity between its gradient and the average validation gradient at step t is lower bounded by
cos (< (m) Gv) 2 K,
(28)
4h where = e
|80ll22 + 312 B
1 and eç 0 - 0.
B
20

---

Published in Transactions on Machine Learning Research (11/2025)
Proof outline. Compare gradients at IC and 8's using L-smoothness to bound deviations by 611 Expand the inner product and bound cross-terms via Cauchy-Schwarz and the gradient-norm bound B. Plug the base alignment from Lemma [ at 0g to transfer alignment to IC yielding the stated 1 - K lower bound with fincrrquadraal dependence on ||óe||.
Proof. Let wm Vol(if' 2m) Vol(fs, Em) and wy = Gf Gv(8)) denote the deviations between gradients evaluated on the coreset trajectory and the full dataset trajectory.
By L-smoothness of f(., Zm) and of the validation-average loss Rl((0) = 22 22g1 e(0, Tj), we have:
|am 112 S [. 1oel22
(29)
|wvll2 S (,el011
(30)
Expanding the inner product:
(Vollde, Gv(de)) = (Voll0's 2m) + wm. Gv(fl)) + wv) (Vol(Os, =m); Gv(0g)) + (wi:: Gv((g)) + (Vol(0s: Zm),uv) + (Wmm wv)
(31)
(32)
Applying Cauchy Schwarz inequality and the bounded gradient norm Voo(( 27122 S B, we have:
(Wim (v(())) 2 || ll | (0))ll2 2 LB |0el|
(33)
(Vol(0s, wv) 2 - //Voe(t's Zm ll2 ||v||2 2 LB 6
(34)
(wm:wV) 2 - |Wim ll2 ||vll22 2 -L3 110.112.
(35)
Thus,
(Vol(O) Gn),Gv((fe))) 2 (Vol(O's Gv(s)) 2LB 1|8ell2 - I.2 115.112.
(36)
The denominator is upper bounded by:
V[le(00, 11, Gv( s (B + L 1
(37)
Combining this result with Lemma 1 we can conclude:
cos (4 (Vol(06 Em), (v(l)))) 21 - ( +
4L
3L2
|dell2 +
|1oel1 =
(38)
B
B
where é captures the initial alignment error when training on S. This completes the proof.
Remark 3. Lemma 2 shows that if a sample's 's gradient is well-aligned with the validation gradient during training on the full dataset (i.e., é is small), then this alignment is preserced when training on a coresel C, long the parameter trajectories 0g and f% remain close. The degradation in alignment is bounded by terms that are linear and quadratic in l| Thus, as long as the coresct trajectory stays near the full dataset trajectory, the coneralization-relevar properties captured by CLD remain slable. This slability is crucial for ensuring that. CLD-based coresets mainlain the training dynamics of the full datasel.
Remark 4 (Influence of Coreset Size on K). It is important to eaplicitly consider how the coreset size k = |CC (as specified in Lemma 2) influences the components of ==
4L #1 116:112 + 33 116.112.
31.2
The lerm é representing the initial alignment error derived from Lemma 1 is affected by k. A smaller coreset size k allows for a more stringent selection criterion for samples based on their CLD scores. Specifically, one can choose only samples with CLD(E,)) very close to 1, which corresponds to smaller in the selection rule CLD(Zm) 2 -e (from Theorem ] and Lemma 2) A smaller naturally leads to a smaller é,.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Conversely, the terms in R. that depend on ||ōe|12 |10c data parameter trajectories) are also influenced by k. While a smaller k allows for higher individual sample quality, very small & might result in coreset that is less representative of the full dataset S. This reduced representativeness can lead to a larger divergence ||ó:ll2 during training, as the oplimization trajectory on the small coreset may differ more subslantially from that on the full data. An increase ||ōellz would, in turn, increase the overall value of K.
flsll2 (the deviation between coreset and full-
Therefore, the selection of an appropriate coresel size k involves an inherent trade-off. A smaller k can be beneficial for the é, component of by enabling the selection of higher-qualitn samples. However, if k is too small, it could adversely affect the components of related to ||xrll2 by making the coreset insufficiently rep- resentative. The stability discussed in Remark 3 relies on ||6e1l2 remaining small, highlighting the importance of being chosen to adequately approzimate the full dataset's training dynamics while leveraging the benefits of high CLD scores.
Lemma 3 (Subset-Gradiec Gradient Approximation). Suppose the conditions in Theorem hold, including L-- smoothness, bounded gradients, and validation representativeness as described in Section
Define the average coreset gradient at step t
l0 =
(39)
C
MEC
Then, for every training step 1, we have
Ihé VeRD(O')ls S BV2x* +
(40)
where k. captures the alignment error and satisfies K + 0 + 0.
Proof oulline. Split the error as lh - Gvll + |Gv - V The second term is ö by validation repre- sentativeness. For the first, average the per-sample deviations and apply Jensen to get a mean of squared distances; combine with Lemma to bound each by 2B2x, yielding I - Gvll s Bv2k.
Proof. We decompose the error using the triangle inequality:
1 Vo (D())l < Ihéè - Gv(o)))l - +[[Gv(0E) - V Vo
(41)
The second term (@ - (O is bounded by by the validation representativeness assumption. To bound the first term In (v(Oe))l 2 apply Jensen's inequality:
Ihé - Gv(00))1
C (Vel(!', =m) ( -
Σ
(42)
1
S
Vol(O' Z,,) Gv(00))l2
(43)
|C|
meC
Define otn as the angle between Vol(Os =m) and Gv()))) By Lemma Expanding the squared distance:
2 we have cOs ot, 2 1 - K for all m.
Vollfe, zm) Gv(ob)ll== = Vol(O', [el((t [[v 2(V Zm). Gv(()))
(44)
= Vol((0' 2m 2 ++ (1 2/Vool =m) (v((0)))l cos pi. (45)
Since Vol(Oe, Em))l2 [lll/llGv@@@)@)) < B and cos 5t 2 - we have
Vol(O' Zm) Gv(06))1 & 2
(46)
22

---

Published in Transactions on Machine Learning Research (11/2025)
Substituting back into the Jensen bound,
I Gv(lc) 2B²k.
(47)
Taking square roots gives Thus, combining the two bounds,
1 Gv(08)ll - BV2K.
(48)
Ihé VollD((C))12 - 2V2x+00
(49)
as claimed.
0
Remark 5. Lemma 3 shows that under mild conditions, the average gradient computed over coreset selected based on CLD remains close to the true risk gradient throughout training. The deviation is controlled by two sources: the aligument error k. arising from the selection of high-CLD samples, and the validation approzimation error 8 due finite sample size. Consequently, optimization over CLD-coresets closely tracks the gradient flow of the full dataset, ensuring that convergence and generalization properties are preserved. This result is crucial for connecting loss trajectory dynamics with practical coresct construction.
B.3 Proof for Theorem I
Proof outline. Apply the T-smooth descent. lemma to RD(()) with the update 0 = 0 777 to obtain a one-step incquality involving RVR.. Decompose 76 = V RD + (né - VRD) and bound the crror term using Lemma 3 E. = l - VBDll || < Bv2x + 6. Control the mixed term by Cauchy Schwarz + Young; bound lhell by B. Sum over t to telescope R. R1+1 and isolate min, [( yielding the stated bound with (Bv2k + 5)2 and an LnB² residual.
Proof. Define
R₂ RD(Oc), = VoRD(()))
Ve = C
Σ MEC
Vel(Oc,.m)
(50)
By L-smoothness of ((-), and in extension Rp(:), and 17 5 1/L,
I
R4t1 S R4 + (G,, 01+ - 0cc) 2
|0+11 00.
(51)
Substituting the model update 0f+1 f = -
R441 S Ri n/Gr, 1e) + Ln²
2 l1
(52)
Rearranging,
Lnj²
(Gr,) R4 Ret 2 1l
(53)
Decomposing the inner product using the true gradient G. we get,
(G.,TT) = (G,, G. + (76 G.)) = ||G.1l2 (Gr, G.).
(54)
Substituting this back,
n |1Gol < R₁ R4t1 - n d6 - Gt) + Ln² l 1
(55)
2
Let E, = Inc - Gell.
Lemma 3 under the stated assumptions gives the bound E, < Bv2k + 8. By Cauchy-Schwarz Schwarz inequality,
Gn. G.) n/IGrll G.ll2 [|rrlll Fa.
(56)
23

---

Published in Transactions on Machine Learning Research (11/2025)
Young's incquality states that ab - (2)) +2b2/2, Va, 2 0 and y > 0. By using this inequality with y == 1,
(G,, 26 - GGt) S
? 2
|G + 7 E?
(57)
2
Substituting this into the inoquality in Equation 55
Taj² hnel2
n |G S Re Re41 + |G + 2 E
(58)
Since all gradients are bounded by B, their average Incell2 is also bounded by B.
2 |Grl2 S
1 E + R. Reti
LTi
B².
(59)
2
2
Summing from t = to T 1 1:
Tap
1
GGil < (Ro- RT)
E
B²
(60)
2
2
1=
Let Run = infg (0). Then Ro - RT S Ro - Ryn--
Substituting the bound Fg S Bv2n + 6:
Ln² B².
GGll < (D( - Ramt + + (Bv (Bv2x + 8)2 +
(61)
2
The sum on the left is lower bounded by T times the minimum term, i.e., T-- |G f 2 T-T-mine minosiee |Gr1
||G.122 s (D( Pint + (B + 8)2 T LTj² B².
min
(62)
2
osKT
Since >> 0,
2 RD (0)) - Rini ||Gr122
min 0st<T
+ (BB2x + LnB²
(63)
1/T
This proves the theorem.
C Datasets, Models, and Experimental Details
We evaluate our method on two standard image classification benchmarks: C1F (Krizhevsky et al. 2009) and ImageNet-1k (Russakovsky al. 2015). CIFAR-100 consists of 50,000 training and 10,000 test images across 100 classes and is publicly available without licensing restrictions. For ImageNet-1k,, use the official release from research license and requires user agreement, to the terms of access.
net wttp:/ageeeeeetorr//////ooooo/ee php/ which is provided under standard academic
Architectures. Our experiments employ the following CNN backbones:
ResNet-118 ResNet-34, and ResNet-50 He et al. (016)) from httpes//pptorrch.or//viiio stable//ooeels/rennt..tte (BSD-3-Clause license).
VGG-199ith batch normalization Simonyan & Zisserman 2014 from httpss//pytorcc..org
vission/stablle/mdeels///ss html
BOSD-3-Clause license).
DenseNeet-121 Huang et al. 2017 from https///gyaarraooo///ssssssststb
htmml (BSD-3 C-Cusse license).
For baseline comparisons, we use the DeepCore library Guo et al. 2002 (nttps://githul http://iithbb.ccooPttttiizz//// com/Patrickk//
DeepCore), which provides standardized implementations of several coreset selection techniques and is censed under MIT. All training and evaluation code was implemented in PyTorchh dependencies including torchvision MIT/BSD licensed.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Training Setup for CIFAR-100. All networks were trained using SGD Bottou (2010) for 164 epochs with initial learning rate of 0.1, decayed by factor of 0.1 at epochs 81 and 121. Nesterov momentum (Stskever
et al. (013)) with momentum 0.9 was used, along with weight decay 5 x 10 Standard augmentations included resizing to 32 x 32, random cropping with padding = 4, random horizontal flips, and normalization.
Training Setup for ImageNet-1k. Training followed standard ImageNet protocols: all models were trained with SGD for 90 epochs, with learning rate of 0.1 docayed by 0.1 at epochs 30 and 60. Nesterov momentum with coefficient 0.9 and weight. decay 10-1 were used. Data augmentations included random rosized cropping to 224 224, horizontal flipping, and normalization.
Reproducibility. No fine-tuning or additional regularization was applied to any method, including ours, ensuring fairness in coreset comparisons. All methods used the same validation split as the validation proxy, and the same train split as the full training set available for each seed when scoring training samples. Each experiment was repeated across 5 independent runs with distinct seeds; reported results reflect the mean and standard deviation.
All experiments were conducted on private compute cluster with access to NVIDIA A40 GPUs (48 GB memory, 300W TDP). All training and evaluation runs were performed in full precision using PyTorch.
Results.
Performance results for CIFAR-100 and ImageNet-Ik coreset experiments are shown in Tables 2 Cross-architecture transferability results for CLD coresets, as discussed in Section 6 are shown
and
in Table i
25

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Performance (top-1 accuracy) of score-based coreset methods on CIFAR100 trainsplit. The coresets were selected and finetuned on ResNet-18. The full trainset performance was 70.95 + 0.68. accuracy over 5 runs, along with their standard deviation, is reported.
The mean
Coreset Sizes
CLD
Random
Herding
Forgetting
Cal
EL2N
Moderate
CCS()UM)
D2-Pruning
(Ours)
3.66
2.57
3.52
5.24
3.9
3.8
4.1
4.6
5.67
0.2%
0.41
1..52
1.116
10.41
10.45
10.47
0.5
1.052
10.36
6.03
3.42
5.12
7.46
6.2
6
6.9
7.4
7.51
0.4%
0.28
0.49
0.33
#0.28
#0.11
#0.4
+0.42
0.46
+.59
6.8
4.07
6.8
9.12
8.7
8.4
8.2
8.6
8.91
0.6%
0.27
#0.41
0.188
#0.27
#044
#0.4
+.38
+.43
#0.41
7.96
5.14
8.42
10.19
9.8
10.2
10.5
10.5
10.56
0.8%
0.79
007
0.38
0.779
0.48
#0.49
0.55
#0.5
#0.24
9.38
5.32
11.53
13.73
13.2
12.7
12.1
13.1
13.04
1%
10.41
10.33
10.44
10.41
1.466
10.45
10.49
10.47
10.54
12.74
8.29
15.9
16.45
17
16.3
16.4
17.5
17.05
2%
10.28
10.45
10.21
10.28
10433
10.43
10.46
10.45
10.22
16.58
9.23
18.24
20.05
21.1
20
20.3
21.3
21.85
3%
1.955
10.52
10.32
10.95
10.5
10.46
10.52
1..49
10.85
19.99
11.52
23.82
22.97
24.5
23.2
23.9
24.1
24.01
4%
1.04
1116
0.88
1.04
0.47
0044
0.5
0.46
+0224
22.41
13.66
26.38
24.37
27.8
26.1
27.1
28.4
27.26
5%
0.54
1.35
0.85
00.54
0.44
0041
0.44
0.43
+0883
23.66
15.49
28.16
26.93
29.9
28.4
29.4
30.8
31.24
6%
0.49
0.92
0.62
0049
0.4
0.39
0.4
004
00199
28.36
18.52
30.95
27.37
32
30.3
31.5
31.7
32.37
7%
1.885
10.56
10666
10.85
10.38
10336
1.038
10.37
10.71
30.75
18.52
31.84
28.32
33.2
31.6
32.6
32.9
33.31
8%
11122
10.39
10.23
11.12
10.36
10.34
1.036
10.35
10.68
32.12
18.52
32.79
29.19
34.4
33
33.8
34.1
34.04
9%
1.48
1..98
10.94
11448
10.34
10.33
1.335
10.34
10.74
32.75
19.54
33.04
31.02
35.8
34.2
35
35.4
35.81
10%
1.02
0.85
0.65
+ 1.02
0.32
+0331
0.33
+0332
0.21
35.63
35.14
37.12
35.24
39.6
38.7
40.1
41.2
39.15
20%
0.99
1.06
0.85
0.81
0.28
+027
0.29
+0288
0.57
43.17
44.14
45.78
42.18
47.2
46
48.3
49
46.18
50%
= 1.02
0.4
0.41
0.69
0.24
0.23
0.25
+0.24
0013
63.21
66.12
66.12
63.05
65.8
65.1
66.9
67.8
68.01
75%
0.0.
10.46
1.661
10.41
10.
1.1.9
1.0.1
1.0.2
1.882
26

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: Performance (top-1 accuracy) of optimization and training property-based coreset methods on C1FAR100 trainsplit. The coresets were selected and finetuned on ResNet-18. The full trainset performance was 70.95 0.68. The mean accuracy over runs, along with their standard deviation, is reported.
Coreset Sizes
CLD
Random
CRAIG
Glister
GraphCut
SloCurv
TDDS
Dyn
Unc
DUAL
(Ours)
3.66
3.5
3.43
5.8
3.62
3.7
2.1
2
5.67
0.2%
10.41
1.442
1..32
1.224
10.4
10.44
1.555
1..56
1.336
6.03
5.8
4.91
7.07
5.46
6
3.5
3.4
7.51
0.4%
0.288
0.4
0.37
0.39
+.36
+
#0.5
+.55
0.59
6.8
7.9
7.49
8.96
7.44
8.2
5.1
8.91
0.6%
0.27
0.39
0.61
00331
#0.2
004
0.48
#
0.41
7.96
9.8
8.65
9.39
9.96
10.1
7.2
10.56
0.8%
+0.79
0.47
0.47
00.81
#0.42
0.48
0.53
00.54
#0.44
9.38
12.2
9.04
11.86
12.17
12.8
9.5
10.4
13.04
1%
1041
10.43
10.47
10.3
10.45
10.5
10.51
1.554
12.74
15.9
14.54
16.95
13.35
16.5
14.2
15.8
17.05
2%
10.28
10.42
10.41
1.555
1042
10.43
10.47
10.48
1..22
16.58
19.6
17.47
19.21
22.67
20.3
18.9
20.5
21.85
3%
10.55
10.46
10.32
1.557
10.8
10.47
10.5
10.5
1..85
19.99
23
23.99
21.33
21.97
23.7
22.7
24.3
24.01
4%
1.04
00.4
0.58
0.71
0.55
0.45
0.48
+0.47
0.24
22.41
25.8
24.83
26.31
23.44
26.6
25.8
27.5
27.26
5%
00.54
0.4
0.73
0.58
0.71
0.42
+ 0.44
+0443
0.83
23.66
28.1
26.57
30.35
25.41
28.9
28.2
30
31.24
6%
+0.49
0.38
0.69
0.68
+043
0.39
0.41
004
0.19
28.36
27.57
31.63
27.45
30.8
30.3
32.1
32.37
7%
10.85
10.36
10.84
10.71
10.6
10.37
L0.8
10.38
1.771
30.75
31.3
28.79
32.22
29.17
32
31.7
33.4
33.31
8%
11122
10.34
1.0.8
10.48
10.47
10.35
0.36
10.35
10.68
32.12
32.7
30.22
33.02
30.71
33.4
34.8
34.04
9%
1.448
1..32
10.32
10.83
10.57
10.33
1 L0.34
10.33
10.74
32.75
34.1
31.22
34.41
33.17
34.7
34.4
36.2
35.81
10%
1.02
= 0.3
+ 1.33
0.96
1.16
+ 0.31
0
+0.31
0.21
35.63
38.2
36.49
38.02
37.23
39.1
38.8
40.5
39.15
20%
+099
+ 0.26
+0.7
0052
0.81
0.27
0028
+0227
0.57
43.17
45.6
41.81
45.23
45.19
46.5
45.8
47.5
46.18
50%
+ 1.02
+ 0.22
+ 0.88
0.42
+
0.23
0024
+0223
0.13
63.21
64.8
63.85
65.18
66.53
65.9
64.5
66.2
68.01
75%
0.5
1.1.9
1 1.02
1 1..32
10.61
1.2
1.2.
1.0.9
0.82
27

---

Published in Transactions on Machine Learning Research (11/2025)
Table Performance (top-1 accuracy) of score-based coreset methods on ImageNet-1k trainsplit. The coresets were selected and finetuned on ResNet-18. The full trainset performance was 69.91 = 0.01. The mcan accuracy over runs, along with their standard deviation, reported.
Coresel
Random
Herding
Forgetting
Cal
EL2N
Moderate
CCS()UM)
De-Pruning
CLD
Sizes
0.7
0.31
0.64
1.13
0.88
0.85
1.52
1.95
1.96
0.1%
0.00
1.0.1
1.001
1.122
10.25
10.5
10.4
10.7
3.98
1.39
4.78
6.84
5.83
4.75
7.04
7.2
7.16
0.5%
0.19
0.17
1.01
0.13
+01
10.5
#0.1
+0.25
+0663
7.86
4.32
12.67
13.17
15.2
14.32
14.86
16.01
15.92
1%
0.43
0.62
0.51
#022
0.5
1.03
0.25
#0.4
#0.11
39.78
15.36
44.86
37.65
40.43
38.95
44.04
45.75
46.5
5%
0.23
0.18
0.74
1.3
1.003
1.25
10.1
1.3
1.19
51.24
26.84
53.19
44.16
45.16
44.56
52.01
50.65
53.81
10%
0.04
0.05
0.00
0.78
0.4
#0.1
0.2
0.3
+.22
60.87
46.61
60.9
54.41
53.22
55.29
61.84
60.75
62.91
30%
0.13
= 0.87
0.05
0.44
+.25
+0.1
0.5
0.1
0.51
62.13
53.88
62.39
58.45
56.45
60.08
62.48
61.04
63.51
40%
+ 0.38
= 0.23
0.91
0.92
0.1
0.15
0.02
0.5
#..75
64.11
59.14
63.18
60.11
59.46
62.58
64.31
64.92
65.78
50%
0.12
0.41
0.05
0.6
00.45
+.25
0.04
0.66
1.08
65.21
64.23
65.24
64.42
61.28
64.98
65.17
67.01
68.02
65%
1..03
+02
0.02
0.01
+.225
1011
101.1
0.1.1
0.388
68.81
65.22
67.85
66.57
64.23
65.18
68.81
68.91
68.91
70%
10.04
0.02
1..9
10.03
1..80
10.03
10.11
10.01
1..01
68.41
67.42
68.01
67.12
65.45
67.13
69.01
69.42
69.42
75%
00002
= 0.01
001
0.02
102
+.00
+
10.05
0.05
68.12
68.02
68.81
68.15
66.95
68.9
69.93
69.93
69.93
80%
1.003
1..02
10.5
10.03
10.01
10.02
10.02
10.02
68.75
68.01
68.81
68.91
67.17
68.9
69.91
69.93
69.93
85%
10.05
0.03
1.5.5
10.04
+01
102
+.225
+0.02
1.02
69.1
69.92
70.04
69.23
68.81
69.01
70.12
70.12
70.12
90%
10.78
104
1..52
10.5
10.3
10.2
1 10.03
10.03
1.003
69.91
69.91
69.91
70.12
69.91
70.12
69.91
70.12
70.12
95%
10006
0.04
1.004
0.001
10004
40.03
10.1
10.03
1.00
28

---

Published in Transactions on Machine Learning Research (11/2025)
Table Performance (top-1 accuracy) of optimization and training property-based coresel methods on ImageNet-1k trainsplit. The coresets were selected and finetuned on ResNet-18. The full trainset perfor- mance 69.91 0.01. The mean accuracy over 5 runs, along with their standard deviation, is reported.
Coreset Sizes
Random
CRAIG
Glister
GraphCut
SloCurv
TDDS
Dyn
Unc
DUAL
CLD
0.7
0.94
0.86
1.09
1.23
1.04
0.45
0.41
1.96
0.1%
1.00
10011
1.001
10.09
10.06
10.2
1.9
11.06
10.7
3.98
6.41
5.55
7.27
5.89
5.64
1.45
1.34
7.16
0.5%
0.19
0.25
0.05
0.03
0.07
0.08
1.05
1088
0.63
7.86
15.56
12.45
14.27
14.17
14.05
3.92
4.85
15.92
1%
+ 0.43
0.05
0.01
0.31
=0.02
0011
+
00775
00411
39.78
39.95
42.19
39.8
40.1
40.54
15.98
16.2
46.5
5%
0.23
0.01
0.03
1.6
0014
+ 0.05
0.5
0.4
0.19
51.24
46.76
50.1
48.27
46.39
46.45
29.78
50.75
53.81
10%
0.04
0.1
0.01
1.02
0.5
0.6
0.5
=
60.87
55.41
58.53
61.23
57.19
57.24
50.16
60.19
62.91
30%
0.133
0.05
= 0.05
0.01
0.01
#.1
+05
+.00
+
62.13
56.55
61.72
63.23
62.11
62.23
60.25
63.45
63.51
40%
0.38
0.02
0.02
= 0.08
0.67
0.5
0.2
0.25
0.75
64.11
58.56
63.41
65.17
64.78
63.96
63.17
65.21
65.78
50%
0.12
0.5
0.51
0.13
0.81
#01
0.04
1.08
65.21
63.41
65.83
67.91
65.81
65.19
65.98
68.31
68.02
65%
10.03
0.2
0.02
0.54
0.02
0.1
1.25
+.01
0.38
68.81
65.21
67.91
68.52
67.18
67.25
66.32
68.76
68.91
70%
10.04
10.01
0..03
10..4
10.03
10.01
10.1
10.5
0.00
68.41
68.01
68.54
68.81
68.03
68.14
68.19
69.92
69.42
75%
+0.02
0.5
0.46
0.02
0.15
0.25
10011
0.01
+ 0.05
68.12
68.01
68.78
69.12
69.1
68.91
68.19
69.92
69.93
80%
10.03
10.5
1.00
10.05
10.14
L0.
10.01
10.01
0.00
68.75
68.78
68.78
70.01
69.1
68.91
69.93
69.93
69.93
85%
1.055
+ 0.00
0.02
0.28
1000
1.25
+02
+.2
0.02
69.1
68.48
69.18
70.12
69.68
69.68
70.12
70.12
70.12
90%
10,78
1.255
0.34
10.43
10.03
10.03
10.03
1 10.03
1 L.003
69.91
69.91
69.91
70.12
69.91
69.68
70.12
70.12
70.12
95%
0.06
0.04
0.04
20433
1.044
1.00
+.003
1.003
0.03
29

---

Published in Transactions on Machine Learning Research (11/2025)
Table 6: arcsttcrch performance coresets of different sizes ImageNet-Ik identified by CLD. Each cell reports mean test accuracy (top) and standard deviation (bottom) over runs. Minimal accuracy drop (< 1%) is observed when transferring coresets from a smaller ResNet-18 model larger or different architecturos.
Coresel size (% of dataset)
Target Model
Source Model
5%
10%
25%
40%
50%
75%
80%
100%
46.91
54.83
60.21
66.83
68.93
71.12
73.04
73.21
ResNet-34
ResNet-18
0.03
+ 0.05
+ 0.12
+ 0.48
+ 0.01
+ 0.01
+ 0.05
+ 0.01
47.01
54.75
60.98
67.26
69.03
71.06
73.04
73.21
ResNet-34
ResNet-34
0.02
+ 0.35
+ 0.71
+ 0.10
+ 0.04
+ 0.02
- 0.01
+ 0.01
47.19
56.14
62.83
68.14
71.15
73.04
74.95
75.81
ResNet-50
ResNet-18
+ 0.01
+ 0.05
+ 0.13
+ 0.01
+ 0.03
0.07
+ 0.02
+ 0.05
48.10
57.03
63.14
68.91
71.25
73.57
74.95
75.81
ResNet-50
ResNet-50
0.05
0.03
+ 0.02
0.05
+ 0.04
0.01
+ 0.02
0.05
45.18
55.18
61.19
67.34
71.02
71.85
73.85
74.90
DenseNet-122
ResNet-18
+ 0.37
0.61
+ 0.01
- 0.04
+ 0.83
0.04
+ 0.02
0.37
46.07
55.88
62.01
67.91
71.36
72.18
74.04
74.90
DenseNet-121
DenseNet-121
+ 0.02
+ 0.03
+ 0.72
- 0.61
0.10
- 0.07
- 0.04
+ 0.37
43.12
53.18
60.12
66.17
70.37
72.01
73.98
74.70
VGG-19(bn)
ResNet-18
+ 0.04
+ 0.03
+ 0.02
+ 0.07
+ 0.01
- 0.04
+ 0.37
+ 0.71
44.01
54.12
61.01
67.09
70.25
72.58
74.05
74.70
VGG-19(bn)
VGG-19(bn)
0.50
+ 0.47
+ 0.31
- 0.16
= 0.25
- 0.02
- 0.04
- 0.71
D Additional Ablations
D.1 Stability across random seeds
We measure sensitivity to random initialization by computing per-example CLD scores across five independent seeds on ImageNet- (ResNet-18). The pairwise mean absolute error (MAE) between score vectors is consistently below 10 indicating negligible variance and high reproducibility; see Figure 7a
Seed 2 Seod 1
0.Ce+000
S.Ge DG
8, De 0G
8.5e
8.5e
x10
Moan Top-1 aceuracy over 5 seeds (9%)
70
S:ie:
0.0e+000
88
Scod 3
8/6--t66
Culle:
8:3c-10
80
.
Razdom
TODS
Sood 1
CCS (AUM)
DUAL
N.5e-tti
0.00-4000
84
F1.20
S1oCurv
Forget t
CLD
Glister
Full acc.
10.5%
Seed
8.5c.06
3.6e-00
8.3c.06
8.de-06
0.0c+000
Prusing
100%
Seed
Seed
2
Seed 3
Seed
Seed
Coreset size (% of full dataset)
(a) Seed reproducibilityy Pairwise MAE of CLD
(b) Subset size vs. accuracy. ImageNet-Ik, ResNet-18. shaded band marks the 0.5% tolerance below full-data accuracy.
The
scores across 5 seeds (lower is better).
Figure 7: Seed-wise stability and subsot-size trade-offs on ImageNet-1k.
30

---

Published in Transactions on Machine Learning Research (11/2025)
D.2 Minimum subset size for full-data accuracy
We quantify the subset fraction required to recover near full-data performance ImageNet-1k (ResNet-18). As shown in Figure 7b] CLD attains test accuracy within 0.5% of the full-data model using only 75% of the training set, on par with Di-Pruning and DUAL, and superior to other baselines we evaluated.
E
Detailed Explanation of Compute and Storage Cost of Coreset Methodologies
We denote the number of training samples by N (S ~ DV) and the number of query (held-out validation) samples by Q (V DQ). The model trained for T epochs. Certain TDA metries have hyperparameters (denoted Xz) used compute the TDA metric T, which may influence computational cost.
Recap of Notation
We measure computation in seatiig-poontt operations (FLOPs). Let flarge be the cost of single-ezample forward pass for the large model with Plarge parameters, and approximate the backward-pass cost as flrgee When a proxy (smaller) model is used, we write its per-example forward cost and parameter count as f and p. with f < flarge and p < Plarge-
R is the number of model retrainings (when applicable). is the coresct size; d the feature dimensionality; and B is the minibatch size (used during training but not appearing in per-example FLOP counts). Some methods perform subset reselection during training; when used, we denote the reselection interval (in epochs) by 7
Reference: full-data training (large model) Training the large model on all N points for T epochs costs 3NT flarge FLOPs and stores Plarge parameters (ignoring optimizer state). Totals below are the end- to-end cost to (i) select coreset size k and (ii) train the large model on that coreset.
Example scenario (used for all plug-in estimates). To further illustrate the computational efficiency of CLD, we provide approximale cost values by substituting the values of the parameters for finding and training 10% coreset on the ImageNet-1k 1k dataset.
N (99% of train), Q212,8 (remaining 1%), d-224x 224x2222224x3, c=100
Size of coreset k=126,836.
Proxy encoder:
ResNet-18 with p=11,689,122 128 and per-example forward FLOPs f=1181,,22,,111
Large model: ResNet-50 with 25 552,032 and 11188~=8,178,000 000,00000
We use the standard ImageNet recipe of prop epochs (Yang. 2017).
When method has additional parameters, the paper's choice (e.g., in Glister, 7=20).
E.1
Score-based Methods
E.1.1 Kernel Herding (Herding)
Herding (Chen et al. 2010) iteratively constructs representative subset by approximating the data distri- bution in an RKHS. At iteration f, it selects
get arg max (W;-1, ((E)),
EE
where (0-) is the kernel feature map and w is an RKHS weight vector. Repeating for k iterations yields size-k coreset.
Execution. One-time selection prior to training the large model (a proxy encoder is used to obtain features):
Train procy encoder: train a proxy model for T Tproxy epochs (forward cost S, parameters p). ii) Encode full dataset: extract features for all points using the trained proxy (Nf FLOPs).
in) Herding selection: for t=1:k, update candidate scores and select get using inner products in feature
space (explicit features of dim. give O(Nd) per iteration, i.c., O(Ndk) total).
iv) Train on coreset: train the large model the selected k points for Thate epochs (here Tia
31

---

Published in Transactions on Machine Learning Research (11/2025)
End-to-end compute (selection + coresct training).
ComputCHarding Harding
3NV proxx f +
NJ feuture extraction
((Nkk)
3k Tinto Starge
train proxy
iterative hording updates
train large coreset
Selection-stage storage overhead. Storing explicit features dominates; caching kxk Gram among selected points is optional:
StorageOOverheddeeddie Herding = O(Nd) (+ O(12) optional Gram)
Example scenario values (ImageNeet-k;; Tiate 990, -90)x
Compute (PFLOPs only) x0 622.663 + 2.306 + 280.061 = 905.031 PFLOPs,
proyy
NS
3k 3k fiarge
(negligible herding updates)
StorageOOeerhcadnaeetn (float32) x Nd 763.692 GB (features only).
E.1.2 Example Forgetting (Forgetting)
Forgetting (Toneva et al., 2018) measures, for each training sample, how many times it transitions from being correctly classified to incorrectly classified during training ("forgetting events"). Examples with higher forgetting counts are ranked as more informative.
Execution. One-time selection prior to training the large model on the coreset. Let Tearly be the number of early epochs run on the full dataset to collect forgetting statistics, and Tiate = T Tearly the remaining epochs used to train on the selected coreset:
i) Train on all N points for Tearly epochs while tracking, per example, the previous correctness bit and a forgetting counter (constant-time update per visit).
ii) Select the top examples by forgetting count; train the large model on this coreset for Tiate epochs.
End-to-end compute (sclection + coresct training).
Comput(rorgetting
3N Tearly ftarge
33 Thate flarge
collect forgetting on full data
train large on coresct
Selection-stage storage overhead. Streaming the metric requires only scalar counter (and correctness bit) per training example:
Storag@Oveheeadepppattt pargaatt O(N) (por-sample counter/bit /bit)
Example scenario values (ImageNeet-lk; Tearly=10, 10, Ta)
Computerorgersingg (PFLOPs only) Rs
311.178
248.943
560.122 PFLOPs,
3NT flarae
3kTTate flare
StorageOOerhheadppgertttgg/ (float32) R N = 5,073,420 B 0.005 GB.
E.1.3 Area Under Margin (AUM)
(Pleiss et al. 2020 scores each training sample by aggregating its margin over training (e.g., logit of the true class minus the max non-true logit), producing the area under the maryin across epochs/ /aahhs/updatss.... Higher absolute AUM indicates more consistently confident predictions; lower AUM can flag ambiguous or noisy samples. We compute AUM with a prozy model and then train the large model on the selected coresct.
AUM
Execution. One-time selection with proxy; the large model then trains the coreset for all T epochs:
32

---

Published in Transactions on Machine Learning Research (11/2025)
Train prozy & log margins: train proxy for T Tproxy epochs all N samples (per-cxample for- ward cost f, parameters p): recording each sample's margin as it appears in training (no extra forward/backward backward beyond training).
ii Compute AUM & select: for each sample, aggregate (c.g., sum/average) its logged margins to obtain AUM and select a size-k coreset according to the desired criterion (e.g., highest AUM, or filter low- AUM points).
ii) Train large on coreset: train the large model on the selected samples for T epochs.
End-to-end compute (selection + coreset training).
Compute
3 T proxy
3kT fiarge
train proxy (margin logging piggybacks)
train large on coreset
Selection-stage storage overhead. AUM can be streamed with rumning sum/count per sample:
StorageOverhheddgnn O(N) (running sums)
Example scenario values (ImageNet-Ilk; 1proxy-900 T=90).
Computeжии (PFLOPs only)
622.663
280.061
902.724 PFLOPs,
3NTyr
3kT ftarge
StorageOverheaduux (float32) N 4 5,073,420 = 0.005 GB.
E.1.4 Contrastive Active Learning (Cal)
Cal (Margatina al.] 2021 acquires unlabeled examples that are near labeled ones in feature space yet
differ in predictive probabilities (contrastive pairs), using nearest neighbors encoder features and simple divergenco-based ranking.
Execution. A one-time selection stage is performed prior to training the large model:
i) train a prozy model from scratch on the (growing) labeled set up to size (per-example forward cost f < fiarge);
ii) encode all unlabeled points with the trained proxy (here U=N); iii) run ktNlstyle neighbor search between the unlabeled pool and the labeled set (size k), and select coresct of size k.
(The divergence compulation is much cheaper than (ii)- and is absorbed into big-O.)
Overall compute (select once, then train-on-coresst).
Compute 3k proxy
+
Uf
3KT fiarge
proxy
encude unlabeled
KNN features
large on coreset
Selection-stage storage overhead. Storage overhead due to the cached features for the unlabeled pool
StorageOverteadga = OUUd) (praxy features)
Example scenario values: ImageNet-Ik; U=N).
Computeca (PFLOPs only) 2 62.267
+ 2.306
+ 280.061
3kT f
Uf (U=N)
3KT Juaze
344.634 PFLOPs.
StorageOverheadduu (float32) x Ud 763.739 GB.
33

---

Published in Transactions on Machine Learning Research (11/2025)
E.1.5
Gradient and Error L2 Norm-based Data Pruning (GraNd, EL2N)
GraNd (Paul et al. 2021 ranks training examples by the (expected) per-cxample gradient norm early in training:
(Z.) Es, Vo, E(O,. Zm 11
averaged over multiple random initializations and early epochs, then retains the top-k examples.
EL2N
Paul et al. 2021 ranks examples by the (expected) L2 error of prodictions carly in training:
EL2N((Em) = Es, (()m) - ymll2:
where Po are class probabilities and ym is the one-hot label. Scores are computed at small t and optionally averaged over R runs.
Execution. One-time selection prior to training the large model. Let Teadl be the number early epochs used for scoring and Tate= = early the remaining epochs used train on the coreset:
For each of R initializations, train on all N points for Tearly epochs (costing 3/ fargo) while logging per-example predictions.
(a) EL2N: compute scores directly from the logged predictions (no extra passes).
(b) GraNd: run an additional scoring sweep to obtain per-sample gradients (one forward one backward pass per example per early epoch).
ii) Average scores across runs; keep the top k; train the large model on the coresct for Thate epochs.
End-to-end compute (selection + train-on-coresee))
ComputegLax = 3N Tearly R fiarge
3k Tiate Suarge
early training (errors reused)
train large coreset
ComputeGrake
3N Tealy R farge
3N TNTaarty Tearly R Starge
3k Thate faarge
carly training
extra per-sample gradient sweeps
train large on coresct.
Selection-stage storage overhead. During scoring, running vector of N scalar scores need to be stored:
StorageOverheeadazzz = O(N),
StorageOverhhendarak O(N).
Example scenario values (ImageNNet-lk; R-10, 110, Tiate
Computezıas (PFT.OPs only) ~ 3.111.782
248.943
= 3.360.725 PFLOPs.
3NTaail R flees
34T Jtarge
Computcarana (PFLOPs only) ~ 3,111.782
3,111.782
+ 248.943
6,472.507 PFLOPs,
early training
extra gradient sweops
3kT Staron
StorageOverhead (float32) R N 4 = 0.005 GB for each.
E.1.6 Using Class Feature Medians (Moderate)
Moderate [iia et al. 2022 builds a representative coreset by selecting, within each class, the samples whose feature-to-center distances are closest to that class's median distance (thus avoiding both easy near-center rodundancios and far-out outliers). We compute class centers and distances proxy fcature space.
Execution. One-time selection with proxy, then train the large model the coreset for all T epochs:
34

---

Published in Transactions on Machine Learning Research (11/2025)
Train prozy: train proxy encoder on all N samples for T proxy epochs (per-example forward cost parameters p).
ii
Encode dataset: extract proxy features for all N samples (cost Nf FLOPs).
iii Class-median selection: for each class, compute the class center and all sample distances, then select. the per-class quota. samples whose distances are closest to the class-wise median (distance computation O(Nd); median/quanten selection O(N log N) or linear-time selection).
iv) Train large on coresed: train the large model on the selected samples for T epochs.
End-to-end compute (selection + coreset training).
Comput@Modorate
3N T. Proxy
f
NS
(Nd+ Nogg
3k T Starge
train proxy
feature extraction
class distances, medians
train large on coreset
Selection-stage storage overhead. The main storage overhead is from caching the features during selec-
tion
StorageOOvehhcadnddenn.. Moderate ((Nd) (proxy features)
Example scenario values (ImageNet-11; proxy -90, T-90).
Computenodarata (PFLOPs only) = 622.663 + 2.306 + 280.061 = 905.031 PFLOPs,
Nf
3KT fuage
StoraggOverrhcadndoaees (float32)= = Nd x 4= 763.739 GB.
E.1.7 Message Passing (D22 - -Prunigg
D'Pruiing MMharanaa et al. 2024 selects a coreset by balancing difficulty and diversity via message passing on a dataset graph built from proxy features. Tnitial per-sample difficulty scores (from the proxy) are diffused over kNN graph so that each example's score incorporates information from its neighbors; a graph- based sampler then selects a subset that covers diverse yet difficult regions.
Execution. One-time selection with proxy, then train the large model on the coreset for all T epochs:
i) Train prozy: train a proxy encoder on all N samples for Tproxy epochs (per-example forward cost f parameters p).
ii) Encode dataset: extract proxy features for all N samples (cost Nf FLOPs).
iii) Build graph: construct kNN graph over the features (e.g., ANN); cost O(N k: (or O(N dlog N)).
iv) Message passing & sampling: run H rounds of (forward/ revers)))) message passing the N x k edges to update difficult--wware scores, then sample size-k coreset (cost O(H N k) plus linear-time sampling).
v) Train large on coresel: train the large model on the selected samples for epochs.
End-to-end compute (selection + coreset training).
Compute Paruning = 3N T Tproxy] f +
Nf
O(Nk))+O(HN) +
3k T flarge
train proxy
feature extraction
graph build & message passing
train large
Selection-stage storage overhead. Caching proxy features dominates; the &NN adjacency is linear in N and smaller in practice:
AlorageOverheeadpppunns O(Nd) + O(Nx) (features + &NN graph)
Example scenario values (ImageNett-kk; T Tproxy=90 -90, 9'=0)).
ComputCnpPreasin (PFLOPs only) 622.663
2.306 + 280.061 905.031 PFLOPs, (excluding graph terms)
/
Ns
3kT furrnn
StoraggOOeereeahpppuitet (float332) Nd 4 763.692 GB.
35

---

Published in Transactions on Machine Learning Research (11/2025)
E.2
Optimization-based Methods
E.2.1 Gradient Matching Optimization (CRAIG)
CRAIG [Mizzasleeiman et al., 2020) selects a subset whose (aggregated) gradients closely match those of the full dataset across training, typically using a submodular (stochastic- objective over per-sample gradient embeddings at selected anchor epochs. We compute embeddings with proxy model and then train the large model the coreset for all T epochs.
Execution. One-time selection with a proxy, then train the large model:
i) Train prozy: train a proxy network on all N samples for Tproxy epochs (per-cxample forward cost f. parameters p).
ii Per-sample gradient embeddings anchors: every Tane epochs (anchors proxy Tane), compute for each sample the last-layer gradient embedding using only forward-pass outputs:
9 (P& 0 (7.) y:)
hg (Tt),
where he is the penultimate representation (dim. F) and yy B is the class class-probability error; this avoids backward passes (piggybacks on training). Selection then runs stochastic-greedy on these embeddings per anchor.
iii) Select & train large: union the anchor-wise selections to a size-k coreset and train the large model on it for all epochs.
End-to-end compute (sclection + coresct training).
ComputecRAra
3N T proxy
log(L/e)) Dett) Nog(/e)))
3k T flargo
train proxy (embeddings piggyback)
stochastie-greedy over anchors
train large on coreset
Here Det F-C' is the embedding dimensionality (penultimate features and class-probability error); the submodular arithmetic is negligible in FLOPs relative to training and is kept in big-O.
Selection-stage storage overhead. We stream anchor processing so only a single anchor's embeddings need be cached once:
Storage@Oeerheadquur (NN(F+C) (per-anchor embeddings, streamed per anchor)
Example scenario values (ImageNet-1k; 990 7'=00, F-512, C²=1000).
Computeceare (PFLOPs only; excluding big-O selection) s 622.663 + 280.061
902.724 PFLOPs,
3NT p
3 Fiare
Storagecurr (float32) s X1F+C)x4+ = 7.671 GB.
E.2.2
Generalization-basel Data Subset Selection for Efficient and Robust Learning (Glister)
Glister (Kilansetly et al. 2021b selects Sj of size k via mixed discrete continuous bi-level objective:
Cv())(S)) where (* = arg max LO;
arg max Sy S, 18, S
Execution. Glister replaces the training loop: it interleaves training on the current subset with periodic re-selection every 7 epochs (using stochastic-greedy with Taylor approximation).
Overall compute (train-on-coreeset). Training on the coreset over epochs costs 3KT flarge- Selection (MQ+N log(1/e)) fuoga T
across T epochs at frequency costs O
Hence
(kQ N log([/e)) flarge T
Compulegisster = 3kT Jarge + O
?
36

---

Published in Transactions on Machine Learning Research (11/2025)
Selection-stage storage overhead. Storage overhead is from validation caches:
StorageOverhenddgiserr = ( (validation cache)
Example scenario values:
Compute R3
280.061
60,017.420
3&T furge (PFLOPs)
(og)/)oo(()/) ftoge (PFLOPs; c=0.01, 7=20)
60,297.481 PFLOPs,
StorageOOverheaddaases
x 4 51,248 50 GB (< 0.05 MB).
E.2.3 GraphCut-based Data Subset Selection (GraphCut)
GraphCut [Iyer et al. 2021 selects Sj via the generalized graph-cut function
Σ s(m,j) mES jES,
Σ
s(1,,2), X 2.
Execution. GraphCut adds one-time selection stage prior to training (it does not replace training). The procedure is:
i) train a proxy model;
fi) extract features for all N training points using the trained proxy (per- Exer-example cost f < flarge); iii run (atochastic-)rredly. selection build a size-k subset..
(Similarity operations are typically much cheaper than (ti (Wi), so absorb them into big-O.)
Overall compute (train-on-cooeeset) One-time selection (including proxy training) + large-model train-
ing:
Computecrapatut 3N T proxy f
+
Nf feature extraction
O(N k)
+
3KT flarge
train proxy
grendy selection
train large coreset
Selection-stage storage overhead. Storage overhead is from storing pairwise similaritics
brarageOverhheaddrappcaa O(N2) (pairwise similarities kernel)
Example scenario values:
Computegrapacus (PFLOPs only) x 622.663
+ 2.306
+ 280.061 3k1 ftarge
3NT prowey
N
905.08 001 PFLOPs,
StorageOOverhexddapppaaa (float32):
full kernel peak 6,434.944 GB,
half kernel peak 3.217.493 GB.
E.2.4 Reconstructing the Decision Boundary (BoundarySet-Cos)
BoundarySet-CCS (Yang et al. 2024] selects samples near the model's decision boundary and then enforces coverage across distance bands. Distance-to-boundarry is approximated per sample by the minimum number of PGD steps required to flip its prediction; CCS (coverage-centric sampling) then allocates the coreset budget across bands to prescrve distribution coverage.
Execution. One-time selection with proxy, then train the large model the coreset for all T epochs:
37

---

Published in Transactions on Machine Learning Research (11/2025)
Thain prozy: train proxy network on all N samples for Tproxy epochs (per-example forward cost J. parameters p)-
ii) distance-to-bounddee (PGD): for each sample, run projected gradient steps until misclassification (cap at Kmax steps). the stopping step is k, define d(z) = k. Each PGD step requires one forward one backward; we use 3f per step our convention. Let K S Kmax be the average sleps per sample.
iii CCS selection: partition samples by d(x) E { m and allocate the size-k budget across bands (linear-imme bucketting and sampling).
iv) Train large on coreset: train the large model on the selected l points for all T epochs.
End-to-end compute (selection + coreset training).
Compute@oundarpe BoundarySet-OCS
3N Tproxy
+
3N K
+
3 Sargg
train proxy
PGD distanceto-bouundayy sweeps
train large on coreset
Selcction-stage storage overhead.
Storage is mainly through the scalar distance per sample during
selection:
pourgggg- ((N) (per-sample distance)
Example scenario values (ImageNet-lk; T 1rooxy=00,
Kmax=50 so K=50, T=90).
SounaytegoudaryySe--oss (PFT.OPs only) 622.663 + 345.924 + 280.061 1,248.648 648 PFL.OP,,
3NT
aNKf
3KT fuage
BenrageOvrrreeddddnnnagggeee (foat(3)) R N 4 0.005 GB.
E.3 Training Property-based Methods
E.3.1
Samples with Low Loss Curvature (SloCurv)
SloCurv (Garg & Roy 2023) scores each training sample by an input-loss curvature proxy computed at the end of (proxy) training. For sample Σm, with model parameters HT and random Rademacher directions u, scaled by h, the score is
R Σ
Curv(zm; = [=(0 Em + hr,) E(OT, 2m 11
R
Samples with the lowest curvature are retained to form size-k coreset.
Exccution. One-time selection prior to training the large model; a prozy model used for scoring:
Train prozy: train a proxy encoder with per-example forward FLOPs and parameters p for Tproxy epochs on all N points.
ii) Curvature scoring: at the end of proxy training, for each sample compute Curv(Em: (T) using R Hutchinson repeats. This requires (R+1) gradient evaluations per sample (one at En and one for each E. thwr), each costing ~ (1 fwd + bwd) x 3f in our convention.
iii) Train on coreset: select the k lowest-curvature samples and train the large model on this coreset for Thae= epochs.
End-to-end compute (selection + coresct training).
Computesiocur = 3N T Tproxy/ f +
3N (R+1)
3k Tiate flarge train large coreset
train proxy
curvature scoring at end of proxy training
Selection-stage storage overhead. Storage overhead is from keeping track of the running curvature values and the directions probed.
StoraggOverheadg.dau glaturr O(N) + O(Rd) (running stats + R probe dirs)
38

---

Published in Transactions on Machine Learning Research (11/2025)
Example scenario values (ImageNet-Ik; 190 190, R=10).
Compulesiocurv (PFLOPs only) ~ 622.663
+ 76.103
+ 280.061
978.827 PFLOPs,
procy
3NV (R+1))
Than finege
StorageOOverhexddsccom (float32) R N x 4+ Rd
x 4 11,094,540 B 0.011 GB.
E.3.2
Temporal Dual-Depth Scoring (TDDS)
TDDS [Zhang et al. 2024) builds a coreset by combining two temporal depths of signal from training with a prozy model. Depth computes, for each epoch, the projection of each sample's per-sample gradient onto the epoch's 's accumulated gradient direction. Depth 2 then aggregates these per-epoch contributions sliding window of length J and emphasizes their temporal variability (e.g., windowed variance). We maintain windowed statistics in streaming manner (constant-tine updates), so full trajectories need not be stored.
Execution. One-time selection with a proxy; the large model then trains on the coreset for the full T epochs:
Train prozy: train proxy for proxy epochs all N samples (per-example forward FLOPs parameters p); accumulate the epoch gradient direction.
ii) Per-sumple gradients: after each proxy epoch, run scoring sweep to compute per-sample gradients and their projections onto the epoch direction (costing one forward backward per sample); update the J-length windowed statistics and TDDS score (streaming).
iii Select & train large: rank by TDDS and keep the top-k; train the large model on these samples for all epochs.
End-to-end compute (selection + coresct training).
Compulerpos = 3N 3N TNToyoxy proxx
J +
3N T Proxy
3k7 T Starge
train proxy
per-sample gradient sweeps for TDDS
train large on coreset
Selection-stage storage overhead. Streaming TDDS requires J-length buffer of scalar contributions per example (and lemporary epoch-direction vector):
StorageOvarhcad = O(NJ) (windowed logs)
Example scenario values (ImageNet-1k; T-90,, T proxx==00, J=10).
Computemms (PFLOPs only) 622.663
622.663
280.061
1,525.387 PFLOPs,
f
per-sample gradient sweeps
3kT fuage
StorageOverhead (foot132) NJ x 4 50,734,200 0.051 GB.
E.3.3 Using Prediction Uncertainty with Proxy (Dyn-Unc, DUAL)
Dyn-Unc (He et al. 2024) measures prediction uncertainty via sliding window of length J over per-example target-class probabilities and averages the windowed uncertainty across proxy training.
DUAL (Cho et al. 2025) combines wncertainty with difficulty (window-mean prediction) and computes scores from an early stage of proxy training.
Execution. One-time selection with a proxy; the large model then trains on the coreset for the full T epochs:
Train prozy: train a proxy with per-example forward FLOPs f and parameters p for Trroyy epochs; maintain sliding-wint window statistics (length J) via O(1) updates per visit.
ii) Score & select:
39

---

Published in Transactions on Machine Learning Research (11/2025)
Dyn-Unc:
use windowed uncertainty (variance over the last // predictions), averaged over all proxy
epochs T proxy.
DUAL: use the product of windowed uncertainty and difficulty (window mean), averaged over the early proxy epochs Tproxy,early I Trrxxy.
Bela sampling (DUAL): apply pruning-ran ratioo adaptive sampling based on a Beta distribution stabilize extreme pruning. This adds negligible compute and storage. Train on coreset (large model): train for all T epochs on the top-k points.
ii)
End-to-end compute (selection + coreset training).
Computopya-ue pyn-une
3N T pruxx f train proxy logging
+
3k T flarge
train large
Computezux
3N T proxxy.early f
3k T flarge
early praxy scoring
triin large
Selection-stage storage overhead. Only require scalar window of values per example.
StorageOveerhcadggn--ee ONJ)),
StoragoOverheadduu. (NNJ).
Example scenario values (ImageNet-1k: T-90, I9roxy=90, T =500 J=10).
Computehyn-tec (PFLOPs only)
622.663
280.061
902.724 PFLOPs,
3NTpoosy'
3kT finrge
Computeuuu(PPLOOP only)
345.924
280.061 3&T fiagge
625.985 PFLOPs,
3N proxy
StorageOverhead (float32) NJ 0.051 GB for each.
E.4
Our Method Correlation of Loss Differences (CLD)
CLD builds a coreset by leveraging only loss values over training: it records the per-epoch losses of all training points and small held-out query set, then ranks training examples using the correlation of loss differences across epochs between train and query. No gradients or Hessians are required.
Execution. One-time selection with a proxy, followed by large-model training:
i) Train prozy: train a proxy model for prroxy epochs on all N samples (per-example forward cost f. parameters p).
ii) Collect losses: during proxy training, record per-epoch losses for all N training samples (no extra compute beyond the training pass), and run forward pass on all Q query samples each epoch to record their losses (Qf FLOPs per epoch).
iii) Score & select: compute CLD scores (corrclations of loss differences over epochs) and select sizc-k coreset. The arithmetic for correlations correlations/raakinng /ranking is linear-time in the number of stored losses and is negligible compared to FLOPs above.
iv) Train on coreset: train the large model on the selected points for T epochs.
End-to-end compute (selection + coreset training).
Compute = 3N T Prooxy/ f + Q Prooxy f
3k T flarge
(FLOPs)
train proxy
query loes collection
train large
40

---

Published in Transactions on Machine Learning Research (11/2025)
Selection-stage storage overhead. We store loss scalars for all N training and Q query samples across T prox epochs:
StorageOverheadda = (((+))) (loss logs)
Example scenario values (ImageNet-Ik;; prroxy=90 990, T-90).
Compute (PFLOPss only) 622.663 + 2.097
280.061
904.821 PFLOPs,
QT prony
SkTTate finegr
StorageOverheadqas (foott2)) (N+Q)T precyy x 4 461,220, B 0.461 GB.
F
Comparison of CLD with Influence
The impact measured by CLD closely aligns with the "inffuence" of individual training samples on model's predictions that are measured by Training Data Attribution (TDA) methods. TDA methods have been widely employed for tasks such as debugging datasets, interpreting models, and optimizing training efficiency Koh & .Liang (017);
Yeh et al. (2018); Feldman & Zhang (020))
The earliest TDA methods utilized Leave-One-OWl One-Out (LOO) training, which involves retraining the model after removing specific data points and observing the changes in performance. While straightforward, LOO retraining is computationally prohibitive for modern deep learning models due to the need for multiple retraining cycles Koh & Liang (2017). Recent TDA metries, such as F"Z-Influence (Infl) Feldman & Zhang (2020) and Datamodels Ilyas et al. (2022), have gained popularity owing to precomputed scores for widely- used datasets in computer vision. These methods, however, face scalability challenges.
A prominent alternative that arose was Influence Functions, which estimated the effect of downweighting individual samples using first-order (gradient) and second-order (Hessian) computations Koh & Liang (2017);
Basu et al. (2021) performed at the end training. Methods like RandSelect Wojnowicz et al. (2016) and Arnoldi iterations Schioppa et al. (2022) improved computational efficiency by approximating the Hessian. Similarly, TRAK Park al. (2023) combined random projections, gradient-based methods, and ensembling to estimate the influence of training samples. However, these approaches often rely on strong assumptions, such convergence to a unique optimal solution, which limits their applicability to neural networks. Ad- ditionally, Hessian computations introduce significant computational overhead. To address these challenges, unrolling-based methods that observe the learning process across training iterations have been proposed. These techniques approximate the impact of samples by differentiating through the optimization trajec- tory Hara et al. (2019). Among these, TracIn Pruthi et al. (2020) is a highly efficient method that estimates inffuence using gradients tracked throughout training. Its practical implementation, TracInCP, uses inter- mediate checkpoints to alleviate computational burdens. While effective, unrolling methods require storing intermediate training states, leading to high storage and computational costs.
In contrast, CLD solely relies on loss trajectories rather than first- or second-order quantities (e.g., gradients and Hessians).
In order to measure the "influence' of a training sample Em on an individual unseen (or query) sample zy we modified Definition slightly to be
CLD (Zm, kg) = p (Am. Bo)
(64)
We will now compare the impact measured by this metric (CLDine1) to the influence measured by TDA metrics, by utilizing the linear datamodeling score (LDS) introduced by Park et al. (2023). LDS measures the correlation between group-level attribution scores (CLDins1 or influence) and their observed impact on model predictions when subsets of training data are used.
LDS definition For a query data point Zg: random subsets {8,) are sampled from the training dataset, where each subset S, contains [aN]] points, with a (0,1) as the sampling ratio. Each subset. Sy used to retrain the model R times with different initializations RR
R and training parameters X, resulting in
41

---

Published in Transactions on Machine Learning Research (11/2025)
0.25
Linear Datamodeling Score (LDS)
TRAK
Infl Dataxodele
Arnoldi aati
TRAK
Infl
Arnoldi Randon
CLD
0.20
TracIn
Frac. of Prediction Flips
11
TracIn
Datamodels
0.15
0.10
0.05
0.3
0.5 0177 0.9
No. of Training Samples Removed
(a) Linear datamodeling scores (LDS) of existing TDA metries compared to CLD The scores were evaluated on CIFAR-10, ResNet-9 with 200 (randomly selected) query samples evaluated over 100 subsets.
(b) Prediction brittleness of CLD and TDA metrics on CIFAR-10 with ResNet-9. The top-k influential training samples were removed, and the average prediction flips for 200 query samples over 5 seeds are shown.
Figure Comparison of CLD to TDA metrics. (Best viewed in color.)
the model 03 Sy.E. This trained model is then used to compute a measurable quantity f(zg, OJ, (,, A group allribution score, (r S), is caleulated U Sj, = Exes, T S, where T 2, is the attribution score for training point 5 with respect to Ey. The LDS is then obtained using Spearman's rank Spearman 1904) correlation (p.):
E (Eg 05 (1
LDS = ps P
(or ( Sy. S),
(65)
We compared the LDS scores of CLDiari against those of TRAK, Arnoldi, TracIn, Infl, and Datamodels. Precomputed scores for Infl and Datamodels were used for the CIFAR-10 dataset Krizhevsky et al. (2009) with ResNet-9 He et al. (2016), while 10 models were trained for TRAK, Arnoldi, TracIn, and CLDinfi The evaluation employed C' 100 random subsets, sampling ratios a rang- ing from 0.3 N query set of 200 samples, and R 10 seeds. The measurable quantity was the
Experimental Setup:
accuracy of query samples.
Results and Observations: The results presented in Figure 8a reveal that while the impact captured by CLD is distinct from the influence measured by traditional TDA metrics, it aligus closely with mot such as TracIn and TRAK in terms of behavior while being resource-efficint.. Notably, the performance gap between these computationally intensive methods and CLDias1 narrows as a increases. The drop in LDS scores al a a= _ due to the stochastic nature of model retraining
Takeaways: Although CLD (and in essence CLD) fundamentally differs from influence-basee based TDA met- rics, it mirrors their trends at higher sampling ratios while maintaining superior computational efficiency, solidifying its utility practical tool for analyzing training dynamics.
F.1 Importance of the Top-k Samples
We demonstrate that the samples identified by CLD are indeed pivotal for generalization, addressing the question: "Are the training samples with the top-k scores truly the most critical for forming a coreset?" This is evaluated using the prediction brittleness metric. This is also mentioned briefly in Section
Experimental Setup: To quantify the influence of top-k samples, we systematically removed the most impactful data points identified by their CLD scores, from the training set and retrained the model. The metric interest was the fraction prediction flips observed in a held-out query set after retraining. If these samples are truly critical for generalization, their removal should cause substantial prediction changes. This experiment also included comparative analysis with the top-k influential samples identified by TDA
2hhis ohservation consistent with the findings previous research Karthikeyan Sogaard 2021)
al. 2024
42

---

Published in Transactions on Machine Learning Research (11/2025)
scores, discussed in this section. Experiments were performed on the CIFAR-10 dataset using ResNet-9 architecture, with randomly selected query set of 200 samples. For each configuration, once the top-k samples were excluded, the model was retrained 5 times to account for randomness, and the average fraction of prediction flips was recorded.
Results and Observations: The results, summarized in Figure illustrate that the top-k samples identified by CLD have a comparable influence on prediction outcomes to those identified by TDA-based metrics such as TracIn and TRAK. Notably, removing the top-800 samples of CIFAR-10, which constitutes just 1.6% of the dataset, results in prediction flips for over half of the query set. This highlights the significant role of the samples identified by CLD in supporting model generalization. While metrics like Datamodels and Infl exhibit greater impact, they are computationally prohibitive, rendering them unsuitable for large-scale coreset generation.
Takcaways: emerges as an effective and computationally efficient approach for identifying training samples critical to generalization, making it practical tool for coreset selection in large-scale machine learning pipelines.
43


Reviews:
[{"review_id": "cO2eAOYTu8", "reviewer": "Reviewer_rkSn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new way to construct coresets when there is access to a validation set which is close to the distribution of the dataset. They construct the coreset by considering samples which have a higher correlation to the loss difference between the training and validation trajectory. \n\nThe strengths of this contribution is enough ablations and empirical evaluation, an attempt to bridge the theory for the proposed method. The primary weakness is that the loss needs to be computed for all samples in the dataset for each epoch which might be an expensive excercise on large models.  Assumption 3 which assumes representativeess of the validation set is also hard to verify in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I like the paper overall especially the empirical evidence provided and the different analyses that the authors have done. \n\nI am a bit iffy about the theoretical results but I don't think the authors overstate their claims. I would request the authors to make the \"general\"ity of the theoretical results more exact - they are general in what sense?\n\nLet me highlight my reservations about the theoretical framework (I might be wrong and please clarify if possible):\n1. How does the size of the coreset, the size of the validation dataset affect the convergence? (Are these effects captured in the quantities $\\kappa$ and $\\delta$) I know the authors have some discussion in the appendix but a small remark might improve interpretability of the result. \n\n2. Further I don't know how reasonable the assumption 3 is -- is there a simpler way to verify it's existence. \n\n3.  Finally, is there a way to come up with a method which evaluates the loss on some fraction of the dataset and then computes an approximation of the metric? Is there a way to quantify it's error?", "audience_yn": "Yes", "audience_explanation": "Coreset selection is a relevant problem both in academia and industry and is especially very useful as we scale model sizes. Therefore I am sure a subset of TMLR audience will be interested in this paper.", "requested_changes": "1. Kindly add proof outlines for the proofs in the appendix to make going through them easier. The techniques are standard but still it helps a lot.\n\n2. I'd recommend putting table 1 in the introduction and referencing the papers for each - it greatly improves the readability (btw the table is super nice and helpful)\n\n3. I like the experiments a lot - but can you combine Section 6, 7, and 8 into a single nice numerical section and add an intro paragraph which explains your methodology and what you evaluate on exactly. This is optional but I feel can improve the readability. \n\n4. In Section 9 I would add the limitation that the loss needs to be computed on all samples which might not be possible with very large datasets. \n\n5. Add a remark on tradeoff in size of validation dataset and epsilon?\n\n6. Highlight in introduction the key motivation for which you provide evidence which is computing the coreset on smaller model and transfering it to a large one. (You mention this in passing but i think this should be highlighted more)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1wt2uQX0B8", "reviewer": "Reviewer_nKmJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes correlation of loss differences (CLD) a metric to assess the representativeness of a training sample. CLD works by comparing the difference in losses observed on a target point over the training trajectory relative to the difference in losses observed on a validation set. Higher similarity indicates a more representative sample. The authors use CLD as a coreset selection metric. Theoretically, they show that coresets with high CLD will also generalize better. Empirically, the CLD-based coreset selection scheme outperforms all baselines in terms of accuracy as well as many baselines in terms of efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, this is a well-written, solid paper with strong theoretical backing and compelling experimental results. Theorectically, the authors seem to make reasonable assumptions (even the strongest one, assumption 3, appears to be typical in the literature). The authors evaluate on Imagenet-1k which indicates applicability to large-scale settings.\n\nMy primary concern is the lack of a more thorough ablation study beyond the one in Figure 5 (unless I missed it). CLD is based on correlations of loss-differences; some questions that readers may have is how this compares to 1) the parameter gradient similarities between a point and the validation set along the training trajectory, 2) the loss values themselves over the trajectory (perhaps normalized in some way), 3) differences in the gradient norm instead of differences in the losses. Some of these do not have the same advantages of CLD: they are not architecture-agnostic and are less efficient, but comparing with these will provide some more insight into why CLD works so well.", "audience_yn": "Yes", "audience_explanation": "Yes; this paper would definitely be of interest to the coreset community, particularly given the strong empirical results.", "requested_changes": "**Would strengthen**\n- Additional ablations as outlined above\n- Text in Figure 6 is a bit small; enlarging the text would help readability", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6yOOyuWcFJ", "reviewer": "Reviewer_iHX7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of coreset selection, a subset of the dataset that preserves full-data performance while avoiding the heavy computation many prior methods need (e.g., gradients, Hessians, pairwise similarities). The paper proposes CLD (Correlation of Loss Differences), a simple coreset-selection metric that ranks each training example by how closely its loss-change trajectory over training correlates with a class-wise average loss trajectory computed on a held-out validation set; it needs only per-sample loss logs at a few checkpoints (no gradients) and then picks top-scoring examples per class. The authors provide a convergence analysis showing that training on high-CLD coresets approaches full-data population risk, with the gap controlled by an alignment term $\\omega$ and the representativeness $\\epsilon$ of the validation set. Empirically on CIFAR-100 and ImageNet-1k, CLD typically matches or outperforms strong baselines across subset sizes, transfers across architectures with minimal accuracy drop, remains stable when computed from early or subsampled checkpoints, and achieves favorable compute/storage trade-offs. Main caveats are the need to log per-sample training losses and to reserve a representative validation set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "> We introduce CLD, a simple and scalable metric for coreset construction based on the correlation between a training sample’s loss differences and the average validation loss trajectory, serving as a proxy for generalization\n\nThe authors discuss the metric in Section 4. The metric is described in sufficient detail with clear notation. The metric captures the correlation between the training _dynamics_ of the loss per sample and the _dynamics_ of the average validation loss. By capturing the correlation between the trajectory of loss evolution rather than say just the final loss, the metric captures the effect of the learning algorithm in addition to the informativeness of the sample. The metric is simple and scalable as it does not rely on additional quantities besides those already computed during training process. \n\n> We develop a general convergence framework showing that training on high-CLD samples yields population risk close to full-data training, with the suboptimality explicitly governed by sample alignment and validation representativeness\n\nIn Section 4, the authors present the central theoretical results showing that high CLD samples approaches the full-data population risk. The assumptions are stated clearly, and fairly reasonable.The proof provided in the Appendix appears to be sound and correct. The authors also explain the implications of the result in sufficient detail. \n\n> We show that on CIFAR-100 and ImageNet-1k, CLD-selected coresets typically outperform or closely match state-of-the-art methods across a wide range of subset sizes, and remain within 1% of more expensive baselines when not leading. \n\nThe experimental results in Section 6 cover CIFAR-100, ImageNet-1k using ResNet-18 as the scoring model. CLD is consistently the top or nearly top-performing approach on both datasets compared to other baselines (some of which use more gradients, etc). The coresets selected with CLD also transfer to other architectures matching the performance of the oracle specific architecture. The ablations in the Appendix further support the consistent results for CLD. \n\n> CLD avoids gradient and curvature computations, incurs minimal compute and storage cost, transfers across architectures via proxy models, and remains stable under checkpoint subsampling and random seeds, making it highly practical for large-scale settings.\n\nThe approach does not require expensive computations. It does require maintaining per sample loss over the course of training, which is discussed in the paper (along with ablations). The authors also provide a very well written summary of the computational and memory cost of the baselines in the Appendix. \n\nOverall, I found the paper to be quite well written and thorough. The claims are generally very specific and well supported.", "audience_yn": "Yes", "audience_explanation": "I am not an expert in coreset selection but the paper studies a well defined and interesting problems and proposes a clean and scalable solution to the problem with well done experiments and meaningful results. Data subset selection can have various applications beyond the multi-class classification results discussed in the paper, so I think the findings would be of interest to the ML community,", "requested_changes": "* (Minor) It would be good to have the full data performance for reference in Figure 2. The numbers are mentioned in the appendix but it would be good to have them in the main result. \n* (Minor) The paper is focused on the multi-class classification task. While that is good to have a focused scope, I think it would interesting to discuss the implications to other settings which are prevalent in the current research landscape (unsupervised learning - e.g. next token prediction, masked prediction, etc). \n* (Minor) The current experiments while thorough focus solely on vision datasets. The paper would be stronger if CLD's effectiveness could be demonstrated on other modalities.", "broader_impact_concerns": "The paper does not have a broader impact section and I don't think it needs one. Although it would be good to mention that procedures like coreset selection can potentially adversely affect performance on minority groups in the data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)