Noise-free Loss Gradients: A Surprisingly Effective Baseline for Coreset Selection
Authors: Saumyaranjan Mohanty
Chimata Anudeep
Konda Reddy Mopuri
Venue: Accepted by TMLR ()

Abstract: The exponential rise in size and complexity of deep learning models and datasets have resulted
in a considerable demand for computational resources. Coreset selection is one of the
methods to alleviate this rising demand. The goal is to select a subset from a large dataset
to train a model that performs almost at par with the one trained on the large dataset
while reducing computational time and resource requirements. Existing approaches either
attempt to identify remarkable samples (e.g., Forgetting, Adversarial Deepfool, EL2N, etc.)
that stand out from the rest or solve complex optimization (e.g., submodular maximization,
OMP) problems to compose the coresets. This paper proposes a novel and intuitive approach
to efficiently select a coreset based on the similarity of loss gradients. Our method
works on the hypothesis that gradients of samples belonging to a given class will point
in similar directions during the early training phase. Samples with most neighbours that
produce similar gradient directions, in other words, that produce noise-free gradients, will
represent that class. Through extensive experimentation, we have demonstrated the effectiveness
of our approach in out-performing state-of-the-art coreset selection algorithms
on a range of benchmark datasets from CIFAR-10 to ImageNet with architectures of varied
complexity (ResNet-18, ResNet-50, VGG-16, ViT).We have also demonstrated the effectiveness
of our approach in Generative Modelling by implementing coreset selection to reduce
training time for various GAN models (DCGAN, MSGAN, SAGAN, SNGAN) for different
datasets (CIFAR-10, CIFAR-100, Tiny ImageNet) while not impacting the performance
metrics significantly. Source code is provided at URL.

Published in Transactions on Machine Learning Research (04/2025)
Noise-free Loss Gradients: A Surprisingly Effective Baseline for Coreset Selection
Saumyaranjan Mohanty
ai23resch04001 1t:.:0000@ttttoo. @iith.oc.it
Deportment of Artificial Intelligence Indian Institute of Technology Hyderabad
Chimata Anudcep
chanuderp00@mmaii.ccom
Deportment of Computer Science BITS Pilani KK Birla Goo Campus
Konda Reddy Mopuri
Armopuri@ai. iith ac
Department of Artificial Intelligence Indian Institute of Technology Hyderabad
Reviewed on OpenReview: https: // openreview. net/ forum? id- 0E4PItW84Q
Abstract
The exponential rise in size and complexity of deep learning models and datasets have re- sulted in a considerable demand for computational resources. Coreset selection one of the methods to alleviate this rising demand. The goal is select subset from large dataset to train model that performs almost at par with the one trained on the large dataset while reducing computational time and resource requirements. Existing approaches either attempt to identify remarkable samples (c.g., Forgetting, Adversarial Decpfool, EL2N, etc.) that stand out from the rest or solve complex optimization (e.g. submodular maximization, OMP) problems to compose the coresets. This paper proposes a novel and intuitive ap- proach to efficiently select a coreset based on the similarity of loss gradients. Our method works on the hypothesis that gradients of samples belonging to given class will point in similar directions during the early training phase. Samples with most neighbours that produce similar gradient directions, in other words, that produce noise-free gradients, will represent that class. Through extensive experimentation, fectiveness of our approach in out-performing state-of-the-ar coreset selection algorithms on a range benchmark datasets from CIFAR-10 to ImageNet with architectures of varied complexity (ResNet-18, 18, ResNett550, VGG-16, ViT). We have also demonstrated the effective- ness of our approach in Generative Modelling by implementing coreset selection reduce training time for various GAN models (DCGAN, MSGAN, SAGAN, SNGAN) for differ- ent datasets (CIFAR-10, CIFAR-100, Tiny ImageNet) while not impacting the performance metrics significantly. Source code is provided al URL
have demonstrated the ef-
1 Introduction
Large-scale datasets have bocome essential for training state-of-the-art deep learning models. All the ap- plication fronts of Artificial Intelligence, such as Computer Vision, Natural Language, and Speech Process- ing, have accumulated massive datasets. Concurrently, the complexity of deep learning models has grown stcadily. Modern deep learning tasks (especially supervised ones) require extensive hyperparameter tuning during training achieve the best performance. These have resulted in an exponential rise computational requirement. (Schwartz et al. 2019) and carbon footprint Heikkill, Killamsetty et al. 2021a).

---

Published in Transactions on Machine Learning Research (04/2025)
Coreset selection aims to mitigate the above issues by finding the most representative data samples from the given training set. Mainly, coreset selection attempts to approximate the learning characteristics of the complete data (e.g,, the loss function) {Fekdman 2020). By obtaining a representative coreset with a cardinality of a fraction of the entire datasct, training duration and computational requirement for end-to-end training can be reduced significantly while delivering the desired generalization performance.
Forgetting (Tonvaa et al.) ) attempts to identify data points that are difficult to learn as the coreset. The idea presented is that model trained on "hard to classify" samples naturally generalizes onto the simpler samples. Similarly, training loss based methods attempt to select samples that contribute more to the training of deep neural networks. Gradient Norm (GraNd) and the Error L2-Norm (EL2N) scores introduced by Paul et al. (2023) prune significant fractions of training data without sacrificing much of the classifier's generalization performance. Section 2 comprehensively summarizes the existing works.
Multiple lines of thought based on loss gradients have emerged to compose corescts. Gradient matching basod methods aim to compose a coreset whose gradients (weighled combination) closely approximate the gradients produced by the entire training dataset. For instance, CRAIG (Mizzasoleiman et al. 2020 suggests finding the optimal coreset by converting the gradient matching problem to submodular function maximization and using a greedy approach to optimize it. Because of the individual weights (or learning rates) on the samples in CRAIG, it may not be straightforward to train the models on the coresets in the mini-batch gradient descent framework. Hence, the training speed gains are not maximal on such coresets.
Similarly, Gradmatch KKlllmsetty al. 2021a constructs an objective for matching the gradients computed over the coreset with that calculated over the complete dataset. They propose minimizing the matching error by casting the objective as weakly submodular maximization problem and solving it using an orthogonal matching pursuit (OMP) based greedy algorithm. As it identifies the coreset by selecting different subsets during model training to perform gradient matching, training must be carried out each time to obtain coresets at various fractions of the entire dataset.
In this paper, we propose a novel methodology for selecting a coreset based on the hypot that most samples from the same class produce gradients in similar directions (particularly in the early stages of the model training, claborated further in Section 3 Figures and Our method proposes that samples with substantial gradient similarity to many others belonging to the same class would be ideal candidates for constituting the representative coreset. This approach differs from the existing coreset ideas driven by 'distinctness' or 'difficulty', which prefer samples that give rise to noisy (or minority) loss gradients. Hence, we refer to the proposed method as 'Noisc-free Gradicnts' for coresct selection. In the case of a multi-class classification task, we identify representative samples wlass-wise and combine them into a coreset.
Our approach uses cosine similarity between gradient vectors to rank data samples according to their ability to represent other samples. Based on the desired coreset size, the top-ranked samples from each class are picked to represent the whole dataset. This makes the proposed approach computationally attractive compared to the optimization--rivon coreset idcas (Refer to Section 2 for detailed review of such works). Moreover, while the existing approaches, such as CRAIG, use a per-sample learning rate while training from the coreset, our method uses single learning rate for all the samples. We also introduce various steps to reduce the computational time. We have applied this coreset selection methodology to two popular and diverse application areas, namely, 1) Image classification and 2) Generative Modelling. The details are discussed in Section 4
In summary, the major contributions of our work can be summarized as:
Contrary to the existing 'diffiullt//distinnee /distintttssss notions, this paper introduces novel, intuitive coreset selection method for identifying class-wise representative samples driven by noise-free loss gradients.
Also, compared to the existing gradient-based corcset selection methods that are computationally demanding (e.g, submodular function optimization), our method is computationally efficient.
We thoroughly evaluate the proposed method over multiple object recognition datasets of varying complexity. Experiments with different deep neural network classifiers (CNN and Vision Trans-
2

---

Published in Transactions on Machine Learning Research (04/2025)
former) and cross-architecture generalization studies demonstrate that method achieves higher accuracy than state-of-the-art coreset selection methods.
We apply the proposed method in the context of GAN training and demonstrate consistent perfor- mance improvement over existing methods.
2 Related Works
Multiple coresct selection methods have recently been proposed. Here, we briefty present the most prominent works.
Toneva et al. 2019 proposed coreset selection through catastrophic forgetting. During the training, they defined a flip in classifying a training example from the correct label to an incorrect one as a 'forgetting" event. Removing unforgettable examples results in minimal performance drop while speeding up the training and hyperparameter tuning.
K-Center Greedy approximation (Sener Savarese, 2018 attempls to solve the minimax facility location problem to select coresets from large dataset such that the maximum distance between points in the non-coresct and its closest point in the coreset is minimized.
Cncerlainty-based methods work on the idea that samples having lower confidence may have a higher impact during training than those with higher confidence. Thus, these methods suggest constituting the coresets with the samples with lower confidence. Commonly used metrics to calculate sample uncortainty are least confidence (Shen al. 2018), entropy (Settles 2012), and margin (Coleman et al. 2020).
Adversarial Deepfool Ducoffe & Precioso 2018 and Contrastive Active Learning Margatina et al. 2021 work to find data points distributed near the decision boundary. Samples that lie close to the classification boundary are treated as difficult samples to learn. Error-based methods try to select samples that contribute more to the training of the neural networks. Two metrics called the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores are introduced by Paul et al. (2023) that help in pruning significant fractions of training data without sacrificing test accuracy. GraNd mcasures the importance of each sample the training loss early epochs. ELΩN approximates the GraNd score, which measures the norm of the error vector, with higher score indicating higher potential influence. The authors conclude that the images with higher scores tend to be harder to learn (forgettable examples). Their method chooses these forgettable samples as the coreset.
RETRIEVE (Killamsetty et al. 2021d formulates coresct selection for Semi-Supervised Learning (SSL) as a bi-level optimization problem. This method considers labelled and unlabeled sets to develop the bi-level optimization problem. It uses a greedy algorithm select the coreset that minimizes the labelled set loss. GLISTER (Killamsetty et al.) (2021b), Generalization--aasd Data Subset Selection for Efficient and Robust Learning, applies bi-level optimization for supervised and active learning. It formulates corcset selection bi-level optimization problem that maximizes the log-likelihood
held-out validation dataset.
Gradient matching- based methods work on the expectation that the optimal coreset can approximate the gradients produced by the entire training dataset. CRAIG [Mirzasoleiman et al. 2020 selects representative subsets that closely approximate the entire gradient. They achieve this by converting the gradient matching problem to optimizing a submodular function using a greedy approach. Gradmatch (Killamsctty et al., 2021a) method follows similar approach. It introduces a squared L2 regularization term over the weight vectors and uses a greedy Orthogonal Matching Pursuit (OMP) algorithm to select the coreset iteratively. After training on the resulting coresct for a pre-determined number of epochs, the algorithm repeats the coreset construction using the latest iterate.
Several methods have utilized submodular functions Liu 2020 to leverage their ability to measure diversity and information, then use greedy algorithms maximize a submodular function for coresct selection. Iyer et al. (2021) have studied various submodular functions such Graph Cut, entropy, and facility location and utilize greedy optimization algorithms for data summarization.
3

---

Published in Transactions on Machine Learning Research (04/2025)
Xia et al. (2023) have proposed a universal method termed 'Moderate Coreset' to tackle the task-specific nature of dataset selection. They calculate the distance between the hidden representation of samples and the representational class centres. Based on these Euclidean distances, they rank the data points in an ascending order and select the data points closest to the distance median as a coreset.
Yang et al. 2023) proposed a parameter influence-based optimization for dataset selection. They argue that not all training data contributes uniformly to the model's parameter learning. So, an influence function- based iterative method can be designed to linearly estimate the impact of omitting a specific subset the model parameters. However, they have not considered lower coreset sizes than 50% of the original dataset size.
Sinha et al. (0199) introduced the application of coreset selection to sub-sample a larger batch to produce a smaller batch to speed up GAN training significantly while not sacrificing performance to a significant level. DeVrics ct al. (2020) introduced instance selection based upon the high-density region of the data manifold
speed up GAN training.
Guo et al. 2022 has developed excellent and comprehensive code library named DeepCore that im- plements current popular and stato-of-the-art coresct selection methods in unified framework based on PyTorch (Paszke et al. 2019 DeepCore framework has provided unified comparative analysis among 15 algorithms for CIFAR-10 and 14 algorithms for ImageNet-1K. The algorithms compared are: Contex-
tual Diversity (Agarwal et al., 2020 Herding (Welling 2012), ; k-Center Greedy (Sener & Savarese 2018)
Least Confidence, Entropy, Margin Coleman et al. 20200) Forgetting (Toneva al.) 2019), GraNd (Paul
et al. 2023), Cal (Margatina et al., 2021), DeepFool (Ducoffe & Precioso 2018), Craig (Mirzasoleiman al.
2020 GradMatch (Killamsetty et al. 2021b), Glister (Killamsctty et al.) (021b), Facility Location and GraphCul (Iyer al.] 2021). Due to its large running time, the DeepFool algorithm is not considered for the ImageNet-1K dataset. We have also utilized publicly available implementation of Moderate Coreset (Xia let al. 2023) for CIFAR-10, CIFAR-100 and Tiny ImageNet. datasots and presented the comparative analysis
in Section
3 Methodology
This section provides a detailed description of the proposed Noise-free Gradients approach. We first focus the supervised classification task of object recognition using Deep Neural Network classifiers (CNNs and Vision Transformers) because of their proven effectiveness. We have also applied our proposed method to speed up GAN training without sacrificing the performance of the model significantly. Table provides the notation used throughout the paper.
Table 1: Notation
Symbol
Description
V
( 1))
Large Training Dataset Coresct of V (target)
C V
Parameters of the classifier
Threshold on the gradient similarity for neighborhood identification Training data belonging to class c
Va
9..
Loss gradients computed for data sample (æt, yi) at the last fully connected layer 12 norm of 4 norm of x Cardinality of set A Indicator function Dot product operator
|x]||
|
1
<..>>
Our objective is to select a representative subset S the complete dataset such that model g trained on $ has generalization performance close to that of the model 0² trained V. Training deep neural networks
4

---

Published in Transactions on Machine Learning Research (04/2025)
is reduced to empirical risk minimization problem often optimized in the gradient descent framework. In practice, the incremental Gradient (TG) methods, such as Stochastic Gradient Descent (SGD), iteratively estimate the gradient on mini-batches of training data that construct the parameter updates.
Existing gradient--asse based coreset selection methods such as CRAIG Mirzasoleiman et al., 2020 and Grad- match Killnnsttl al. 2021ap try to find an optimal coreset such that the weighted sum of the gradients of the coreset elements remains within an error margin of the gradients of the entire dataset. 'hee objective function can be written
arg min F s0. S
Ini
Σ
(1)
V
|
(E
Where is the vector of weights associated with the elements of the coreset, and F is distance metric. It can be re-written as:
Eg
Ix. = Es
Σ
+E
(2)
|||||
V
Where is the error term having the same dimension as the gradient vector, note that equation 2 the expected value of the approximation error in the parameter space.
considers
Our method identifics the representative samples based on their gradients. Intuitively, data samples whose gradients are similar most other samples best approximale the complete dataset. Tn other words, these representative samples result in local minima close the minima achieved by the entire dataset.
Our method thus selects subset S of desired cardinality that closely approximates the whole dataset V. We define the normalized gradient similarity between two samples
De. Izy
plz, Yc, Zz.35.
(3)
192. 11
For each sample x; in the dataset V, we can mcasure its ability to represent V
as
f(x;) = Eg
Σ
plai,
Mer, Di. Tj. Mi 0)
(4)
RYEV. J4i
Using this measure, we compute the suitability of every sample in V become an element of the coreset S. Essentially, this translates to sorting the dataset samples in the decreasing order of this measure and composing the coreset of desired cardinality. The exact steps involved in this process are described in the next paragraph.
We start with a randomly initialized model and update its parameters on the complete dataset for a few epochs. In our experiments, we observe that generally, 5 10 epochs (during which the loss value doesn') 't platcau) are sufficient. We save the checkpoints of the model parameters after each epoch. For cach of these checkpoints, we calculate the gradients of the loss function with respect to the model parameters computed at each data sample. We score the dataset samples based on their ability to represent other samples denoted in equation 4 We aggregate these scores across the saved checkpoints as an approximation to the expectation over the parameter space.
Figure shows the histogram of pairwise cosine similarity gradients of images belonging to airplane class for CIFAR-10 dataset on ResNet-18 architecture for initial four epochs. As we can see, during initial training, the most images have gradients with high cosine similarity values, indicating moving in similar directions. Figure 2 shows the histogram of pairwise cosine similarity of gradients of images belonging to (i) airplane
5

---

Published in Transactions on Machine Learning Research (04/2025)
Epoch:
10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
=10*
Epoch: 2
10
Epoch 3
10
-10 0.1
02
0.3
0.4
0.5
0.8
0.7
0.8
0.9
Epoch:
10
Figure 1: Histogram of pairwise cosine similarity of gradients during the training of ResNet-18 for the initial four epochs. Samples belong to the 'airplane' class of CIFAR-10 dataset. A strong similarity (or alignment) of the gradients can be observed.
Epoch Intra class (Airplane)
-0.8
-0.4
-0.2
100
Epoch 1: between Airplane class and Bird class
-0.2
100
Epoch 1: between Airplane class and Truck class
Figure 2: histogram of pairwise cosine similarity of gradients of images belonging to (i) airplane class (top panel), (ii) airplane and bird class (middle panel), and (iit) airplane and truck class (bottom panel). While intra-class samples display strong similarity, inter-class samples have gradients in different directions.
class (top panel), (ii) airplane and bird class (middle panel), and (iii) airplane and truck class (bottom panel). All the computations are done for the model saved after the first epoch. It indicates that images belonging to the same class will have gradients in similar directions, while images belonging to different classes will have gradients in different directions. This behaviour forms the basis of our method for computation of the coreset.
To improve the time efficiency of our approach, we slightly modify to utilize nearest-neighbor search algorithm. We utilize radius-based neighbour learning to obtain the local density of data points. The per-

---

Published in Transactions on Machine Learning Research (04/2025)
sample scores denoted by equation are measured in terms of the number of dataset samples present nearby in the gradient space. A nearest neighbour search algorithm finds the number of samples within given radius from each sample (in other words, with similarity more than a threshold Φ and assigns it as its score. We then rank the samples according to their aggregated scores across multiple checkpoints. The top-ranked images are selected as the representalive coreset.
The formulation of Noise-free Gradients can be represented as shown in equation 5 For given class e, the top-ranked samples 2j selected based their scores.
Tj arg max ΣΣ (1, +)
(5)
(E. Vo iti
Algorithm
1
presents approach more formally.
Algorithm 1 Noise-free Gradients for Coreset algorithm
Require: Train set: V; Total epochs: T; number of classes: C; number of coreset images per class: N;
Model checkpoint after initial T epochs:
Ensure: Coreset $
for class c in I...., do for (zi, 36) E Ve do for epochs in 1,... T do compute %
3:
4:
5: end for
6:
f(x:) = Σ E
0.) > +)
Oe (e).s))EVoJas
Store f(x;)
end for end for
10: $ =0 0
11: for class e in I,..., C' do
12: S ww
b Descending order
13: end for
4 Implementation of Noise-free Gradients
We have implemented the proposed method Noise-free Gradients using the PyTorch framework.
4.1 Gradients with respect to the Classification layer
Deep learning models such as ResNet (He al.] 2015 2016), VGG (Simonyan & Zisserman 2015 and VIT Small (Lee al.] 2021) have millions of parameters, and it is impractical consider the gradient of loss function with respect to cach of the parameters. It is obscrved that the gradient mostly captures the variation of gradient norm with respect to the parameters of the last (classification) layer of the neural network (Katharopoulos & Fleuret 2019). Similar to the earlier works (Ash et al., 2020 Killamsetty et al, 2021a), we avoid computation of the gradient of the loss function with respect to all the model parameters, restricting to only the final fully connected layer while calculating the score of data samples (equation 4p.
4.2 Nearest Neighbor algorithm
The complexity of similarity score computation (equation 4 among the samples of a particular class of size N O(N2). The complexity and computation time are linear in the number classes in the dataset C. Instead of computing N similarity scores for each sample given class, our approach finds the number of these samples within close neighborhood. This vital modification saves nontrivial complexity. We have utilized the radius-based nearest neighbor algorithm implementation from scikit-learn (Pedregosa et al.

---

Published in Transactions on Machine Learning Research (04/2025)
2011) library with the proposed distance (or similarity) measure mentioned in equation 3 as the metric for identification of neighbors within a given radius. Pairwise cosine similarity function of torchmetrics library is used. This results in a 2 30X speed up of the computation time compared to the naive method. We have used a threshold value (+) of 0.2 in all our experiments. For the ablation experiments on the effect of this threshold, readers can refer to the supplementary document (Section 2).
5
Experiments and Results
5.1
Experimental Setup
Applications.
We have evaluated our coreset selection methodology image classification and GAN
training.
Datasets. For image classification application, we evaluate the effectiveness of our method on four popu- larly used benchmark object recognition datasets, i.e., (FFAR-10(Krizhevsky 2009),
2009), Tiny ImageNet (Le & Yang 2015) and ImageNNt-11K (Russakovsky et al. 2015). CIFAR-10 dataset consists of 50, 000 colour images of dimension 32 32 3 from 10 different classes, each class having 5, 000 images. CIFAR- 100 dataset consists of 50, 000 training images from 100 classes with 500 training images per class. Tiny ImageNet dataset consists of 100, 000 training images from 200 classes with 500 training images per class. ImageNet-1K is a subset of the larger datasel ImageNet, an image dataset organized according to the WordNet hierarchy. ImageNet-1K consists of 1000 classes, with 1, 281, 167 training images and 50, 000 validation images. In Summary, to test the robustness of our method, we have considered four popular classification datasets with the number of classes varying from 10 to 1000.
Classifier. We have used randomly initialized Res (He et al. 2016). ResNet-50 (He et al. 2015), VGG- 16 (Simonyan & Zisserman, 2015) and ViT Small {Lee al., 2021) architectures to carry out comparative performance analysis of existing SOTA algorithms against our method. The first three are CNN architectures that vary in complexity and number of parameters, and the fourth is vision transformer. We have also carried out cross-architecture generalization to demonstrate the effectiveness of our method.
GAN models. For GAN training application, we evaluate the effectiveness of our method on four different GAN architectures i.e. DCGAN (Goodfellow et al. 2014), MSGAN (Mao et al., 2019 SNGAN Miyato
ct al. 2018) and SAGAN Zhang et al. (2019).
GAN Metrics. We have considered the Inception Score (IS) (Salimans et al 2016 and Fretchet Inception
Distance (FID) (Heusel et al. 2018) for comparing the performance of our method against existing coreset selection for GAN methodologics.
Baselines We have considered the results reported by DeepCore benchmark wherever available and utilized DeepCore implementation to carry out experiments wherever results are not available We have utilized publicly available implementation of Moderate-DSe, and instance selection for GAN Abbreviations used in results for different methods which are part of DeepCore are: GC(GraphCut), F(Forgetting), R(Random). FL(Facility Location), GN (NraNdd, CR(CRAIG), H(Herdingg.
Implementation Experiments for CIFAR-10 and CIFAR-100 are carried out for ten individual runs with different random seeds. Experiments for tiny ImageNet were carried out for five separate runs, and Ima- geNet1K were carried out for two individual runs due to substantial computational roquirements. In cach iteration, number of training epochs were set at 200, and an SGD optimizer with a momentum of 0.9 was used with a batch size of 128 and a learning rate of 0.1. For ImageNet- IK, batch size of 256 was used.
For GAN training, we have used default settings for all the GAN models recommended by their respective authors. For instance selection, 50% retention ratio is used. For our method, 50% coreset selection percentage is used. FID scores are lower the better, and Inception scores are higher the better. The number of steps used for full dataset training is 200k, and the steps used for coreset-based training is 100k.
https:
https: htt.://a.ccccmmaaaatatc///////op http:://ihhbbbooo/aarrrdl/l/gggp
httt::///tthbb.ccmm//llaaa///// ICLR MDderate-DS
http:://gthhbb.com//uuueee wiw.rrgggnnnnttneeee seapiieeeeeeeennnnneeeeeeeeeeeee for_gan
8

---

Published in Transactions on Machine Learning Research (04/2025)
5.2 Results for Image Classification
5.2.1 Results for CIFAR-10
A comparison of accuracy values obtained from DeepCore, Moderate-DS and Noise-free Gradients on the CIFAR-10 dataset for various fractions of the original dataset is presented in Table 2 Table 3 Table and Table for ResNet--18, ResNet-50, VGG-16 and VIT Small architecture respectively. The algorithm with the best accuracy at each percentage level reported by DeepCore mentioned within brackets against the accuracy value. The average rank of a method is calculated as the mean of its rank across different fractions of the dataset considered. As can be observed from the values, our method outperforms all the cxisting SOTA algorithms across architectures of diverse complexity and depth.
Table Comparison of results for CIFAR-10 ResNet-18
Table 3: Comparison of results for CIFAR-10 ResNet-50
Size
DoepCore
Moderate-DS
Our Method
Size
DeepCore
Moderate-DS
Our Method
0.5%
34.90 + 2.30 (GC)
33.93 1.10
43.64 0.87
0.5%
32.85 + 1.20 (GC)
13.73 + 4.82
47.32 0.69
1.0%
42.80 + 1.30 (GC)
39.83 0.56
54.06 = 0.92
1.0%
36.58 + 1.50 (GC)
18.86 + 1.55
56.06
0.10
5.0%
65.70 + 1.20 (GC)
69.20 = 1.30
72.96 + 0.28
5.0%
63.31 + 1.53 (GC)
53.89 + 1.03
78.04 = 0.32
10.0%
(GC)
10.0%
(P)
78.06
20.0%
87.10 + 0.50 (R)
86.49 = 0.29
87.18
+ 0.45
20.0%
85.06 + 1.48 (FL)
84.00 1:079
86.11 + 0.48
30.0%
(F)
89.62
30.0%
(F)
89.22
Rank
2.17
2.67
1.17
Rank
1.83
3.00
1.17
Table 5: Comparison results for CIPAR-10 on VIT small
Table Comparison of results CIFAR-10 on VGG-16
Size
DeepCore
Moderate-DS
Our Method
Size
DoepCore
Moderate-BB DS
method
0.5%
30.05 + 1.21 (GC)
23.10 + 1.41
35.22 + 0.31
1.0%
37.31
0.52 (GC)
27.00 4.80
38.50
+ 0.27
0.5%
32.85 + 1.2 (GC)
13.73 #4.82
47.32 = 0.69
5.0%
53.96 : 0.45 (GC)
51.30 = 2.30
56.82
f 0.20
5.0%
1.0%
10.0%
20.0%
63.31 + 1.53 (GC)
85.06
36.58
70.64
1.48 (FL)
1.5 (GC)
(F)
53.89 #103
84.00 +0.79
18.86 1.55
1183
56.06
73.04
86.11 + 0.48
= 0.32
0.10
20.0%
30.0%
10. 0%
64.98 + 0,90 (GC)
67.38 1: 1.01 (GC)
57.43 + 0.09 (GC)
66.10 1.66
54400 = 2.35
70.40 0.28
71.43
67.16
58.32
+ 0.52
+ 0.16
+ 0.49
30.0%
89.83
+0.3 (F)
+0.27 88.74
89.22 +0.2
50.0%
40.0%
71.24 + 0.20 (GC)
72.98 + 2.10 (F)
72.60 1.40
73.10 4.09
78.07
75.45
+ 0.68
1 0.70
Rank
1.67
3
1.33
60.0%
(GC)
76.95
Rank
2.33
2.55
1.11
5.2.2 Results for CIFAR-100
A comparison of accuracy values obtained from DeepCore, Moderate-DS, and Noise-free Gradients on the CIFAR-100 dataset for various fractions of the original dataset is presented in Table 6 Table Table and Table for ResNet-18, ResNet-50, VGG-16 and VIT Small architectures respectively. The algorithm with the best accuracy at each percentage level reported by DeepCore is mentioned within brackets against the accuracy value. The average rank of a method is calculated as the mean of its rank across different fractions of the dataset considered. We outperform all the other methods for training on ResNet-50, VGG-16 and ViT Small architecture while performing better for lower selection percentages (up to 20%) for training on ResNet-18 architecture.
5.2.3 Results for Tiny ImageNet and ImageNet-1K
We believe the effectiveness of coreset selection algorithms must be tested in the face of complex datasets. Hence, evaluate our algorithm with the Tiny ImageNet and ImageNet-1K datasets. Similar to other experiments, we train multiple models on the resulting coresets. Each model is trained for 200 epochs with a random PyTorch seed. The comparative analysis is labulated in Table T0 11 12 and 13 The algorithms with the best accuracy at each percentage level reported by DeepCore are mentioned within brackets against the accuracy value. The average rank of a method is calculated as the mean of its rank across different fractions of the dataset considered. Our method consistently outperforms all the other coresct selection algorithms.

---

Published in Transactions on Machine Learning Research (04/2025)
Table 7: Comparison of results for CIFAR-100 on ResNet-50
Table 6: Comparison of results for CIFAR-100 ResNet-18
Size
DeepCore
Moderate-DS
Our Method
Size
DeepCore
Moderate-DS
Our Method
0.5%
20.0%
1.0%
10.0%
41.98 + 0.88 (GC)
31.18
13.01 + 0.22 (GC)
56.68 + 0.98 (GC)
8.49 + 0.68 (GC)
0.37 (GC)
32.57 1.40
7.78 + 0.20
54. 0.19
5.36 #0.12
46.48 + 0.28
58.06 = 0.34
18.95 = 0.08
12.0 = 0.11
5.0%
0.5%
1.0%
30.0%
10.0%
20.0%
48.14 + 0.23 (GC)
36.91 1: 0.18 (GC)
12.69 + 0.28 (GC)
25.7 = 0.55 (GC)
7.34 + 0.62 (GC)
56.1 = 0.32 (GC)
51.83 +0.52
57.79 #1611
22.74 +1.67
14.70 10:24
3.64 #0.62
5.48 =0.42
31.64 = 0.19
54.18 0.51
41.32 + 0.48
15.85 = 0.16
60.82 = 0.08
10.61 + 0.22
30.0%
64.05
0.28 (GC)
62.97 0.41
62.58 0.24
40.0%
62.52 + 0.23 (GC)
64.92 +0.93
65.01 0.24
40.0%
60.0%
73.80
69.37
+ 0.30 (GC)
0.31 (GN)
67.33 0.22
70.74 0.20
66.31 + 0.16
60.0%
50.0%
68.62 + 1,20 (GC)
73.34 = 0.39 (F)
69.11 #0.62
71.87 #0.91
68.67 + 0.17
71.10 + 0.38
Rank
1.63
2.63
1.75
Rank
2.33
2.33
1.33
Table 8: Comparison of results for CIFAR-100 on VGG-16
Table 9: Comparison of results for C-FAR-100 VIT Small
Size
DeepCore
Moderate-DS DS
Our Method
Size
DeepCore
Moderate-DS
Our Method
0.5%
2.45 + 0.57 (GC)
2.00 + 0.10
8.92 + 0.45
0.5%
7.45
0.17 (GC)
5.20 +: 0.06
11.03 +: 0.23
1.0%
4.47 + 1.28 (GC)
2.10 + 0.23
15.2 : 0.46
1.0%
14.57 + 0.47 (GC)
7.82 + 0.20
14.35
0.18
5.0%
27.23
0.70 (GC)
13.70 0.91
32.12 + 0.58
5.0%
24.50 + 0.71 (GC)
21.12 0.30
24.85
+ 0.30
10.0%
38.17
+ 1.32 (GC)
28.10 + 1.60
11.67 = 0.05
10.0%
32,00 + 0.31 (GC)
27.00 = 0.50
32.60
+ 0.30
20.0%
(GC)
49.90
53.95
20.0%
1.19 (GC)
38.51 0.47
41.30
30.0%
58.87
1: 0.45 (GC)
59.50
1: 0.30
61.23 + 0.17
30.0%
45.50
1: 1.01 (GC)
44.90 1: 0.31
48.12
+ 0.48
40.0%
62.52
(GC)
64.92
0.93
66.17 + 0.23
40.0%
46.72
(GC)
46.20 0.60
51.75
+ 2.23
50.0%
68.62
1.21 (GC)
69.11
0.62
69.47 = 0.19
50.0%
47.11
+ 2.00 (GC)
52.83 0.32
57.62
+ 0.64
60.0%
73.34 + 0.39 (F)
0.91 71.87
71.82 + 0.20
60.0%
49.66 + 0.64 (GC)
0.40 55.00
60.81
+ 0.53
Rank
2.22
2.55
1.22
Rank
2.11
2.77
1.11
Table 10: Comparison of results for Tiny ImageNet ResNet-18
Table 11: Comparison of results for Tiny ImageNet ResNet-50
Size
DeepCore
Moderate-DS
Our Method
Size
DeepCore
Moderate-DS
Our Method
0.5%
3.29 + 0.23 (FL)
3.34 = 0.33
6.91 + 0.24
0.5%
2.26 + 0.32 (GC)
1.71 = 0.27
5.24 0.03
1.0%
5.31 1:31:000 0.45 (GC)
4.11 = 0.24
11.09 + 0.45
1.0%
5.36 = 0.37 (F)
3.38 = 0.13
10.55 + 0.41
0.71 (GC)
14.03 1.01
22.83 +
5.0%
20.56
0.33 (GC)
12.39 0.02
26.07
0.19
10.0%
21.23 + 0.16 (GC)
21.40 + 0.41
28.05 + 0.82
10.0%
29.83 + 0.49 (GC)
23.21 + 0.23
34.08 + 0.21
20.0%
29.97
+ 0.72 (GC)
30.99 + 0.93
34.91 + 0.43
20.0%
39.60 + 1.32 (GC)
34.71 + 0.36
43.95 + 1.03
30.0%
33.73
: 0.37 (GC)
37.18 + 0.8T
40.36 = 0.43
30.0%
45.80 + 0.23 (F)
41.65 - 0.20
50.82 + 0.03
40.0%
35.74
0.37 (F)
40.21 1 0.24
40.41 + 0.17
40.0%
50.00
= 1.72 (GC)
48.72 1 0.61
55.23 + 0.08
60.0%
39.74
0.73 (F)
46.19 + 0.61
41.71 #0.41
60.0%
56.97
0.12 (F)
56.10 0.13
61.38 #0.11
Rank
2.75
2.12
1.12
Rank
2.00
3.00
1.00
Table 12: Comparison of results Tiny ImageNet on VTT small
Table 13: Comparison results for ImageNNtt-1K ResNet-18
Size
DeepCore
Moderate-DS
Our Method
2.88 4: 0.04 (FI.)
0.12
Size
DeepCore
Noise-free Gradients
1.0%
4.23 + 0.25 (F)
4.34 + 0.13
9.18 + 0.25
5.0%
8.41
0.15 (GC)
13.56 0.23
18.10 = 0.15
0.1%
1.29 + 0.09 (CAL)
1.88 + 0.10
10.0%
11.53 : 1.23 (GC)
15.84 = 0.22
22.58 = 0.08
0.5%
7.66 + 0.43 (GraphCut)
11.58 + 0.15
20.0%
12.54
+: 0.06 (GC)
22.03
0.32
28.26: 0.31
1.0%
18.10 1: 0.22 (GraNd)
22.82 1 0.07
30.0%
14.09
+ 0.11 (GC)
25.35
0.31
33.66 + 0.21
5.0%
47.64 + 0.03 (Forgetting)
43.50 0.41
40.0%
15.69
1 0.45 (GC)
28.40
- 0.22
37.29 + 0.32
10.0%
55.12 + 0.13 (Forgetting)
49.51 = 0.26
50.0%
17.34
(GC)
30.68
40.92
0.36
Rank
1.4
60.0%
19.23
+ 0.32 (GC)
32.74
= 0.27
43.80 +0.38
Rank
3.00
1.00
We visualize the top and bottom-ranked samples in this subsection according to coresct selection al- gorithm. We chose the "Zebra" class from the 1000 ImageNet classes. Figure g presents the top-ranked and bottom-ranked 12 images of the "Zebra" class. The selected samples clcarly emphasize the fact that the images with top ranks are represenlative of the class. In contrast, images at the bottom positions are ambignously labelled (have parts of objects from other classes) or difficult to label as "Zebra". As our ob- jective to select the most representative images from each class, the visualization provides evidence for the intuition over which the proposed method is founded. We have provided additional examples in the supplementary material.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Top ranked images
Bottom ranked images
Figure 3: Top-ranked and bottom-ranked 12 images from the "zebra" class from ImageNet- dataset by the proposed "Noiso-free Gradicnts" approach. Images are ranked per the number neighbors within a threshold gradient similarity value. Top-ranked images unambiguous representative of the class, while bottom-ranker images either mislabeled or ambiguously labeled.
5.2.4 Robustness against image noise
In realistic scenarios, training data may be polluted by corrupted images et al. 2023). We have randomly added five different types of noise to the training subset of the original dataset, namoly Gaussian Noise, random occlusion, resolution change, fog and motion blur. We have conducted coreset selection on the corrupted train set and observed its generalization performance on the uncorrupted test set.
We have adapted the implementation provided by Moderate-DD((Xie et al. 2023 available in public github repository[ Table 14 compares accuracy obtained by all the methods available in DeepCore library and Moderate with our Noise-free Grudients method for a 30% of the training dataset impacted by noise. ResNet- 18 architecture is used for evaluation. As can be seen, for this significant amount of noise added to the original dataset, our method outperforms all the other methods for most of the coreset selection. As our proposed method composes the coreset with images having gradient similarity with a higher number images from the same class, the coresct able to perform better than existing methods even when the training datasct is corrupted by significant image noise.
Table 14: Comparison resulta for CIFAR-100 with 30% corruption ResNet-18
Table 15: Comparison of results for CIFAR-100 with 5% corruption ResNet-50
Size
DeepCore
Moderate-DD DS
Our Method
Size
DeepCore
Moderate-DS
Our Method
0.5%
6.16
0.23 (CR)
5.07 +0.57
11.1 + 0.18
1.0%
8.74 # 0.32 (CR)
(GC)
17.56 10.61
33.56
+ 0.65
20.0%
10.0 0%
42.50
22.2 + 0.13 (H)
#1.27 (H)
35.1 #0.31
46.78 +1.9
53.48 0.22
42.0 + 0.24
20.0%
30.0%
10.0%
28.14
24.40
20.7
+0.98 (GC)
70.76 (GC)
0.89 (GC)
57.58
29.20 +0.23
41.27 +0.56
#0.16
31.24
41.89
51.93
+ 0.40
+ 0.11
=0.25
40.0%
60.0%
30.0%
60.54
70.22 + 0.22 (F)
53.88
10.37 (H)
0.94 (H)
71.46 +0.19
65.4 #0.39
57.36 11.22
70.27 = 0.16
59.9 10.13
64.96 +0.2
Rank
1.17
Rank
3.00
1.60
1.40
Table 15 and Table 16 show accuracy comparison for 5% and 20% of images from CIFAR-100 dataset
impacted by noises, trained on ResNet-50 architecture. Table 17 compares accuracies obtained for 20% noise
httpp:///gichubb.cmm/mma//a ICLR Moderate-DS

---

Published in Transactions on Machine Learning Research (04/2025)
Table 16: Comparison of results for CIFAR-100 with 20% corruption on ResNet-50
Table 17: Comparison of results for CIFAR-100 with 20% corruption on VIT Small
Size
DoreCore
Moderate-
Method
Size
DeepCore
Moderate- DS
Our Method
10.0%
24.78
0.45 (F)
27.05 20.8
42.32 = 0.23
1.0%
10.02 + 0.29 (CR)
7.20 #0.31
10.44 0.10
20.0%
44.42
10.46 (H)
42.98 10.87
54.63 = 0.26
5.0%
17.58 i 0.36 (GC)
19.03 10.8
23.65
0.15
30.0%
53.37
+0.31 (H)
55.80 #0.96
60.58
=0.20
10.0%
20.7 + 0.89 (GC)
29.20 20.23
31.24 = 0.11
40.0%
(H)
61.84 11.96
00.35
20.0%
(GC)
10.56
41.89
0.40
60.0%
69.10
+ 1.73 (H))
70.05
+1.29
70.51 + 1.31
30.0%
28.14 +0.76 (GC)
57.58 10.16
51.93
+0.25
Rank
2.80
2:20
1.00
Rank
2.00
1.20
ratio in the CIFAR-100 dataset, trained on transformer architocture(ViI Small used). In all four studies, we can that our method significantly outperforms all the SOTA methods.
5.2.5 Robustness against label noise
Practical datasets may also involve label noise, where some images are mislabeled. We have adapted the implementation provided by Moderate-DS (Xia et al. 2023) available in public GitHub repository to gen- erate mislabeled data. Table I and Table 19 compare accuracy obtained by all the methods available in the DeepCore library and Moderate with our Noise-free Gradients method for a 20% of the training dataset impacted by label noise ResNet-50 architecture and VIT Small architecture respectively. As be seen, for this significant amount of noise added to the original dataset labels, our method outperforms all the other methods. As our proposed method composes the coreset with images having gradient similarity with a higher number of images from the same class, the coreset is able to perform better than existing methods even when the training datasct is corrupted by significant label noise.
Table 18: Comparison of results on CIFAR-10M 100 Dataset with 20% label noise ResNet-50 architecture
Table 19: Comparison of results on CIFAR-101 Dataset with 20% label noise on ViT Small architecture
Size
DeepCore
Moderate- DS
Method
DeepCore
Moderate-DS DS
Our Method
10.0%
38.59 +1.29 (GC)
25.02
+0.27
40.71 + 0.34
0%
28.66 #0.52 (GC)
23.02 #0.32
31.39
0.36
20.0%
47.15
11.66 (GC)
42.75 #1.20
52.99 + 0.51 60.77 11117
20.0%
38.79
10.55 (GC)
34.92 10.10
36,90
0.21
30.0%
57.04
+0.88 (GC)
56.28 +1.11
30.0%
40.22
+0.92 (GC)
43.91 +0.28
44.71 #:0.07
Rank
2.00
3,00
1.00
Rank
2.00
3.00
1.00
5.2.6 Impact on class-wise accuracy
We visualize the class-wise accuracy of training ResNet-18 model on the CIFAR-100 dataset, comparing the performance of the full dataset and coreset selection with 20% of the dataset in Figure 4 We observe Spearman rank-order correlation coefficient of 0.87, indicating a high correlation between class-wise accura- cies obtained with the full dataset and 20% of the full dataset as coreset. It can be seen that our method does not adversely impact any particular class during coreset selection. The coreset composed by our method results in tlass-wise accuracies strongly correlated to that of the full dataset.
5.2.7 Cross-architecture generalization
To study the robustness of our coreset selection method, we have carried out cross-architecture generalization performance comparison at 1% and 10% fractions of CIFAR-10 and Tiny ImageNet for two CNN architectures ResNet-18 (He et al. 2016), VGG-16 (Simonyan & Zisserman, 2015) and one transformer based architecture ViT-Small. The comparative analysis is tabulated in Table 20 and Table 21 The "Source" column denotes the architecture on which the coreset is selected, and the "Target" row indicates the architect ure on which its performance evaluated. As can be observed, our proposed method able to perform consistently better than existing methods in cross-architccture sctting. Cross-architecture analysis the Tiny-ImageNet dataset is provided in the supplementary material.
htttp:://githubb.cmm//ee IC____CRR_____ Moderate-DS
12

---

Published in Transactions on Machine Learning Research (04/2025)
0.0
0.8
Class index
Figure Class-wise accuracy comparison between full dataset and coreset with 20% selection percentage. A high correlation is observed indicating coreset composed by proposed method not impacting any particular class adversely.
Table 20: Cross-archieccture comparison for 1% coreset of CIFAR-10
Table 21: Cross-architecture comparison for 10% coreset of the CIPAR-10
Target -
ResNet-18
Target +
ResNet-18
DeepCore
Moderate-DS
Our Method
Source
DeepCorc
Moderato-DS
Method
ResNet-18
42.78 + 1.30 (GC)
41.44 10.34
48.00 2.10
ResNet-18
76.65 1.48 (GC)
75.68 10 62
78.62 + 1.05
VGG-16 VIT Small
43.02 + 1.30 (GC)
42.12 10.27
46,27 0.33
VGG-16 VIT Small
78.66 0.55 (GC)
74.27 10.72
78.75 +\r 0.16
26.01 + 2.00 (GC)
41.76 +0035
45.05 +: 0.51
70.21 +1.16 (GC)
73.66 4-0.29
75.15 4: 0.42
Target
VGG-16
Target
VGG-16
Source
DeepCore
Moderate-DS
Our Method
Source
DeepCore
Moderate-DS
Our Method
ResNet-18
+ 3.63 (GC)
44.81 40.33
47.21
ResNet-18
73.29 1.05 (GC)
10.86
1 0.29 78.84 + 0.33 75.47 + 0.35
VGG-16
27.47 + 4.00 (GC)
44.35 +:0.5
47.64 0.71
VGG-16 VIT Small
77.91 + 0.71 (GC)
76.62 +0.52
VIT Small
35.29 = 2.82 (GC)
34.44 +0.30
38.43 0.40
70.38 0.87 (GC)
73.81 #0.68
Target
VIT Small
Target
VIT Small
Source
DeepCore
Moderate-DS
Our Method
Source
DeepCore
Moderate-DS
Our Method
18
29.06 + 0.75 (GC)
34.71 +.0.33
40.35
0.90
ResNet-18
62.99 0.12 (GC)
61.69 4-0.23
63.66 :: 0.35
VGG-16
42.06 + 0.90 (GC)
31.25 +0.73
44.89 + 0.55
VGG-16 VIT Small
63.14 + 0.18 (GC)
61.49 +0.33
66.88 + 0.41
VIT Small
22.89
1.45 (GC)
31.11 +0.30
38.43 0.40
49.30 1.30 (GC)
54.40 #2.35
58.32 0.16
5.3 GAN Analysis
So far, we have presented the effectiveness of our proposed method in the context of training discriminative model (classifier); however, in this subsection, we investigate the effectiveness of the proposed method in a different context, which is training generative model. Similar to our approach for the classification task, we train the generative model for the initial few epochs with the full dataset and then select the class-wise with a 50% selection percentage. We utilize the discriminator as classifier for the purpose of computation of gradient similarity and ranking of images. Thercon, the generative model is trained with the coresct.
wise coreset
5.3.1 Results on CIFAR-10
We have trained four different GAN architectures, namely DCGAN, MSGAN, SAGAN, and SNGAN, on the CIFAR-10 dataset. Table 22 compared the FID and scores obtained by training with a full dataset and coreset composed by small GAN, instance selection, and our method. As can be observed, our method consistently performs better than the other methods in the FID score, which a key metric for measuring the quality of generated images.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Table 22: Comparison of results on CIFAR-10 Dataset, 4 means lower is better, f means higher is better
Method
DCGAN
MSGAN
SAGAN
SNGAN
FID 1
IS
FID 1
18
FID 4
IS
FID
IS
Full Dataset
28.80
5.84
33.16
5.81
33.73
5.56
22.44
7.36
Small GAN
35.53
45.76
Instance Sclection
31.26
5.62
39.05
5.71
36.96
5.48
27.81
6.98
Our Method
31.14
35.65
5.51
34.60
5.42
24.46
7.22
5.3.2 Results on CIFAR-100
We have trained four different GAN architectures, namely DCGAN, MSGAN, SAGAN and SNGAN, on the CIFAR-100 dataset. Table 23 compared the FID and IS scores obtained by training with full dataset and coresct composed by small GAN, instance selection and our method. As can be observod, our method consistently performs better than the other two methods in FID score, which is key metric for measuring the quality of generated images.
Table 23: Comparison results on CIFAR-100 Dataset. 4 means lower is better, means higher better
Method
DCGAN
MSGAN
SAGAN
SNGAN
FID |
I$
FID 1
18 f
FID 1
18 1
FID
I8
Dataset
27.98
Small GAN
32.81
6.11
71.13
6.75
41.53
5.76
39.71
5.51
Instance Selection
30.48
6.07
53.88
7.10
39.17
5.60
28.88
6.13
Our Method
30.08
6.08
50.22
6.64
36.15
5.61
27.71
6.14
5.3.3 Results on Tiny ImageNet
We have trained SAGAN architecture the Tiny ImageNet dataset. Table 24 compared the FID and IS scores obtained by training with full dataset and coreset composed by small GAN, instance selection and our method.
Table 24: Comparison of results on Tiny ImageNet Dataset. | means lower is better, 1 means higher is better
Method
SAGAN
FID
18 f
Full Dataset
35.67
7.12
Small GAN
75.12
6.65
Instance Selection
69.16
6.46
Method
5.4
Computation time analysis
5.4.1 Image Classification
Our method does consider all images a given dataset in a single coreset selection step. Instead, it processes the dataset class-wise, implements multiprocessing and optimizes the nearest neighbour algorithm to parallelize the ranking process. Hence, the methodology can be easily scaled to large-scale dataset beyond ImageNet-IK. Also, the computation of gradients and nearest neighbours is onc-time process for a given dataset and architecture (per selection epoch). After that, aggregating the individual image ranks across the training epochs takes only 225 seconds (steps 11-13 in Algorithm [ As we directly utilize the gradicnts of the loss function with respect the weights of the last fully connected layer, there no requirement for storage of the gradient values. Table 25 provides timing analysis for the ImageNet dataset ResNet-18 architecture 48 GB RTX A6000 GPU.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Table 25: Execution time comparison
Process
Execution Time
Single epoch full dataset training
4.46 bours
Gradient Calculation (per epock)
3.19 hours
Ranking for the coresct
225 seconds
5.4.2 GAN training
By using 50% of the total dataset, we train the GAN model with coreset selection for half the number of update steps compared to training with the entire dataset. As shown in section 5.3 our method can achieve comparable FID score to training with a whole dataset while utilizing half of the available dataset. Table
26 provides the total training time required for various methods, and the lotal time taken significantly. At the same time, degradation in performance is not significant.
that method reduce
Table Execution time comparison GAN training CIFAR-100 Dataset (In hours)
Method
DCGAN
MSGAN
SAGAN
SNGAN
Dataset
2.90
9.48
Small GAN
9.82
15.10
16.22
19.35
Instance Selection
1.32
3.00
1.56
3.95
Our Method
1.34
3.53
1.80
3.98
5.5 Accuracy vs training speed tradeoff
Figure 5 and i shows classification accuracy vs computational time trade-off for full dataset training coreset training using 10%, 20% and 30% of the entire dataset. As it can be seen, with small reduction in accuracy, we can improve the training speed, and it shows the effectiveness of the proposed coreset met hod during hyper-parameter tuning, which may require evaluating diffcrent values for them over multiple runs.
Dataset: CIFAR10, Model: ResNet18
Dataset: CIFAR1D, Model:
5000
100
Cerrruattonnnnimm time secones)
4000
80
7200
72
Training (In
3000
Cosunno (nn
5400
2000
40
3000
30
36
1000
20
1800
dataset
coreset @20%
coreset
coreset @:10%
9900%
-training time
test
-taining acturary
Figure 5: Accuracy vs training speed trade-off. Tt can be observed that using 30% of coreset. results in similar accuracy while the training time is reduced more than 3 times.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset: Tiny ImageNet, Model: ResNet18
Dataset: Tiny ImageNet, Model: VIT Small
8000
60
9000
60
Compubitonal time (in seconds
compurattonsttime/// seconds
7500
50
4000
40
8000
40
Accuraty (In
6500
30
Accuvcy
2000
20
3000
20
1000
1500
Full dataset
coresat @30N
coreset 8220% coreset 6910%%
detaset
B30%
820% t--tst accuracy
coreset
-tariiiig
---test eccuracy
-etranning tima
Figure 6: Accuracy vs training speed trade-off. For a complex dataset like Tiny ImageNet, the reduction in training time is not insignificant for applications like hyperparameter luning where model needs to be trained multiple times.
6 Conclusions
In this paper, introduced Noise-free Gradienls, an intuition-driven, gradient similarity-basee coreset selection method for identifying representative instances from large training datasets. We studied its effec- tiveness through our extensive experiments on popular object recognition datasets (CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNet-IK) and sophisticated model architectures (CNNs and Vision Transformer) at various coreset sizes. We demonstrated superior generalization performance of classifiers trained on the re- sulting coresets. We have also demonstrated that our model achieves consistently higher performance in the case of cross- moss-architeccure generalization. Thus, strongly propose method as an essential bascline to benchmark the forthcoming coreset selection methods.
A branch of existing methods, such as Forgetting (Toneva et al. 2019). utilizes noisy gradients to compose coresets. The argument put forward by these methods is the samples that are hard to learn will be able to provide better generalization on the test set. However, our approach explores an opposing notion. Samples that derive loss gradients similar to many other samples can effectively represent the whole dataset and thus form coreset. This hypothesis naturally exploits the redundancy in large datasets compose extremely small coresets. Our method outperforms other coreset selection methods even in the presence of noise in the original dataset. We have also applied our coreset method to improve computational time significantly for GAN training without sacrificing much performance.
6.1 Future Directions
While one can appreciate the simplicity and intuition behind the proposed coreset selection method, it must be studied further, particularly concerning the approximation error between the gradients computed by the coreset and the entire dataset. Inducing diversity in the selected samples along with individual importance weights (or par-sample step size) would further improve the effectiveness of the proposed approach. One may consider these aspects for future study.
There have been few reported applications of coreset selection, namely Active Learning (Sener & Savarese 2018) and semi-supervised learning (Killamsetty et al.) 2021c). It would be interesting to study the effec- liveness of our Noise-free Gradients approach concerning these application areas.
16

---

Published in Transactions on Machine Learning Research (04/2025)
References
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning, 2020.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds, 2020.
Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy; Elficient data selection for deep learning, 2020.
Terrance DeVries, Michal Drozdzal, and Graham W. Taylor. Instance selection for gans, 2020. URL https:
//arxiv.org/abs//0007. 15255
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach, 2018.
Dan Feldman. Introduction to corc-scts: an updatod survey, 2020.
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL
1406.2661
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprchensive library for coresct selection in deep learning, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Compuler Vision and Pallern Recognilion, CVPR '16, pp.
770-778.10 IEEE, June 2016. doi: 10. 10.1199/CPPR.2016. URL http:://eeeepploreeeeee ott.uuuuuuoor////muue///////////
7780459
Melissa Heikkilä. technologyreview..com
We're
getting a better idea of AT's true carbon footprint
http:://www.eehhollogrreeeeee http://wwwrrrreooo//////////////
were
algorithmkutn wttp:iuuttmmmiiuuueeemem the www.rriithmuumiii..mmmmmmm wwwagementtuutmmcommm
12-04-2023
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium, 2018. URL https://arxiv. org/ /tt//abs11700.050000
Rishabh Iyer, Ninad Khargonkar, Jeff Bilmes, and Himanshu Asnani. Submodular combinatorial information measures with applications in machine learning, 2021.
Angelos Katharopoulos and François Flcuret. Not all samples are created equal: Deep learning with impor- tance sampling, 2019.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training, 2021a
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Gen- eralization based data subset selection for efficient and robust. learning, 2021b.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 11488-14500 Curran Associates, Inc., 2021c. URL httpp::/pproeeddinnne wtw.eedddiinnnn...ii.....e.eee.e cctpaccrcppppr.rreo pplee__files/ faper//oorrttrraaaaaaassssstttie
17

---

Published in Transactions on Machine Learning Research (04/2025)
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning, 2021d.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009.
URL https:
7/www. cs. toronto. wtt.t/otttooooootooooeeee//////e
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL httpss//apl
semanticscholar. httpihollarrrgoooooooooee I:: 16664790
Soung Hoon Lee, Scunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets, 2021.
Siwen Liu. A Review for Submodular Optimization on Machine Scheduling Problems, pp. 252-267. Springer International Publishing, Cham, 2020. ISBN 978-3-000-41672-0.
doi: 10.1007/9775-3-00--46777 16.
URL http://0000r///0000000//072----0
Qi Mao, Hsin-Ying Lee, Hung-Yi Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adver- sarial networks for diverse image synthosis, 2019. URL https://arxiv wwwppiv....rb/////e/aeaaeoo hwtg/abs/1903..00
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples, 2021.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks, 2018. URL https://arxiv, org/ wwtps/wwwwrg/bb//02...... habbs//8820009555
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http::/papeer: http:///aapprr..uiiss..comm cesurips.cccpappeee. 9015 htp.rrrdraaaiiipeerreeeeeeeteaaa
deep- learning- library. pdf
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples carly in training, 2023.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(0ct):2825-25)) 2830,, 2011.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Fei-Fei. Imagenet large scale visual recognition challenge, 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016. URL https://arxxv.org/obbl wttp:/www..w//////6//////// 1606. 03498
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach, 2018.
Burr Settles. Active Learning. Springer International Publishing, 2012. ISBN 9783031015601. doi: 10.1007/ 978-3-031-01560-11 URL http://dx doi wtt..///./..1/////......... 1-11//777-----01111---------
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition, 2018.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. Small- gan: Speeding up gan training using core-sets, 2019.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J. Gordon. An empirical study example forgetting during deep neural network learning, 2019.
Max Welling. Herding dynamic weights for partially observed random field models, 2012.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2023. URL https:://penrrevee. teee/foormmm11775E00000
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence, 2023.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks, 2019. URL hhttps://arx: www.xivvorrg/aae 08318
19


Reviews:
[{"review_id": "qSCxXkEhYw", "reviewer": "Reviewer_1AJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed method is well-motivated and clearly explained, with experimental results demonstrating its effectiveness.\n\n2. The authors conduct extensive experiments to validate the method's effectiveness, robustness to sample and label noise, generalization across architectures, and applicability to GANs for image generation.\n\n3. The paper is well-structured and easy to follow.\n\n**Weaknesses:** The term \"noise-free gradients\" is inadequately justified. Although the authors argue that existing coreset methods based on 'distinctness' or 'difficulty' favor samples that give rise to noisy loss gradients, this claim lacks experimental support or a clear explanation. Providing a more convincing rationale or additional experiments to demonstrate how prior methods introduce noise and why the proposed method avoids it would significantly strengthen the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates coreset selection, motivated by the observation that gradients of data within the same class exhibit strong cosine similarity. The authors propose selecting the top-ranked gradient cosine similarity within a class to represent the coreset. They validate the effectiveness of their method on image classification tasks using CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1k datasets, employing ResNet, VGG, and ViT architectures. Additionally, the method is tested on GANs for image generation, demonstrating superior performance over other data selection techniques. The paper also includes a detailed experimental analysis, highlighting the method's robustness against sample and label noise, as well as its generalization across different architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The proposed method is well-motivated, supported by comprehensive experiments and analysis, and the paper is well-organized. However, I have a concern regarding the naming of \"noise-free gradients.\" A more convincing explanation or additional experiments to substantiate this term would strengthen the paper.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xYUjsiDVr2", "reviewer": "Reviewer_EDLA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed method is very intuitive, simple, and easy to implement. The computational overhead of constructing the coreset is low. This is shown in the experiments in Sec 5.4.\n\n- Despite the simplicity, the proposed method is very effective. Experiments in the submission show that it achieves better accuracy (for the same coreset size) than previous methods on a variety of tasks (image classification and GAN training) across different datasets. It significantly reduces the computation time needed for training while maintaining good accuracy (Sec 5.5).\n\n- As an extra benefit, the proposed method achieves good robustness when a small part of the dataset is corrupted. A similar idea for achieving robustness has been explored in [this paper](https://arxiv.org/abs/2003.10647).\n\n**Weaknesses**\n\n- The authors claim that their methodology of using gradient similarity to build coresets is novel. However, based on my understanding of the related work section, the Moderate Coreset by Xiao et al. (2017) is also similarity-based, though their similarity is computed on hidden representations rather than gradients. It would be great if the authors could include a more detailed comparison with their work in the introduction, and explain the intuition behind choosing gradients rather than hidden representations to evaluate similarity.\n\n- The authors use the cosine similarities among the gradients to choose the coreset. My understanding is that the length/norm of each gradient is not taken into account when calculating the cosine similarity. That is, the cosine similarities only depend on the directions of the gradients and not on the lengths of the gradients. I'm not sure how this will affect the execution of SGD with momentum (the optimization algorithm used by the authors). (My understanding is that the lengths of the gradients would affect the result of running SGD with momentum.) It would be great if the authors could clarify this and justify the design choice of ignoring the gradient lengths when forming the coreset.\n\n- The proposed method chooses the same number N of data points from each class. I'm wondering what is the idea behind this design choice. Does this lead to better class-wise balance/fairness when the original dataset is imbalanced across label classes?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes a new method for coreset selection in deep learning. Instead of running an entire training process on a full dataset which may become prohibitively inefficient/expensive, the goal of coreset selection is to pick a small representative subset of data (i.e. coreset) with the hope that training only on the coreset can achieve comparable accuracy with training on the full dataset.\n\nMany previous coreset selection approaches used the idea of selecting data points that are distinct/hard to learn. The method proposed in this submission is based on a different intuition of selecting data points with high similarity to other data points. Specifically, within each label class, the proposed method selects the N data points with the most neighbors, where the neighbors of a given data point z are other data points within the class whose gradients are close to the gradient at z.\n\nThe authors show an efficient way of implementing the proposed coreset selection method based on nearest neighbor search. The authors also perform extensive experiments on image classification and GAN training to demonstrate the effectiveness of the proposed approach. They also test the robustness provided by the proposed coreset selection method when the training data is corrupted.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Minor**\n\n- Second bullet point on Page 1: demanind -> demanding\n- In Figures 5 and 6: I wonder if it is better if the x-axis is arranged in decreasing order: Full dataset, coreset @ 30%, coreset @ 20%, coreset @ 10%.", "broader_impact_concerns": "I don't see any major broader impact concerns caused by the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FuzjCfeqjX", "reviewer": "Reviewer_R3WH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written, with clear notations, and motivations;\n- The proposed approach is simple yet intuitive;\n- The authors proposed an efficient practical implementation to reduce the computational cost;\n- The choice of baselines is comprehensive and covers well the literature;\n- The experiments are large-scale and enough ablation is provided;\n- The authors provide an open-source implementation;\n- The authors show significant improvement both for image classification and speeding GAN training.\n\n**Weaknesses**\n- The authors could provide additional ablation on the benefits of nearest neighbors (decrease of computation time v.s. performance compared to computing the gradients on all the samples of a given class);\n- The authors could provide additional ablation on the threshold value for the $\\textit{Noisy-free Gradients}$ algorithm to compute the similarity measure of each training sample.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a $\\textit{Noise-free Gradients}$ approach for coreset selection. The method addresses the challenge of reducing training computational time and resource requirements by selecting a subset from a larger dataset to train models without hindering the training quality. The core idea of the method is to rely on the similarity between gradients of the training loss during the early training phase. \n\nFirst, the authors show experimentally that gradients of samples belonging to the same class have a high similarity (measured in terms of cosine similarity) while samples from different classes are not well aligned. The authors then propose a $\\textit{Noise-free Gradients}$ algorithm that builds on this idea by first computing the loss gradients for all the samples and then computing a similarity measure that assesses the ability of each sample to represent the other samples in the same class using the cosine similarity above-mentioned.  The final selected subset of size $N$ aggregates the $N$ individual samples that have the highest similarity measure. \n\nThe authors propose two implementation simplifications to reduce the computational cost (namely computing the gradients only w.r.t. the classification layer and computing the similarity measure using the nearest neighbors algorithm to avoid calculating cosine similarity between gradients for all the samples of a given class). The authors demonstrate the effectiveness of their approach with extensive experiments for image classification and to speed up Generative Adversarial networks (GAN) training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall the paper is interesting with an intuitive and efficient approach for coreset selection. The experiments are extensive and showcase significant improvement. The implementation details are given and the authors provide an open-source implementation, which I think is an additional strength of the proposal.\n\nI would like to recommend acceptance for this work.\n\nThe changes proposed below would strengthen the work in my opinion. Regarding the weaknesses, could the authors provide\n1) Additional ablation on the benefits of nearest neighbors (decrease of computation time v.s. performance compared to computing the gradients on all the samples of a given class)?\n2) Additional ablation on the threshold value for the $\\textit{Noisy-free Gradients}$ algorithm to compute the similarity measure of each training sample?\n3) The anonymized Github link given by the authors has expired by the time of the review. Could the authors make sure that a working link is provided either for camera-ready if the paper is accepted or for a potential resubmission? Indeed, I believe that open-sourcing the code is a strength of this work and could benefit the community.", "broader_impact_concerns": "I do not have any ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple, intuitive and computationally cheap method that works very well in the experiments. The authors also show an impressive number of ablation results, going beyond a standard evaluation in a typical paper. The reviewers were also unanimous in recommending acceptance, and all of the major questions they had were addressed by the authors.
Decision: Accept (poster)