Title: Data Pruning via Moving-one-Sample-out
Venue: NeurIPS 2023
Accepted: yes
Keywords: Data Valuation

Deep Learning

Data Pruning

Coreset Selection.
Abstract: In this paper, we propose a novel data-pruning approach called moving-one-sample-out (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples.  
Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and outperforms state-of-the-art methods by a large margin across various settings.

Abstract

In this paper, we propose a novel data-pruning approach called moving-one-sampleout (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings.

1 Introduction

The recent advances in AI have been largely driven by the availability of large-scale datasets [31, 3, 6, 34, 41, 10, 37], which enable the training of powerful models [51, 3, 45, 6, 1, 4, 36]. However, such datasets also pose significant challenges in terms of computational and storage resources. It is important to note that these datasets may contain redundant or noisy samples that are either irrelevant or harmful to the model's performance. Data pruning techniques aim to address these issues by removing such samples and retaining a smaller, more compact core set of training samples [11, 28, 42, 27, 48, 38, 15]. This can not only reduce the costs of model training and data storage but also maintain the performance of the model compared to the original dataset.

Existing approaches can be broadly categorized into three major groups: pruning by importance criteria [11, 42, 28, 27, 26, 46], coverage or diversity-driven methods [48, 29, 33, 15], and optimizationbased methods [38, 8, 23, 24, 21, 44]. Among these, the first group of studies is the most effective and popular. These studies assume that hard samples are critical and informative core-set samples, and thus, they design difficulty-based metrics to assess sample importance. Such metrics include prediction entropy [11], forgetting [28] or memorization [46] score, gradient norm [27], E2LN (variance of prediction) [27], self-supervised prototype distance [42], diverse ensembles [26], and others.

Limitations and Motivations. The methods we discussed have some major drawbacks: (i). Hard samples are not necessarily important samples. For example, noisy samples [48] and outliers [39] often lead to high losses, which makes it difficult for importance criteria [11, 27] to distinguish them from truly important samples. (ii). Training dynamics are rarely considered. The mainstream methods [27, 46, 26, 11, 42, 15, 48, 38] do not possess the awareness of training dynamics as they generally utilize a converged surrogate network for data selection. This may favor samples that are difficult or influential in the later stages of training, but not necessarily in the earlier stages or the whole training process [50, 12].

Our Method. In this paper, we propose a new data pruning algorithm, which involves the newly proposed Moving-one-Sample-out (MoSo) score with an efficient and error-guaranteed estimator. To address the first limitation, MoSo utilizes the change of the optimal empirical risk when removing a specific sample from the training set to measure sample importance instead of only focusing on sample difficulty. By doing so, MoSo can better separate important samples from harmful noise samples, as the former tends to lower the empirical risk, while the latter may increase it. However, MoSo is too costly to compute exactly as it needs brute force leaving-one-out-retraining. Therefore, we propose an efficient first-order approximator with linear complexity and guaranteed approximation error. The proposed approximation is simple: samples whose gradient agrees with gradient expectations at all training stages will get higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. The second limitation is addressed since MoSo comprehensively considers information from different training stages.

Weevaluate our MoSo on CIFAR-100 [5], Tiny-ImageNet [49], and ImageNet-1K [31] under different pruning ratios. As shown in Figure 1, our MoSo significantly surpasses the previous state-of-the-art methods, especially for high pruning ratios. Besides, experimental results demonstrate that the coreset selected by our MoSo under one network (such as ResNet) can generalize well to other unseen networks (such as SENet and EfficientNet) (refer to Figure 3(a) and Figure 3(b)). Additionally, we study the robustness of our MoSo on datasets with synthetic noisy labels (refer to Figure 3(c) and Figure 3(d)). It can be seen that our MoSo performs best on average, and surpasses the previous methods based on difficulty-based importance criteria by a large margin.

2 Related Work

Finding important samples is not only the purpose of data pruning, but also the core step in many machine learning tasks and problems, like active learning [7, 52, 32, 39, 19], noisy learning [9], and continuous learning [16]. In data-efficient deep learning, there are also some related topics like data distillation [13, 40, 53] and data condensation [17, 47, 20]. Unlike data pruning, they focus on synthesizing a small but informative dataset as an alternative to the original large-scale dataset. Existing data selection/pruning approaches could be broadly divided into several categories, pruning by importance criteria [11, 42, 28, 27, 26, 46], coverage or diversity driven methods [48, 29, 33, 15], optimization-based methods [38, 8, 23-25, 21, 44, 43].

Pruning by importance criteria. This group of studies is the most popular. Generally, they assume that hard samples are critical and informative core-set samples and thus design difficulty-based metrics to assess sample importance. The EL2N score [27] measures the data difficulty by computing the average of the ℓ 2 -norm error vector from a set of networks. GraNd [27] measures the importance by calculating the expectation of the gradient norm. The Forgetting score [28] counts how many times a model changes its prediction from correct to incorrect for each example during the training process. Memorization [46] assigns a score to each example based on how much its presence or absence in the training set affects the model's ability to predict it correctly. Diverse ensembles [26] gave a score to each sample based on how many models in a group misclassified it. However, hard samples are not necessarily good for model training [42]. For example, noisy samples [48] and outliers [39] often lead to high losses, which makes it difficult for importance criteria [11, 27] to distinguish them from truly important samples. As a comparison, our MoSo score measures sample importance instead of only focusing on sample difficulty by calculating the change of the optimal empirical risk when removing a specific sample from the training set . By doing so, it can better separate important samples from harmful noise samples, as the former tends to lower the empirical risk, while the latter may increase it.

Coverage or diversity driven methods. Sener et. al. [32] applied greedy k-center to choose the coreset with good data coverage. BADGE [19] is a diversity-based selection method in active learning that clusters the gradient embeddings of the current model using k-means++ and selects a subset from each cluster. CCS [15] balances the data distribution and the example importance in selecting data points. Moderate [48] chooses data points with scores near the median score. Note that some diversity-driven methods, such as CCS [15] and Moderate [48], can use any selection criterion, such as EL2N score [27], as a basis.

Optimization-based methods. In addition, a line of recent works proposed to select data by optimization, such as gradient matching [8, 23], bi-level optimization [24, 25], submodularity [21, 44, 43]. One of the most advanced methods, the optimization-based dataset pruning [38], builds an algorithm over the sample-wise influence function [35] to remove samples with minimal impact and guarantees generalization. However, like mainstream methods [27, 46, 26, 11, 42, 15, 48, 38], it does not account for the effect of samples on the training dynamics, as it only uses the information from the final model. This may favor samples that are difficult or influential in the later stages of training, but not necessarily in the earlier stages or the whole training process [50, 12]. In our work, the proposed method is fully training-dynamic-aware since the MoSo's approximator comprehensively considers information from different training stages.

3 Method

In the following, we will first present the background knowledge in Section 3.1. Following that, we will elaborate on the proposed MoSo score for assessing sample importance in Section 3.2. Furthermore, we will introduce an efficient approximator of MoSo in Section 3.3. Section 3.4 shows the along with its complexity analysis and error guarantees.

3.1 Background

In this work, we focus on the classification task, where S ' tp x i , y i q| N i ' 1 u denotes the training set, drawn i.i.d from an underlying data distribution P , with input vectors x P R d and one-hot label vectors y P t 0 , 1 u K . Let l p¨q denote the widely used cross-entropy loss function for classification tasks. Given a pruning ratio δ and a parameterized deep network f w , the data pruning task aims to find the most representative training subset ˆ S Ă S while pruning the remaining samples. This can be formulated as:

where p| S | ´ | D |q{| S | ' δ , | ¨ | represents the cardinality of a set, and w ˚ D indicates the optimal network parameter trained on D with the stochastic gradient descent (SGD) optimizer. The SGD optimizer updates the parameters as follows:

where t P t 1 , ..., T u , η t is the learning rate at the t -th step, and B t represents the mini-batch, ∇ is the gradient operator with respect to network parameters, L p¨q is the average cross-entropy loss on the given set/batch of samples.

3.2 Definition for Moving-one-Sample-out

Here, we will describe the details of Moving-one-Sample-out (MoSo) score.

Definition 1. The MoSo score for a specific sample z selected from the training set S is

where S { z indicates the dataset S excluding z , L p¨q is the average cross-entropy loss on the considered set of samples, w ˚ S is the optimal parameter trained on the full set S , and w ˚ S { z is the optimal parameter on S { z .

The MoSo score measures the importance of a specific sample z by calculating how the empirical risk over S { z changes when removing z from the training set. Specifically, with a representative

(important and with proper annotation) sample z , retaining it can promote training and result in a lower empirical risk while removing it could be harmful to the training and result in a higher empirical risk. Hence, M p z q ą 0 . On the contrary, when z is unrepresentative , M p z q « 0 . Moreover, if the selected data point z is harmful (e.g. noisy samples) , the retention of z is a hindrance to the learning process on S { z , so the risk would be high; removing the harmful z could result in a lower risk value. Hence, M p z q ă 0 .

3.3 Gradient-based approximator

The exact calculation of MoSo, as shown in Eq.(3), has a quadratic time complexity of O p Tn 2 q , considering a dataset with n samples and a total of T training iterations required to obtain the surrogate network. However, this is practically infeasible; for instance, it may take more than 45 years to process the ImageNet-1K dataset using a Tesla-V100 GPU. To address this issue, we propose an efficient first-order approximator for calculating the MoSo score, which reduces the complexity to O p Tn q . This approximation not only significantly decreases the computational requirements but also maintains the effectiveness of the MoSo score in practical applications.

Proposition 1.1. The MoSo score could be efficiently approximated with linear complexity, that is,

where S { z indicates the dataset S excluding z , l p¨q is the cross-entropy loss function and L p¨q means the average cross-entropy loss, ∇ is the gradient operator with respect to the network parameters, and tp w t , η t q| T t ' 1 u denotes a series of parameters and learning rate during training the surrogate network on S with the SGD optimizer. T is the maximum time-steps and N is the training set size.

The MoSo score approximator in Eq.(4) essentially represents the mathematical expectation of the inner product between the gradient with respect to network parameters considering only sample z and the gradient using the training set excluding z (denoted as S { z ) over T learning iterations. A sample z will be assigned a higher MoSo score if the mathematical expectation of the inner product is larger. This can be intuitively understood as follows: if the gradient ∇ l p z, w q from sample z is consistent with the average gradient vector ∇L p S { z, w q , it implies that optimizing the network using sample z will yield a similar effect on reducing the empirical risk as using all remaining samples. This indicates that sample z is an important and representative sample. Concurrently, according to Eq.(4), it is also assigned a high MoSo score, which aligns with the intuition.

3.4 Theoretical analysis of tightness and complexity

First, we provide a rigorous mathematical justification to show that there is a theoretical guarantee for the error between the approximator we provide and the exact score by the brute-force leave-one-out retraining.

Proposition 1.2. By supposing the loss function is ℓ -Lipschitz continuous and the gradient norm of the network parameter is upper-bounded by g , and setting the learning rate as a constant η , the approximation error of Eq. (4) is bounded by:

where T is the maximum iterations.

The proposition shows that approximation error is positively correlated with many factors such as training duration T , gradient norm g , and learning rate η . In order to control the impact of approximate errors, in practice, we will not train the surrogate network to complete convergence, instead, we will only update a small number of epochs.

Complexity analysis. We show that Eq.(4) efficiently approximates the MoSo score with linear complexity. Specifically, calculating the overall gradient information requires a time complexity of O p n q . Additionally, computing the expectation of the gradient over different training iterations in Eq.(4) takes T steps, resulting in a total complexity of O p Tn q . In practice, we can randomly sample a few time steps rather than considering all T steps to calculate the mathematical expectation, reducing the overall complexity to be less than O p Tn q . Moreover, the optimized gradient calculation operator,

Algorithm 1: Data Pruning with MoSo.

implemented by advanced deep learning frameworks such as PyTorch [2], further accelerates the computation, making it more feasible for practical applications.

3.5 Data Pruning with MoSo

This section presents the pipeline for utilizing the approximated MoSo score in data pruning and coreset selection. The pseudocode is provided in Algorithm 1. In the MoSo scoring step (see line 10 of Algorithm 1), we employ Eq.(4) from Proposition 1 to calculate the MoSo score. In practice, there is no need to sum all the time steps t 1 , ..., T u when calculating the mathematical expectation. Instead, an efficient approach is randomly sampling several time steps for calculating the average or expectation, reducing the overall computational cost. In the data pruning step, samples with low scores are pruned, and the proportion of pruned data corresponds to the predefined ratio δ .

Parallel acceleration by dataset partitioning. To enhance the practical applicability of MoSo on large-scale datasets, we propose a parallel acceleration scheme that can be employed when multiple GPU devices are available. Specifically, before training the surrogate network, we initially divide the original full dataset into a series of non-overlapping subsets. This approach enables efficient processing by leveraging the parallel computing capabilities of multiple GPUs, where the number of subsets should be no more than the available computing devices. We select a subset from S without replacement for each device and then perform training and scoring within the chosen subset, following Algorithm 1. Finally, MoSo scores from different devices are combined together. As long as the number of samples in a subset is large enough to approximately represent the overall statistics of the dataset, this partitioning scheme will not compromise performance while significantly reducing computation time by a factor of I . This approach is particularly useful for large-scale datasets.

4 Experiments

Datasets and baselines. We evaluate our method on three well-known public benchmarks: the CIFAR-100 [5], which contains 50,000 training examples of 100 categories; the Tiny-ImageNet [49], which has 100,000 images of 200 classes; and the ImageNet-1K [31], which covers 1000 classes with more than 1M training images. We compare our method with a range of baselines, including: (1). Random selection; (2). Herding [29]; (3). Forgetting [28]; (4). GraNd [27]; (5). EL2N [27];

0.0

0.2

0.4

0.6 0.7 0.8

Pruning Ratio

40

60

80

Top-1 Accuracy

Random

Herding

Forgetting

GraNd

EL2N

OPT

SSP

Moderate

MoSo (ours)

(a) CIFAR-100

0.0

0.2

0.4

0.6 0.7 0.8

Pruning Ratio

20

30

40

50

Top-1 Accuracy

Random

Herding

Forgetting

GraNd

EL2N

OPT

SSP

Moderate

MoSo (ours)

(b) Tiny-ImageNet

0.0

0.2

0.4

0.6 0.7 0.8

Pruning Ratio

30

40

50

60

70

Top-1 Accuracy

Random

Herding

Forgetting

GraNd

EL2N

SSP

Moderate

MoSo (ours)

(c) ImageNet-1K

(6). Optimization-based Dataset Pruning (OPT) [38]; (7). Self-supervised pruning (SSP) [42]; (8). Moderate [48].

Implementation details. We implement our method in Pytorch [2]. All the experiments are run on a server with 8 Tesla-V100 GPUs. Unless otherwise specified, we use the same network structure ResNet-50 [22] for both the coreset and the surrogate network on the full data. We keep all hyperparameters and experimental settings of training before and after dataset pruning consistent. We train the surrogate network on all datasets for 50 epochs. To estimate the mathematical expectation in Eq.(4) from Proposition 1, we randomly sample 10 time steps. Thus, MoSo can compute gradients from multiple epochs without increasing the overall time cost significantly, compared to methods that need to train a network fully ( e.g. 200 epochs for CIFAR-100) before calculating the scores.

4.1 Main Results

In the following subsections, we present the detailed performance comparison of our method and baselines on three experiments: data pruning, generalization to unseen structures, and robustness to label noise.

Data pruning. As Figure 1(a) and 1(b) show, our method significantly surpasses the SOTA method [48] on CIFAR-100 and Tiny-ImageNet at high pruning ratios. Note that some selected baselines perform worse than the random selection, especially when with high pruning ratios, e.g. the results of the classic EL2N on CIFAR100, while our method doesn't suffer from this problem. Furthermore, in Figure 1(c), our method achieves satisfactory performances on ImageNet1K across different pruning rates, showing its effectiveness on largescale and complex datasets. These results also indicate that our method can capture the sample importance more accurately and robustly than the existing methods.

Class index

1.0

Fullset

MoSo

0.8

0.6

0.4

0.2

0.0

0

20

40

60

80

100

To study whether our algorithm improves/hurts certain classes, we visualize the class-wise accuracy before and after applying our MoSo data pruning approach in Figure 2. We observe a significant correlation between the two, with a Spearman correlation coefficient of 0.913 and a P value of

Herding

Forgetting

GraNd

OPT

SSP

Moderate

MoSo

30

35

40

45

50

55

60

(a) ResNet to SENet

Herding

Forgetting

GraNd

OPT

SSP

Moderate

MoSo

(b) ResNet to EfficientNet

Herding

Forgetting

GraNd

OPT

SSP

Moderate

MoSo

(c) Pruning-ratio = 70%

Herding

Forgetting

GraNd

OPT

SSP

Moderate

MoSo

(d) Pruning-ratio = 80%

0.0295. This indicates that the performance before and after pruning with MoSo is consistent across categories, and no significant improvement or harm to any particular category is observed.

Generalization test. To test whether the pruning results are overfitting to the specific network architecture, we evaluate MoSo's generalization ability to unseen architectures. Following the protocol in [48], we use ResNet-50 as the surrogate network for scoring, and training different network architectures, SENet [18] and EfficientNet-B0 [30], on the selected data. Figure 3(a) and Figure 3(b) show the experimental results on different network architectures. MoSo exhibits a satisfying generalization ability to unseen models and consistently outperforms or matches the state-of-the-art methods such as SSP [42], and OPT [38].

Robustness test. Label noise is a common challenge in real-world applications. Therefore, how to improve the model robustness to label noise is an important and popular problem. In this section, we study the robustness of MoSo to label noise by conducting comparative experiments on CIFAR-100 and Tiny-ImageNet with synthetic label noise. Specifically, we inject label noise [14] into the two datasets by randomly replacing the labels for a percentage of the training data with all possible labels. The noise rate is set to 20% for all the experiments. We use ResNet-50 as the network architecture and keep all experimental settings consistent with the previous data pruning experiments. The results are shown in Figure 3(c) and Figure 3(d). We observe that some difficulty-based importance criteria don't work very well, such as Forgetting, GraNd on both settings, while Herding [29], Moderate [48] and our MoSo, perform well and significantly outperform other baselines in all settings. Our MoSo achieves comparable performance with the best baseline, Herding [29], only lagging behind Herding by less than 1% Top-1 acc.

4.2 Computational Efficiency

We evaluated MoSo and the other baseline methods on a server with 8 Tesla V100 GPUs. We used the CIFAR-100 dataset and the ResNet50 backbone for our experiments. It should be noted here that we also take into account the training time of the surrogate network. This is because not all datasets will have a community-provided network to calculate scores, and private datasets will require practitioners to train a surrogate network. MoSo achieves the best trade-off between computational requirements and performance, making it the best-performing model with rea-

0

50

100

150

200

250

300

Time-cost (m)

GraNd

OPT

Moderate

MoSo

Method

> 1 day

Time-cost (surrogate training)

Time-cost (scoring)

sonable computational demands. Notably, it outperforms the state-of-the-art method, Moderate, while being more efficient. Because of the use of large-scale linear programming in the scoring phase, OPT is significantly more time-consuming than the other methods.

0.0

0.2

0.4

0.6

0.7

0.8

Pruning Ratio

50

60

70

Top-1 Accuracy

Random

MoSo without TDA

MoSo

(a)

0.1

0.15

0.2

0.4

Sampling Rate

65

66

67

68

69

Top-1 Accuracy

(b)

4.3 Further Study

In this subsection, we perform additional ablation experiments to investigate the effect of the awareness of training dynamics, the effect of time step sampling, the effect of the parallel speed-up scheme (dataset partitioning), and the effect of the number of epochs in the surrogate training stage.

Effect of the awareness of training dynamics. Here we investigate the effect of incorporating the awareness of training dynamics into our MoSo score. To do this, we compare our method with a variant that removes this awareness by only considering the gradient from the very last epoch of the surrogate network. This variant is equivalent to using the gradient norm as a measure of sample importance. The results are shown in Figure 5(a). We can clearly see that our method outperforms the variant on both CIFAR-100 across different pruning rates. This indicates that the awareness of training dynamics is crucial for capturing the impact of a sample on the model performance and that the gradient norm alone from a converged network is not sufficient for measuring sample importance.

Effect of time step sampling. We then investigate the effect of time step sampling on the accuracy and efficiency of our method. Time step sampling is a technique that we use to reduce the computational cost of calculating Eq.(4) by randomly selecting a subset of epochs to estimate the MoSo score. However, this technique also introduces variance into the estimation and may even affect the quality of the selected coreset with a too-small sampling rate. To study this trade-off, we conduct experiments with different sampling rates and measure the performance of the final coreset on CIFAR-100. The results are shown in Figure 5(b). As expected, we observe that the mean performance decreases as the sampling rate decreases, and the variance also increases as the sampling rate decreases. This suggests that time-step sampling is a useful technique for improving the efficiency of our method, but it should be used with caution to avoid sacrificing too much accuracy.

Effect of the parallel speed-up scheme. The most time-consuming aspect of data pruning is training the surrogate network. Our framework utilizes a parallel speed-up scheme, explained in line 3 of Algorithm 1 in the main text. Essentially, we partition the original full set into several non-overlapping subsets with equivalent size S Ñ t S 1 , ..., S I u , where I represents the number of computing devices. On each device,

we can train a surrogate network on S i . Then, for each sample z P S i , we perform MoSo scoring within the current set by using M p z | S i q to approximate M p z | S q . Implementing this approach reduces the overall time overhead by I fold. Table 1 demonstrates that partitioning S into more subsets can improve data pruning's performance. This means that if the training set is vast, a single sample's impact may be buried, making it challenging to measure. In a relatively small training set, however, the effect of a single sample can be more sensitively reflected and easily captured.

Effect of the number of epochs in the surrogate

training stage. To investigate the impact of surrogate network training epochs on the ultimate data pruning performance, we augmented the number of training epochs and presented the experimental outcomes on CIFAR-100, as shown in Table 2. However, it is evident that augmenting the training duration does not lead to a uniform improvement in performance. For instance, lengthening the train-

ing duration from 50 epochs to 200 epochs only resulted in a meager 0.82 Top-1 accuracy gain, while the time consumed quadrupled. Note that this is consistent with our conclusion in Proposition 1.2 that the approximation error between the approximated MoSo and the exact MoSo value by leave-one-out-retraining increases with time (T), leading to no improvement in DP performance with longer training time.

5 Conclusion

This paper introduces a novel metric for measuring sample importance, called the Moving-oneSample-out (MoSo) score. It quantifies sample importance by measuring the change of the optimal empirical risk when a specific sample is removed from the training set. By doing so, MoSo can better distinguish important samples that contribute to the model performance from harmful noise samples that degrade it, as the former tends to lower the empirical risk, while the latter may increase it. Moreover, we propose an efficient estimator for MoSo with linear complexity and approximation error guarantees. The estimator incorporates the awareness of training dynamics by considering the gradient difference across different epochs. We conduct extensive experiments on various data pruning tasks and demonstrate the effectiveness and generalization of our method.

Limitations and future work. First, the MoSo score is actually the agreement between the gradient of a single sample and the mathematical expectation of the gradient. The higher the agreement, the sample will be given a higher score. In fact, this is based on the important assumption that the amount of information in the data set is much greater than the amount of noise. However, if the amount of noise is dominant, the usefulness of MoSo is not guaranteed. Therefore, we believe that in the future, it is very necessary to propose a variant of MoSo to adapt to high noise conditions with theoretical guarantees. Secondly, in terms of application, this paper only evaluates the performance of MoSo on classification tasks. Many practical tasks, e.g. large-scale multimodal learning, are worth considering in future work.

Social Impact. MoSo has potential influences in important applications such as data collection and data-efficient AI. Moreover, it is beneficial for reducing the computational workload during training and the cost of storing datasets, which is of great significance for environmentally friendly and energy-friendly economies. But it may be deployed for inhumane web-data monitoring. The potential negative effects can be avoided by implementing strict and secure data privacy regulations.

6 Acknowledgment

This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), Theme-based Research (T45-702/22-R), and RGC Matching Fund Scheme (RMGS). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.

References

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et. al: Palm: Scaling language modeling with pathways. arXiv:2204.02311 (2022)

Adam Paszke, Sam Gross, Soumith Chintala, G Chanan, E Yang, Zachary Devito, Zeming Lin, Alban Desmaison, L Antiga, A Lerer, et.al.: Automatic differentiation in pytorch. In: Advances in neural information processing systems Workshop (2017)

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever: Language models are unsupervised multitask learners. OpenAI blog (2019)

Alec Radford, Jong Wook Kim, Chris Hallacy, ditya Ramesh, Gabriel Goh, Sandhini Agarwa, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever: Learning transferable visual models from natural language supervision. CoRR abs/2103.00020 (2021)

Alex Krizhevsky: Learning multiple layers of features from tiny images. Technical report (2009)

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick: Segment anything. arXiv:2304.02643 (2023)

Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Bottou: Fast kernel classifiers with online and active learning. Journal of Machine Learning Research 6 (54), 1579-1619 (2005)

Baharan Mirzasoleiman, Jeff Bilmes, Jure Leskovec: Coresets for data-efficient training of machine learning models. In: International Conference on Machine Learning (2020)

Baharan Mirzasoleiman, Kaidi Cao, Jure Leskovec: Coresets for robust training of deep neural networks against noisy labels. In: Advances in Neural Information Processing Systems (2020)

Christoph Schuhmann, Romain Beaumont, Richard Vencu, et. al: LAION-5b: An open largescale dataset for training next generation image-text models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022)

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia: Selection via proxy: Efficient data selection for deep learning. In: International Conference on Learning Representations (2019)

Daphna Weinshall, Dan Amir: Theory of curriculum learning, with convex loss functions. Journal of Machine Learning Research 21 (222), 1-19 (2020)

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, Jun-Yan Zhu: Dataset distillation by matching training trajectories. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu: Making deep neural networks robust to label noise: A loss correction approach. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017)

Haizhong Zheng, Rui Liu, Fan Lai, Atul Prakash: Coverage-centric coreset selection for high pruning rates. In: International Conference on Learning Representations (2023)

Jaehong Yoon, Divyam Madaan, Eunho Yang, Sung Ju Hwang: Online coreset selection for rehearsal-based continual learning. In: International Conference on Learning Representations (2022)

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, Hyun Oh Song: Dataset condensation via efficient synthetic-data parameterization. In: International Conference on Machine Learning (2022)

Jie Hu, Li Shen, Gang Sun: Squeeze-and-excitation networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018)

Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, Alekh Agarwal: Deep batch active learning by diverse, uncertain gradient lower bounds. In: International Conference on Learning Representations (2019)

Justin Cui, Ruochen Wang, Si Si, Cho-Jui Hsieh: Dc-bench: Dataset condensation benchmark. In: Advances in Neural Information Processing Systems (2022)

Kai Wei, Rishabh Iyer, Jeff Bilmes: Submodularity in data subset selection and active learning. In: International Conference on Machine Learning (2015)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep residual learning for image recognition. In: IEEE / CVF Computer Vision and Pattern Recognition Conference (2016)

Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, Rishabh Iyer: Grad-match: Gradient matching based data subset selection for efficient deep model training. In: International Conference on Machine Learning (2021)

Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Rishabh Iyer: Generalization based data subset selection for efficient and robust learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)

Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, Rishabh Iyer: Retrieve: Coreset selection for efficient and robust semi-supervised learning. In: Advances in Neural Information Processing Systems (2021)

Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, Felix A. Wichmann: Trivial or impossible-dichotomous data difficulty masks model differences (on imagenet and beyond). In: International Conference on Learning Representations (2022)

Mansheej Paul, Surya Ganguli, Gintare Karolina Dziugaite: Deep learning on a data diet: Finding important examples early in training. In: Advances in Neural Information Processing Systems (2018)

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, Geoffrey J Gordon: An empirical study of example forgetting during deep neural network learning. In: International Conference on Learning Representations (2018)

Max Welling: Herding dynamical weights to learn. In: International Conference on Machine Learning (2009)

Mingxing Tan, Quoc Le: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning (2019)

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, others: Imagenet large scale visual recognition challenge. International journal of computer vision 115 (3), 211-252 (2015)

Ozan Sener, Silvio Savarese: Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 (2017)

Ozan Sener, Silvio Savarese: Active learning for convolutional neural networks: A coreset approach. In: International Conference on Learning Representations (2018)

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Pete Clark, Ashwin Kalyan: Learn to explain: Multimodal reasoning via thought chains for science question answering. In: Advances in Neural Information Processing Systems (2022)

Pang Wei Koh, Percy Liang: Understanding black-box predictions via influence functions. In: International Conference on Machine Learning (2017)

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer: Highresolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752 (2021)

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, et. al.: Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108 (2023)

Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, Ping Li: Dataset pruning: Reducing training data by examining generalization influence. In: International Conference on Learning Representations (2023)

Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, Christopher D Manning: Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. arXiv preprint arXiv:2107.02331 (2021)

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, Xinchao Wang: Dataset distillation via factorization. In: Advances in Neural Information Processing Systems (2022)

Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut: Conceptual 12M: Pushing webscale image-text pre-training to recognize long-tail visual concepts. In: IEEE / CVF Computer Vision and Pattern Recognition Conference (2021)

Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., Morcos, A.: Beyond neural scaling laws: beating power law scaling via data pruning. In: Advances in Neural Information Processing Systems (2022)

Suraj Kothawade, Nathan Alexander Beck, Krishnateja Killamsetty, Rishabh Iyer: Similar: Submodular information measures based active learning in realistic scenarios. In: Advances in Neural Information Processing Systems (2021)

Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, Rishabh Iyer: Prism: A unified framework of parameterized submodular information measures for targeted data subset selection and summarizationg. In: Proceedings of the AAAI Conference on Artificial Intelligence (2022)

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et. al: Language models are few-shot learners. In: Advances in Neural Information Processing Systems (2020)

Vitaly Feldman, Chiyuan Zhang: What neural networks memorize and why: Discovering the long tail via influence estimation. In: Advances in Neural Information Processing Systems (2020)

Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, Neil Shah: Graph condensation for graph neural networks. In: International Conference on Learning Representations (2022)

Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, Tongliang Liu: Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In: International Conference on Learning Representations (2023)

Ya Le, Xuan S. Yang: Tiny imagenet visual recognition challenge. CS 231N (2015)

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston: Curriculum learning. Journal of the American Podiatry Association 60 , 6 (2009)

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Liu, Baining Guo: Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021)

Zeyad Ali Sami Emam, Hong-Min Chu, Ping-Yeh Chiang, Wojciech Czaja, Richard Leapman, Micah Goldblum, Tom Goldstein: Active learning at the imagenet scale. arXiv preprint arXiv:2111.12880 (2021)

Zhiwei Deng, Olga Russakovsky: Remember the past: Distilling datasets into addressable memories for neural networks. In: Advances in Neural Information Processing Systems (2022)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method called moving-on-sample-out (MoSo) to remove less informative samples from the training data. The criteria for removed samples is based on the change in the optimal empirical risk when the sample is removed. As exact calculation of the criterion is computationally challenging, the authors propose an estimator based on gradient information. MoSo shows empirical success compared to baseline methods in data pruning, generalization between networks, and robustness to label noise.\n\nSTRENGTHS:\n- The method is based on a simple yet effective idea.\n- The computational complexity problem of MoSo is effectively tackled with the proposed estimator.\n- In the experimental section a wide range of comparison methods are used as baselines.\n- An ablation study is performed to detail the necessity of sub-elements of the method.\n- The paper is fairly easy to follow.\n- MoSo is shown to outperform baseline methods in data pruning, generalization between networks, and robustness to label noise in terms of accuracy.\n\nWEAKNESSES:\nMain points:\n- The justification for the gradient based estimator of MoSo is solely based on intuition. A more rigorous justification would be desirable. \n- The initial motivation in the introduction (line 26) states that data pruning ideas can be used to reduce training time. However, within the experimental section, only the accuracy is compared to baseline methods. It would be interesting to see how MoSo compares to baseline methods in terms of training time. Especially because random seems to perform so good and takes essentially no time to score and sample.\n- It is mentioned that MoSo is aware of training dynamics (line 57). However, it is not entirely clear what is meant by *awareness* and what part of the method is responsible for this. Furthermore, it is not entirely clear how this is different from other methods that use gradients to prune data and why it should be utilized.\n- The conclusion of Proposition 1.2 is not discussed in the main text. It would be interesting to see how close the estimator is to the true criterion. This could even be done in an experimental setting comparable to the ablation in Figure 2(b).\n\nMinor points:\n- While Tables 1,2,3 are detailed, the same results are already summarized in Figure 1. Hence, those tables seem redundant if the error bars are included in Figure 1 and the tables could be moved to the supplementary material.\n- Initially, it is unclear that MoSo requires to train a surrogate model to estimate the criterion. This should be highlighted earlier and compared to baseline methods.\n- Within the experimental section, (all tables and Figure 2(b)) it is unclear how many runs the results are averaged over and if the errors are standard deviations or some other measure of uncertainty.\n- Within the experimental section, it is unclear how the hyperparameters were chosen for the baseline methods.\n- Algorithm 1, line 1: $\\phi$ is not defined/discussed.\n- The statement \"this suggests that time step sampling is a useful technique for improving the efficiency of our method\" in line 305 is not supported by the results in Figure 2(b).\n\nAfter the Rebuttal:\nI have read all other reviews and all rebuttals. Furthermore, I thank the authors for their answers. I keep my original score.\n\nQUESTIONS:\nQuestions:\n- Q1: In Algorithm 1, line 1: What is $\\phi$?\n- Q2: How many runs were conducted? Are the aggregated statistics averages over multiple runs? What do the error bars represent? Can you clarify?\n- Q3: What is the implication of Proposition 1.2? How close is the gap of the estimator? \n- Q4: Regarding the paragraph about robustness to label noise. It is claimed that MoSo \"select informative and clean samples\" (line 288). However, it is unclear how MoSo is able to distinguish between informative and clean samples as only test accuracies are compared in Table 5. Is it possible to analyze which fraction of the pruned samples are clean and which are informative?\n- Q5: Does MoSo suggest an optimal pruning ratio, e.g., by a meaningful cut-off in the distribution of scores? \n- Q6: Is it possible to analyze whether MoSo actually selects \"representative\" samples? For example by comparing how a network trained on the pruned data performs on the full test set.\n- Q7: In Figure 2(a), there is a method called \"random\". It is unclear what this method does. I assume random data points get pruned, i.e., the score is uniform among all data points and so is the selection distribution. Can you clarify?\n- Q8: The exact scenario in which training ImageNet would take 45 years (line 152) in unclear. Can you clarify?\n\n\nPreferential comments:\n- When listing many references as in the introduction it is easier to read if they are sorted [1,2,3,4] instead of [4,1,3,2].\n- In the results tables bold the best numbers, as well as those with standard errors that overlap with the best method for fair highlighting of the best results.\n- Inconsistent presentation of tables. While in Tables 1-3 there are vertical lines between methods, results and avg rank, those lines are missing in Tables 4 and 5.\n- The statement \"famous influence function\" (line 188) may be inappropriate.\n\nTypos/grammar/other:\n- Potentially missing related work:\n\t- Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., ... & Gal, Y. (2022, June). Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning (pp. 15630-15649). PMLR.\n- References [33] and [34] are the same paper.\n- Inconsistent spelling: \"core set\", \"core-set\", \"coreset\"\n- Why are there full stops after enumeration marks, i.e., (i). and (ii).?\n- Whenever I read \"our MoSo\" I have the feeling a word is missing, like \"our MoSo score\" or \"our MoSo approach\" especially if I expand MoSo (what it stand for).\n- lines 117-121, 136, 187: I am not sure why *will* appears here. This is not the case later on, for example in lines 211-215.\n- line 123: we focus on **a** classification task?\n- line 124: i.i.d. (last dot is missing)\n- line 126: I would appreciate if you could specify the spaces of newly introduced variables, i.e., $\\delta \\in (0,1)$. Same holds for $\\eta$.\n- line 126: Please introduce $\\mathbf{w}$.\n- Equations (2) and (6): Is there an empty line in LaTeX before the equation? That would explain the spacing above the equation.\n- line 137: Please state once that by $z$ you mean a pair $(x,y)$.\n- line 157: Why \"could\" and not \"can\"?\n- Equation (4) and more: I think the transpose symbol and $T$ are too close. Especially in the supplement where $T$ also appears as an superscript. I would use another transpose symbol, i.e., `\\newcommand*{\\transpose}{{\\mkern-1.5mu\\mathsf{T}}}` (looks like $\\newcommand*{\\transpose}{{\\mkern-1.5mu\\mathsf{T}}}$) which follows the (DIN) EN ISO 80000-2:2013 standard\n- line 187: Why are 1.1 and 1.2 not clickable references?\n- line 193: There is no Proposition 1.\n- line 203: Please name $I$ as the number of subsets.\n- line 252: additional blank space\n- line 256: remove 'when'\n- line 257: doesn't -> does not\n- Sometimes I see $z:(x,y)$ or $\\epsilon:0$ in which I read and interpret the colon symbol as an equal sign and I think this notation is confusing for some people. \n- It would have helped to start the counters (line numbers, equations, etc) in the supplementary material from the counters of the main paper.\n- Appendix Equation (5): $\\mathcal{L}$ is a function, not a set (below the sum symbol). I guess $\\mathcal{S}$ is meant here. Also, consider putting brackets around the terms to clarify that both terms are within the sum! As mentioned earlier, I consider the notation $\\epsilon:0$ confusing.\n\nLIMITATIONS:\nLimitations are only mentioned in the supplementary material but could be more rigorously discussed in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a framework for data pruning that retains important samples while considering the training dynamics. While the overall methodology relies on analyzing the change in empirical risk from removing individual points, the paper introduces a first-order approximation algorithm that can be efficiently computed. Numerical results demonstrate the effectiveness of the method.

STRENGTHS:
-	The motivating idea is intuitive.
-	Numerical results on CIFAR100 and TinyImagenet suggest the method is effective.

WEAKNESSES:
-	The proof for Proposition 1.1 seems to be incorrect. Specifically $L(S/z, w)$ is defined as $L(S,w) – l(z, w) / N$, however this ignores that the empirical risk when we remove a point also needs to be re-normalized to $1/(N-1)$. I believe this breaks the proof steps from page 4 onwards. I suspect that this mis-specification carries on to the remaining theoretical analysis as well.
-	The mathematical presentation is also generally unclear. 
 - For example, in the Proof to Proposition 1.1., $L^t$ is used without definition.
 - In Proposition 1.2, the loss function is assumed Lipschitz in parameters for fixed data, but what the loss function is Lipschitz in is not clear.
-	Minor comment: there are also general presentation issues and errors in the main paper, although the authors have caught revisions in the Appendix.

QUESTIONS:
Please see comments about Proposition 1.1

LIMITATIONS:
The paper includes a discussion on limitations in the Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MoSo, a method to identify and remove the least informative samples from a large dataset. The underlying idea is to consider the impact of each sample on the optimal empirical risk. Quantifying this exactly requires leave-one-out retraining for every point, which is intractable. So, the authors provide an approximation of this score that is more efficient in that it does not require leave-one-out retraining for every point. They provide bounds on the quality of this approximation and present empirical comparisons of the data pruning approach with contemporary baselines.\n\nSTRENGTHS:\n* The proposed approximation of the MoSo score and the accompanying analysis are sound and novel to the best of my knowledge.\n* The authors present theoretical results establishing the accuracy of their approximation.\n* Empirical evaluations on benchmark vision tasks are provided that support the effectiveness of the method relative to state-of-the-art baselines.\n\nWEAKNESSES:\n* The authors state that data pruning can address the computational challenges in the introduction. However, it is not clear to me how the method can provide a computational speedup given that it needs to train a surrogate model on the entirety of the dataset (Line 8 of Algorithm 1) to compute the MoSo score approximation.\n* The claimed asymptotic computation times are difficult to understand. Upon looking at the algorithm, it seems that the power of the approximation in Eq. (4) comes from the fact that the surrogate network needs to only be trained once, rather than once per each point (as required by Eq. 3). This should be clarified. Please see the Questions section for more details.\n* The claim that randomly sampling a few time steps rather than considering all $T$ steps “reduces the overall complexity to be less than $\\mathcal O(Tn)$” does not seem sound since the expectation of a uniform sample from $\\\\{1, \\ldots, T\\\\}$ is $(T+1)/2$.\n* Only the ResNet50 architecture is used exclusively throughout the experiments to construct the pruned datasets. Diversity of architecture in the evaluations would have strengthened the method’s appeal.\n* The computational complexity of the method is quite high as it requires training the model on the entire dataset relative to the compared methods.\n\nQUESTIONS:\n1. It is not clear to me why the original MoSo score “takes $\\mathcal O(Tn^2)$ time.” More generally, the asymptotic complexities are confusing throughout because neither the batch size of SGD, nor the dimensionality of the data points $d$ are accounted for in the asymptotic analysis. Are we assuming that the batch size = $n$, i.e., regular GD and dimensionality = 1 for the samples?\n2. Related to the above, I don’t understand how the approximation in Eq. (4) is faster in an asymptotic sense than the original MoSo score in Eq. (3). Assuming that batch size = $n$ and ignoring the dimensionality of the points as the authors do, two full rounds of training to obtain $w_{\\mathcal S}^*$ and $w_{\\mathcal S \\setminus z}^*$ take $\\mathcal O(n T)$ time overall (based on the way the authors express training time). Once we have those two models, computing the difference of losses in Eq. (3) takes $\\mathcal O(n)$ time. Where is the quadratic in $n$ coming from? It would help to clarify that Eq. (4) requires only training the model once (as in Line 8 of Alg. 1), unlike Eq. (3) which requires computing $w_{\\mathcal S \\setminus z}^*$ for each $z$.\n3. What is the appeal of the data pruning method from a practical efficiency perspective, given that it requires training the surrogate network for $(T+1)/2$ iterations in expectation on the full dataset?\n\nLIMITATIONS:\nYes, they are mentioned in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper presents moving-one-sample-out (MoSo) for the purpose of coreset selection. This algorithm measures how empirical loss changes when excluding individual points during training. The paper introduces an approximation and other tricks to make this method computationally feasible. Experimentally, MoSo outperforms other methods on standard datasets, and also has nice properties like generalization to other architectures and robustness to label noise.

STRENGTHS:
- Good presentation: method is clear and contextualized properly through related work and comparison experiments
- Good results: MoSo does better on all datasets evaluated than the comparison baselines
- Method seems new and is differentiated from related work
- Interesting ablations on architecture generalization and label noise

WEAKNESSES:
- Lack of analysis on the computational cost of this method. Comparisons use same amount of samples seen during training, but this does not take into account the extra cost of calculating the coreset for some of these methods. This can be addressed with end-to-end training time or a similar metric. Though the authors mention that this is substantially cheaper than methods that require training a full-network for scoring, this information is still important, especially when considering scaling up this method to larger datasets.

QUESTIONS:
See weaknesses

Is there a reason why experiments aren't done on CIFAR-10? This seems to be a standard benchmark for this line of work.

Would also be interested in seeing this method evaluated on a larger dataset or other tasks (e.g. large-scale multi-modal learning mentioned in supplementary)

LIMITATIONS:
Addressed in weaknesses - lack of analysis on computational cost.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes a data-pruning method. The authors argue that sample importance should not be determined by sample difficulty. Alternatively, they present the MoSo score, which quantifies the changes in empirical risk upon excluding a single data point. An efficient approximation for MoSo is proposed to calculate the score with some theoretical guarantee. Experiments are conducted with common image classification benchmarks.

STRENGTHS:
[1] The proposed measure of sample importance is intuitive, and the approximation appears novel.

[2] The paper is overall well-written, with a clear description of the insights and adequate related work.

WEAKNESSES:
[1] The authors propose an approximation for the MoSo score.  However, it is unclear from the experiment how expensive the proposed method is compared to the baselines.

QUESTIONS:
[1] I appreciate the set of experiments with label noise, as it could provide evidence for the authors' insight on difficult data. I would like to see the overlapping ratio between samples with injected noise and pruned data. This overlapping can be more informative than simple test accuracy in demonstrating whether the proposed method can exclude hard but noisy data.

[2] The authors mainly report the trade-off of top-1 accuracy and data pruning ratio, which could demonstrate the effectiveness of a method. It would be interesting to see the data pruning ratio across classes to better understand the method. Does the pruning change the data sample distribution, and does the method improve/hurt certain classes?

LIMITATIONS:
The authors didn't discuss limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06