SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Generalization test. To test whether the pruning results are overfitting to the specific network architecture, we evaluate MoSo's generalization ability to unseen architectures. Following the protocol in [48], we use ResNet-50 as the surrogate network for scoring, and training different network architectures, SENet [18] and EfficientNet-B0 [30], on the selected data. Figure 3(a) and Figure 3(b) show the experimental results on different network architectures. MoSo exhibits a satisfying generalization ability to unseen models and consistently outperforms or matches the state-of-the-art methods such as SSP [42], and OPT [38].
Robustness test. Label noise is a common challenge in real-world applications. Therefore, how to improve the model robustness to label noise is an important and popular problem. In this section, we study the robustness of MoSo to label noise by conducting comparative experiments on CIFAR-100 and Tiny-ImageNet with synthetic label noise. Specifically, we inject label noise [14] into the two datasets by randomly replacing the labels for a percentage of the training data with all possible labels. The noise rate is set to  $20\%$  for all the experiments. We use ResNet-50 as the network architecture and keep all experimental settings consistent with the previous data pruning experiments. The results are shown in Figure 3(c) and Figure 3(d). We observe that some difficulty-based importance criteria don't work very well, such as Forgetting, GraNd on both settings, while Herding [29], Moderate [48] and our MoSo, perform well and significantly outperform other baselines in all settings. Our MoSo achieves comparable performance with the best baseline, Herding [29], only lagging behind Herding by less than  $1\%$  Top-1 acc.
4.2 Computational Efficiency
We evaluated MoSo and the other baseline methods on a server with 8 Tesla V100 GPUs. We used the CIFAR-100 dataset and the ResNet50 backbone for our experiments. It should be noted here that we also take into account the training time of the surrogate network. This is because not all datasets will have a community-provided network to calculate scores, and private datasets will require practitioners to train a surrogate network. MoSo achieves the best trade-off between computational requirements and performance, making it the best-performing model with rea
Figure 4: Time-core comparison between our MoSo and other baselines. Please note that when implementing the GraNd method, we don't take the summation of the gradient norm from all epochs, instead, we use the same time-step sampling scheme as MoSo.
sonable computational demands. Notably, it outperforms the state-of-the-art method, Moderate, while being more efficient. Because of the use of large-scale linear programming in the scoring phase, OPT is significantly more time-consuming than the other methods.
(b)
Figure 5: Ablation study on the effect of (a) incorporating the training-dynamic-awareness (TDA) into the MoSo score, and (b) using different time step sampling rates on the accuracy of the selected coreset (with a pruning ratio of 0.4). The experiments are conducted on CIFAR-100 with ResNet-50 as the network architecture.
4.3 Further Study
In this subsection, we perform additional ablation experiments to investigate the effect of the awareness of training dynamics, the effect of time step sampling, the effect of the parallel speed-up scheme (dataset partitioning), and the effect of the number of epochs in the surrogate training stage.
Effect of the awareness of training dynamics. Here we investigate the effect of incorporating the awareness of training dynamics into our MoSo score. To do this, we compare our method with a variant that removes this awareness by only considering the gradient from the very last epoch of the surrogate network. This variant is equivalent to using the gradient norm as a measure of sample importance. The results are shown in Figure 5(a). We can clearly see that our method outperforms the variant on both CIFAR-100 across different pruning rates. This indicates that the awareness of training dynamics is crucial for capturing the impact of a sample on the model performance and that the gradient norm alone from a converged network is not sufficient for measuring sample importance.
Effect of time step sampling. We then investigate the effect of time step sampling on the accuracy and efficiency of our method. Time step sampling is a technique that we use to reduce the computational cost of calculating Eq.(4) by randomly selecting a subset of epochs to estimate the MoSo score. However, this technique also introduces variance into the estimation and may even affect the quality of the selected coreset with a too-small sampling rate. To study this trade-off, we conduct experiments with different sampling rates and measure the performance of the final coreset on CIFAR-100. The results are shown in Figure 5(b). As expected, we observe that the mean performance decreases as the sampling rate decreases, and the variance also increases as the sampling rate decreases. This suggests that time-step sampling is a useful technique for improving the efficiency of our method, but it should be used with caution to avoid sacrificing too much accuracy.
Effect of the parallel speed-up scheme. The most time-consuming aspect of data pruning is training the surrogate network. Our framework utilizes a parallel speed-up scheme, explained in line 3 of Algorithm 1 in the main text. Essentially, we partition the original full set into several non-overlapping subsets with equivalent size  $S \rightarrow \{S_1, \dots, S_I\}$ , where  $I$  represents the number of computing devices. On each device,
Table 1: The effect of dataset partitioning on the final data pruning performance, where the pruning ratio is 0.2. The bold one represents the settings used in this work.
Subsets number I	1	2	5	10
MoSo (ours)	74.35	75.11	75.76	75.81
we can train a surrogate network on  $S_{i}$ . Then, for each sample  $z \in S_{i}$ , we perform MoSo scoring within the current set by using  $\mathcal{M}(z|S_i)$  to approximate  $\mathcal{M}(z|S)$ . Implementing this approach reduces the overall time overhead by  $I$  fold. Table 1 demonstrates that partitioning  $S$  into more subsets can improve data pruning's performance. This means that if the training set is vast, a single sample's impact may be buried, making it challenging to measure. In a relatively small training set, however, the effect of a single sample can be more sensitively reflected and easily captured.
Effect of the number of epochs in the surrogate training stage. To investigate the impact of surrogate network training epochs on the ultimate data pruning performance, we augmented the number of training epochs and presented the experimental outcomes on CIFAR-100, as shown in Table 2. However, it is evident that augmenting the training duration does not lead to a uniform improvement in performance. For instance, lengthening the train
ing duration from 50 epochs to 200 epochs only resulted in a meager 0.82 Top-1 accuracy gain, while the time consumed quadrupled. Note that this is consistent with our conclusion in Proposition 1.2 that the approximation error between the approximated MoSo and the exact MoSo value by leave-one-out-retraining increases with time (T), leading to no improvement in DP performance with longer training time.
Table 2: The effect of surrogate network training epochs on the final data pruning performance, where the pruning ratio is 0.2.
Training epochs	50	100	150	200
MoSo	75.76	76.41	76.19	76.58
This paper introduces a novel metric for measuring sample importance, called the Moving-one-Sample-out (MoSo) score. It quantifies sample importance by measuring the change of the optimal empirical risk when a specific sample is removed from the training set. By doing so, MoSo can better distinguish important samples that contribute to the model performance from harmful noise samples that degrade it, as the former tends to lower the empirical risk, while the latter may increase it. Moreover, we propose an efficient estimator for MoSo with linear complexity and approximation error guarantees. The estimator incorporates the awareness of training dynamics by considering the gradient difference across different epochs. We conduct extensive experiments on various data pruning tasks and demonstrate the effectiveness and generalization of our method.
Limitations and future work. First, the MoSo score is actually the agreement between the gradient of a single sample and the mathematical expectation of the gradient. The higher the agreement, the sample will be given a higher score. In fact, this is based on the important assumption that the amount of information in the data set is much greater than the amount of noise. However, if the amount of noise is dominant, the usefulness of MoSo is not guaranteed. Therefore, we believe that in the future, it is very necessary to propose a variant of MoSo to adapt to high noise conditions with theoretical guarantees. Secondly, in terms of application, this paper only evaluates the performance of MoSo on classification tasks. Many practical tasks, e.g. large-scale multimodal learning, are worth considering in future work.
Social Impact. MoSo has potential influences in important applications such as data collection and data-efficient AI. Moreover, it is beneficial for reducing the computational workload during training and the cost of storing datasets, which is of great significance for environmentally friendly and energy-friendly economies. But it may be deployed for inhumane web-data monitoring. The potential negative effects can be avoided by implementing strict and secure data privacy regulations.
6 Acknowledgment
This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), Theme-based Research (T45-702/22-R), and RGC Matching Fund Scheme (RMGS). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.