SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoru Tan $^{1,3}$
Sitong  $\mathbf{W}\mathbf{u}^{2,3}$
Fei  $\mathbf{Du}^{3,4}$
Yukang Chen
Zhibin Wang<sup>3,4</sup>
Fan Wang $^{3,4}$
$^{1}$ HKU  $^{2}$ CUHK  $^{3}$ DAMO Academy, Alibaba Group  $^{4}$ Hupan Lab, Zhejiang Province
Official Implementation
In this paper, we propose a novel data-pruning approach called moving-one-sample-out (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings.
The recent advances in AI have been largely driven by the availability of large-scale datasets [31, 3, 6, 34, 41, 10, 37], which enable the training of powerful models [51, 3, 45, 6, 1, 4, 36]. However, such datasets also pose significant challenges in terms of computational and storage resources. It is important to note that these datasets may contain redundant or noisy samples that are either irrelevant or harmful to the model's performance. Data pruning techniques aim to address these issues by removing such samples and retaining a smaller, more compact core set of training samples [11, 28, 42, 27, 48, 38, 15]. This can not only reduce the costs of model training and data storage but also maintain the performance of the model compared to the original dataset.
Existing approaches can be broadly categorized into three major groups: pruning by importance criteria [11, 42, 28, 27, 26, 46], coverage or diversity-driven methods [48, 29, 33, 15], and optimization-based methods [38, 8, 23, 24, 21, 44]. Among these, the first group of studies is the most effective and popular. These studies assume that hard samples are critical and informative core-set samples, and thus, they design difficulty-based metrics to assess sample importance. Such metrics include prediction entropy [11], forgetting [28] or memorization [46] score, gradient norm [27], E2LN (variance of prediction) [27], self-supervised prototype distance [42], diverse ensembles [26], and others.
arXiv:2310.14664v1 [cs.LG] 23 Oct 2023
Limitations and Motivations. The methods we discussed have some major drawbacks: (i). Hard samples are not necessarily important samples. For example, noisy samples [48] and outliers [39] often lead to high losses, which makes it difficult for importance criteria [11, 27] to distinguish them from truly important samples. (ii). Training dynamics are rarely considered. The mainstream methods [27, 46, 26, 11, 42, 15, 48, 38] do not possess the awareness of training dynamics as they generally utilize a converged surrogate network for data selection. This may favor samples that are difficult or influential in the later stages of training, but not necessarily in the earlier stages or the whole training process [50, 12].
Our Method. In this paper, we propose a new data pruning algorithm, which involves the newly proposed Moving-one-Sample-out (MoSo) score with an efficient and error-guaranteed estimator. To address the first limitation, MoSo utilizes the change of the optimal empirical risk when removing a specific sample from the training set to measure sample importance instead of only focusing on sample difficulty. By doing so, MoSo can better separate important samples from harmful noise samples, as the former tends to lower the empirical risk, while the latter may increase it. However, MoSo is too costly to compute exactly as it needs brute force leaving-one-out-retraining. Therefore, we propose an efficient first-order approximator with linear complexity and guaranteed approximation error. The proposed approximation is simple: samples whose gradient agrees with gradient expectations at all training stages will get higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. The second limitation is addressed since MoSo comprehensively considers information from different training stages.
We evaluate our MoSo on CIFAR-100 [5], Tiny-ImageNet [49], and ImageNet-1K [31] under different pruning ratios. As shown in Figure 1, our MoSo significantly surpasses the previous state-of-the-art methods, especially for high pruning ratios. Besides, experimental results demonstrate that the coreset selected by our MoSo under one network (such as ResNet) can generalize well to other unseen networks (such as SENet and EfficientNet) (refer to Figure 3(a) and Figure 3(b)). Additionally, we study the robustness of our MoSo on datasets with synthetic noisy labels (refer to Figure 3(c) and Figure 3(d)). It can be seen that our MoSo performs best on average, and surpasses the previous methods based on difficulty-based importance criteria by a large margin.
Finding important samples is not only the purpose of data pruning, but also the core step in many machine learning tasks and problems, like active learning [7, 52, 32, 39, 19], noisy learning [9], and continuous learning [16]. In data-efficient deep learning, there are also some related topics like data distillation [13, 40, 53] and data condensation [17, 47, 20]. Unlike data pruning, they focus on synthesizing a small but informative dataset as an alternative to the original large-scale dataset. Existing data selection/pruning approaches could be broadly divided into several categories, pruning by importance criteria [11, 42, 28, 27, 26, 46], coverage or diversity driven methods [48, 29, 33, 15], optimization-based methods [38, 8, 23-25, 21, 44, 43].
Pruning by importance criteria. This group of studies is the most popular. Generally, they assume that hard samples are critical and informative core-set samples and thus design difficulty-based metrics to assess sample importance. The EL2N score [27] measures the data difficulty by computing the average of the  $\ell_2$ -norm error vector from a set of networks. GraNd [27] measures the importance by calculating the expectation of the gradient norm. The Forgetting score [28] counts how many times a model changes its prediction from correct to incorrect for each example during the training process. Memorization [46] assigns a score to each example based on how much its presence or absence in the training set affects the model's ability to predict it correctly. Diverse ensembles [26] gave a score to each sample based on how many models in a group misclassified it. However, hard samples are not necessarily good for model training [42]. For example, noisy samples [48] and outliers [39] often lead to high losses, which makes it difficult for importance criteria [11, 27] to distinguish them from truly important samples. As a comparison, our MoSo score measures sample importance instead of only focusing on sample difficulty by calculating the change of the optimal empirical risk when removing a specific sample from the training set. By doing so, it can better separate important samples from harmful noise samples, as the former tends to lower the empirical risk, while the latter may increase it.
Coverage or diversity driven methods. Sener et. al. [32] applied greedy k-center to choose the coreset with good data coverage. BADGE [19] is a diversity-based selection method in active learning that clusters the gradient embeddings of the current model using k-means++ and selects a subset from each cluster. CCS [15] balances the data distribution and the example importance in selecting data points. Moderate [48] chooses data points with scores near the median score. Note that some diversity-driven methods, such as CCS [15] and Moderate [48], can use any selection criterion, such as EL2N score [27], as a basis.
**Optimization-based methods.** In addition, a line of recent works proposed to select data by optimization, such as gradient matching [8, 23], bi-level optimization [24, 25], submodularity [21, 44, 43]. One of the most advanced methods, the optimization-based dataset pruning [38], builds an algorithm over the sample-wise influence function [35] to remove samples with minimal impact and guarantees generalization. However, like mainstream methods [27, 46, 26, 11, 42, 15, 48, 38], it does not account for the effect of samples on the training dynamics, as it only uses the information from the final model. This may favor samples that are difficult or influential in the later stages of training, but not necessarily in the earlier stages or the whole training process [50, 12]. In our work, the proposed method is fully training-dynamic-aware since the MoSo's approximator comprehensively considers information from different training stages.
In the following, we will first present the background knowledge in Section 3.1. Following that, we will elaborate on the proposed MoSo score for assessing sample importance in Section 3.2. Furthermore, we will introduce an efficient approximator of MoSo in Section 3.3. Section 3.4 shows the along with its complexity analysis and error guarantees.
In this work, we focus on the classification task, where  $\mathcal{S} = \{(x_i,y_i)|_{i = 1}^N\}$  denotes the training set, drawn i.i.d from an underlying data distribution  $P$ , with input vectors  $x\in \mathbb{R}^d$  and one-hot label vectors  $y\in \{0,1\} ^K$ . Let  $l(\cdot)$  denote the widely used cross-entropy loss function for classification tasks. Given a pruning ratio  $\delta$  and a parameterized deep network  $f_{\mathrm{w}}$ , the data pruning task aims to find the most representative training subset  $\hat{S}\subset S$  while pruning the remaining samples. This can be formulated as:
$$
\hat {S} = \underset {D \subset S} {\arg \min } \mathbb {E} _ {z: (\boldsymbol {x}, y) \sim P} \left[ l (z, \mathbf {w} _ {D} ^ {*}) \right], \tag {1}
$$
where  $(|S| - |D|) / |S| = \delta, |\cdot|$  represents the cardinality of a set, and  $\mathbf{w}_D^*$  indicates the optimal network parameter trained on  $D$  with the stochastic gradient descent (SGD) optimizer. The SGD optimizer updates the parameters as follows:
$$
\mathbf {w} _ {t} = \mathbf {w} _ {t - 1} - \eta_ {t} \nabla \mathcal {L} \left(\mathcal {B} _ {t}, \mathbf {w} _ {t - 1}\right), \tag {2}
$$
where  $t \in \{1, \dots, T\}$ ,  $\eta_t$  is the learning rate at the  $t$ -th step, and  $\mathcal{B}_t$  represents the mini-batch,  $\nabla$  is the gradient operator with respect to network parameters,  $\mathcal{L}(\cdot)$  is the average cross-entropy loss on the given set/batch of samples.
3.2 Definition for Moving-one-Sample-out
Here, we will describe the details of Moving-one-Sample-out (MoSo) score.
Definition 1. The MoSo score for a specific sample  $z$  selected from the training set  $S$  is
$$
\mathcal {M} (z) = \mathcal {L} \left(\mathcal {S} / z, \mathbf {w} _ {\mathcal {S} / z} ^ {*}\right) - \mathcal {L} \left(\mathcal {S} / z, \mathbf {w} _ {\mathcal {S}} ^ {*}\right), \tag {3}
$$
where  $S / z$  indicates the dataset  $S$  excluding  $z$ ,  $\mathcal{L}(\cdot)$  is the average cross-entropy loss on the considered set of samples,  $\mathbf{w}_S^*$  is the optimal parameter trained on the full set  $S$ , and  $\mathbf{w}_{S / z}^*$  is the optimal parameter on  $S / z$ .
The MoSo score measures the importance of a specific sample  $z$  by calculating how the empirical risk over  $\mathcal{S} / z$  changes when removing  $z$  from the training set. Specifically, with a representative
(important and with proper annotation) sample  $z$ , retaining it can promote training and result in a lower empirical risk while removing it could be harmful to the training and result in a higher empirical risk. Hence,  $\mathcal{M}(z) > 0$ . On the contrary, when  $z$  is unrepresentative,  $\mathcal{M}(z) \approx 0$ . Moreover, if the selected data point  $z$  is harmful (e.g. noisy samples), the retention of  $z$  is a hindrance to the learning process on  $S/z$ , so the risk would be high; removing the harmful  $z$  could result in a lower risk value. Hence,  $\mathcal{M}(z) < 0$ .
3.3 Gradient-based approximator
The exact calculation of MoSo, as shown in Eq.(3), has a quadratic time complexity of  $\mathcal{O}(Tn^2)$  considering a dataset with  $n$  samples and a total of  $T$  training iterations required to obtain the surrogate network. However, this is practically infeasible; for instance, it may take more than 45 years to process the ImageNet-1K dataset using a Tesla-V100 GPU. To address this issue, we propose an efficient first-order approximator for calculating the MoSo score, which reduces the complexity to  $\mathcal{O}(Tn)$ . This approximation not only significantly decreases the computational requirements but also maintains the effectiveness of the MoSo score in practical applications.
Proposition 1.1. The MoSo score could be efficiently approximated with linear complexity, that is,
$$
\hat {\mathcal {M}} (z) = \mathbb {E} _ {t \sim \text {u n i f o r m} \{1, \dots , T \}} \left(\frac {T}{N} \eta_ {t} \nabla \mathcal {L} \left(\mathcal {S} / z, \mathbf {w} _ {t}\right) ^ {\mathrm {T}} \nabla l (z, \mathbf {w} _ {t})\right), \tag {4}
$$
where  $S / z$  indicates the dataset  $S$  excluding  $z$ ,  $l(\cdot)$  is the cross-entropy loss function and  $\mathcal{L}(\cdot)$  means the average cross-entropy loss,  $\nabla$  is the gradient operator with respect to the network parameters, and  $\{(w_t, \eta_t)|_{t=1}^T\}$  denotes a series of parameters and learning rate during training the surrogate network on  $S$  with the SGD optimizer.  $T$  is the maximum time-steps and  $N$  is the training set size.
The MoSo score approximator in Eq.(4) essentially represents the mathematical expectation of the inner product between the gradient with respect to network parameters considering only sample  $z$  and the gradient using the training set excluding  $z$  (denoted as  $S / z$ ) over  $T$  learning iterations. A sample  $z$  will be assigned a higher MoSo score if the mathematical expectation of the inner product is larger. This can be intuitively understood as follows: if the gradient  $\nabla l(z, \mathbf{w})$  from sample  $z$  is consistent with the average gradient vector  $\nabla \mathcal{L}(S / z, \mathbf{w})$ , it implies that optimizing the network using sample  $z$  will yield a similar effect on reducing the empirical risk as using all remaining samples. This indicates that sample  $z$  is an important and representative sample. Concurrently, according to Eq.(4), it is also assigned a high MoSo score, which aligns with the intuition.
3.4 Theoretical analysis of tightness and complexity
First, we provide a rigorous mathematical justification to show that there is a theoretical guarantee for the error between the approximator we provide and the exact score by the brute-force leave-one-out retraining.
Proposition 1.2. By supposing the loss function is  $\ell$ -Lipschitz continuous and the gradient norm of the network parameter is upper-bounded by  $g$ , and setting the learning rate as a constant  $\eta$ , the approximation error of Eq. (4) is bounded by:
$$
\left| \mathcal {M} (z) - \hat {\mathcal {M}} (z) \right| \leqslant \mathcal {O} \left(\left(\ell \eta + 1\right) g T + \eta g ^ {2} T\right), \tag {5}
$$
where  $T$  is the maximum iterations.
The proposition shows that approximation error is positively correlated with many factors such as training duration  $T$ , gradient norm  $g$ , and learning rate  $\eta$ . In order to control the impact of approximate errors, in practice, we will not train the surrogate network to complete convergence, instead, we will only update a small number of epochs.
Complexity analysis. We show that Eq.(4) efficiently approximates the MoSo score with linear complexity. Specifically, calculating the overall gradient information requires a time complexity of  $\mathcal{O}(n)$ . Additionally, computing the expectation of the gradient over different training iterations in Eq.(4) takes  $T$  steps, resulting in a total complexity of  $\mathcal{O}(Tn)$ . In practice, we can randomly sample a few time steps rather than considering all  $T$  steps to calculate the mathematical expectation, reducing the overall complexity to be less than  $\mathcal{O}(Tn)$ . Moreover, the optimized gradient calculation operator,
Algorithm 1: Data Pruning with MoSo.
Require: Dataset  $\mathcal{S} = \{(x_i,y_i)|_{i = 1}^N\}$  , pruning ratio  $\delta$    
Require: Random initialized the parameter  $\mathbf{w}_0$  of a network;   
Require: cross-entropy loss  $l(\cdot)$  , SGD optimizer;   
Require: Learning-rate scheduler  $\{\eta_1,\dots,\eta_T\}$  , maximum iteration  $T$  . 1: Initialize the sample-wise score set  $\nu = \phi$  as a null set; 2: if multiple computing devices are available then 3: Partitioning  $\mathcal{S}$  into  $\mathbb{S}: \{S_1, \ldots, S_I\}$ ; //With parallel acceleration. 4: else 5:  $\mathbb{S} = \{\mathcal{S}\}$  //Without acceleration. 6: end if 7: for  $S_i \in \mathbb{S}$  do 8:  $\{(w_t, \eta_t)|_{t=1}^T\} = \mathrm{SGD}\Big(\mathcal{L}(S_i, w_0), T, \{\eta_1, \dots, \eta_T\}\Big)$ ; //Train the surrogate network. 9: for  $z \in S_i$  do 10:  $\mathcal{M}(z) \gets \mathbb{E}_{t \sim \text{uniform}[1, \dots, T]} \big( \eta_t \nabla \mathcal{L}(\mathcal{S}/z, w_t)^{\mathrm{T}} \nabla l(z, w_t) \big)$ ; //MoSo scoring. 11: Merge into the score set  $\mathcal{V} \gets \mathcal{V} + \{\mathcal{M}(z)\}$  12: end for 13: end for 14: return  $\hat{S} \gets \text{Pruning}(\mathcal{S}|\mathcal{V}, \delta)$  //Data pruning.
implemented by advanced deep learning frameworks such as PyTorch [2], further accelerates the computation, making it more feasible for practical applications.
3.5 Data Pruning with MoSo
This section presents the pipeline for utilizing the approximated MoSo score in data pruning and coreset selection. The pseudocode is provided in Algorithm 1. In the MoSo scoring step (see line 10 of Algorithm 1), we employ Eq.(4) from Proposition 1 to calculate the MoSo score. In practice, there is no need to sum all the time steps  $\{1,\dots,T\}$  when calculating the mathematical expectation. Instead, an efficient approach is randomly sampling several time steps for calculating the average or expectation, reducing the overall computational cost. In the data pruning step, samples with low scores are pruned, and the proportion of pruned data corresponds to the predefined ratio  $\delta$ .
Parallel acceleration by dataset partitioning. To enhance the practical applicability of MoSo on large-scale datasets, we propose a parallel acceleration scheme that can be employed when multiple GPU devices are available. Specifically, before training the surrogate network, we initially divide the original full dataset into a series of non-overlapping subsets. This approach enables efficient processing by leveraging the parallel computing capabilities of multiple GPUs, where the number of subsets should be no more than the available computing devices. We select a subset from  $\mathbb{S}$  without replacement for each device and then perform training and scoring within the chosen subset, following Algorithm 1. Finally, MoSo scores from different devices are combined together. As long as the number of samples in a subset is large enough to approximately represent the overall statistics of the dataset, this partitioning scheme will not compromise performance while significantly reducing computation time by a factor of  $I$ . This approach is particularly useful for large-scale datasets.
Datasets and baselines. We evaluate our method on three well-known public benchmarks: the CIFAR-100 [5], which contains 50,000 training examples of 100 categories; the Tiny-ImageNet [49], which has 100,000 images of 200 classes; and the ImageNet-1K [31], which covers 1000 classes with more than 1M training images. We compare our method with a range of baselines, including: (1). Random selection; (2). Herding [29]; (3). Forgetting [28]; (4). GraNd [27]; (5). EL2N [27];
(a) CIFAR-100
(b) Tiny-ImageNet
Figure 1: Performance comparison of our proposed MoSo and other baseline methods on three image classification datasets: CIFAR-100 [5], Tiny-ImageNet [49], and ImageNet-1K [31]. The results show that our approach outperforms most of the baselines, especially for the high pruning rate (e.g.,  $70\%$ ,  $80\%$ ).
(c) ImageNet-1K
(6). Optimization-based Dataset Pruning (OPT) [38]; (7). Self-supervised pruning (SSP) [42]; (8). Moderate [48].
Implementation details. We implement our method in Pytorch [2]. All the experiments are run on a server with 8 Tesla-V100 GPUs. Unless otherwise specified, we use the same network structure ResNet-50 [22] for both the coreset and the surrogate network on the full data. We keep all hyperparameters and experimental settings of training before and after dataset pruning consistent. We train the surrogate network on all datasets for 50 epochs. To estimate the mathematical expectation in Eq.(4) from Proposition 1, we randomly sample 10 time steps. Thus, MoSo can compute gradients from multiple epochs without increasing the overall time cost significantly, compared to methods that need to train a network fully (e.g. 200 epochs for CIFAR-100) before calculating the scores.
In the following subsections, we present the detailed performance comparison of our method and baselines on three experiments: data pruning, generalization to unseen structures, and robustness to label noise.
Data pruning. As Figure 1(a) and 1(b) show, our method significantly surpasses the SOTA method [48] on CIFAR-100 and Tiny-ImageNet at high pruning ratios. Note that some selected baselines perform worse than the random selection, especially when with high pruning ratios, e.g. the results of the classic EL2N on CIFAR100, while our method doesn't suffer from this problem. Furthermore, in Figure 1(c), our method achieves satisfactory performances on ImageNet1K across different pruning rates, showing its effectiveness on largescale and complex datasets. These results also indicate that our method can capture the sample importance more accurately and robustly than the existing methods.
Figure 2: We show the class-wise accuracy before (bars named Fullset) and after (bars named MoSo) applying our MoSo approach. The experiment is conducted on CIFAR-100. We chose ResNet-18 as the network architecture and set the pruning ratio to be 0.8.
To study whether our algorithm improves/hurts certain classes, we visualize the class-wise accuracy before and after applying our MoSo data pruning approach in Figure 2. We observe a significant correlation between the two, with a Spearman correlation coefficient of 0.913 and a P value of
(a) ResNet to SENet
(b) ResNet to EfficientNet
Figure 3: In (a) and (b), we study the generalization performance of MoSo and other baselines on CIFAR-100 from ResNet-50 to SENet (R to S) and ResNet-50 to EfficientNet-B0 (R to E). In (c) and (d), we show the robustness against label noise of MoSo and other baselines on CIFAR-100, where the labels are randomly replaced with any possible label with a  $20\%$  probability.
(c) Pruning-ratio  $= 70\%$
(d) Pruning-ratio  $= 80\%$
0.0295. This indicates that the performance before and after pruning with MoSo is consistent across categories, and no significant improvement or harm to any particular category is observed.