SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pruning the dataset does reduce the training time without significantly degrading the accuracy [13, 14]. However, these techniques require fully training the model on the whole dataset to identify the samples to be removed, which is comput intensive. While most of the proposed solutions perform well on small datasets such as CIFAR, many fail to maintain accuracy on larger datasets like ImageNet [27].
Selective-Backprop [17] combines importance sampling and online data pruning. It reduces the number of samples to train on by using the output of each sample's forward pass to estimate the sample's importance and cuts a fixed fraction of the dataset at each epoch. While this method shows notable speedups, it has been evaluated only on tiny datasets without providing any measurements on how accuracy is impacted. In addition, the authors allow up to  $10\%$  reduction in test error in their experiments. EIF [44] is similar to Selective-Backprop: it reduces the computation cost of training by filtering out the samples with the lowest loss.  $\mathrm{E}^2$ -Train [45] shows that the combination of randomly dropping samples during training with selective layer update in CNNs can significantly reduce the training time, while slightly degrading the accuracy. However,  $\mathrm{E}^2$ -Train targets edge environments and is evaluated only on very small datasets.
GRAD-MATCH [18] is an online method that selects a subset of the samples that would minimize the gradient matching error, where the error of the gradients of a matched subset samples (and their weights) becomes minimum. To avoid the impractical storing and computation of the optimization of the gradients of all instances, the authors approximate the gradients by only using the gradients of the last layer, use a per-class approximation, and run data selection every  $R$  epochs, in which case, the same subsets and weights will be used between epochs. The infrequent selection, however, means the model is limited in its capacity to learn in intermediate epochs - where selection occurs - since it trains on the same limited subset of samples. This often leads to a longer numbers of epochs needed to converge to the same validation accuracy that can be achieved by the baseline or the baseline reaching much higher accuracy [29]. Another important point worth mentioning is that GRAD-MATCH is impractical in distributed training, which is a de facto requirement in large dataset and models (e.g., the DeepCAM model/dataset). That is since the approximation of the classes would require very expensive high-volume collective communication operations to gather the gradients scattered across different samples belonging to the same class. The communication cost would be  $O(N.R.G)$  where  $N$  is the number of samples,  $R$  is the frequency of selection, and  $G$  is the gradients (of the last layer, if gradient approximation is to be used). Distributed GRAD-MATCH would require a scatter communication to collect the class approximations and a collective all-reduce of the gradients to then do the matching optimization. This is practically a very high cost for communication per epoch that could even exceed the average time per epoch. Finally, the mini-batch variant of GRAD-MATCH can only be effective for small mini-batches. However, since in distributed training the mini-batch grows with the scale (i.e., the mini-batch aggregates the local mini-batch of all workers), the cost of communication amplifies by B (where B is mini-batch size).