SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We run our experiments on a supercomputer with 1000s of compute nodes, each equipped with 2 Intel Xeon Gold 6148 CPUs, 384 GiB of RAM, 4 NVidia V100 GPUs, and Infiniband EDR NICs  $(100\mathrm{Gbps}\times 2)$ . We run 4 MPI ranks per compute node so that each rank has a dedicated access to a GPU.
Table 7: Datasets and Models Used in Experiments (* Down-stream training using the pre-trained model).
Model	Dataset	#Samples	#Epoch	#GPUs	minibatch (per GPU)	Task
Resnet50 [36]	ImageNet-1K [19]	1.2M	100	32	64	Image Classification
EfficientNet-b3 [37]	32
WideResNet-28-10 [39]	CIFAR-100 [41]	50K	200	32	32	Image Classification
DeepCAM [4]	DeepCAM [4]	~ 122K	35	1024	1	Image Segmentation
DeiT-Tiny-224 [42]	Fractal-3K [6]	3M	80	32	16	Image Classification
(*) CIFAR-10 [41]	50K	1000	8	96
(*) CIFAR-100 [41]	50K	1000	8	96
B.2. Model training method details and dataset information:
Table 7 summarizes the models and datasets used in this work. In details, we evaluate KAKURENBO using several models on various datasets as the following:
- ImageNet-1K [19]: We use the subset of the ImageNet dataset containing 1000 classes each containing around 1300 images (1,282,048 images in total). We also test the trained model on the validation set of 50,000 samples. We train ResNET-50 and EfficientNet-b3 provided by 'torchvision v0.12.0' on ImageNet-1K dataset.
- CIFAR-10/CIFAR-100 [41]: The CIFAR-10/CIFAR-100 dataset dataset consists of 60,000 colour images. It has 100 categories each containing 600 images. The dataset provides 50,000 training images and 10,000 test images with a size of  $32 \times 32$  pixels. CIFAR-100 dataset is available at https://www.cs.toronto.edu/ kriz/cifar.html.
- DeepCAM [4]: DeepCAM dataset for image segmentation, which consists of approximately 122K samples and requires 8.8TBs of storage. We use the settings in [4] to train DeepCAM with the top learning rate of 0.0055.
- Fractal-3K [6] A rendered dataset from the Visual Atom method [6]. Fractal-3K dataset comprise of 3 million images of visual atoms, where the number of classes is  $\mathrm{C} = 3000$  and the number of images per class is  $\mathrm{N} = 1000$ . We train the DeiT-Tiny-224 model on Fractal-3K dataset and fine tune it with CIFAR-10 and CIFAR-100 datasets.
B.3. Hyper-parameters
It is worth noting that we follow the hyper-parameters reported in [38] for training ResNet-50, [39] for training WideResNet-28-10, [37] for training EfficientNet-b3, and [4] for DeepCAM. We also use the setting in [6] for both pretrain and finetune tasks in Fractal-3K. Table 8 shows the detail of our hyper-parameters. Specifically, We follow the guideline of 'TorchVision' to train the ResNet-50 that uses the CosineLR learning rate scheduler  $^4$ , auto augments, and random erasing, etc [38]. We also set the weight decay to  $1e - 05$  and crop the input image to  $176\times 176$  pixels and train for a long number of epochs, i.e., 600 (The ResNet-50 setting). We train the WideResNet-28-4 on the CIFAR-100 dataset in 200 epochs following the setting in [39]. Specifically, we use the base learning rate of  $0.025\times k$ , momentum 0.9, and weight decay 0.0005. For EfficientNet-b3, we use RMSProp optimizer with momentum 0.9; batch norm momentum 0.99 weight decay  $1e - 5$  (following [37]). We use an initial learning rate of 0.016 that decays by 0.9 every 2 epochs. We set the minibatch size
4implemented by timm https://github.com/huggingface/pytorch-image-models/tree/main/timm
Table 8: Hyper-parameters used for different training in the paper and the baseline top-1 testing accuracy. We also consider different hyper-parameters for ResNet-50 model on ImageNet-1K dataset.
ImageNet-1K	CIFAR-100	Fractal-3K	CIFAR-10	CIFAR-100
ResNet-50	ResNet-50 (A)	ResNet-50 (B)	EfficientNet-b3	WideResNet-28-10	DeiT-Tiny-224
Train Res	224	224	224	224	32	224	224	224
Test Res	232	224	232	224	32	-	32	32
Epochs	600	100	600	100	200	80	1000	1000
Number of workers	32	32	32	32	32	32	8	8
Batch size	2048	1024	1024	1024	1024	512	768	768
Optimizer	SGD	SGD	SGD	SRMSProp	SGD	adamw	SGD	SGD
Momentum	0.9	0.9	0.9	0.9	0.9	-	0.9	0.9
LR	0.11	0.0125	0.125	0.01	0.025	0.001	0.01	0.01
Weight decay	1e-5	5e-5	2e-5	5e-5	5e-4	0.05	1e-4	1e-4
LR decay	cosineLR	step	cosineAnnealing	step	step	Cosine_iter	Cosine_iter	Cosine_iter
Decay rate	-	0.1	-	0.9	0.2	-	-	-
Decay epochs	-	[30, 60, 80]	-	2	[60, 120, 160]	-	-	-
Warmup epochs	5	5	5	5	1	5	5	5
Warmup method	linear	linear	linear	linear	linear	linear	linear	linear
Label Smoothing	0.1	-	-	-	-	-	0.1	0.1
H.flip	YES	YES	YES	YES	YES	YES	YES	YES
Erasing prob.	0.1	-	0.1	-	-	0.5	0.5	0.5
Auto augument	taWide	-	taWide	-	-	rand-m9-mstd0.5-inc1
Interpilation	bilinear	-	bilinear	-	-	bicubic	bicubic	bicubic
Train crop	176	-	176	-	-	224	224	224
Test crop	224	-	224	-	-	-	-	-
EMA	YES	-	-	-	-	-	-	-
EMA steps	32	-	-	-	-	-	-	-
EMA decay	0.99998	-	-	-	-	-	-	-
Loss	Cross	Cross	Cross	Cross	Cross	Cross	Soft Target
Entropy	Entropy	Entropy	Entropy	Entropy	Entropy	Cross Entropy
Baseline acc.	74.89	73.68	76.58	76.63	77.49	-	95.03	79.69
Max fraction	0.3	0.3	0.3	0.3	0.3	0.3	-	-
Max fraction decay	[1, 0.8, 0.6]	[1, 0.8, 0.6]	[1, 0.8, 0.6]	[1, 0.8, 0.6]	[1, 0.8, 0.6]	[1, 0.8, 0.6]	-	-
Fraction decay epoch	[200, 400, 600]	[30, 60, 80]	[200, 400, 600]	[30, 60, 80]	[60, 120, 160]	[30, 60, 80]	-	-
KAKURENBO acc.	75.15	73.52	76.62	76.23	77.21	-	95.28	79.35
per worker (GPU) to  $b$ , e.g., the global batch size of  $b \times p$  in the case of  $p$  GPUs. The minibatch size per GPU and the number of GPUs in each experiments are shown in Table 7.
To show the robustness of KAKURENBO, we also train ResNet-50 with different settings, e.g., marked as (A) and (B) in the Table 8 and discuss the result in Appendix C.3. For example, in ResNet-50 (A) setting, we follow the hyper-parameters reported in [34]. Specifically, we use using the Stochastic Gradient Descent (SGD) optimizer with a Nesterov momentum of 0.9 and weight decay of 0.00005. We trained all the models for 100 epochs and apply the linear scaling rule with the base learning rate of  $0.0125 \times k$  where  $k$  is the number of workers. We reduce the learning rate by 0.1 at the 30th, 60th, and 80th epoch. We gradual warmup which starts with 0 and is linearly increased to the base learning rate over 5 epochs. We also use scale and aspect ratio data augmentation. The input image is a  $224 \times 224$  pixel random crop from an augmented image or its horizontal flip.
B.4. Implementation
It is worth noting that KAKURENBO merely hides samples before the input pipeline. As a result, KAKURENBO can be easily implemented with simple extensions to PyTorch and TensorFlow implementations<sup>5</sup>. Using KAKURENBO with new models and datasets can be added to any training code by indicating so in the model launch parameters.
$^{5}$  Our PyTorch implementation is available at https://github.com/TruongThaoNguyen/kakurenbo
Appendix C. Ablation Studies
C.1. Analysis of the Factors Affecting KAKURENBO's Performance
In this section, we present an analysis of the factors affecting KAKURENBO's performance, e.g., the lagging loss and the prediction confidence.
The loss. Figure 5 shows the histogram of the loss as the number of epochs increases when training ResNet-50 (A) on the ImageNet-1K dataset. At the first few epochs, the histogram of the loss follows a Gaussian distribution. As the number of epochs increases, the number of samples with small loss increases significantly. For example, starting from epoch 30, more than  $50\%$  of the samples have a loss which is lower than  $5\%$  of the highest loss. As a result, there is an increase in the number of samples that provide about the same absolute contribution to the update, e.g., in the latter epochs. Hiding a fraction of (fixed)  $F$  samples during training in this case may lead to a relatively higher negative impact on the accuracy than that at some early epochs. Thus, we reduce the maximum hidden fraction at the epoch number increases (as mentioned in Section 3.4 in the main manuscript).
Figure 5: Histogram of the lagging-loss as the number of epoch increases during training (ResNet-50 w/ ImageNet-1K).
Figure 6: Number of hidden samples of each class in KAKURENBO (ResNet-50, ImageNet-1K). The figure shows the result of the first 50 classes. The number on top of each column shows the rank over 1000 classes (a lower rank indicates a higher number of hidden samples).
In addition, as the number of samples with the same absolute loss increases, there is a high probability that samples classified as important are in fact unimportant. To this end, we propose moving back samples from the hidden set based on their prediction confidence score (as per Section 3.2). Unlike the ahead-of-time method proposed by the authors in [13], instead of computing the loss of all the
samples before training and selecting samples to be removed from the training process, we compute the loss of the samples on the fly. With this method, at each epoch, a dynamic hiding fraction  $F^{*}$  is applied. Figure 6 shows the number of hidden samples of each class in KAKURENBO (ResNet-50, ImageNet-1K). The figure shows the result of the first 50 classes. The number on top of each column shows the rank over 1000 classes (a lower rank indicates a higher number of hidden samples). The result shows that our method could dynamically hide the samples at each epoch. For example, fewer samples in the class 25 are hidden while more and more samples in class 13 are selected to hide as epochs increase. Figure 7 shows that the impact of each class remains different during the training. Easy classes such as class 1, and 2 are hidden during the training more than the class 31 and 47. The result in Figure 8 also shows that in two continuous epochs, only (around)  $30\%$  of samples are hidden again. This result infers that the importance (or contribution level) of samples is changing epoch by epoch.
Figure 7: Number of hidden samples of each class in KAKURENBO (ResNet-50, ImageNet-1K). The figure shows the result of the first selected classes for a better presentation. The impact of each class (or each sample) is different in training. Easy classes such as class 1, 2 are hidden during the training more than the class 31 and 47.
Figure 8: Number of hidden samples per epoch. Max. hidden samples presents the number of samples considered for hiding in each epoch, e.g., which is in proportional to the fraction  $F$ . Hidden samples is the actual fraction of hidden samples in each epoch (after moving samples from the hidden list back to the training list). Hidden again presents the number of samples that were hidden in an epoch  $i$  and also hidden in the epoch  $i - 1$ . In general, only around  $30\%$  of the samples are hidden again in each epoch. The number of samples that are moved back becomes smaller when the epoch increases because the prediction confidence becomes higher during the training.
The calibration of the softmax prediction confidence As mentioned in the main manuscript, only samples that have low loss and sustain correct prediction with high confidence in the current epoch are hidden in the following epoch. A sample is correctly predicted with high confidence at an epoch  $e$  if it is predicted correctly (PA) and the prediction confidence (PC) is no less than a certain threshold  $\tau$ . As shown in Figure 8, the number of samples moved back becomes smaller when as the number of epochs increases because the prediction confidence becomes higher during the training, thus the actual hidden samples become similar to the max. hidden samples in the latter epoch. Results in
Table 9: Max testing accuracy (Top-1) of KAKURENBO in the comparison with Baseline. Reported Acc. represent the accuracy reported in the main manuscript. New Acc. represent the accuracy achieved in 3 different runs with different random seeds.
Setting	CIFAR-100
WRN-28-10	ImageNet-1K
ResNet-50(A)
Reported Acc.	New Acc.	Reported Acc.	New Acc.
Baseline	77.49	77.26 ± 0.55	73.68	74.06 ± 0.08
KAKURENBO	77.21	77.20 ± 0.63	73.52	73.65 ± 0.07
Random	-	76.82 ± 0.43	-	-
Table 10: Test accuracy (Top-1) in percentage and total training time in seconds of KAKURENBO in the comparison with those of the baseline.
Setting	ResNet-50(A) + ImageNet-1K	ResNet-50(B) + ImageNet-1K
Accuracy	Time (sec)	Accuracy	Time (sec)
Baseline	73.68	16118	76.58	64060
KAKURENBO-0.2	-	-	76.11	61723
KAKURENBO-0.3	73.52	12984	76.17	59063
KAKURENBO-0.4	-	-	75.62	57582
Table 5 show that when we increase the threshold  $\tau$  we obtain better accuracy (i.e., fewer hidden samples) at the cost of smaller performance improvements. However, the gaps remain small.
C.2. Evolution of the Hiding Fraction
Figure 4 shows how KAKURENBO adapts the size of the hidden set during the training of EfficientNet-b3. At the beginning of the training, the maximum hiding fraction is set to  $30\%$ . This fraction is progressively reduced after a few epochs followed by our fraction adjustment rule. The figure also reports the effective proportion of samples that are hidden at each epoch (Hiding rate in the Figure). As described in Section 3 in the main manuscript, KAKURENBO first cuts a part of the dataset before moving back samples that are mispredicted or correctly predicted but with low confidence. Figure 4 shows that the moving back strategy mostly impacts the beginning of the training when the model is still inaccurate.
Figure 4 also reports the measured speedup per epoch as compared to the baseline epoch duration. The speedup follows the same trend as the hiding rate. This is because reducing the number of samples in the training set impacts the speed of the training. The measured speedup does not reach the maximum hiding rate because of additional hidden sample selection and due to the need for computing the forward pass on samples in the hidden list.
C.3. Robustness of Our Method
In this section, we first confirm that the results of our implementation are stable by running experiments multiple times. Due to resource limitations, we could not run experiments multiple times for each data point reported in the paper, especially for models and datasets that are big in this work. The result for CIFAR-100 and ImangetNet-1K is reported in Table 9. The result shows that KAKURENBO is stable with different random seeds.
We now demonstrate the robustness of KAKURENBO with different settings during training, e.g. (1) when using different techniques to improve accuracy and (2) the batch size is changed.
We first measure the robustness of KAKURENBO when using SoTA techniques in training, the ResNet-50 (A) and (B) described in Table 8. The result in Figure 9 and Table 10 show that our proposed method is also stable with different learning techniques. For example, KAKURENBO could reduce the total training time to  $19.5\%$ $(7.8\%)$  with only  $0.2\%$ $(0.41)$  percent of accuracy reduction when the maximum hidden fraction is set to  $30\%$  for RESNET-50 (A) and (B), respectively.
Figure 9: Convergence and speedup of KAKURENBO with different settings of ResNet-50 including [LEFT] ResNet-50 (A) and [RIGHT] ResNet-50 (B).
Table 11: Test accuracy (Top-1) in percentage of KAKURENBO in comparison with those of the baseline when the batch size changes.
Setting	ResNet-50 (A) + ImageNet-1K
#GPUs	32	64	128	256
Batch size	1024	2048	4096	8192
Baseline	73.68	73.98	73.59	73.81
KAKURENBO-0.4	73.60	73.21	73.03	72.84
We now fix the mini-batch size per worker to 32 and then increase the number of workers (GPUs), i.e., we increase the global batch size in the case of ResNet-50 (A). Table 11 shows the top-1 testing accuracy of ResNet-50 (A) on the ImageNet-1K dataset when the batch size changes from 1024 to 8192. The result shows that KAKURENBO can maintain the accuracy (or with a trivial reduction of accuracy) even with large batch sizes. KAKURENBO could help with large-scale training which has become common when training DL models on a large supercomputer or cluster.
C.4. Comparison with other methods
We provide extra results in Table 9 to evaluate the training accuracy of random hiding with the CIFAR-100 dataset and WRN-28-10 model. As seen, accuracy is only  $76.82\%$  which is lower than that of both KAKURENBO and Baseline. In fact, randomly hiding samples has been investigated before in the GradMatch paper and it's been reported that accuracy is low. This drove us originally not to evaluate this method.
It is important to note that for method [28], the reported speedup is for a specific training regime (that uses a particular optimizer: LARS). The specific training regime described in the paper leads to a slow baseline (see Table 2 in [28], which shows the baseline that trains ImageNet-1K/ResNet-50 on 8 A100 GPUs for 90 epochs to be 3-4x slower than the typical number of hours to train ImageNet-1K/ResNet-50 on 8 A100 GPUs, as reported by many sources, including Nvidia NGC catalog $^6$ ). That means while InfoBatch reports  $26\%$  in speedup, the baseline setting for which InfoBatch is demonstrated to be effective is slow.
Basically the proposed method in [43] has a high overhead of visiting each sample to find the coreset. In addition, it is only demonstrated on small datasets such as CIFAR and TinyImageNet, at which the overheads for a large number of samples would not appear. KAKURENBO on the other hand is demonstrated on ImageNet-1K, DeepCAM, and ImageNet-1K-Fractal.
$^{6}$ https://catalog.ngc.nvidia.com/orgs/nvidia/teams/dle/resources/resnet_pyt/performance
Figure 10: The impact of different components of KAKURENBO on testing accuracy (DeepCAM). v1000: Hiding  $F\%$  lowest-loss samples only (HE). v1001: HE + LR (Adjusting Learning Rate). KAKURENBO: our proposed method with  $\mathrm{HE} + \mathrm{LR} + \mathrm{MB}$  (Moving Back) + FR (Reducing the Fraction by epoch). We also consider the version in which we cut  $2\%$  of the highest-loss samples at each epoch (DropTop).
Appendix D. Discussion on DeepCAM
We have shown how KAKURENBO's internal strategies, and their combination, affect the testing accuracy of a neural network in the case of ResNet-50 and the ImageNet-1K dataset. Figure 10 presents the same result on the DeepCAM dataset. In this experiment, we evaluate two combinations: v1000 and v1001. For v1000 we hide  $F\%$  lowest-loss samples only (Hiding Example or HE for short). For v1001 we combine HE and learning rate adjustment techniques. It is worth noting that our proposed method, KAKURENBO, is the combination of HE, LR, MB (Moving Back sample), and FR (Reducing the Fraction by epoch). The result with different maximum hidden fractions, e.g.  $F$  from 0.2 to 0.4, shows that using LR helps to improve the training accuracy by a significant amount, and KAKURENBO achieves the best accuracy which is very close to the baseline. This result is similar to what we observed with ResNet-50 and the ImageNet-1K dataset.
For DeepCAM, we also observed that the loss of the samples with the highest loss does not decrease significantly during the last few epochs of training and remain substantially above the rest of other samples. Those samples may be hard to learn or represent noise in the data. Figure 11 demonstrates this phenomena showing the loss distributions of the full, bottom  $98\%$  and top  $2\%$  of the dataset according to the loss values, respectively. As seen, the top  $2\%$ ’s loss distribution remains high until the very last epoch.
This observation motivated us to consider a version in which we cut  $2\%$  of the highest-loss samples at each epoch (DropTop). Interestingly, it helps to improve the testing accuracy of DeepCAM, e.g., from  $77.16\%$  in KAKURENBO to  $77.37\%$  with a maximum fraction of 0.3. For version v1001, Droptop increases the accuracy by  $0.82\%$ .
Appendix E. Related work
As the size of training datasets and the complexity of deep-learning models increase, the cost of training neural networks becomes prohibitive. Several approaches have been proposed to reduce this training cost without degrading accuracy significantly.
Biased with-Replacement Sampling has been proposed as a method to improve the convergence rate in SGD training [11, 12]. Importance sampling is based on the observation that not all samples are of
Figure 11: Loss distributions of DeepCAM training samples (full dataset, bottom  $98\%$  and top  $2\%$ ) in the last 10 epoch of training.
equal importance when it comes to training, and accordingly replaces the regular uniform sampling used to draw samples from datasets with a biased sampling function that assigns a likelihood to a sample being drawn proportional to its importance; the more important the sample is, the higher the likelihood it would be selected. The with-replacement strategy of importance sampling maintains the total number of samples the network trains on.
Several improvements over importance sampling have been proposed. Reducible Holdout Loss Selection (RHO-LOSS) [12] is a selection function that quantifies by how much each sample would reduce the loss on unseen data had it been trained on. Mercury uses an importance-aware data sharding technique in order to speed up distributed training [22]. It distributes important samples across workers between iterations. This allows important samples to be uniformly distributed between workers, and it reduces the number of samples to communicate for each epoch since non-important samples are kept local.
The importance of a sample can be estimated with several methods. In [23], authors use distance weighted sampling to determine the importance of samples. [24] uses stochastic optimization to reduce the stochastic variance. [25] selects each coordinate with a probability proportional to the square root of its smoothness parameter (applied to accelerated coordinate descent). RAIS [26] proposes approximating the ideal sampling distribution, which introduces little computational overhead.
Overall, biased with-replacement sampling aims at increasing the convergence speed of SGD by focusing on samples that induce a measurable change in the model parameters, which would allow a reduction in the number of epochs. While these techniques promise to converge in fewer epochs on the whole dataset, each epoch requires computing the importance of samples which is time-consuming; and the actual speedup in terms of time-to-solution remains unclear. Moreover, existing studies [11, 12, 22] only evaluate small datasets. Our experiments show that the biased with-replacement, importance sampling [11], the algorithm does not speedup the training when applied to large-scale datasets (demonstrated in the evaluation section in the paper).
Data Pruning techniques are used to reduce the size of the dataset by removing less important samples. Pruning the dataset requires training on the full dataset and adds significant overheads for quantifying individual differences between data points [27]. However, the assumption is that the advantage would be a reduced dataset that replaces the original datasets when used by others to train. Several studies investigate the selection of the samples to discard from a dataset. In [13], authors detect unforgettable samples that are correctly classified during the course of training. EL2N [15] uses the loss gradient norm of samples to identify the important ones and prune the unimportant samples from the dataset after a few epochs. While this work does not require fully training the model before pruning, it remains unclear if EL2N reduces the total training time. Another work uses memorization to identify outliers or mislabeled samples in a given dataset [14]. Removing these atypical samples accelerates the training without altering the trained model accuracy. Ensemble Active Learning [16] trains an ensemble of networks and uses ensemble uncertainty to identify which samples are hard to learn. They manage to reduce the ImageNet dataset by  $20\%$  without degrading
the accuracy of the trained model, but again, their method is prohibitive for models and datasets that require excessive resources for training.