KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training
Authors: Thao Nguyen Truong
Balazs Gerofi
Edgar Josafat Martinez-Noriega
François Trahay
Mohamed Wahib
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep learning
Sampling
TL;DR: we adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process, without significantly degrading accuracy.

Abstract: This paper proposes a method for hiding the least-important samples during the training of deep neural networks to increase efficiency, i.e., to reduce the cost of training. Using information about the loss and prediction confidence during training, we adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process, without significantly degrading accuracy. We explore the converge properties when accounting for the reduction in the number of SGD updates.  Empirical results on various large-scale datasets and models used directly in image classification and segmentation show that while the with-replacement importance sampling algorithm performs poorly on large datasets,  our method can reduce total training time by up to 22\% impacting accuracy only by 0.4\% compared to the baseline.

KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training
Truong Thao Nguyen' Balazs 2.4 Edgar Josafat Martinez-Noriega'
François Trahayss Mohamed 4ahib
1 National Institute of Advanced Industrial Science and Technology (AIST), Japan Intel Corporation, USA
Télécom SudParis, Institut Polytechnique de Paris, France
4 RIKEN Center for Computational Science, Japan
(nguyen. truong, edgar mattaaammainnezooieesa @aaiattgoo...................... go.
balazs. balazs. wew.zz..gerrfiiiine....cmm francois, wtw..aaattttteeece--eeaaa...a.aa
mohamed.attia@ jp
Abstract
This paper proposes a method for hiding the least-important samples during the training of deep neural networks to increase efficiency, i.e., reduce the cost of training. Using information about the loss and prediction confidence during training, we adaptively find samples exclude in a given epoch based on their contribution to the overall learning process, without significantly degrading accuracy. We explore the converge properties when accounting for the reduction in the number of SGI) updates. Empirical results on various large-scale datasets and models used directly in image classification and segmentation show that while the with- replacement importance sampling algorithm performs poorly on large datasets, our method can reduce total training time by up to 22% impacting accuracy only by 0.4% compared to the baseline. Code available at https: /tt///tttttae thub com/ TruoonThaoNquyenn /aaaar/akaaararraa
Introduction
Empirical evidence shows the performance benefits of using larger datasets when training deep neural networks (DNN) for computer vision, as well as in other domains such as language models or graphs [1]. More so, attention-bian based models are increasingly employed pre- pre-trained models using unprecedented dataset sizes, e.g. the JFT-3B dataset consists of nearly three billion images, annotated with class hierarchy of around 30K labels [2], LIAON-5B provides 5,85 billion CLIP--iltered image- text pairs that constitute over 240TB [3]. A similar trend is also observed in scientific computing, e.g., DeepCAM,, climate simulation dataset, is over 8.8TB in size [4]. Furthermore, the trend of larger datasets prompted efforts that create synthetic datasets using GANS [5] or fractals [6]. The downside of using large datasets is, however, the ballooning cost of training. For example, it has been reported that training models such as T5 and AlphaGo cost $1. 1.33M [7] and S35M [8], respectively. Additionally, large datasets can also stress non-compute parts of supercomputers and clusters used for DNN training (c.g., stressing the storage system due to excessive I/O requirements [9, 10]).
In this paper, build our hypothesis on the following observations on the effect of sample quality on training: a) biased with-replacement sampling postulates that not all samples biased, with-replacement sampling method can lead to faster convergence [11, 12], b) data pruning methods show that when select samples are pruned away from a dataset, the predication accuracy
focusing accelerating DNN training over large datasets and models. We
of the same importance and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table I: Summary of related works. Complexity on the number of samples N, number of Epochs M. and ensemble size E.
Appreach
Methed
Demerits (-)
Merits (+)
Online' Offiine
Practical Overhead
(Bottleneck)
Complexity (Cost)
Theuretically faster cuprergence
Biased Replacement
Sampling
Emportunce
speclup large datesets (Section
Sorting samples
OoN.tog(("))
Noodeteministic
Sampling
Theonetically faster convergence
RHOLLOSS [12]
No demonstrated speedup
Held-oit approx.
mondeieministi
Data Prunlug (prone dutavet
Scores [13]
Forgivesess
Full needed identily samples poune
Robust
prediction
Tracking change
OLY-AF))
oftiine save oost future training)
ELEN [:5]
needed identify samples prune scalaiiliyy
specdlip
Offine
Sorting samples
OCY")
Menovization [14]
Full tmining reoded to deeniiyy samples to poine
Demonstrated speodup
Tracking cross-sample prediction
Limited scalabiliyy
Ensemble Active
Fell training needed to identify samples geune
Lemonitrated speciupp
Midel uncertainty 4pproximation
OLY3)
Leaming [16]
Limited scaabiliiy
(DDD) [20]
Ensembles
Pull training needed to idemtity samples to goune
Demunsirated speclup
Tracking cross ensemble prediction:
O(V1. F)
Limited scalahility
Hiding Samples
Selective Backprop [17]
No demoonttattad spealup while maintaining securacy
No comergence guarantes
Arburey hidlng samples
Online
Sorting samples
OLV (og()))
GRAD MATCH [2]]
Adpalive
single CPU (distribuied training impractical)
Matching samples gradients
O(N.M))
convergeace guarantee skipping selection
Scalable
This Work
Efficiendl hiding samples
Theorctically convergence gurrantee
Sorting samples
OKV (lg())
Demonstratee speedup while maintaining accuracy
that can be achieved by training from scratch using the pruned dataset similar to that of the original dataset [13, 14, 15, 16]. Our hypothesis is that if samples have varying impact on the learning process and their impact decreases as the training progresses, then we can in real-time, adaptively, exclude samples with the least impact from the dataset during neural network training.
In this paper, we dynamically hide samples in dataset to reduce the total amount of computing and the training time, while maintaining the accuracy level. Our proposal, named KAKURENBO, is built upon two pillars. First, using combined information about the loss and online estimation of the historical prediction confidence (see Section 3.1) 1. of input samples, we adaptively exclude samples that contribute the least to the overall learning process on a per-epoch basis. Second, in compensation for the decrease in the number of SGD steps, we derive a method to dynamically adjust the learning rate and the upper limit on the number of samples to hide in order to recover convergence rate and accuracy.
We evaluate performance both in terms of reduction in wall-clock time and degradation in accuracy. Our main results are twofold: first, we show that decaying datasets by eliminating the samples with the least contribution to learning has no notable negative impact on the accuracy and convergence and that the overhead of identifying and eliminating the least important samples is negligible. Second, we show that decaying the dataset can significantly reduce the total amount of computation needed for DNN training. We also find that ofate-of-the-ar methods such as importance sampling algorithm [1 I]. pruning [13), or sample hiding techniques [17, 18] performs poorly on large-scale datasets. To the contrary, our method can reduce training time by 10.4% and 22.4% on ImageNet-IK [19] and DeepCAM [4], respectively, impacting Top-1 I accuracy only by 0.4%.
2 Background and Related Work
As the size of training datasets and the complexity of deep-learning models increase, the cost of training neural networks becomes prohibitive. Several approaches have been proposed to reduce this training cost without degrading accuracy significantly. Table summarizes related work against this proposal. This section presents the main saate-of-the-nee techniques. Related works are detailed in the Appendix-E.
Biased with hith-meelaaemmnt Sampling has been proposed as a method to improve the convergence rate in SGD training [11, 12]. Importance sampling is based on the observation that not all samples are of equal importance for training, and accordingly replaces the regular uniform sampling used to draw samples from datasets with biased sampling function that assigns likelihood to sample being

---

Input
Hidden Selection
C. Training Proces Traininx
D. Updatt Los & Precliction
(1) Seriing
F.
(3) Mooing sumples
Unifoens sempling
Truining
(21 Use legping for Trining
by lows
samples
back
buach
PW
PA
PC Lowi
Training Lise
NW
00.0
1:4
0
1
More bosk
MCU
ont, FOG,M)
Pr. Sofossnnnnnn
0:01 1:32
BWW
0.2
0
0
F2-04
P.A y
Ikr
W1
Loos A(oe,)))
0.11
ce
000
200
(2) Leaning rite iquameest
FW
Param: Max. fraction
Iraction scbodiling
1-Fr
Perfier for samples is List
PC: Predictio
Low
Law
High
Correctly predicted samppee
Truining sumplee
FW
Forward
WU
Weight Upsaac
Confielance
Wiceefully predicted samples
Hidcen sample
Hackword
Figure I: Overview of KAKURENBO At each epoch, samples are filtered into two different subsets, the training list and the hidden list, based on their loss, prediction accuracy (PA), and prediction confidence (PC), with maximum hidden fraction of F. PA and PC are used to drive sample move back decisions. Samples in the training list processed using uniform sampling without replacement. The loss and the prediction accuracy. calculated from the training process, are reused to filter samples in the next epoch. For samples on the hidden list, KAKURENBO only calculates the loss and PA by performing the forward pass at the end each epoch.
drawn proportional to its importance: the more important the sample is, the higher the likelihood it would be selected. The with-replacement strategy of importance sampling maintains the total number of samples the network trains on. Several improvements over importance sampling have been proposed for distributed training [22], or for estimating the importance of samples [12, 23, 24, 25, 26].
Overall, biased with-replacement sampling aims at increasing the convergence speed of SGD by focusing on samples that induce a measurable change in the model parameters, which would allow a reduction in the number of epochs. While these techniques promise to converge in fewer epochs on the whole dataset, each epoch requires computing the importance of samples which is time-consuming,
Data Pruning techniques are used to reduce the size of the dataset by removing less important samples. Pruning the dataset requires training on the full dataset and adds significant overheads for quantifying individual differences between data points [27]. However, the assumption is that the advantage would be reduced dataset that replaces the original datasets when used by others train. Several studies investigate the selection of the samples to discard from a dataset[13, 15, 14, 16] [28].
Pruning the dataset does reduce the training time without significantly degrading the accuracy [13, 14]. However, these techniques require fully training the model on the whole dataset to identify the samples to be removed, which is compute intensive.
Selective-Backproy [17] combines importance sampling and online data pruning. reduces the number of samples to train on by using the output of each sample's forward pass to estimate the sample's ' importance and cuts fixed fraction of the dataset at each epoch. While this method shows notable speedups, it has been evaluated only on tiny datasets without providing any measurements on how accuracy is impacted. addition, the authors allow up to 10% reduction in test error in their experiments.
Grad-Match [18] is an online method that selects a subset of the samples that would minimize the gradient matching error. The authors approximate the gradients by only using the gradients of the last layer, use per-class approximation, and run data selection every R epochs, in which case, the same subsets and weights will be used between epochs. Due to the infrequent selection of samples, Grad-Match often needs a larger number of epochs to converge to the same validation accuracy that can be achieved by the baseline [29]. Moreover, Grad-Match is impractical in distributed training, which is a de facto requirement in large dataset and models. Distributed Grad-Match would require very costly collective communication to collect the class approximations and to do the matching optimization. This practically a very high cost for communication per epoch that could even exceed the average time per epoch.
3

---

3 KAKURENBO: Adaptively Hiding Samples
In this work, we reduce the amount of work in training by adaptively choosing samples to hide in each epoch. We consider a model with loss function f(w, X=; Ya) where (xn, yn is a dataset of N sample-label pairs (In, E X), and G : X - X is a function that is applied to hide certain samples during training, e.g., by ranking and cut-off some samples. Using SGD with learning-rate 7 and batch size of B, the update rule for each batch when training with original full dataset is
Wo41 = Wt BB
Σ nEB(((())
V (Wes Xn.Yn)
where k (t) is sampled from [N/B] -
B (k) is the set of samples in batch i (to simplify, B is divisible by N). We propose to hide M examples by applying the a hiding function G.
We modify the learning rule to be
1 Wi+1 = W: 7
E
(we, (we, y.
(2)
REB(A(O)
using B batch at each step, which is composed of N/B steps. Since we exclude M samples, the aggregate number of steps is reduced from N/B to become (N MM/B, i.e. fixing the batch size and reducing the number of samples reduces the number of SGD iterations that performed for each epoch.
Sample hiding happens before presenting the input to each epoch. The training set that excludes the hidden samples (N M) is then shuffled for the training to process with the typical w/o replacement uniform sampling method.
Based on the above training strategy, we propose KAKURENBO, a mechanism to dynamically reduce the dataset during model training by selecting important samples. The workflow of our scheme is summarized in Figure 1. First, (B.1) we sort the samples of a dataset according to their loss. We then (B.2) select a subset of the dataset by hiding fixed fraction F of the data: the samples with the lowest loss are removed from the training set. Next, (B.3) hidden samples that maintain a correct prediction with high confidence (see Section 3.1) are moved back to the epoch training set. The training process (C) uses uniform sampling without replacement to pick samples from the training list. KAKURENBO adapts the learning rate (C.2) to maintain the pace of the SGD. At the end of the epoch, we perform the forward pass on samples to compute their loss and the prediction information on the up-to-date model (D). However, because calculating the loss for all samples in the dataset is prohibitively compute intensive [1 I, we propose to reuse the loss computed during the training process, which call lagging loss (D.2). We only recompute the loss of samples from the hidden list I). In the following, we detail the steps of KAKURENBO.
3.1 Hidden Samples Selection
We first present our proposed algorithm to select samples to hide in each epoch. We follow the observation in [11] that not all the samples are equal so that not-too-importan samples can hidden during training. An important sample is defined as the one that highly contributes to the model update, e.g., the gradient norm V w (W,, Xn.y)) yn)) in Equation 1. Removing the fraction F' of samples with the least impact the training model from the training list could reduce the training time, i.e., the required computing resource, without affecting the convergence of the training process. Selecting the fraction F is arbitrary and driven by the dataset/model. If the fraction F is too high, the accuracy could drop. In contrast, the performance gained from hiding samples will be limited if F is small, or potentially less than the overhead to compute the importance of samples. In this work, to design an adaptive method to select the fraction F in each epoch. We start from a tentative maximum fraction F at the beginning of the training process. We then carefully select the hidden samples from F based on their importance and then move the remaining samples back to the training set. That is, each epoch dynamic hiding fraction F* is applied.
aim
is worth noting that the maximum fraction number F does not need to be strictly accurate in our design; it is a maximum ceiling and not the exact amount of samples that will be hidden. However,
Dee | W/ X.. becomes too high, could ΣN (VWT((e....
if the negative impact of hiding samples, i.e.

---

significantly affect the accuracy. For example, when a high maximum fraction F is set and/or when most of the samples have nearly the same absolute contribution to the update, e.g., at the latter epoch of the training process. We investigate how to choose the maximum hiding fraction in each epoch in Section 3.3.
Moving Samples Back: since the loss is computed in the forward pass, it is frequently used as the metric for the importance of the sample, ie. samples with high loss contribute more to the update and are thus important [11, 22]. However, the samples with the smallest loss do not necessarily have the least impact (i.e., gradient norm) on the model, which is particularly true at the beginning of the training, and removing such high-impact samples may hurt accuracy. To mitigate the misselection of important samples as unimportant ones, we propose an additional rule to filter the low-loss samples based on the observation of historical prediction confidence [13]. The authors in [13] observed that some samples have a low frequency of toggling back from being classified correctly to incorrectly over the training process. Such samples can be pruned from the training set eternally. Because estimating the per-sample prediction confidence before training (i.e., offline) is compute-intensive, in this work, we perform an online estimation to decide whether an individual sample has a history of correct prediction with high confidence or not in a given epoch. Only samples that have low loss and sustain correct prediction with high confidence in the current epoch are hidden in the following epoch.
A sample is correctly predicted with high confidence at an epoch e if it is predicted correctly (PA) and the prediction confidence (PC) is no less than a threshold т, which we call the prediction confidence threshold, at the previous epoch. In addition to the prediction confidence of a given sample (r, y) is the probability that the model predicts this sample to map to label y:
out model (wa: 2, y) PC max(o (outt))
where a is sigmod (softmax) activation function. In this work, unless otherwise mentioned, we set the prediction confidence threshold to r = 0.7 as investigated in Section 4.3.
3.2 Reducing the Number of Iterations in Batch Training: Learning Rate Adjustment
After hiding samples, KAKURENBO uses uniform without replacement sampling to train on the remaining samples from the training set. In this section, we examine issues related to convergence when reducing the number of samples and we provide insight into the desirable convergence properties of adaptively hiding examples.
Implicit bias in the SGD training process may lead to convergence problems [30]: when redueing the total number of iterations at fixed batch sizes, SGD selects minima with worse generalization. We examine the selection mechanism in SGD when reducing the number of iterations at a fixed batch size. For optimizations of the original datasets, i.e., without example hiding, we use loss functions of the form
1 N Σ (w.
f f(w)
(4)
N where {xn, vn is a dataset of N data example-label pairs and l' is the loss function. We use SGD with batch of size B and learning-rate 7) with the update rule
1
Vwl Xn. Σ REB(k(H))
W = W. "B
(5)
for without replacement sampling, B divisible by N (to simplify), and l: (4) sampled uniformly from (L.....N/B), When using an werrpaaameeerrize model as is the case with deep neural networks, we typically converge to a minimum w that is a global minimum on all data points N in the training set [31, 14]. Following Hoffer et al. [32], linearizing the dynamics of Eq. 5 near w
(vn: V (w", Xn, y.)) = 0) gives
W(+1 = We - "
Σ H,W,, REB(K(())
(6)
5

---

where we assume w" = 0 since the models we target are over parameterizee (i.e., deep networks) leading to converge to a minimum w" We also assume H. - V (w. represents the per-example loss Hessian. SGD can select only certain minima from the many potential different global minima for the loss function of given the full training set N (and without loss of generality, for the training dataset after hiding samples N M). The selection of minima by SGD depends on the batch sizes and learning rate through the averaged Hessian over batch k
H
4
1 B
Σ H, nEB(k)
and the maximum over the maximal eigenvalues of { (
N/B
Amax
max v ( v||=1
(7)
kE
This Amax affects SGD through the Theorem proved by Hoffer et al. [32]: the iterates of SGD (Eq. 6) will converge if
2 T]
Amax
The theorem implies that a high learning rate leads to convergence to be for global minima with low Amax and low variability H,, Since this work we are fixing the batch size, we maintain Amax. the variability of (H) Therefore, certain minima with high variability in H, will remain accessible to SGD. Now SGD may converge to these high variability minima, which were suggested exhibit worse generalization performance than the original minima [33].
We mitigate this problem by reducing the delta by which the original learning rate decreases the learning rate (after the warm-up phase [34]). That way we make these new minima inaccessible again while keeping the original minima accessible. Specifically, KAKURENBO adjusts the learning rate at each epoch (or each iteration) by the following rule:
Tee TJbase,e
8
Fe
where Moase.. is the learning rate epoch in the non-hiding scenario and Fe is the hiding fraction at epoch e. By multiplying the base learning rate with a fraction T-F. KAKURENBO is independent of the learning rate scheduler of the baseline scenario and any other techniques related to the learning rate.
3.3 Adjusting the Maximum Hidden Fraction F
Merely changing the learning rate may not be sufficient, when some minima with high variability and low variability will eventually have similar Amax so SGD will not be able discriminate between these minima.
To account for this, we introduce a schedule to reduce the maximum hidden fraction. For the optimum of the set of hidden samples, WM = G(x,) and an overall loss function F(.) that acts as a surrogate of non-convex losses fa f.(w), where each is individually non-convex
loss for problems which in w. With Lipschitz continuous gradients with constant L2 we can assume
[V[((w)) (s(( < L || [[w w2ll
Since we are hiding samples when computing the overall loss function F(.), we assume each of the functions f((.) shares the same minimum value min fe(w) min (w) V i,j. We extend the proof of the theorem on the guarantees for a linear rate of convergence for smooth functions with strong convexity [35] to the non-convex landscape obtained when training with hidden samples (proof Appendix A)
Lemma 1. Let F(w) [[(i(w)] be non-convex. Set 0² = E[| V V with w
argminF Suppose 1) S
Let A, Wt W.-w- -.. After T iterations, SGD satisfies:
sup; L,
E || 1
(1 -2n A + R2
(9)
d2
where C = X(1 nusu, L.) and R. O

---

Since the losses f(()) are effectively dropping for individual samples, driven by the weight update, we thus drop the maximum fraction that can be hidden satisfy Eq. 9. Specifically, we suggest selecting a reasonable number that is not too high at the first epoch, e.g. F = 0.3. We then adjust the maximum fraction per epoch (denoted as Fe) to achieve Fe. We suggest using step scheduling, le.. to reduce the maximum hiding fraction gradually with factor of a by the number of epochs increases. For example, we set a as [1. 0.8, 0.6, 0.4] at epoch [0, 30, 60, 80] for ImageNet- IK and [0, 60, 120, 180] for CIFAR-100, respectively.
3.4 Update Loss and Prediction
Our technique is inspired by an observation that the importance of each sample of the local data does not change abruptly across multiple SGD iterations [22]. We propose to reuse the loss and historical prediction confidence, computed during the training process, and only recompute those metrics for samples from the hidden list. Specifically, the loss and historical prediction confidence of samples are computed only one time at each epoch, i.e., when the samples are fed to the forward pass. is not re-calculated at the end of each epoch based the latest model. Therefore, only samples of the last training iteration of a given epoch have an up-to-date loss. Furthermore, if we re-calculate the loss of hidden samples, i.e., only skip the backward and weight update pass of these samples, the loss of hidden samples is also up-to-date. For instance, if we cut off 20% of samples, we have nearly 20% up-to-date losses and 80% of not-up-to-date losses at the end of each epoch As the result, in comparison to the baseline scenario, KAKURENBO helps to reduce the total back ward and weight update time by a fraction of Fe while it does not require any extra forward time
Evaluation
We evaluate KAKURENBO using several models various datasets. We measure the effectiveness of our proposed method on two large datasets. We use Resnet50 [36] and EfficientNet [37] on ImageNet-IK [19], and DeepCAM [4], scientific image segmentation model with its accompanying dataset. To confirm the correctness of the baseline algorithms we also use WideResNet-288- 10 on the CIFAR- 100 dataset. Details of experiment settings and additional experiments such as ablation studies and robustness evaluation are reported in Appendix-B B and Appendix-C. We compare the following training strategies:
Baseline: We follow the original training regime and hyper-parameters suggested by their authors using uniform sampling without replacement.
Importance Sampling With Replacement [11] (ISWR): In each iteration, each sample is chosen with probability proportional to its loss. The with-replaacmmet strategy means that sample may be selected several times during an epoch, and the total number of samples fed to the model is the same as the baseline implementation.
FORGET is an online version of pruning technique [13]: instead of fully training the model using the whole dataset before pruning, we train it for 20 epochs, and a fraction F of forgettable samples (i.e. samples that are always correctly classified) are pruned from the dataset'. The training then restarts from epoch O. We report the total training time that includes the 20 epochs training with the whole dataset, and the full training with the pruned dataset.
Selective Backprop (SB) [17] prioritizes samples with high loss at each iteration. It performs the forward pass on the whole dataset, but only performs backpropagation on a subset of the dataset.
Grac-Match [18] trains using a subset of the dataset. Every R epoch, a new subset selected so that it would minimize the gradient matching error.
KAKURENBO: our proposed method where samples are hidden dynamically during training.
is worth noting that we follow the hypee-parameters reported in [38] for training ResNet-50, [39] for training WedeResNee-28-- 10, [37] for training EfficientNet-b3. and [4] for DeepCAM. We show the detail of our hyper-parameters in Appendix B. We configure ISWR, and FORGET to remove the
same fraction F as KAKURENBO. For SB, we use the 3 parameter that results in removing 50% of samples. Unless otherwise mentioned, our default setting for the maximum hidden fraction
We choose the samples remove by increasing number of forgetting events in [13].
7

---

WiddeRea CIFAR-100
RESNET T-50, ImageNet-1K
DeepCAM
EfficieneNat-33, ImageNut-1K
80
8D
Test Ancuney (%)
50
176
Test Accumncy (%)
Toss Acouuaoy (8)
Accuusuu (S)
ED-
33 8%:
0.75
Drawkine
201
KAKGMENOU
Bawdine PORGET
KAKURENBO
IOVR
78
Besslina KAKURENBO
20
KAKURENBO
Beselin
1000
2000
30.00
4000
22000
46000
00000
00000
1200
1000
8000
16000
24000
38000
Seconds
Seconds
Seconuta
Becondo
Figure 2: Convergence and speedup of KAKURENBO and importance sampling (ISWR).
Table 2: Max testing accuracy (Top-1) in percentage of Table 3: Comparison with Grad- Match in a single GPU (cutting
KAKURENBO the comparison with those the Baseline and other SOTA methods. Diff. represent the gap to the Baseline.
fraction is set to 0.3.
Setting
WRN-28-10
CIFAR-100
ResNet-50
ImageNet-IK
EllicienNNet-h3
DeepCAM
Setting
CIFAR-100
ResNet-18
Acc.
DHI..
Acc.
Acc.
DifL.
DHI..
Ace.
Time (sec)
Baseline
|
77.49
74.89
76.63
78.14
Baseline
77.98
8556
ISWR
76.51
(-0.98)
74.91
(+0.02)
N/A
75.75 (-2.39)
76.87
8104
FORGET
76.14
(-1.35)
73.70
(-1,20)
N/A
N/A
Grad-Match-O.3
(-1.11)
(-5,3%)
SB
77.03
(-0.46)
(-3.52)
N/A
77.05
8784
KAKURENBO-0.3
KAKURENBO
77.21
(-0.28)
75.15
(+0.26)
76.23
(-0.5)
77:42
(-0.9)
(-0,93)
(+2.7%)
F for KAKURENBO is 30%, except for the CIFAR-100 small dataset, for which we use 10% (see below).
To maintain fairness in comparisons between KAKURENBO and other state-of-the-art methods, we use the same model and dataset with the same hyper-parameters. This would mean are not capable of using state-of-the-art
tuning methods to improve the accuracy of ResNet- ResNet-50/ImageNet (e.g., as in [40]). That is since the tfte-of-the--a hyper-parametedd tuning methods are not applicable to some of the methods we compare with. Particularly, we can not apply GradMatch for training with a large batch size multiple GPUs. Thus, compare KAKURENBO with GradMatch using the setting reported in [18], i.e., CIFAR-100 dataset, ResNet-18 model.
hhyer-parameterd
4.1 Accuracy
The progress in the top-1 1 test accuracy with a maximum hiding fraction of 0.3 is shown in Figure 2. Table 2 summarizes the final accuracy for each experiment. We present data on the small dataset of CIFAR- to confirm the correctness of our implementation of ISWR, FORGET, and SB. Table reports the single GPU accuracy obtained with Grad-Match because cannot work on distributed systems. For CIFAR-100, we report similar behavior as reported in the original work on ISWR [11], SB [17], FORGET [13], and Grad-Match [18]: ISWR, FORGET, and Grad-Match degrade accuracy by approximately 1 %% while SB and KAKURENBO roughly perform as the baseline. KAKURENBO on CIFAR-100 only maintains the baseline accuracy for small fractions (e.g. = 0.1). When hiding a larger part of the dataset, the remaining training set becomes too scarce, and the model does not generalize well.
On the contrary, on large datasets such as ImageNet- K, ISWR and KAKURENBO slightly improve accuracy (by 0.2) in comparison to the baseline, while FORGET and SB degrade accuracy by 1.2% and 3.5%, respectively. On DeepCAM, KAKURENBO does not affect the accuracy while ISWR degrades by 2.4% in comparison the baseline Table 4 reports the accuracy obtained for transfer learning. We do not report Grad-Match results because we could not apply it to this application. Using SB significantly degrades accuracy compared the baseline, while ISWR, FORGET, and KAKURENBO maintains the same accuracy as the baseline. Especially, as reported in Figure 3, the testing accuracy obtained by KAKURENBO are varied when changing the maximum hiding fraction. We observe that for small hiding fractions, KAKURENBO achieves the same accuracy as
We confirm the same behaviors of KAKURENBO and other methods with different parammerssssrsss (as shown in Appendix-C).

---

RESNET-50
EfficiensNet-b3, ImagoNet-1K
DeepCAM
WiddRessNet28-100 CIFAR-100
75
78
80
80-
+:
Deseline
Teat Aaluurrray (15)
Teat Accuracy (8)
Teal Coneraen (8
26
76
Teat Accuracy
60
-3
69-
74
72
KARLRE-6D-03
150
Senaline
NAKUREHEO--0
07
740
7/3
20
40
60 Epoch
80
100
56
70
85
100
26
20
32
35
0
40
120
160
200
Epoch
Epoch
Epoch
Figure 3: Test accuracy vs. epoch of KAKURENBO with different maximum hiding fractions F.
Table 4: Impact of KAKURENBO in transfer learning with DeiT--Tiy-
Table 5: Impact of r (pre- diction confidence thresh- old) on the performance of KAKURENBO.
224 model.
Dataset
Metries
Baseline
ISWR
FORGET
SB
KAKUR.
Up
Fractal-3K
3.26
3.671
4.18
CIFAR-100 WRN-28-10
Time (min)
623
719
533
414
529
Setting
Impe.
(+15.4%%
(-14.4%)
(-33,5%)
(-15.1%)
Acc.
Time (sec)
Acc. (%) Diff.
95.03
95.79
95.85
93.59
95.28
Down
CIFAR 10
(+0.76)
(+0.82)
(-1.44)
(+0.25)
0.5
76.37
753.9
stream
Acc. (56) Diff.
79.69
79:62
79.95
76.98
79.35
0.7
76.81
758.9
CIFAR- 100
(-0.07)
(+0.26)
(-2.71)
(-0.34)
= 0.9
76.92
760.7
the baseline. When increasing hiding fractions, as expected, the degradation of the testing accuracy becomes more significant.
4.2 Convergence Speedup and Training Time
Here we discuss KAKURENBO's 's impact on training time. Figure reports test accuracy as the function of elapsed time (note the X-axis), and reports the training time to target accuracy. Table 4 reports the upstream training time of DeiT- SirT--Tin-2211 224. The key observation of these experiments is that KAKURENBO reduces the training time of Wide-ResNet by 21.7%, of ResNet-50 by 23%, of EfficientNet by 13,7% of DeepCAM by 22.4%, and of DeiT-Tiny by 15.1% in comparison to the baseline training regime.
Surprisingly, Importance Sampling With Replacement (ISWR) [11] introduces an overhead of 34.8% on WideNet, of 41% on ImageNet-1K 1K and offers only slight improvement of 2.5% on DeepCAM. At each epoch, ISRR processes the same number of samples as the baseline. Yet, it imposes an additional overhead of keeping track of the importance (i.e., the loss) of all input samples. While on DeepCAM achieves a modest speedup due to its faster convergence, these experiments reveal that ISWR's behavior is widely different on large datasets than on the smaller ones previously reported [11, 17].
FORGET increases the training time of WideResNet by 46.1% because of the additional 20 epochs training on the whole dataset needed for pruning the samples. When the number of epoch large, such as for ResNet50 that runs for 600 epochs, FORGET decreases the training time by 17.9%, and for DeiT by 14.4%. However, this reduction of training time comes at the cost of degradation of the test accuracy. On WideResNet and ResNet, SB performs similarly to KAKURENBO by reducing the training time without altering the accuracy. However, SB significantly degrades accuracy compared to the baseline for ImageNet and DeiT.
is worth noting that KAKURENBO has computation overheads for updating the loss and prediction (Step D in Figure 1), and sorting the samples based on the loss (Step A in Figure 1). For example, Figure reports the measured speedup per epoch as compared to the baseline epoch duration. The speedup follows the same trend as the hiding rate. This is because reducing the number of samples in the training set impacts the speed of the training. The measured speedup does not reach the maximum hiding rate because of the computation overhead. The performance gain from hiding samples will be limited the maximum hiding fraction F is small, or potentially less than the overhead to compute the importance score of samples. In experiments using multiple GPUs, those operations are performed in parallel to reduce the running time overhead. When using single GPU on CIFAR-100 with ResNet-18 18 (Table 3), the computational overhead is bigger than the speedup gained from hiding

---

Table 6: The impact of different components of KAKURENBO on testing accuracy including HE: Hiding F% % lowest-loss exam- ples, MB: Moving Back, RF: Reducing the Fraction by epoch, LR: Adjusting Learning Rate. Numbers inside the (.) indicate the gap in percentage compared the full version of KAKURENBO.
EfficlentNet-b3, ImageNet-1K
40
Max. Fraction
Hiding Rate
30
Speed Up
Ratio (%)
20-
Component MB
HE
RF
LR
Accuracy
10
Baseline
X
X
X
x
73.68
v1000
V
X
X
x
72.25 (-1.8%)
0
v1001
73.08 (-0.7%)
0
20
40
60
80
100
Epoch
v1010
X
72.81 (-1.1%)
v1011
V
V
V
73.27 (-0,4%)
Figure Reduction of hiding fraction, per epoch, and the re- sulting speedup.
v1
V
V
X
72.37 (-1.7%)
v1101
73.09 (-0.7%)
v1110
72.96 (-0.9%)
KAKUR (vIIIL)
V
V
73.6
samples. Thus, KAKURENBO takes more training time in this case. In short, KAKURENBO is optimized for large-scale training and provides more benefits when running on multiple GPUs.
4.3 Ablation Studies
Impact of prediction confidence threshold т. Higher prediction confidence threshold leads to a higher number of samples being moved back to the training set, i.e., fewer hidden samples at the beginning of the training process. At the end of the training process, when the model has is well-trained, more samples are predicted correctly with high confidence. Thus the impact of the prediction confidence threshold on the number of moved-back samples becomes less (as shown in Figure 4). The result in Table 5 shows that when we increase the threshold т, we obtain better accuracy (fewer hidden samples), but at the cost of smaller performance gain. We suggest set T 0.7 in all the experiments as a good trade-off between training time and accuracy.
Impact of different components of KAKURENBO. We evaluate how KAKURENBO's individual internal strategies, and their combination, affect the testing accuracy neural network. Table reports the results we obtained when training ResNet-50 on ImageNet-1K3 with a maximum hiding fraction of 40% The results show that when only HE (Hiding Examples) of the 40% lowest loss samples is performed, accuracy slightly degrades. Combining HE with other strategies, namely MB (Move-Back) RF (Reducing Fraction), and LR (Learning Rate adjustment) gradually improves testing accuracy. In particular, all combinations with RF achieve higher accuracy than the ones without For example, the accuracy of v1 v1110 10 is higher than that of v1 100 by about 0.59%. We also observe that using LR helps to improve the training accuracy by significant amount, i.e., from 0.46% to 0.83%. The MB strategy also improves accuracy. For example, the accuracy of v1010 is 72.81%, compared to v10 which is 72.96%. This small impact of MB the accuracy is due to moving back samples at the beginning of the training, as seen Appendix C.3. By using all the strategies, KAKURENBO achieves the best accuracy of 73.6%, which is very close to the baseline of 73.68%.
5
Conclusion
We have proposed KAKURENBO, mechanism that adaptively hides samples during the training of deep neural networks. It assesses the importance of samples and temporarily removes the ones that would have little effect on the SGD convergence. This reduces the number of samples process at each epoch without degrading the prediction accuracy. KAKURENBO combines the knowledge of historical prediction confidence with loss and moves back samples to the training set when necessary. It also dynamically adapts the learning rate in order to maintain the convergence pace. We have demonstrated that this approach reduces the training time without significantly degrading the accuracy on large datasets.
3 We use the 50 (A) configuration in this evaluation shown in Appendix-B
10

---

6 Acknowledgments
This work was supported by JSPS KAKENHI under Grant Numbers JP21K17751 and JP22H03600. This paper is based on results obtained from project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This work was supported by MEXT as "Feasibility studies for the next-generation computing infrastructure" and JST PRESTO Grant Number JPMJPR20MA.
We thank Rio Yokota and Hirokatsu Kataoka for their support on the Fractal-3K dataset.
References
[1] Y. Tay, M. Dehghani, S. Abnar, H. W. Chung, W. Fedus, J. Rao, S. Narang, V. Q- Tran, D. Yogatama, and D. Metzler, "Scaling laws vs model architectures: How does inductive bias influence scaling?" 2022. [Online]. Available: htpps//axxix.or//ax2222.....0
[2] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, "Scaling vision transformers." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022, pp. 12 104-
12 113.
[3] L. scale Artificial Intelligence Open Network (LAION), "LAION 5B Dataset," http:://liio..t 1o0-----ww-er--o--opee---- cetee-off--poe-----eeeee lawg---e------eee-e-ee-------eee scale-Amulti-nneeeeeee woweii-ooddddedaeee retrieved 29 april 2022.
[4] T. Kurth, S. Treichler, J. Romero, M. Mudigonda, N. Luehr, E. Phillips, A. Mahesh, M. Math- eson, J. Deslippe, M. Fatica er al., "Exascale deep learning for climate analytics," " in SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2018. pp.
1, 2. 7, 15
[5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, pp.
139- 144, 2020.
[6] H. Kataoka, R. Hayamizu, R. Yamada, K. Nakashima, S. Takashima, X. Zhang, E. J. Martinez- Noriega, N. Inoue, and R. Yokota, "Replacing labeled real-image datasets with auto-generated contours," " in Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2022, pp. 21 232-21 241. 15
[7] O. Sharir, B. Peleg, and Y. Shoham, "Thh cost of training nlp models: A concise overview," arXiv preprint 2rXiv:2004.08800, 2020. 1
[8] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, "Green AI," Commun. ACM, vol. 63, no. 12. p. 54-63, nov 2020. [Online]. Available: http://doi.org// http///...1//38888333333 1
[9] T. Nguyen, F. Trahay, J. Domke, A. Drozd, Vatai, Liao, M. Wahib, and B. Gerofi, "Why globally re-shuffle? revisiting data shuffling in large scale deep learning." in JEEE International Parallel and Distributed Processing Symposium (IPDPS), 2022, pp. 1085-1096.
[10] N. Dryden, R. Böhringer, T. Ben-Nun, and T. Hoefler, "Clairvoyant prefetching for distributed machine learning I/O," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, pp. 1-15.
[11] A. Katharopoulos and F. Fleuret, "Not all samples are created equal: Deep learning with importance sampling," in International conference on machine learning. PMLR, 2018, pp. 2525-2534. 1, 2, 5, 7, 9, 21, 22
[12] S. Mindermann, J. M. Brauner, M. T. Razzak, M. Sharma, A. Kirsch, W. Xu, B. Höltgen, A. N. Gomez, A. Morisot, S. Farquhar, and Y. Gal, "Prioritized training on points that are learnable, worth learning, and not yet learnt," in Proceedings of the 39th International Conference Machine Learning. PMLR, Jun 2022. p. 15630-15649. 1, 2, 3, 21,22
[13] M. Toneva, A. Sordoni, R. des Combes, A. Trischler, Y. Bengio, and G. Gordon, "An empirical study of example forgetting during deep neural network learning," in 7th International Conference on Learning Representations (ICLR), 2019. [Online]. Available:
http:///ppccceiee..o//founiiiei 3idlmm 2, 3,5, 8, 17, 22, 23
[14] V. Feldman and C. Zhang, "What neural networks memorize and why: Discovering the long tail via influence estimation," in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS'20, 2020. 2, 3, 5, 22,

---

[15] M. Paul. S. Ganguli, and G. K. Dziugaite. "Deep learning on a data diet: Finding important examples early in training." Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, 2021, pp. 20 596-20 607. [Online]. Available: (atri//eeeeaeeea3366999110f11139 Pasper5---aper.p.fff. 2, 3,
[16] K. Chitta, J. M. Álvarez, E. Haussmann, and C. Farabet, "Training data subset search with ensemble active learning," " IEEE Transactions on Intelligent Transportation Systems, 2021. 2, 3,
22
[17] A. H. Jiang, D. L. K. Wong, G. Zhou, D. G. Andersen, J. Dean, G. R. Ganger, G. Joshi, M. Kaminksy, M. Kozuch, Z. C. Lipton, and P. Pillai, "Accelerating deep learning by focusing on the biggest losers," 2019. [Online]. Available:
2, 3, 7, 8, 9,
23
[18] K. Killamsetty, S. Durga, G. Ramakrishnan, A. De, and R. Iyer. "Grad-match: Gradient matching based data subset selection for efficient deep model training." in International Conference on Machine Learning. PMLR, 2021, pp. 5464 5474. 2, 3, 7, 8, 23
[19] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Conf. comp. vision and pattern recogn. IEEE, 2009, pp.
248-255. 2, 7, 15
[20] K. Meding, L. M. S. Buschoff, R. Geirhos, and A. Wichmann, "Trivial impossible- dichotomous data difficulty masks model differences (on ImageNet and beyond)," arXiv preprint
a..522 2021. 2
[21] K. Killamsetty, D. Sivasubramanian, G. Ramakrishhan, A. De, and R. K. Iyer, "GRAD-MATCH: gradient matching based data subset selection for efficient deep model training," " in ICML, ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 5464-5474. 2
[22] X. Zeng, M. Yan, and M. Zhang, "Mercury: Efficient On-Device Distributed DNN Training via Stochastic Importance Sampling." in Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, 2021, pp. 29-41. 3, 5, 7, 22
[23] C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krahenbuhl, "Sampling matters in deep embedding learning," in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp.
3, 22
[24] P. Zhao and T. Zhang, "Stochastic optimization with importance sampling for regularized loss minimization," in international conference on machine learning. PMLR, 2015, pp. 1-9. 3, 22
[25] Z.. Allen-Zhu, Z., Qu, P. Richtárik, and Y. Yuan, "Even faster accelerated coordinate descent using non-uniform sampling," " in International Conference on Machine Learning. PMLR, 2016, pp. 1110-1119; 3,22
[26] T. B. Johnson and C. Guestrin, "Training deep models faster with robust, approximate im- portance sampling." Advances Neural Information Processing Systems, vol. 31, 2018.
22
[27] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos, "Beyond neural
scaling laws: beating power law scaling via data pruning," 2022. [Online]. Available: htpps//arxiv.or//abs/2200
14486 3, 22, 23
[28] Z. Qin, K. Wang, Z. Zheng, Gu, X. Peng. D. Zhou, and Y. You, "Infobatch: Lossless training speed up by unbiased dynamic data pruning," ArXiv, vol. abs/2303.0494. 2023. 3, 20
[29] O. Pooladzandi, D. Davini, and B. Mirzasoleiman, "Adaptive second order coresets for data- efficient machine learning." in International Conference on Machine Learning. PMIR, 2022,
pp. 17 848-17 869. 3, 23
[30] Q. Meng, W. Chen, Y. Wang, Z. Ma. and T. Liu, "Convergence analysis of distributed stochastic gradient descent with shuffling." Neurocomputing, vol. 337, pp. 46-57, 2019. 5
[31] C. C.ahang. S. Bengio, M. Hardt, B. Recht, and O. Vinyals, "Understanding deep learning requires rethinking generalization." in 5th International Conference on Learning Representations (ICLR), 2017. [Online]. Available:
[32] E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry, "Augment your batch: Improving generalization through instance repetition," in CVPR. Computer Vision Foundation
/ IEEE, 2020, pp. 8126-8133 8135..1.. 6,
12

---

[33] L. Wu, C. Ma, and W. E.. "How SGD Selects the Global Minima in Pver-Parameterized Learn- ing: Dynamical Stability Perspective," in Proceedings of the 32nd International Conference on Neural Information Processing Systems, ser. NIPS' 18, 2018, p. 2889-8298. 6
[34] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour," CoRR, vol.
abs/7777. 2017. 6, 16
[35] D. Needell, N. Srebro, and R. Ward, "Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm," in Proceedings of the 27th International Conference Neural Information Processing Systems Volume 1, ser. NIPS' 14, 2014, p. 1017-1025. 6
[36] K. He, X. Zhang, Ren, and Sun, "Deep residual learning for image recognition," in CVPR, 2016, pp. 770-778. 7, 15
[37] M. Tan and Q. Le. "EfficientNet: Rethinking model scaling for convolutional neural networks," in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09-15 Jun 2019. pp. 6105-6114. [Online]. Available: https//ppocceddnnnniiippes/////p 19..html 7,
15
[38] V. Vryniotis, "How to train state-of-the-art models using torchvision's latest primi- tives." " [Online]. Available:
using-torchhi panchvisionnlateste primitives/ 7, 15
[39] S. Zagoruyko and N. Komodakis, "Wide residual networks," in BMVC, September 2016, pp.
87.1-87.12. 7, 15
[40] R. Wightman, H. Touvron, and H. Jégou, "Resnet strikes back: An improved training procedure timm," 2021.
[41] A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," Master's thesis, Dep. of Comp. Sci. Univ. of Toronto, 2009. 15
[42] H. Touvron, M. Cord, M. Douze, Massa, A. Sablayrolles, and H. Jegou, "Training data-efficient image transformers & distillation through attention," in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18 24 Jul 2021, pp. 10347-10357. [Online]. Available:
http://pooceediing..l...eesssss 139ttouvron2 a.hmml 15
[43] Y. Yang, H. Kang, and B. Mirzasoleiman, "Towards sustainable learning: Coresets for data- efficient deep learning," in Proceedings of the 40th International Conference on Machine
Learning, ser. ICML'23. JMLR.org. 2023. 20
[44] Y. Wu, Z. Wang, Y. Shi, and Hu, "Enabling on-device enn training by supervieed instance filtering and error map pruning." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 39, no. 11, pp. 3445-3457, 2020. 23
[45] Y. Wang, Z. Jiang, X. Chen, P. Xu, Y. Zhao, Y. Lin, and Z. Wang, "EE2-triin: Training state-
of-the-art cnns with over 80% energy savings," Advances in Neural Information Processing
Systems, vol. 32, 2019. 23
13

---

APPENDIX
Appendix A. Proof of Lemma 1
Lemma 1. Let F(w) = [[f((w)] be non-convex. Set o² = Blll ((wa)l) with w
argminF(w) Suppose 1i S sup; Li
Let A. - W--W. -w. After T iterations, SGD satisfies:
E ||
(1- 21-)n) +
o² C
where C = =A111 - suuu I,) and R₂
Proof. (i((() )| = 0 in the noiseless setting, and so == 0. For 2% being the input at i random
index for iteration k, there exists parameter Xw,, for Amaz (Eq. 7), and w wx. we have for step size y
[|] [2]]
[Cs - v. [V/((=)11
[xx; x.) 7VV (xx) Vf((x.)) - Vf(x)))1
|| F. 2 2 a. Vf(((E)) + + ([()) - V.((x.) + Vii(=))1
| 2| 2 I. (f(())) f(xz)) +
|| x. _|2 2y x. (
+ 27 L x. + Vfi (f(()))) + 27 [
where we employ Jensen's ss inequality the first inequality for o² [[IV/[(ww)]())) Then fx))] Fxx). and we obtain
19]
a =.11² 2
x. F(xx) 2,2E σ., Vxx( vf((x.)]
+ 2 V
| - x111 1|2 2
x. F(xx) + + sup L,Ex, x., (z(())) vf((x,)
+ 2 2
= =|,0, x. 2| 2 x. (xx) - 27² sup Lo x.. F(xx) - F(x.) + 22
when 7 S I. Recursively applying this bound over the first k iterations yields the desired result
k-1
|| || s (1-27111 1-2:/(11 - -1)) ||000 1i.0-x.112+ x.112+2 +2
Σ
2-x((1-7))
(000-11) (aa(((-))) -))))² 7202
5=
(1-22110 2m1(1 - -)) 11:00 x.112 +
702
1 - 7)
Appendix B. Experiments Details
B.1. System detail
We run our experiments on a supercomputer with 1000s compute nodes, each equipped with Intel Xeon Gold 6148 CPUs, 384 GiB of RAM, 4 NVidia V100 GPUs, and Infiniband EDR NICs (100Gbpsx2). We run MPI ranks per compute node so that each rank has dedicated access to GPU.
14

---

Table Datasets and Models Used in Experiments (* Down- stream training using the pre-traine pre-trained model).
minibatch
Model
Dataset
#Samples
#Epoch
#GPUs
Task
(per GPU)
Resnet50 [36]
64
ImageNct-1K, [19]
1.2M
100
32
Image Classification
EfficientNet-h3 [37]
32
WideResNet-28-10
CIFAR-100
50K
200
32
32
Image Classification
[39]
[41]
DeepCAM [4]
DeepCAM [4]
122K
35
1024
1
Image Segmentation
Fractal-3K [6]
3M
80
32
16
DeiT- Teiy--in [42]
(*) CIFAR-10
Image Classification
50K
1000
8
96
[41]
(*) CIFAR-100
50K
1000
8
96
[41]
B.2. Model training method details and dataset information:
Table summarizes the models and datasets used this work. In details, we evaluate KAKURENBO using several models on various datasets as the following:
ImageNet-1K [19]: We use the subset of the ImageNet dataset containing 1000 classes each containing around 1 1300 images (1 (1,282,048 images in total). We also test the trained model the validation set of 50, 000 samples. We train ResNET-50 and EfficientNet-h3 provided by 'torchvision 12.12.0' on ImageNet- K dataset.
CIFAR-10/CIFAR- 100 [41]: The CIFAR-IO/CIFARE 100 dataset dataset consists of 60,000 colour images. It has 100 categories each containing 600 images. The dataset provides 50,000 training images and 10,000 test images with a size of 32x32 pixels. CIFAR-100 dataset is available at https::/www.cc www.::.wwww...ttooooooo/oooo .du/ kriz/cifar.h htmml.
DeepCAM [4]: DeepCAM dataset for image segmentation, which consists of approximately 122K samples and requires 8.8 of storage. We use the settings in [4] to train DeepCAM with the top learning rate 0.0055.
Fractal-3K [6] A rendered dataset from the Visual Atom method [6]. Fractal-3K dataset comprise of million images of visual atoms, where the number of classes is C = 3000 and the number of images per class is 1000. We train the DeiT-Tiny-224 model on Fractal-3K dataset and fine tune it with CIFAR-10 and CIFAR-100 datasets.
B.3. Hyper-parameters
It is worth noting that we follow the hyper-parameters reported in [38] for training ResNet-50, [39] for training WideResNee-22-10, [37] for training EfficientNet-b3, and [4] for DeepCAM. We also use the setting in [6] for both pretrain and finetune tasks in Fractal-3K. Table 8 shows the detail of our hyper-parameter
Specifically, We follow the guideline of "TorchVision to train the ResNet-50 that uses the CosineLR learning rate scheduler 4, auto augments, and random erasing, etc [38]. We also set the weight decay to - 05 and crop the input image to 176 176 pixels and train for long number of epochs, i.e., 600 (The ResNet-50 setting). We train the WideResNeet28-4 on the CIFAR- 100 dataset in 200 epochs following the setting in [39]. Specifically, we use the base learning rate of 0.025 x k, momentum 0.9, and weight decay 0.0005. For EfficientNet-b3,
RMSProp optimizer with momentum 0.9; batch norm momentum 0.99 weight decay le 5 (following [37]). We an initial learning rate of 0.016 that decays by 0.9 every epochs. We set the minibatch size
'implemented by timm / httpss//gggtabb..ccc/ggggggggeee pytprccttttttecco image- maddge-moelll models/ tree/main/ten
15

---

Table 8: seper-parameters used for different training the paper and the baseline top-1 testing accuracy. We also considers different hyper- parameters for ResNet-50 model on ImageNet- IK
dataset.
ImageNet-1K
CIFAR-100
Fractal-3K CIFAR-10 CIFAR-100
ResNet-50
ResNet.50 (A)
ResNer-S0
EfficientNet-hh3
WideResNet228-U1
DeiT-Tiny-224
Timin Res
224
224
224
234
32
234
224
224
Test Res
232
232
224
32
32
32
Epochs
600
100
200
Number of workers
32
32
32
32
32
8
Batch size
2048
1024
1024
1024
1024
512
768
768
Optimizer
SGD
SGD
SGD
SRMSPmpp
SGD
adamm
SGD
SGD
Momentum
LR
0.11
0.0125
0.125
0.01
0.025
0.001
0.01
0.01
Weight decay
le-5
Se-3
26-5
Se-5
Sc-4
0.05
1e-4
1o-4
LR deeay
cosinel.
step
cosineAmealling
sep
step
Cosine_ite Heer Cosine
Cosine Heer
Decay rate
Decay spochs
[30, 60), 80)
2
(60, 120, 160)
Warmup epochs
5
5
5
5
5
Wammup method
linear
linear
linear
Lahel Smoothing
0.1
H.flip
YES
YES
YES
YES
YES
YES
YES
YES
Eraiin proh.
0.1
0.1
0.5
0.5
0.5
Auto suguneett
wide
wide
rand-mo--mitlU
Interpilation
bilincar
biliner
bicubic 224
bicubic 224
bicubic 224
Train crop
176
176
Test crop
224
224
EMA seps
32
EMA decay
0.99998
Cross
Cross
Cross
Cross
Cross
Cross Entropy
Soft Target Entropy
Loss
Entropy
Entropy
Entropy
Entropy
Entropy
Baseline
74.89
73.68
76.58
76.63
77.49
95.03
79.69
Max fraction decay
[L. 0.8. 0.61
[L. 0.8.8 0.6]
[1. 0.8, 0.6] (200), 400, 600)
[L. 0.8. 0.6]
[1, 0.8, 0.6] (60, 120, 160)
0.8,0.00 0.6]
Frection decay spoch
(2000 400, 600]
[30, 60, NDj
[30, 60, 80J
[30, 60, 8I]
KAKURENBO RENBO acc.
75.15
73.52
76.62
76.23
77.21
95.28
79.35
worker (GPU) to b. e.g., the global batch size of b x p in the case of GPUs. The minibatch size per GPU and the number of GPUs in each experiments are shown in Table 7.
To show the robustness of KAKURENBO, we also train ResNet-50 with different settings, e.g.. marked (A) and (B) in the Table 8 and diseuss the result in Appendix C.3. For example, in ResNett50 (A) setting, we follow the prpee-pprameters reported in [34]. Specifically, we use using the Stochastic Gradient Descent (SGD) optimizer with Nesterov momentum of 0.9 and weight decay of 0.00005. We trained all the models for 100 epochs and apply the linear scaling rule with the base learning rate of 0.0125 & where k the number of workers. We reduce the learning rate by 0.1 at the 30th, 60th, and 80th epoch. We gradual warmup which starts with 0 and is linearly increased to the base learning rate over epochs. We also use scale and aspect ratio data augmentation. The input image is a 224 x 224 pixel random crop from an augmented image or its horizontal flip.
B.4. Implementation
It is worth noting that KAKURENBO merely hides samples before the input pipeline. As result, KAKURENBO can be easily implemented with simple extensions to PyTorch and TensorFlow implementations? Using KAKURENBO with new models and datasets can be added to any training code by indicating so in the model launch parameters.
5 Our PyTorch implementation is available at https://githhn com/ htw.ha/......ooooooooaaaatggaaaa kakurenbo
16

---

Appendix C. Ablation Studies
C.1. Analysis of the Factors Affecting KAKURENBO's Performance
In this section, present analysis of the factors affecting KAKURENBO' performance, c.g., the lagging loss and the prediction confidence.
The loss. Figure 5 shows the histogram of the loss as the number of epochs increases when training ResNet-50 (A) on the ImageNet-Il IK dataset. At the first few epochs, the histogram of the loss follows Gaussian distribution. As the number of epochs increases, the number of samples with small loss increases significantly. For example, starting from epoch 30, more than 50% of the samples have loss which is lower than 5% of the highest loss. As result, there is an increase in the number of samples that provide about the same absolute contribution to the update, e-g., the latter epochs. Hiding fraction of (fixed) F samples during training in this case may lead to relatively higher negative impact on the accuracy than that at some early epochs. Thus, we reduce the maximum hidden fraction at the epoch number increases (as mentioned in Section 3.4 in the main manuscript).
80.0%
Epoch
Normalized Data Amount
60.0%
50.0%
40.0%
30.0%
10.0%
0.00E+00
1.00E 01
2.00E 01
3.00E 02
4.00E
5.00E
6.00E
Loss
Figure 5: Histogram of the lagging-loss as the number of epoch increases during training (ResNet-50 w/ ImageNet-1K K)).
5
220
260
270
40
Number of hidden samples
200
10
1559
Epoch
228
42
40g
35
205
%0
Epoch
200
20
Class Index
Figure 6: Number hidden samples of each class KAKURENBO (ResNet-50, ImageNet- K). The figure shows the result of the first 50 classes. The number on top of each column shows the rank over 1000 classes (a lower rank indicates a higher number of hidden samples).
addition, as the number samples with same absolute loss increases, there a high probability that samples classified as important are in fact unimportant. To this end, we propose moving back samples from the hidden set based on their prediction confidence score (as per Section 3.2). Unlike the ahead-of timmeee- method proposed by the authors in [13], instead of computing the loss of all the
17

---

samples before training and selecting samples to be removed from the training process, we compute the loss the samples on the fly. With this method, each epoch, a dynamic hiding fraction F is applied. Figure 6 shows the number of hidden samples of each class in KAKURENBO (ResNet-50, ImageNet- K). The figure shows the result of the first 50 classes. The number on top of each column shows the rank over 1000 classes (a lower rank indicates higher number of hidden samples). The result shows that our method could dynamically hide the samples at each epoch. For example, fewer samples in the class 25 are hidden while more and more samples in class 13 are selected to hide as epochs increase. Figure shows that the impact each class remains different during the training. Easy classes such as class 1, and are hidden during the training more than the class 31 and 47. The result in Figure 8 also shows that in two continuous epochs, only (around) 30% of samples are hidden again. This result infers that the importance (or contribution level) of samples is changing epoch by epoch.
1000
Number of hidden samples
800
600
400
10
15
20
25
30
36
40
45
50
56
80
85
70
75
80
85
90
Epoch
Cless
Class
Class
Class
Figure 7: Number of hidden samples of each class in KAKURENBO (ResNet-50, ImageNet- K). The figure shows the result of the first selected classes for better presentation. The impact of each class (or each sample) is different in training. Easy classes such as class 1, 2 are hidden during the training than the class 31 and 47.
WideResNNt-ZB-10 CIFAR-100
0000
Number of samples (8)
4000
2000
Max. hidden samples samples
Hiddan again
2000
Soely Pwafs
1000
120
160
Epoch
Figure 8: Number of hidden samples per epoch. Max. hidden samples presents the number of samples considered for hidding each epoch, e.g., which proportional to the fraction F. Hidden samples is the actual fraction of hidden samples in each epoch (after moving samples from the hidden list back to the training list). Hidden again presents the number of samples that were hidden in an epoch and also hidden in the epoch - 1. In general, only around 30% of the samples are hidden again each epoch. The number of samples that are moved back becomes smaller when the epoch increases because the prediction confidence becomes higher during the training.
The calibration of the softmax prediction confidence As mentioned in the main manuscript, only samples that have low loss and sustain correct prediction with high confidence in the current epoch are hidden in the following epoch. A sample is correctly predicted with high confidence at an epoch it is predicted correctly (PA) and the prediction confidence (PC) is no less than certain threshold As shown in Figure 8, the number of samples moved back becomes smaller when as the number of epochs increases because the prediction confidence becomes higher during the training, thus the actual hidden samples become similar to the max. hidden samples in the latter epoch. Results in
18

---

Table 9: Max testing accuracy (Top-1) of KAKURENBO in the comparison with Baseline. Reported Acc. represent the accuracy reported in the main manuscript. New Acc. represent the accuracy achieved in 3 different runs with different random seeds.
CIFAR-100
ImageNet-1K ResNet-50(A)
Setting
WRN-28-10
Reported Acc.
New Acc.
Reported Acc.
New Acc.
Baseline
77.49
77.26 + 0.55
73.68
74.06 + 0.08
KAKURENBO
77.21
77.20 0.63
73.52
73.65 + 0.07
Random
76.82 0.43
Table 10: Test accuracy (Top-1) 1) in percentage and total training time in seconds of KAKURENBO in the comparison with those of the baseline.
Setting
ResNet-50(A) ImageNet-IK
ResNet-50(B) + ImageNet-1K
Accuracy
Time (sec)
Accuracy
Time (sec)
Baseline
73.68
16118
76.58
64060
KAKURENBO-02
76.11
61723
KAKURENBO
73.52
12984
76.17
59063
KAKURENBO-D.4
75.62
57582
Table 5 show that when we increase the threshold 7 we obtain better accuracy (i.e., fewer hidden samples) at the cost of smaller performance improvements. However, the gaps remain small.
C.2. Evolution of the Hiding Fraction
Figure shows how KAKURENBO adapts the size of the hidden set during the training of EfficientNet-b3. At the beginning of the training, the maximum hiding fraction is set 30 %. This fraction is progressively reduced after a few epochs followed by our fraction adjustment rule. The figure also reports the effective proportion of samples that hidden each epoch (Hiding rate in the Figure). As described in Section part of the dataset before moving back samples that mispredicted correctly predicted but with low confidence. Figure 4 shows that the moving back strategy mostly impacts the beginning of the training when the model is still inaccurate.
the main manuscript, KAKURENBO first cuts a
Figure also reports the measured speedup per epoch as compared to the baseline epoch duration. The speedup follows the same trend as the hiding rate. This is because reducing the number of samples in the training set impacts the speed of the training. The measured speedup does not reach the maximum hiding rate because of additional hidden sample selection and due to the need for computing the forward pass on samples the hidden list.
C.3. Robustness of Our Method
In this section, we first confirm that the results of our implementation are stable by running experi- ments multiple times. Due resource limitations, could not experiments multiple times for each data point reported in the paper, especially for models and datasets that are big in this work. The result for CIFAR- 100 and ImangetNet-IK is reported in Table 9. The result shows that KAKURENBO is stable with different random seeds.
We now demonstrate the robustness of KAKURENBO with different settings during training, e.g. (1) when using different techniques to improve accuracy and (2) the batch size is changed.
We first measure the robustness of KAKURENBO when using SoTA techniques in training, the ResNet-50 (A) and (B) described in Table 8. The result Figure 9 and Table 10 show that our proposed method is also stable with different learning techniques. For example, KAKURENBO could reduce the total training time to 19.5% (7. (7.8)) with only 0. 0.2% (0.41) percent of accuracy reduction when the maximum hidden fraction is set to 30% for RESNET-SE 50 (A) and (B), respectively.
19

---

RESNET-50, ImageNet-1K
RESNET-50v2, ImageNet-1k
80
80
Test Acouracy (%)
60
19.5%
Test Test Accuracy (%)
60
-0..1 1%
+16.7%
-7.8%
40
40
-3.6%
Baseline
Baseline KAKURENBO ISWR
KAKURENBO-0.2
20
20
KAKURENBO-O.3
KAKURENBO-0.4
0
0
4000
8000 12000
16000 20000
0
17000
34000
51000
68000
Seconds
Seconds
Figure Convergence and speedup KAKURENBO with different settings of ResNet-50 including [LEFT] ResNet-50 (A) and [RIGHT] ResNet-50 (B).
Table 11: Test accuracy (Top-1) in percentage of KAKURENBO in comparison with those of the baseline when the batch size changes.
Setting
ResNet-50 (A) ImageNet-Ik
#GPUs
32
64
128
256
Batch size
1024
2048
4096
8192
Baseline
73.68
73.98
73.59
73.81
KAKURENBO-04
73.60
73.21
73.03
72.84
We now fix the mini-batch size per worker to 32 and then increase the number of workers (GPUs), i.e. we increase the global batch size in the case ResNet-50 (A). Table 11 shows the top-1 testing accuracy of ResNet-50 (A) on the ImageNet- IK dataset when the batch size changes from 1024 to 8192. The result shows that KAKURENBO can maintain the accuracy (or with a trivial reduction of accuracy) even with large batch sizes. KAKURENBO could help with large-scale training which has become common when training DL models on a large supercomputer or cluster.
C.4. Comparison with other methods
We provide extra results in Table to evaluate the training accuracy of random hiding with the CIFAR-100 100 dataset and WRN-28- 10 model. As seen, accuracy is only 76,82% which is lower than that of both KAKURENBO and Baseline. In fact, randomly hiding samples has been investigated before in the GradMatch paper and it's been reported that accuracy is low. This drove us originally evaluate this method.
is important to note that for method [28], the reported speedup for a specific training regime (that particular optimizer: LARS). The specific training regime described in the paper leads to slow baseline (see Table 2 in [28], which shows the baseline that trains ImageNet- K/ResNet-50 on
8 A100 GPUs for 90 epochs be 3-4x slower than the typical number of hours to train ImageNet- KetesNet-50 on 8 A GPUs, as reported by many sources, including Nvidia NGC catalog") That means while InfoBatch reports 26% in speedup. the bascline setting for which InfoBatch demonstrated to be effective is slow.
Basically the proposed method in [43] has high overhead of visiting each sample to find the coreset. In addition, it is only demonstrated on small datasets such as CIFAR and Tiny ImageNet, at which the overheads for large number of samples would not appear. KAKURENBO the other hand is demonsrated on ImageNet-1K DeepCAM, and ImageNet- 1--oNeett-K-Frccall
'nttps://catall nge. http:////oo//e/aaaa////////////a ressneeeeerrrreeeeee... pyt/performance
20

---

DeepCAM (F=0.2)
DeepCAM (F:0.3)
80
80
Easefro
Acouracy
78
Basekte
v1000.0.2
Accuracy
78
v1000-.2
KAOURENBO 0.34DropTpp
76
74
74
35
29
31
Epoch
Epoch
DeepCAM (F-0.4)
80
78
Basekra
Test Accuracy
KMRREENBO
125
29
33
Epoch
Figure 10: The impact of different components of KAKURENBO on testing accuracy (DeepCAM). v1000:
Hiding 1% lowest-loss samples only (HE). v1001: HE + LR (Adjusting Learning Rate), KAKURENBO: our proposed method with HE + LR + MB (Moving Back) + FR (Reducing the Fraction by epoch). We also consider the version which we cut 2% of the highest-loss samples at each epoch (DropTop).
Appendix D. Discussion on DeepCAM
We have shown how KAKURENBO's internal strategies, and their combination, affect the testing accuracy neural network the case of ResNet-50 and the ImageNet-IK dataset. Figure 10 presents the same result on the DeepCAM dataset. In this experiment, v1000 and v1001. For v1000 we hide F lowest-loss samples only (Hiding Example or HE for short). For v1001 we combine HE and learning rate adjustment techniques. It is worth noting that proposed method, KAKURENBO, is the combination of HE, LR, MB (Moving Back sample), and FR (Reducing the Fraction by epoch). The result with different maximum hidden fractions, e-g. F from 0.2 to 0.4, shows that using LR helps to improve the training accuracy by significant amount, and KAKURENBO achieves the best accuracy which very close to the baseline. This result is similar to what we observed with ResNet-50 and the ImageNet- K dataset.
evaluate two combinations:
For DeepCAM, we also observed that the loss of the samples with the highest loss does not decrease significantly during the last few epochs of training and remain substantially above the rest other samples. Those samples may be hard to learn represent noise in the data. Figure demonstrates this phenomena showing the loss distributions of the full, bottom 98% and top 2% of the dataset according to the loss values, respectively. As seen, the top 2% 's loss distribution remains high until the very last epoch.
This observation motivated us to consider version in which we cut 2% of the highest-loss samples at each epoch (op) Interestingly, it helps to improve the testing accuracy of DeepCAM, e.g., from 77.16% in KAKURENBO to 77.37% with maximum fraction of 0.3. For version v1001, Droptop increases the accuracy by 0.82%.
Appendix E. Related work
As the size of training datasets and the complexity of deep-learning models increase, the cost of training neural networks becomes prohibitive. Several approaches have been proposed to reduce this training cost without degrading accuracy significantly.
Biased with Rephemeeta Sampling has been proposed as a method improve the convergence rate in SGD training [11, 12]. Importance sampling is based the observation that not all samples are of
21

---

Loss diveribution
Epoch
Figure 11: Loss distributions of DeepCAM training samples (full dataset, bottom 98% and top 2%) in the last 10 epoch of training.
equal importance when comes to training, and accordingly replaces the regular uniform sampling used to draw samples from datasets with biased sampling function that assigns likelihood to sample being drawn proportional to its importance; the more important the sample is, the higher the likelihood it would be selected. The with with-replacement strategy of importance sampling maintains the total number of samples the network trains on.
Several improvements over importance sampling have been proposed. Reducible Holdout Loss Selection (RHO-LOSS) [12] is selection function that quantifies by how much each sample would reduce the Joss on unseen data had it been trained on. Mercury uses importance-awarc data sharding technique order to speed up distributed training [22]. It distributes important samples across workers between iterations. This allows important samples to be uniformly distributed between workers, and reduces the number of samples to communicate for each epoch since non- -ommiportannt samples are kept local.
The importance
sample can be estimated with several methods. In [23], authors use distance weighted sampling to determine the importance of samples. [24] uses stochastic optimization to re- duce the stochastic variance. [25] selects each coordinate with a probability proportional the square root of its smoothness parameter (applied to accelerated coordinate descent). RAIS [26] proposes approximating the ideal sampling distribution, which introduces little computational overhead.
Overall, biased with-replacement sampling aims at increasing the convergence speed of SGD by focusing on samples that induce measurable change in the model parameters, which would allow reduction in the number of epochs. While these techniques promise to converge in fewer epochs on the whole dataset, each epoch requires computing the importance of samples which is time- consuming; and the actual speedup in terms of time-to-solution remains unclear. Moreover, existing studies [11, 12, 22] only evaluate small datasets. Our experiments show that the biased with- replacement, importance sampling [1 1]. the algorithm does not speedup the training when applied to large-scale datasets (demonstrated the evaluation section the paper).
Data Pruning techniques are used to reduce the size of the dataset by removing less important samples. Pruning the dataset requires training on the full dataset and adds significant overheads for quantifying individual differences between data points [27]. However, the assumption that the advantage would be reduced dataset that replaces the original datasets when used by others to train. Several studies investigate the selection of the samples to discard from a dataset. In [13]. authors detect unforgettable samples that correctly classified during the course of training. EL2N [15] uses the loss gradient norm of samples to identify the important ones and prune the unimportant samples from the dataset after a few epochs. While this work does not require fully training the model before pruning, remains unclear if EL2N reduces the total training time. Another work uses memorization to identify outliers or mislabeled samples
given dataset [14]. Removing these atypical samples accelerates the training without altering the trained model accuracy. Ensemble Active Learning [16] trains an ensemble networks and uses ensemble uncertainty to identify which
samples are hard to leamn They manage to reduce the ImageNet dataset by 20% without degrading
22

---

the accuracy of the trained model, but again, their method is prohibitive for models and datasets that require excessive resources for training.
Pruning the dataset does reduce the training time without significantly degrading the accuracy [13, 14]. However, these techniques require fully training the model on the whole dataset identify the samples to be removed, which is compute intensive. While most of the proposed solutions perform well on small datasets such as CIFAR, many fail maintain accuracy on larger datasets like ImageNet [27].
Selective- Barekioo [17] combines importance sampling and online data pruning. reduces the number of samples to train on by using the output of each sample's forward pass to estimate the sample's 's importance and cuts a fixed fraction of the dataset at each epoch. While this method shows notable speedups, it has been evaluated only tiny datasets without providing any measurements on how accuracy is impacted. In addition, the authors allow up to 10% reduction in test error in their experiments. HIF [44] is similar to Srtective-Backprog it reduces the computation cost of training by filtering out the samples with the lowest loss. E²-Train [45] shows that the combination of randomly dropping samples during training with selective layer update in CNNs can significantly reduce the training time, while slightly degrading the accuracy. However, E²-Train targets edge environments and is evaluated only on very small datasets.
BRAD-MATCH [18] is an online method that selects a subset of the samples that would minimize the gradient matching error, where the error of the gradients of matched subset samples (and their weights) becomes minimum. To avoid the impractical storing and computation of the optimization of the gradients of all instances, the authors approximate the gradients by only using the gradients of the last layer, use a per-class approximation, and run data selection every R epochs, in which case, the same subsets and weights will be used between epochs. The infrequent selection, however, means the model is limited in its capacity to learn in intermediate epochs- where selection occurs since it trains on the same limited subset of samples. This often leads to a longer numbers of epochs needed to converge to the same validation accuracy that can be achieved by the baseline or the baseline reaching much higher accuracy [29]. Another important point worth mentioning is that GRAD-MATCH is impractical in distributed training, which is de facto requirement in large dataset and models (e.g.. the DeepCAM model/dataset). That is since the approximation of the classes would require very expensive high-volume collective communication operations to gather the gradients scattered across different samples belonging to the same class. The communication cost would be O(N. R.G) where N is the number of samples, R is the frequency of selection, and G is the gradients (of the last layer, gradient approximation is be used). Distributed GRAD-MATCH would require scatter communication to collect the class approximations and collective all-reduce of the gradients to then do the matching optimization. This is practically a very high cost for communication per epoch that could even exceed the average time per epoch. Finally, the mini-batch variant of can only be effective for small mini-batches. However, since distributed training the mini-batch grows with the scale (i.e., the mini-batch aggregates the local mini-batch of all workers), the cost of communication amplifies by B (where B is mini-batch size).
23


Reviews:
[{"review_id": "DiYCHvxBY2", "reviewer": "Reviewer_wMxv", "summary": "\nSummary: This paper proposes a method of dynamically selecting training data during the training process to improve the efficiency of the model training. Specifically, the authors optimize the training process by dynamically adjusting sample weights, learning rates, and the upper limit of sample numbers. While the method is considered reasonable, it lacks surprising and novel features. The absence of experimental validation on the claimed jft-3b and lain-5b datasets somewhat diminishes the urgency for readers to accept and publish its findings.\n", "questions": "\nSee *Weaknesses", "limitations": "\nNo. See *Weaknesses\n\n", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "\n(Positive) Table 1 demonstrates the differences between this paper and related literature effectively, which is commendable. Through comparisons and analysis, Table 1 vividly illustrates the innovation and differences of this paper's method from other approaches, helping readers gain a clearer understanding of the research's value and contributions.\n\n(Positive) The dynamic adjustment of learning rate and sample number limit is interesting and reasonable. By dynamically adjusting these hyperparameters, the model can better adapt to changes in data and training progress, further optimizing the model's performance.\n\n(Positive) The method in Section 3.3 is reasonable and also inspiring. This is a positive acknowledgment of the method in this section, indicating its value and providing readers with new ideas and insights.\n\n\n(Positive and Negative) I am pleased to see the authors conducting experiments on DeepCAM. I hope to see the results of experiments on jft-3b and lain-5b datasets. This is an expectation for the authors' work as it will present a more comprehensive demonstration of the method's applicability and performance in different datasets and application scenarios.\n", "weaknesses": "\n(Negative) Please discuss the difference between this paper and data distillation. Data distillation typically focuses on using an auxiliary model to soften and distill training data to generate more representative data for optimizing the model's generalization ability. In contrast, the method proposed in this paper dynamically allocates weights to samples based on their losses and confidences during the training process, adjusting the influence of training data to enhance the model's training efficiency and performance.\n\n\n(Negative) The authors mentioned the importance of their method on jft-3b and lain-5b datasets. I agree with this motivation. Therefore, I believe the authors should validate their method on these two datasets. Mentioning the importance of the method on datasets like jft-3b and lain-5b indeed highlights their motivation and research drive. Conducting thorough experiments on these datasets will strengthen the persuasiveness of the results, demonstrating the performance of the method in different datasets and application scenarios. Especially for jft-3b and lain-5b datasets, validating and verifying the method's effectiveness will further enhance the study's credibility and applicability.\n\n\n(Negative) The authors' method only achieves a 22% improvement compared to traditional methods, which, in my view, is not significant. If the authors could increase the training time by 10 or 100 times, I think it would be very meaningful. Although a 22% improvement may not be considered remarkable, it is essential to consider that optimizing training time is not the sole objective. Achieving substantial speedup in larger training settings would still be meaningful and worthy of further exploration and research.\n\n(Negative) Dynamically allocating weights based on loss and confidence for each sample is intuitive. Similar approaches exist in existing literature. Therefore, I do not find it particularly amazing. While the method of dynamically allocating weights based on loss and confidence may be intuitive and has similarities with existing literature, it might not be considered astonishing or groundbreaking in the field.\n\n\n(Negative) Comparing Formula 2 and Formula 1, it is best for the authors to differentiate the contents of the summation subscripts to better represent their distinction between the two sampling methods. This is a specific suggestion for the authors' writing, ensuring accuracy and clarity in expression, making it easier for readers to understand the differences between the two sampling methods.\n\n\n(Negative) As Sections 3.1 and 3.2 are so intuitive, I am not sure how to evaluate them. In summary, the methods in Sections 3.1 and 3.2 are reasonable but may not be awe-inspiring and admirable. This represents a neutral viewpoint on the paper's methods, acknowledging their rationality but suggesting that they may lack innovation and breakthroughs.\n\n(Positive and Negative) I am pleased to see the authors conducting experiments on DeepCAM. I hope to see the results of experiments on jft-3b and lain-5b datasets. This is an expectation for the authors' work as it will present a more comprehensive demonstration of the method's applicability and performance in different datasets and application scenarios.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.1. Difference between this paper and data distillation**\nData distillation reportedly introduces high overhead and is thus used mainly as a means to improve the model's generalization ability, and not reducing the amount of work. In comparison, the work in this paper aims to reduce the work by dynamically hiding samples, while sustaining the accuracy.\n\n**1.2. Validate KAKURENBO on jft-3b and lain-5b datasets**: JFT-3B is a closed dataset and LAION-5B is only links to files and not a dataset; there is no direct way to download it. We mention JFT-3B and LAION-5B in this paper as examples to motivate the work. \n\n**1.3 Comparing Formula 2 and Formula 1, it is best for the authors to differentiate the contents of the summation subscripts to better represent their distinction between the two sampling methods.}. We are happy to improve the text based on the suggestions. Specifically, we edit the term  $k\\left(t\\right)$ in the formation 2 in the manuscript to  $k_G\\left(t\\right)$ where $k_G\\left(t\\right)$ is sampled from samples that are not hidden.\n\n**1.4 No significant improvement in training time**: \nIt is unattainable to speed-up the training process by 10-100X just by hiding samples during training as it would require 90-99\\% of samples to be hidden without reducing accuracy. However, our method can be combined with other computation/communication optimization methods that have been proposed for speeding up training [1],[2]). Our method is helpful in pre-training a model with a huge dataset (e.g., in transfer learning). Training in such cases may have very high computing resource costs, e.g., 100s-1000s of GPU hours. Even a 22\\% of reduction in training time would be helpful in reducing the cost of training in this case.\n\n[1] Nguyen, Truong Thao, François Trahay, Jens Domke, Aleksandr Drozd, Emil Vatai, Jianwei Liao, Mohamed Wahib, and Balazs Gerofi. \"Why globally re-shuffle? Revisiting data shuffling in large scale deep learning.\" In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 1085-1096. IEEE, 2022.\n\n[2]Tang, Zhenheng, Shaohuai Shi, Xiaowen Chu, Wei Wang, and Bo Li. \"Communication-efficient distributed deep learning: A comprehensive survey.\" arXiv preprint arXiv:2003.06307 (2020)\n"}, {"review_id": "GNojo07EzZ", "reviewer": "Reviewer_QXg1", "summary": "This paper introduces a method called KAKURENBO to increase the efficiency of training deep neural networks by reducing sample complexity while maintaining performance accuracy. The method encourages adaptive sample selection for each training epoch based on empirical criteria such as prediction accuracy (PA), prediction confidence (PC), and loss. It then adjusts the learning rate and maximum hidden fraction for each iteration to address convergence issues. \n\nAdditionally, the method reuses the empirical criteria (PC and loss) computed by the forward pass from the previous iteration as the basis for running the same selection mechanism in the next iteration.\n\nThe empirical evaluation of the proposed method against existing strategies on a range of datasets shows that KAKURENBO is effective in reducing total training time by up to 22%, with only a 0.4% impact on accuracy compared to the baseline.\n", "questions": "No more except the one that I was asking in the “weaknesses” part.", "limitations": "Failure to generate more efficient training using 1 GPU should be properly addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "$\\textbf{Strong empirical result}$:\nThe results demonstrating the efficacy of KAKURENBO are compelling, as it outperformed other training strategies of the same type in terms of reducing training time while maintaining accuracy. The implementation details, including the source code, are thoroughly reported and appear to be reproducible. The ablation study is also conducted extensively. I believe that the proposed method would make a significant contribution to this subfield of research.\n", "weaknesses": "$\\textbf{More complexity using a single GPU}$: \nI notice that KAKURENBO takes more training time than the baseline, also with worse accuracy, using a single GPU on CIFAR-100 with ResNet-18 (Table 3). Could the authors provide the reasoning in more detail?\n\nMinor redactional issues:\n- L243: missing \".\" before \"As the result ...\"\n- L287: accidental white space\n- L276 - 284: please revisit the grammar\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**More complexity using a single GPU:** \nIt is worth noting that KAKURENBO has computation overheads for updating the loss and prediction (Step D in Figure 1 in our main manuscript) and sorting the samples based on the loss (Step A in Figure 1 in our main manuscript). The performance gain from hiding samples will be limited if F is small, or potentially less than the overhead to compute the importance score of samples. In experiments using multiple GPUs, those operations are performed in parallel reducing the running time overhead. When using a single GPU on CIFAR-100 with ResNet-18 (Table 3 in our manuscript), the computational overhead is bigger than the speedup gained from hiding samples. Thus, KAKURENBO takes more training time in this case. In short, KAKURENBO is optimized for large-scale training and provides more benefits when running on multiple GPUs. "}, {"review_id": "LwTkJ5QXBI", "reviewer": "Reviewer_apYw", "summary": "This paper proposes a framework called KAKURENBO to accelerate DNN training by adaptively hiding samples. In the proposed framework, sample of low loss are hidden (i.e, not used) in the next training epoch, so that the total training steps are reduced. To compensate for the reduction in optimization steps, the authors propose to adjust the learning rate based on the ratio of hidden samples. Models trained with KAKURENBO achieves faster convergence with a little performance drop.", "questions": "In Line 123, it is stated that \"hidden samples that maintain a correct prediction with high confidence are moved back to the epoch training set.\". Is it against the statement in Line 165 that \"Only samples that have low loss and sustain correct prediction\nwith high confidence in the current epoch are hidden in the following epoch\"?", "limitations": "Technical and experimental limitations as stated in weaknesses. No obvious societal negative impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and the logic is clear.\n2. The proposed method achieves better convergence-accuracy tradeoff than some baseline methods.", "weaknesses": "1. The hidden factor F needs to be manually adjusted based on different datasets and training schedules.\n2. The baseline of ImageNet-1K is a bit too weak for both ResNet and EfficientNet. EfficientNet-B3 achieves 81.6 Top-1 Acc in the original paper [1].\n3. Lack of comparison with randomly hidding samples.\n4. Lack of ablation study of not adjusting learning rate.\n5. Lack of comparison with more advanced methods. For example, [2] also achieves fast convergence (saving ~30% training time), but without performance drop. The baseline of ImageNet-1K is also stronger in [2] (ResNet-50 76.4% in [2] vs 74.9 in this paper).\n\n- [1] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\n- [2] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**3.1. The hidden factor F needs to be manually adjusted based on different datasets and training schedules**. We consider F as a hyperparameter, similarly to other methods in this domain. In our method, F is the maximum fraction, which can be adapted and changed during the training process.\n\n**3.2. The baseline of ImageNet-1K is a bit too weak for both ResNet and EfficientNet**. The baseline depends on hyper-parameters and the training technique used. We report the effectiveness of KAKURENBO with different training settings and accuracies in APPENDIX C.3 (Table 3 and Figure 4).\n    \n**3.3. Lack of comparison with randomly hiding samples**\nWe provide extra results in Table 1 to evaluate the training accuracy of random hiding with the CIFAR-100 dataset and WRN-28-10 model. As seen, accuracy is only 76.82\\% which is lower than that of both KAKURENBO and Baseline. In fact, randomly hiding samples has been investigated before in the GradMatch paper and it's been reported that accuracy is low. This drove us originally not to evaluate this method.\n\n**3.4. Lack of ablation study of not adjusting learning rate.** Ablation study for not the adjusting learning rate is shown in APPENDIX C.4 (Table 5). The results show that if adjusting learning rate is not applied, the accuracy becomes lower.\n\n**3.5. Lack of comparison with more advanced methods** \nThe paper in [1] appears after our submission. We are happy to add those papers to the related work. \nIt is important to note that for method [1], the reported speedup is for a specific training regime (that uses a particular optimizer: LARS). The specific training regime described in the paper leads to a slow baseline (see Table 2 in [1], which shows the baseline that trains ImageNet-1K/ResNet-50 on 8 A100 GPUs for 90 epochs to be 3-4x slower than the typical number of hours to train ImageNet-1K/ResNet-50 on 8 A100 GPUs, as reported by many sources, including Nvidia NGC catalog [2]). That means while InfoBatch reports 26\\% in speedup, the baseline setting for which InfoBatch is demonstrated to be effective is slow.\n\n[1] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning)\n[2] \\url{https://catalog.ngc.nvidia.com/orgs/nvidia/teams/dle/resources/resnet_pyt/performance}"}, {"review_id": "dPDM10yZle", "reviewer": "Reviewer_gHmy", "summary": "The paper proposes to accelerate DNN training by dynamically selecting a subset of the training set. The method records the per-sample loss during training, and hides (a heuristically set proportion of) samples with the least loss and correct prediction, then trains the model with an adjusted learning rate. The authors also provide a theoretical justification for the convergence. Experiments show that the proposed method reduces computation with negligible impact on the accuracy.", "questions": "1.\tHow does the hidden sample vary between epochs? It would be interesting to study how the contribution (to training) of samples goes ups and downs during learning.\n2.\tDo you think that per-sample gradients are a better metric for sorting? The gradient is also freely obtainable in training using back-propagation as the loss value.\n3.\tCan the method be combined with acceleration beyond data selection for a better performance? For example, Low dimensional trajectory hypothesis is true: DNNs can be trained in tiny subspaces, TPAMI 2022.", "limitations": "The authors do not discuss potential limitations in the manuscript.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe overall design is intuitive and technically sound. It is interesting to see this elegant framework using the simple hint of loss value.\n2.\tThe authors provide a thorough summary of previous work, so it is clear for a wide community of readers to know this field.\n3.\tThe manuscript is well organized. I like the logical overview before Sec 3.1.", "weaknesses": "1.\tThe hyperparameters (\\tao, F_e) are heuristically set without analysis and discussion. It is unclear how many additional trials are required to obtain the claimed 20% acceleration. So I cannot make sure how it generalizes to untested datasets.\n2.\tWe cannot see clearly (from the results) the superiority of the proposed method. It would be helpful to plot the accuracy-time trade-off of different baselines.\n3.\tThere is a missing reference (Towards Sustainable Learning: Coresets for Data-efficient Deep Learning, ICML 2023) that targets the same problem with dynamic hiding, which claims a very good acceleration performance. A comparison would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**4.1. Discussion on the hyperparameters ($\\tau$, $F_e$):** Higher prediction confidence threshold $\\tau$ leads to a higher number of samples moved back, i.e., fewer hidden samples at the beginning of the training process. At the end of the training process, when the model is well-trained, more samples are predicted correctly with high confidence. Thus the impact of the prediction confidence threshold on the number of moved-back samples becomes less (as shown in Figure 1 of the extra result in the rebuttal).  We also provide ablation study on the impact of prediction confidence threshold $\\tau$ in Table 2 (extra result in the rebuttal). The result shows that when we increase the threshold $\\tau$, wo obtain better accuracy (fewer hidden samples), but at the cost of smaller performance gain. We suggest to set $\\tau = 0.7$ in all the experiments as a good trade-off between training time and accuracy.\n\nWe consider the maximum fraction per epoch ($F_e$) as a hyperparameter and we suggest using step scheduling, i.e., to reduce the maximum hiding fraction gradually with a factor of $\\alpha$ as the number of epochs increases. Specifically, we match the epoch at which $\\alpha$ is reduced to the reduction of the learning rate in the literature. For example, we set $\\alpha$ as [1, 0.8, 0.6, 0.4] at epoch numbers [0,30,60,80] for ImageNet-1K and [0, 60, 120, 180] for CIFAR-100, respectively.\n\n\n\n**4.2. Missing reference (Towards Sustainable Learning: Coresets for Data-efficient Deep Learning, ICML 2023)** The paper appear after our submission. We are happy to add this paper in the related work. Basically the proposed method in this paper has a high overhead of visiting each sample to find the coreset. In addition, it is only demonstrated on small datasets such as CIFAR and TinyImageNet, at which the overheads for a large number of samples would not appear. KAKURENBO on the other had is demonsrated on ImageNet-1K, DeepCAM, and ImageNet-1K-Fractal.\n\n**4.3. How does the hidden sample vary between epochs?** \nIt is difficult to present how the contribution of each sample to the training process changes during training due to the large number of samples (in large datasets). Instead, in this study, we showed the contribution of each class to the training, which we partially discussed in Appendix C.1 (Figure 2) for ImageNet-1K datasets with ResNet-50. We also added new experimental results in Figure 2 (extra result in the rebuttal). The results show that the impact of each class remains different during the training. Easy classes such as class $1$, and $2$ are hidden during the training more than the class $31$ and $47$. The result in Figure 1 (extra result in the rebuttal) also shows that in two continuous epochs, only (around) 30\\% of samples are hidden again. This result infers that the importance (or contribution level) of samples is changing epoch by epoch.\n\n**4.4. Do you think that per-sample gradients are a better metric for sorting?** \nYes. However, the computation overhead for calculating the gradients of the hidden samples (performing both forward and backward pass on those samples) is not trivial. To reduce the overhead, we use the loss of samples that requires performing only the forward pass.\n\n**4.5. Can the method be combined with acceleration beyond data selection for a better performance?** \nTo the best of our knowledge, we believe that KAKURENBO can be combined with the other loss-based optimizer and machine-learning methods. It also can be combined with other communication/computation optimization that have been proposed for speeding up the training process. More specifically, the referred paper by the reviewer proposed a quasi-Newton-based algorithm which is also based on the loss of samples for optimizing the model parameters. We believe that KAKURENBO can be combined with this method.\n\n**4.6. We cannot see clearly (from the results) the superiority of the proposed method. It would be helpful to plot the accuracy-time trade-off of different baselines.** \nWe do plot the accuracy-time trade-off over different baselines (ISWR, FORGET, SB) in Figure 2 in the original submission."}, {"review_id": "Fwfy0IsavE", "reviewer": "Reviewer_Zijr", "summary": "The paper introduces an approach to reduce training time of deep neural\nnetworks on large datasets by excluding samples on an epoch basis with a\nminimal reduction of the accuracy. In each epoch, a scheduled fraction of the\nsamples with the lowest loss is excluded, unless its prediction is incorrect\nor its softmax confidence is below a fixed threshold. In addition, the learning\nrate is scaled to account for the fraction of excluded samples.\nThe wall clock time as well as the accuracy of the introduced approach is\ncompared to four recent related approaches, as well as the unmodified training.", "questions": "- The only major issue I see is that the experimental results seem to be based\n  only on a single setup per trial. While the reason for this was most likely\n  the runtime of the training, it is very important to demonstrate the\n  significance of the results. If this cannot be adressed, I fear that this\n  could be a reason to reject this otherwise very good work.\n\n- The calibration of the softmax confidence should be discussed.\n\n\n### Minor\n- the textual references are odd (u.a. instead of et al., und instead of and)", "limitations": "The authors do not report any limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The approach is novel, simple but seemingly effective.\n\n- The approach is introduced very well.\n\n- All figures are very high quality, especially the overview in Fig. 1.\n\n- The paper provides a very good overview and contextualization over recent related methods (Table 1).", "weaknesses": "- The confidence value used for moving back samples is based on the model\n  softmax, which is may not be very well calibrated.\n\n- The experiments seem to be single trial. While training time makes this\n  problematic, it is important to demonstrate that the results are significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**5.1. Single setup per trial issue:** \nWe confirm that the results of our implementation are stable by running experiments 3 times for CIFAR-100 and ImangetNet-1K. The new result is reported in Table 1 (extra-result in the rebuttal). As for our original results, due to resource limitation, we could not run experiments multiple times for each data point reported in the paper, especially for models and datasets that are big in this work.\n\n**5.2. The calibration of the softmax confidence:**\nAs mentioned in the main manuscript, only samples that have low loss and sustain correct prediction with high confidence in the current epoch are hidden in the following epoch.  A sample is correctly predicted with high confidence at an epoch $e$ if it is predicted correctly (\\textbf{PA}) and the prediction confidence (\\textbf{PC}) is no less than a certain threshold \\textbf{$\\tau$}.\nAs shown in Figure 1 (extra result in the rebuttal), the number of samples moved back becomes smaller when as the number of epochs increases because the prediction confidence becomes higher during the training, thus the actual hidden samples become similar to the max. hidden samples in the latter epoch. Results in Table 2 (extra result in the rebuttal) show that when we increase the threshold $\\tau$ we obtain better accuracy (i.e., fewer hidden samples) at the cost of smaller performance improvements. However, the gaps remain small.\n\n**5.3. Textual references:** Thank you for the comments. We are happy to improve the text based on the suggestions. \n"}]

Author rebuttal:
We thank all reviewers for their constructive comments and questions.

**1. Single setup per trial issue (Reviewer Zijr)**
We confirm that the results of our implementation are stable by running experiments 3 times for CIFAR-100 and ImangetNet-1K. The new result is reported in Table 1 (extra-result in the rebuttal). As for our original results, due to resource limitations, we could not run experiments multiple times for each data point reported in the paper, especially for models and datasets that are big in this work.

**2. Discussion on the hyperparameters (Reviewer gHmy and Zijr)**
Higher prediction confidence threshold $\tau$ leads to a higher number of samples being moved back to the training set, i.e., fewer hidden samples at the beginning of the training process. At the end of the training process, when the model has is well-trained, more samples are predicted correctly with high confidence. Thus the impact of the prediction confidence threshold on the number of moved-back samples becomes less (as shown in Figure 1 of the extra result in the rebuttal).  We also provide ablation study on the impact of prediction confidence threshold $\tau$ in Table 2 (extra result in the rebuttal). The result shows that when we increase the threshold $\tau$, we obtain better accuracy (fewer hidden samples), but at the cost of smaller performance gain. We suggest to set $\tau = 0.7$ in all the experiments as a good trade-off between training time and accuracy.

We consider the maximum fraction per epoch ($F_e$) as a hyperparameter and we suggest using step scheduling, i.e., to reduce the maximum hiding fraction gradually with a factor of $\alpha$ as the number of epochs increases. Specifically, we match the epoch at which $\alpha$ is reduced to the reduction of the learning rate in the literature. For example, we set $\alpha$ as [1, 0.8, 0.6, 0.4] at epoch numbers [0,30,60,80] for ImageNet-1K and [0, 60, 120, 180] for CIFAR-100, respectively.

**3. Comparison with other methods (Reviwer apYm and Zijr)**
We provide extra results in Table 1 %(extra results in the rebuttal)
to evaluate the training accuracy of random hiding with the CIFAR-100 dataset and WRN-28-10 model. As seen, accuracy is only 76.82\% which is lower than that of both KAKURENBO and Baseline. In fact, randomly hiding samples has been investigated before in the GradMatch paper and it's been reported that accuracy is low. This drove us originally not to evaluate this method. 

For the works in [1], and [2], both of the papers appear after our submission. We are happy to add those papers to the related work. 
It is important to note that for method [1], the reported speedup is for a specific training regime (that uses a particular optimizer: LARS). The specific training regime described in the paper leads to a slow baseline (see Table 2 in [1], which shows the baseline that trains ImageNet-1K/ResNet-50 on 8 A100 GPUs for 90 epochs to be 3-4x slower than the typical number of hours to train ImageNet-1K/ResNet-50 on 8 A100 GPUs, as reported by many sources, including Nvidia NGC catalog [3]). That means while InfoBatch reports 26\% in speedup, the baseline setting for which InfoBatch is demonstrated to be effective is slow. The proposed method in [2] has a high overhead of visiting each sample to find the coreset.

[1] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning)

[2] (Towards Sustainable Learning: Coresets for Data-efficient Deep Learning, ICML 2023)

[3] \url{https://catalog.ngc.nvidia.com/orgs/nvidia/teams/dle/resources/resnet_pyt/performance}

**4. Improvement in training time (Reviewer wMxv, QXg1, and gHmy)**
It is worth noting that KAKURENBO has computation overheads for updating the loss and prediction (Step D in Figure 1 in our main manuscript) and sorting the samples based on the loss (Step A in Figure 1 in our main manuscript). The performance gain from hiding samples will be limited if F is small, or potentially less than the overhead to compute the importance score of samples. In experiments using multiple GPUs, those operations are performed in parallel reducing the running time overhead. When using a single GPU on CIFAR-100 with ResNet-18 (Table 3 in our manuscript), the computational overhead is bigger than the speedup gained from hiding samples. Thus, KAKURENBO takes more training time in this case. In short, KAKURENBO is optimized for large-scale training and provides more benefits when running on multiple GPUs. 

It is unattainable to speed-up the training process by 10-100X just by hiding samples during training as it would require 90-99\% of samples to be hidden without reducing accuracy. However, our method can be combined with other computation/communication optimization methods that have been proposed for speeding up training [4],[5]). Our method is helpful in pre-training a model with a huge dataset (e.g., in transfer learning). Training in such cases may have very high computing resource costs, e.g., 100s-1000s of GPU hours. Even a 22\% of reduction in training time would be helpful in reducing the cost of training in this case.

[4] Nguyen, Truong Thao, François Trahay, Jens Domke, Aleksandr Drozd, Emil Vatai, Jianwei Liao, Mohamed Wahib, and Balazs Gerofi. "Why globally re-shuffle? Revisiting data shuffling in large scale deep learning." In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 1085-1096. IEEE, 2022.

[5]Tang, Zhenheng, Shaohuai Shi, Xiaowen Chu, Wei Wang, and Bo Li. "Communication-efficient distributed deep learning: A comprehensive survey." arXiv preprint arXiv:2003.06307 (2020).

**5. The baseline of ImageNet-1K is a bit too weak (Reviewer apYw)**
The baseline depends on hyper-parameters and the training technique used. We report the effectiveness of KAKURENBO with different training settings and accuracies in APPENDIX C.3 (Table 3 and Figure 4).



Meta-review: The paper proposes a method to adaptively remove samples based on their prediction accuracy, confidence and loss. The ratings are 1 accept, 2 weak accepts, 1 borderline accept and 1 borderline reject. The leaning negative reviewer has the remaining concern about the mentioning of LAION and JFT datasets as motivating examples in their final comment, but the AC views this as fine. Reviewers agree the results are favorable compared with existing methods, and most reviewers agree that the method is intuitive and clear. The AC also agrees that the addressed problem about data selection is important. These outweigh the potential issues brought by reviewers, but the authors are also encouraged to take these suggestions to improve the paper. The AC is pleased to recommend the acceptance of this paper.
Decision: Accept (poster)