InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning
Authors: Ziheng Qin
Kai Wang
Zangwei Zheng
Jianyang Gu
Xiangyu Peng
xu Zhao Pan
Daquan Zhou
Lei Shang
Baigui Sun
Xuansong Xie
Yang You
Venue: ICLR 2024 oral (2024)
Keywords: Dynamic Data Pruning; Training acceleration
Primary area: general machine learning (i.e., none of the above)

Abstract: Data pruning aims to obtain lossless performances with less overall cost. A common approach is to filter out samples that make less contribution to the training. This could lead to gradient expectation bias compared to the original data. To solve this problem, we propose InfoBatch, a novel framework aiming to achieve lossless training acceleration by unbiased dynamic data pruning. Specifically, InfoBatch
randomly prunes a portion of less informative samples based on the loss distribution and rescales the gradients of the remaining samples to approximate the original gradient. As a plug-and-play and architecture-agnostic framework, InfoBatch consistently obtains lossless training results on classification, semantic segmentation, vision pertaining, and instruction fine-tuning tasks. On CIFAR10/100, ImageNet-
1K, and ADE20K, InfoBatch losslessly saves 40% overall cost. For pertaining MAE and diffusion model, InfoBatch can respectively save 24.8% and 27% cost. For LLaMA instruction fine-tuning, combining InfoBatch and the recent coreset selection method (DQ) can achieve 10 times acceleration. Our results encourage more exploration on the data efficiency aspect of large model training. Code is publicly available at NUS-HPC-AI-Lab/InfoBatch.

Published as a conference paper at ICLR 2024
INFOBATCH: LOSSLESS TRAINING SPEED UP BY UNBI- ASED DYNAMIC DATA PRUNING
Ziheng Qin Kai Wang's o Zangwei Zheng' Jianyang Gu'
Xiangyu Peng' Zhaopan Xu Yang You
Daquan Zhou' Lei Shang* Baigui Sun Xuansong Xie²
1 National University of Singapore AAlibba Group {zihengg, kai.wang, youy)0comp.. . edu.sg
ABSTRACT
Data pruning aims to obtain lossless performances with less overall cost. A com- mon approach is to filter out samples that make less contribution to the training. This could lead to gradient expectation bias compared to the original data. To solve this problem, we propose InfoBatch, novel framework aiming to achieve lossless training acceleration by unbiased dynamic data pruning. Specifically, InfoBatch randomly prunes portion of less informative samples based on the loss distribution and rescales the gradients of the remaining samples to approximate the original framework, InfoBatch con-
gradient. As plug-and-pe and-play and architecture-agnssii sistently obtains lossless training results on classification, semantic segmentation, vision pertaining, and instruction fine-tuning tasks. On CIFAR 10/100, ImageNet- IK, and ADE20K, InfoBatch losslessly saves 40% overall cost. For pertaining MAE and diffusion model, InfoBatch can respectively save 24.8% and 27% cost. For LLaMA instruction fine-tuning, combining InfoBatch and the recent coreset selection method (DQ) can achieve 10 times acceleration. Our results encourage more exploration on the data efficiency aspect of large model training. Code is publicly available at NUS-HIC-AA--Labintooaa
INTRODUCTION
In the past decade, deep learning has achieved remarkable progress the computer vision area (Doso- vitskiy et al., 2020; Touvron al., 2020). Most tfateoof-the-at art methods (Dosovitskiy et al., 2020; Touvron et al., 2020) are trained on ultra- ultra-large-scal datasets, but the heavy training cost is hardly affordable for researchers with limited computation resources. Reducing the training effort for large- scale datasets has become urgent for broader computer vision and other deep-learning applications.
An intuitive solution is to reduce the training sample amount. Dataset distillation (Nguyen et al., 2021; Zhao & Bilen, 2021; Wang et al., 2022) and coreset selection (Har-Peled & Mazumdar, 2004; Park al., 2022; Xia et al., 2023) respectively synthesize choose informative dataset/subset from the original large one. Although the sample amount is reduced, the distillation and selection algorithms lead to extra costs. Besides, these two methods are also hard to achieve lossless performance.
The other solution is weighted sampling methods (Zhao & Zhang, 2014; Csiba & Richtárik, 2016; Johnson Guestrin, 2018) that aim to improve the sampling frequency of certain samples. It improves the convergence speed, but their accelerations are sensitive to models and datasets (Zhao & Zhang, 2014). In another way, LARS (You et al., 2017) and LAMB (You al., 2019) enable super large batch size to improve data parallelism during training to reduce the overall training time. However, more computation units are required and the total training cost not reduced, which limits the effect under constrained computation resources.
Most recently, a series of works propose to accelerate training by reducing the total training iterations. Toneva et al. (2018); Paul et al. (2021) estimate score for each sample and accordingly prune less informative samples. Since those pruned samples are never added back during training, we refer
"Equal Contribution.
Prooject Lead.
coorrespnndigg author

---

Published as a conference paper at ICLR 2024
Hard
Bias
Num
Prune
O(logN)
Sorting
Iterations
EL2N Score
EL2N Score
Acc. 78.2% vs 71.0%
(a) Previous methods prume samples via setting some heuristic metrics (e.g. EL2N score). The hard pruning operation results in biased gradient expectation (green lines) and sub-optimal training results.
Soft
Rescale
Num.
Prune
Num
(
Compare
K Iterations
Loss Value
Loss Value
Acc. 78.2% 78.2%
(b) InfoBatch soft prunes samples with small loss values and rescales the updates. Therehy InfoBatch maintains approximately the same gradient expectation direction (blue lines) as training on the original dataset.
Figure 1: Illustration of difference between InfoBatch and EL2N (Paul et al., 202 static pruning method). InfoBatch achieves lossless acceleration performance, while EL.2N has a performance drop of 7.2%. Experiments conducted with ResNet- 18 under the same pruning ratio of 50% CIFAR-100 The semi-transparent triangles represent the variance range of gradient estimation.
to these methods as static pruning methods. However, these methods usually need several trials estimate more accurate scores, which requires extra (TM overhead (M is the model size, N is the dataset size, and T is the trial number), sometimes even longer than the training time on large-scale datasets (e.g. ImageNet- 1K as Tab. 2). Therefore, is cumbersome apply those static pruning methods (Toneva et al., 2018; Paul et al., 2021) on large-scale datasets.
To reduce this heavy overhead, Raju al. (2021) dynamically prunes the samples based on easily attainable scores, e.g., loss values, during the training without trials. This work sorts the whole dataset by the score in each pruning cycle, which leads to (log.N) per sample time complexity on sorting. It would cost much less ImageNet-Ik than static pruning, while the overhead would still be non-negligible on larger datasets. Mcanwhile, directly pruning data may lead to biased gradient estimation as illustrated in Fig. 1a, which affects the convergence result. This is crucial factor that limits their performance, especially under a high pruning ratio (corresponding results in Tab. 1).
To tackle these issues, propose InfoBatch, novel unbiased dynamic data pruning framework based on the idea of maintaining the same expected total update between training on the pruned and original datasets. We refer this idea expectation rescaling for simplicity. Specifically, given dataset, we maintain a score of each sample with its loss value during forward propagation. We randomly prune certain portion of small-score (i.e. well-learned) samples each epoch. Different from previous methods where well-learned samples are dropped directly, as shown in Fig. lb, we scale up the gradient of those remaining small-scou samples to keep an approximately same gradient expectation as the original dataset.
Compared to previous works (Toneva et al., 2018; Paul et al., 2021: Raju et al., 2021), the gradient expectation bias in optimization between InfoBatch and standard training is reduced, as illustrated Fig. la and lb. further improve the performance stability and reduce the variance during convergence, full dataset is used for training in the last few epochs. Detailed analysis is provided in Sec 2.3 and Appendix B to prove the unbiasedness of InfoBatch.
InfoBatch is compatible with various deep-learning tasks. In this work, we investigate its effect classification, semantic segmentation, vision pertaining, and language model instruction finetuning. With the same hyperparameters in most cases, InfoBatch achieves lossless training performances with 20% ~ 40% less overall cost across various tasks and architectures. helps mitigate the heavy computation cost for training ultra-large-scale datasets. The time complexity of InfoBatch is O(1)
2

---

Published as a conference paper at ICLR 2024
per sample, which further accelerates from previous (loggN) dynamic pruning methods, costing only seconds to reduce hours of training on datasets like ImageNet-1K.
HOW WE CAN ACHIEVE LOSSLESS RESULT
2.1 PRELIMINARIES
Static Pruning. Given dataset = {-+) ((r,, ye) H score H(=) can defined for each sample. For the pruning process, samples discarded by pruning probability P defined on top of H. Static pruning directly discards all samples satisfying certain condition before training, resulting in (== H) {0, 1]. For examples, Toneva al. (2018) defines:
P(z; H) = (+H(2)) H),
(1)
where H is a threshold and ((() is indicator function. A subset S is formed by pruning samples with P(:; H) = Fig. briefly illustrates the whole process of static pruning. more detailed theoretical analysis is provided in Appendix B.
Dynamic Pruning. For dynamic pruning, pruning is done across training, and the score H, can change along training steps, where denotes the temporal status. The probability is step-dependent:
P.(=) = P(2; H.).
and forms dynamically pruned dataset St. Compared to static pruning, dynamic pruning has access all the original data during training. Thus the gradient expectation bias should be much smaller than static pruning. However, such scheme still has the following limitations: i). As claimed in Raju et all (2021), low-score samples of different f during the training could easily overlap. Directly pruning them every time may still lead to a bias (see in Fig. la). ii). Pruning samples leads to reduced number of gradient updates. Under the premise of saving training cost, existing dynamic pruning methods Raju et al. (2021) hardly achieve lossless results compared to training on the original dataset. iii. Scoring and sorting operations dynamic pruning are conducted repeatedly, the overhead of which is still non-negligible on large-scale datasets.
Dataset
Backward update
Soft Pruning
D1
D2
D3
Expectation Rescaling
D
Arch.
Hi
Avg. Loss
D1
D2
Score update for Epoch
Figure Illustration the proposed InfoBatch framework. InfoBatch mainly consists of two operations, named soft pruning and expectation rescaling. H. denotes the adaptive thresholds of scores samples. Soft pruning randomly prunes some samples from D, with relatively small scores. For remaining samples from D,, expectation rescaling scales up the losses to keep the approximately same gradient expectation as the original datasct.
2.2 OVERVIEW OF INFOBATCH
Based on the above observation and analysis. we propose InfoBatch, a novel framework for achieving lossless training acceleration based on unbiased dynamic data pruning. As illustrated in Fig. 2, score is maintained for each sample with its loss value during forward propagation. The mean of these values are set as the pruning threshold. A certain portion of small-score samples accordingly pruned in each epoch. Then, to obtain the same expectation of gradient as the original dataset in each epoch, the gradients of the remaining small-score samples are scaled up. By doing this, compared previous static and dynamic pruning methods, InfoBatch mitigates the performance differences between training on the pruned dataset and the original dataset. order to further reduce the remaining gradient expectation bias, train with the full dataset in the last few epochs.
3

---

Published as a conference paper at ICLR 2024
2.3 UNBIASED PRUNE AND RESCALE
InfoBatch adopts the dynamic pruning process as in Eqn. 2. Previous methods using a deterministic pruning operation could cause bias. In contrast, InfoBatch introduces randomness into the pruning process. Given a dataset DD, in I-th epoch, a pruning probability is assigned to each sample based on its score. Such soft pruning policy is formulated as:
We first define its pruning policy Pt.
Pt(z)
r, 0, (2)2) 2 H.
H2(2) H.
where Hi is the mean value of all the scores Hi and r E (0, 1) is a predefined hyper- parameter as the pruning probability. This new prune policy has the following benefits: i). Soft pruning allows each small-score sample to be utilized for training, which reduces the bias caused by hard pruning in previous dynamic pruning methods. ii). The proposed strategy is based on the comparison with H4, with no requirement to sort the whole training samples, which reduces the time complexity from NloogN) to ((1). It indicates that InfoBatch could be more efficient large-scale datasets.
Then, we utilize loss values [()) of each sample as the corresponding score based on the following two reasons: i). loss values be obtained without extra cost, ii). loss values reflect the learning status of samples (Cilimkovic, 2015). Specifically, before the (-th (t > 0) epoch, we utilize the soft pruning policy to prune samples based on their scores. Then for the pruned samples, their scores remain unmodified as previous. For the remaining samples, their scores are updated by the losses in the current epoch. Mathematically, (z) would be updated by the latest losses to He+1(2) by:
(He(z),
2 D),
(2) ((2),
= E
Note that, for the first epoch, we initialize the scores with {1} provided no previous loss.
There are several benefits of the soft pruning policy, yet it still cannot avoid the influence caused by the reduced number of gradient updates. To address this issue, we scale up the gradients of the remaining samples. Specifically, given a remaining sample with score H(=) Hr, whose corresponding pruning probability is r, its gradient is scaled to 1/(1 - r) times of original. For the samples with scores no less than He, the loss is not modified. Thereby the gradient update expectation is approximately equal to training on the original dataset. Besides, as the rescaling is operated on small-score samples, it further refines the direction of gradient update expectation. We provide the following theoretical analysis to demonstrate the necessity and advantages of expectation rescaling.
Theoretical Analysis. We can interpret the training objective minimizing empirical risk L. Assuming all samples = from D are drawn from continuous distribution p(z), we can establish the training objective
arg min E [C(2, [[ / ((=)0))(z)d
(5)
lee =ED
After applying proposed pruning, we sample according to normalized (1-P (=))p)) In back- propagation, rescaling loss is equivalent to rescaling the gradient. By rescaling the loss of each sample with a factor (2)) 1/(1 Pz(z)) Vaz E D, Pz(2) = 0 = (x)) 1), the training
objective (expanded proof in Appendix B) on S, becomes:
deo C₁ / [(2,0)
arg min E. [T(=)) 0)] arg min
(6)
dEe sES,
where c = Ee>ppp P+(()] = J t. Then the objective in Eqn. 6 is a constant-rescaled version of the original objective in Eqn. 5. Therefore, training on S₁ with rescaled factor (2) could achieve similar result training on the original dataset. Furthermore, these operations also leverage the problem of reduced iterations. In real-world applications, considering the dataset as discrete ones, we force the sample number as:
et E (0, 1) is constant for temporal status
D
D
D S
o(V,)( ~
E1ED(1 ((())
S,
For each epoch, the iteration number, i.e. gradient update number becomes | of the original one, while our method scale the expected gradient to [D|/|S])- As result, this would leverage the influence of reduced gradient update number. The approximation will hold when the pruning ratio is not too high and the learning rate is not too big. We provide a detailed analysis in Appendix B.
4

---

Published as a conference paper at ICLR 2024
2.4 ANNEALING
Based on the theoretical analysis above, the objectives and updating expectations between InfoBatch and training on the original dataset are approximately the same. However, there still exist minor differences between the optimization on 'D and St. During training, if a sample is pruned in the middle stage, it is still likely be revisited afterward. However, in the last few epochs, the revisiting probability drastically drops, resulting in remaining gradient expectation bias. Therefore, given training epoch C, we define ratio hyper-parameter ó E (0, 1). The pruning is only conducted in the first 5 C' epochs (in practice, 5 is close to I so InfoBatch saves at a reasonable ratio). After that, we train on the full dataset till the end. The corresponding operation can be interpreted as
P((2) r, 10, Ha(x) 2 Hi V128.C
H1(2) < H4 A
(8)
Combining the above components, InfoBatch achieves lossless training performance with fewer iterations compared with training on the original dataset.
EXPERIMENTS
3.1 DATASETS AND IMPLEMENTATION DETAILS
We verify the effectiveness of our method on multiple datasets: CUFAR-10/100 (Krizhevsky et al., a;b), ImageNet-IK (Deng et al., 2009), ADE20K (Zhou al., 2017) and FFHQ (Karras et al., 2019).
Implementation Details. For InfoBatch, default value r = 0.5 and 6 = 0.875 are used if not specified. For classification tasks, we train ResNet18 and ResNet-50(He al., 2016), ViT-Ba (He et al., 2021) and Swin-Tiny (Liu et al., 2021) for evaluation. On CIFAR-10/100 and ImageNet-1K, all models trained with OneCyele scheduler (cosine annealing) (Loshchilov & Hutter, 2016: Smith & Topin, 2017) using default setting and SGD/LARS optimizer ( (You et al., 2017) with momentum 0.9, weight decay 5e-4. All images are augmented with commonly adopted transformations, i.e. normalization, random crop, and horizontal flop if not stated otherwise. The implementation is based on PyTorch (Paszke et al., 2019) and Timm (Wightman et al., 2021). On CIFAR 100, 100, ImageNet- and ADE20K, a more aggressive (0.75) is utilized for smaller loss samples (20%). For the semantic segmentation task, we conduct experiments on ADE20K (Zhou et al., 2017). The chosen network is UperNet (Xiao et al., 2018) 8) with backbone ResNet-50. We follow the default configuration of the mmsegmentation (Contributors, 2020). All other details can be found in Appendix.
3.2 COMPARISONS WITH SOTA METHODS
Performance Comparisons. We compare our proposed InfoBatch with static and dynamic data pruning methods in Tab. 1 on CIFAR-10, 10, CIFAR- 100. In the first part, we introduce static pruning methods, whose simplest baseline is random selection before training. Influence (Koh & Liang, 2017) and EL2N (Paul et al., 2021) are two classical static pruning methods that prune samples based on Influence-score and E1.2N-ccree. DP (Yang et al., 2023b) conducts pruning with consideration of generalization. To make a wider comparison, we include 10 coreset selection methods. These methods select coreset of data via their predefined score function or heuristic knowledge. Then, we introduce 3 dynamic pruming methods in the second part. Following (Raju et al., 2021) 1), we also construct dynamic pruning baseline, termed as Random". which conducts random selection each epoch. Compared to Random operation in Tab. 1, the diversity of training samples in Random" is much better than Random. Therefore, as shown in Tab. 1, Random" usually performs better than Random and many static methods. e- e-greedy (Raju et al., 2021) is inspired by reinforcement learning. UCB (Raju et al., 2021) proposes to prune samples using the upper confidence bound of loss.
Based on the comparisons in Tab. 1, we have the following observations: 1). At 30% pruning ratio, only InfoBatch achieves lossless performances on both datasets. Other methods only obtain near- lossless results CIFAR-10 10 dataset while encountering large performance drops CIFAR- 100000.22 2). As the pruning ratio increases, InfoBatch continuously outperforms other methods by an even larger accuracy margin. It validates the effectiveness of the proposed unbiased dynamic pruning strategy.
Efficiency Comparisons. In addition to the performance comparison, we compare the efficiency between Infobatch and other methods. Although the motivations of these methods are diverse, their
5

---

Published as a conference paper at ICLR 2024
Table 1: The accuracy (%) comparison state-of-the-art methods. All methods are trained with ResNet-18. 18. As InfoBatch has a self-adaptive ratio, we mark the results with where the same forward propagation number during training is matched by controlling epoch number. Random" denotes dynamic random pruning. Details are available in Appendix A.
Dataset
CIFARIO
CIFAR100
Prune Ratio
30
50
30
Random
94.611.0
93.3
90.2
73.8 .4.4
72.1
69.7
Herding (Welling, 2009)
92.2
88.0,7.6
g15.5
73.1 J6.1
71.8
69.618.0
Influence (Koh & Liang, 2017)
g3.1
91.3
88.3
74.4.a.8
72.0
68.9.9.5
K- K-Center (Sener & Savarese, 2018)
94.7
93.9.1.7
90.9
J4.1
72.2,6.0
70.2
DeepFool (Ducoffe & Precioso, 2018)
95.1
94.1.1.5
90.0g5.6
74.2.4.0
73.215.0
69.816.4
Forgetting (Toneva al., 2018)
94.7
94.1
91.7 13.0
75.3
73.1
48.9
EL2N-2 (Toneva al., 2018)
94.4
93.2,j24 J2.4
89.8
74.1
71.0,7.2
68.5
EL2N-2 (Toneva et al., 2018)
95.3g0.3
95.1
91.9 18.7
77.2.1.0
72.1
Statie
Least Confidence (Coleman et al., 2019)
95.0
94.5,1.1
90.3
74.2.4.0
72.315.9
69.8
Margin (Coleman et al., 2019)
94.9
94.3
90.9
74.0
72.2
70.2
CD (Agarwal al., 2020)
95.0gn.6
94.3,1..
90.8
74.2.4.0
72.3
70.3
Craig (Mirzasoleiman al., 2020)
94.8,0.8
93.3
88.4
J4.4
71.9.6.3
69.7
GraNd-4 (Paul et al., 2021)
95.3
94.6,1.0
91.2
74.6.3.6
71.4
68.8
Glister (Killamsetty et al., 202
95.2.0.4
94.0,1.6
90.9
74.6.3.6
73.2
70.4
DP (Yang et al., 2023b)
94.9
93.8
90.8
27.2.1.0
73.1
Dynamic
Random
94.8
94.5
93.0
77.3.0.9
75.3
Full Data Reduced Epoch
94.8
94.6
92.7
77.0.1.2
76.9
76.3
é-greedy (Raju et al., 2021) UCB (Raju al., 2021)
95.2.0.4
94.1 94.9.0.7
76.4
74.8
95.3
94.7.0.9
93.9 11.7
77.3 _0.9
75.3
InfoBatch
95.6
195.1
194.7 .0.9
78.2 10.0
18.1 Ln.1
16..5.
Whole Dataset
95.6_n.1
78.2_o.1
Table 2: Comparison of performance and time cost on ImageNet-IK: K. Results are reported with ResNet-50
Table 3: Experiments on ImageNet-IK, ViT-
Base(MAE) is pretrained with InfoBatch for
under 40% prune ratio for 90 epochs on an 8-A 100-
300 epochs and fine-tuned 100 epochs. Swin-
GPU server. "Time" is wall clock time; "Total (n*h)"
Tiny is trained from scratch InfoBatch.
is the total node hour.
Model #
Prune Ratio
Orignial
InfoBatch
GC
EL2N-20
UCB
Ours
Full Data
Acc (%)
73.6
75.8
76.5
76.4
R-50pyTorch
40.5%
76.4
16.5
R-50Tumm
20.3%
78.4
78.3
Overhead (h)
Time (h)
22
10.5
>17.
10.5
0.03
10.5
0.0028
10.5
17.5
0.0
Swin-T
19.0%
81.5
81.4jo.1
Total (x*h)
108
>234
140.0
ViT-B(MAE)
24.8%
82.8
82.9
main goal to save training costs. Thus, we report training time, extra cost, and total GPU hours of these methods in Tab. 2. Under the same computational condition, static pruning methods GC and EI,2N require extra preprocessing time even longer than the actual training run to process the pruning. Previous ffate-oftthe-a art dynamic data pruning method UCB (Raju et al., 2021) shows far better efficiency in the pruning process, while still not comparable to InfoBatch. InfoBatch further reduces the extra time cost to 1/10 of UCB, taking only 10 seconds for 90 epochs of pruning. More importantly, InfoBatch achieves lossless performance on ImageNet- IK at the pruning ratio of 40%.
3.3 IMAGENET-II K RESULTS
As InfoBatch is proposed to scale data pruning to large-scale datasets, we train ResNet-50, VIT- Base(MAE) (He et al., 2021), l), Swin-Tiny (Liu et al., 2021) l) on ImageNet-IK. K. The results are shown in Table 3. Evidently, InfoBatch is able to achieve lossless performance on both CNN-based net- works and Transformer- based networks with substantial cost savings. The experimental result with Timm (Wightman et al., 2021) indicates that InfoBatch is compatible with existing acceleration meth- ods like mixed-precision training and sugmentation/regulaizziiio methods including Random Erase, MixUp, CutMix (Zhong al., 2017; Zhang al., 2018; Yun et al., 2019) etc. (see appendix A.4).

---

Published as a conference paper at ICLR 2024
Table 4: Ablation of proposed operations in the
Table 5: Comparison of pruning conditions. Ran- dom here using rescaling and annealing, ad- justing the value to control the pruning ratio.
proposed framework on CIFAR-100. We set r 0.5 in the experiments. Random" is same as Tab.
Experiments are conducted on CIFAR- 100 R-50.
Operation
Acc.
P Res
Ann
R-18
R-50
Prune Condition
Acc. (%)
Prune. (%)
Random"
77.3+0.2
79.7+0.2
Soft
77.5+0.2
79.9+0.2
H.(2) < Hr
80.6+0.2
33
Soft
78.1
80.3+0.3
H.(2) > Hr
80.5+0.2
16
Soft
77.7+00.2
80.0=0.2
Random Random
80.5.10.1
16
Soft
78.2+-0.2
80.6+0.2
80.5+0.2
33
Full Dataset
78.240..
80.6-1.11
Full Dataset
80.6+0.1
The overhead of InfoBatch is visualized in Fig. 3d. Results are suggesting that InfoBatch can greatly improve training costs on various computer vision tasks using large-scale datasets like ImageNet-IK. with negligible overhead compared to its acceleration.
3.4 ABLATION EXPERIMENTS
We perform extensive ablation experiments to illustrate the characteristics of InfoBatch. If not stated, the experiments conducted on CIFAR-100 by default.
Evaluating the Components of InfoBatch. We design an ablation study to investigate the soft pruning policy, expectation rescaling, and annealing operations in Tab. 4. Dynamic random pruning, serving as the baseline method, fails to achieve lossless performance on all the architectures. It can be explained that due to pruning, the gradient update number is reduced compared training on the original dataset. Similarly, only applying soft pruning obtains marginally better results than dynamic random pruning as expected.
contrast, the proposed rescale and anneal operations are consistently complementary and achieve lossless performances in both settings, which aligns with our theoretical analysis. Moreover, rescaling obtains better correction the gradient expectation bias on these two tasks than annealing. Using rescaling alone may sometimes achieve lossless results but with higher performance variance and lower mean performance; annealing improves the stability of rescaling. Annealing alone doesn't It improve performance significantly.
Exploring Where to Prune. In default setting, samples with He(z) < He are pruned. Other possible rules are to prune samples with (z) H. or totally random. We compare the performance and cost of these different prune strategies in Tab. 5. Compared to training on the original dataset, the two other strategies also achieve near-lossless performance, while the default pruning condition is better. Hz(x) >> He has a lower pruning ratio because of the skewness of loss distribution (visualized in Appendix E). The ablation results align with the theoretical analysis that 1. rescaling and annealing can contribute to keeping the performance; 2. pruning low-loss samples is better because it risks less of potential scaling limit. More discussion is in Appendix B.
Evaluation of Pruning Ratio. We define as the probability to prune a sample when He(z) < He- In Fig. 3a, we evaluate different pruning ratios for ResNet- 18 (R-18) and ResNet-50 (R-50) on CIFAR- 100. Setting r < 0.5 obtains lossless performance on both architectures, which indicates InfoBatch is relatively robust on hyper-parameter r. Another finding is that further increase r(2 0.6) leads to degraded performances. This can primarily be attributed to the increased variance and reduced steps when too many samples are pruned. Considering the efficiency and performance, we set 0.5 by default. Another benefit is that this would not affect float number precision.
Evaluation of Annealing Ratio. To reduce the remaining gradient expectation bias, we introduced an annealing operation in the last epochs. ó represents a quantile of given training epoch number C. We evaluate from 0.75 to 0.95 and report the performances in Fig. 3b. A larger ó means fewer annealing epochs, leaving more remaining gradient bias, which may result in degraded average performance and increased performance variance. When 8 < 0.875, lossless performance can be achieved with maximized overall cost saving. Limited tuning effort is required to obtain lossless results.
7

---

Published as a conference paper at ICLR 2024
803
Save 40%
Acc. (%%
200 rojan
-1577
110
+
Backa
(a) Acc. vs pruning ratio. (b) Acc. vs annealing ratio. (c) mloU on ADE20K. (d) Overhead and savings.
Figure 3: (a)(b) Evaluation curves of tion and overall cost, InfoBatch achieves lossless performance on ADE20K semantic segmentation. (d) InfoBatch saves the overall cost by 40% on IN-IK. + means costs and means savings.
and d on R-18 and R-50. (c) With 60% itera-
Table 6: architerch robustness results of Info- Table 7: Comparison of accuracy (%) and Batch. 'Full Dataset' denotes training on the original saved cost (%) on CIFAR-10 when trained dataset without pruning.
with R-50 using different optimizers. All the results are obtained from the same hardware.
CIFAR-10
CIFAR- 100
ImageNet- R-18
R-18
R-50
R-18
R-50
R-50
SGD
AdamW
LARS
LAMB
Full Dataset
95.6
95.6
78.2
80.6
70.5
76.4
Full Dataset
95.6
94.3
95.5
95.0
InfoBatch
95.5
95.6
78.2
80.6
70.4
76.5
InfoBatch
95.6
94.4
95.5
95.0
Saved (%)
41.3
41.3
33.8
41.3
26.1
40.9
Saved (%)
41.3
41.2
41.3
41.2
3.5
GENERALIZATION EVALUATION
Cruss-architecture Robustness Evaluation. InfoBatch not correlated to specific model architec- tures, thereby it is model-agnostic framework. Beyond reported ViT-hased results in Tab. 3, we train R-18 and R-50 on CIFAR-10, CIFAR- 100, and ImageNet- K. As illustrated in Tab. 6, we achieve lossless training performances under all settings, which indicates strong generality of InfoBatch. We also report the overall cost saving in Tab. 6. One can observe that InfoBatch can save 26% 440% overall cost on training these datasets.
Cross-optimizer Robustness Evaluation. In the deep-learning area, there are various optimizers (Bottou al., 1991; Kingma & Ba, 2014: You et al., 2017; 2019; Loshchilov & Hutter, 2019) that are broadly adopted. We verify InfoBatch's 'ss robustness across these optimizers. As shown in Tab. 7, InfoBatch achieves lossless performance with four popular optimizers, and the overall cost saving is stable across optimizers. Specifically, we apply InfoBatch to large-batch training optimizers LARS and LAMB. As InfoBatch accelerates the data dimension, InfoBatch further speeds up training by 1.67 times without extra cost performance drop. It shows that InfoBatch has the potential to be combined with acceleration methods in other dimensions.
Cross-task Robustness Evaluation. To verify the generality of InfoBatch on different tasks other than classification, we apply InfoBatch to the seman-
Table 8: Diffusion.
tie segmentation task. We train UperNet on ADE20K and report the mloU
Method
FID
curves in Fig. 3c. Our implementation is based on mmsegmentation (Con- tributors, 2020). InfoBatch uses 60% of original iterations achieve lossless performance (mloU: InfoBatch 41.122 12%, original 40.7%). We further apply InfoBatch in the training of Latent Diffusion (Rombach al., 2021) on FFHQ as demonstrated in Tab. 8 (and generated images in Appendix E). It saves the pretraining cost of Latent Diffusion by ~27, and the image quality is not affected. The result indicates InfoBatch's generality across segmentation and generation tasks.
Original InfoBatch
7.83
7.70
Beyond computer vision tasks, InfoBatch can also be applied to instruction fine-tuning task. We utilize InfoBatch to accelerate LLaMA (Touvron et al., 2023) instruction fine-
Table 9: InfoBatch on instruction tuning with LLaMA-7B.
Method
Time(m)
BBH
DROP
MMLU
Human-Eval
Avg.
DQ+InfoBatch
14.4
31.0
27.5
35.3
11.6
26.3
DQ (2% data)
18.0
32.9
27.6
36.6
8.5
26.3
tuning Alpaca (Taori al., 2023) (originally 52k samples). To further reduce training costs,
8

---

Published as a conference paper at ICLR 2024
utilize dataset quantization (Zhou al., 2023) to select subset of instructions (Ik samples). The results are demonstrated in Tab. 9. It is shown that InfoBatch is able to accelerate language model instruction finetuning tasks and is compatible with static dataset compression methods.
4 RELATED WORKS
In this section, we discuss the related works in static data pruning and dynamic data pruning. These works are most related InfoBatch. Other related works like dataset distillation, and large batch training are discussed in Appendix C.
The motivation for static data pruning methods is to utilize fewer samples while achieving comparable results the original dataset. Almost all the methods are based on predefined or heuristic metrics. These metrics can be roughly divided into the following categories: geometry- boosetty-bbsed (Sener & Savarese, 2018; Agarwal al., 2020), incertainty-basee (Coleman et al., 2019), error-based (Toneva et al., 2018; 88; Paul et al., 2021), Precioso, 2018), gradient-matching (Mirzasoleiman et al., 2020; Killamsetty et al., 2021a), a), bilevel optimization (Killamsetty et al., 2021b) and submodularity-basee methods (Iyer et al., 2021). Con- textual Diversity (CD) (Agarwal et al., 2020), Herding (Welling, 2009), and k-Center remove the redundant samples based on their similarity to the rest of the data. Cal (Margatina et al., 2021) and Deepfool (Ducoffe & Precioso, 2018) select samples based on their difficulties for learning. FL (lyer et al., 2021) and Graph Cut (GC) (lyer et al., 2021) consider the diversity and information simultaneously by maximizing the submodular function. GraNd and EL2N (Paul et al., 2021) propose to estimate sample importance with gradient norm and error-L2-norm. AL (Park et al., 2022) propose to use active learning methods to select a coreset. Moderate (Xia et al., 2023) proposed to use the median of different scores less heuristic metric. Coverage-centric Coreset Selection (Zheng et al., 2023) additionally considers distribution coverage beyond sample importance. The main limitations of these works can be summarized 1). The predefined or heuristic metrics hardly work well across architectures or datasets. 2). The extra cost of these methods not negligible. As illustrated in Tab. 2, the overhead of EL2N is even longer than the training time on ImageNet-IK.
Static Data Pruning.
(Ducoffe &
Dynamic Data Pruning. Dynamic data pruning aims to save the training cost by reducing the number of iterations for training. The pruning process is conducted during training and sample information can be obtained from current training. (Raju et al., 2021) proposes two dynamic pruning methods called UCB and e-greedy. An uncertainty value defined and the estimated moving average is calculated. Once for every pruning period, c-greedy/UCB is used to select given fraction of the samples with the highest scores and then trains on these samples during the period. Under this dynamic pruning setting, it achieves a favorrblee performance compared to static pruning methods on CIFAR-10/100 10/1000KKrzheevss
et al., a;b), while saving the overhead of assessing samples and pruning before training. He et al. (2023) share similar framework, while taking the dynamic uncertainty as score. Mindermann et al. (2022) propose Reducible Holdout Loss Selection which prioritizes samples neither too easy nor too hard. It emphasizes training learnable samples. Two of the works (Raju et al., 2021; He et al., 2023) adopt sorting operation (cost (log.N) per sample on dataset size N) for obtaining samples with the highest scores, which could be an overhead for extra-large datasets (e.g. ImageNet- 1K has 1.28 million pictures) as it is called multiple times.
5 CONCLUSION
We present InfoBatch, a novel framework for lossless training acceleration by unbiased dynamic data pruning. InfoBatch shows its strong robustness on various tasks and datasets, achieving lossless training acceleration on classification, segmentation, vision pertaining, and instruction finetuning. InfoBatch reduces the extra overhead by at least 10 times compared to previous state-of- -ftt----r-- methods, thus being practical for weal-world applications. We provide extensive experiments and theoretical analysis in this paper and hope it can help the following research in this area.
Limitations and Future Work. Removing samples may cause bias in model predictions. is advisable to consider this limitation when applying InfoBatch ethically sensitive datasets. Currently, we haven' 't found obvious evidence of bias. We will report publicly if we find any. 2. The current version of InfoBatch relies multi-epoch training schemes. However, GPT-3 (Brown et al., 2020) and ViT-22B (Dehghani et al., 2023) usually train with limited epochs. InfoBatch may need further adaptation on these tasks. We going explore new strategies for these tasks in the future.

---

Published as a conference paper at ICLR 2024
This work is supported by Alibaba Group through Alibaba Innovative Re- search Program. This work is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD--22 20nD-202100--000 Yang You's s research group being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDaace grant, ARCTIC grant, SMI grant (WBS number: A-8001 104-00-00), Alibaba grant, and Google grant for TPU usage. We thank Yong Liu, Wangbo Zhao, Zelin Zang, and Zirui Zhu for valuable discussions and feedback.
Acknowledgements.
REFERENCES
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In ECCV, pp. 137-153. Springer, 2020.
Léon Bottou al. Stochastic gradient learning in neural networks. Proceedings of Neuro-Niess, 91 (8):12, 1991.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurIPS, 33:1877-1901, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. CVPR, 2022.
George Cazenavette, Tongzhou Wang. Antonio Torralba, Alexei A Efros, and Jun Yan Zhu. General- izing dataset distillation via deep generative prior. In CVPR, pp. 3739-3777 3748, 2023.
Mirza Cilimkovie. Neural networks and back propagation algorithm. Institute of Technology Blanchardstown, Blanchardstown Road North Dublin, 15(1), 2015.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In /CLR, 2019.
MMSegmentation
Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: thub. com oo//oppennmmmmm ab/ wtt.///mnttttaaaimmmmmmmmm.m..aa on, 2020.
Dominik Csiba and Peter Richtárik. Importance sampling for minibatches, 2016. URL https:
//arxiv. org/ aaaaaaa//12/ 02283.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In ICML, pp. 6565-6590, 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprini 230.00022.2...44.....
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVRP, pp. 248-255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale, 2020. URL https:
pestts::/araee //tp///aavvvv///////////.../awav org. abs 22022010000 11929.
Jiawei Du Yidi Jiang, Vincent T. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. CVPR. pp. 3749-3758, 2023.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach. arXiv preprint arXiv:/802.000 880200982, 2018
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint / 2017.
10

---

Published as a conference paper at ICLR 2024
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang. Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In CVPR, 2024.
Ziyao Guo, Kai Wang. George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via dufficullt-alignedd trajectory matching. ICLR, 2024.
Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the 36th annual ACM symposium on Thhoory of computing, pp. 291-300, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty, 2023.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. NeurlPS, 30, 2017.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. information measures with applications in machine learning. Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Submodular combinatorial
Tyler В Johnson and Carlos Guestrin. Training deep models faster with robust, ap- proximate importance sampling. S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett (eds.), NeurlPS, volume 31. Curran Asso- ciates, Inc., 2018. URL https: httttppe//proceeeinge neurips.eeppp... www.rpp..cpppppeeeeeeeeeeeeeeeee wttpppppeee//////eeeeeeeee pdf.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In ICML, pp. 5464-5474, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh lyer. Glister: Generalization based data subset selection for efficient and robust learning. In AAAI, 2021b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL
https: /arxivv///riivaa/aa///a/ea org/ hwtgg//b////// //s// 6980.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1885-1894 JMLR. org, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). a. URL http: wwww cs. toronto. Saltu//kri//iffae html.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. URL http: www. toronto.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Yanqing Liu, Jianyang Gu, Kai Wang. Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. ICCV, 2023a.
Yanqing Liu. Jianyang Gu, Kai Wang, Zheng Zhu, Kaipeng Zhang, Wei Jiang, and Yang You. DREAM+: Efficient dataset distillation by bidirectional representative matching. arXiv preprint arXiv:2310.1500 15052, 2023b.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
11

---

Published as a conference paper at ICLR 2024
Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2016. URL atttps://ariia org/ a 116866888 03983.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. arXiv preprint arXiv:2109.03760 03770774,, 2021.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Dota Team, An empirical model of large- batch training. CoRR, foss/181.066 2018. URL htt.htpp/// wwx.v. wawivoorg/bbas 188 2. 061 62.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training points that learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15630-15649. PMLR, 17-23 Jul 2022. URL mi mwndediann2n222 html.
https: https//ppccceeiieeeeemm ww..mm..rreeee press
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In ICMI.. PMLR, 2020.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. NeurIPS, 34:5186-5198, 2021.
Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for data subset selection. In NeurlPS Workshop, 2022. URL /tttppse/ net/ forum? ? 2ii-AggyyArrrsssss.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In NerulPS, pp. 8024-8035. Curran Associates, Inc., 2019.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early training. 2021. URL https: /atp::///axxaee http///axvvvv////// orgbbbs//////oooo 2atobb/2207.000000
Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning, 2021. URL ht /awxivivvvvvv/////////a//////av/ org/ abs//bb////// /ttbs//111/ 2111.111111 12621.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Ahmad Sajedi. Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstantinos N. Plataniotis. DataDAM: Efficient dataset distillation with attention matching. ICCV, pp. 17097- 17107, 2023.
Maximilian Seitzer. pytorch-fid: fid:: FID Score for PyTorch. pytorch-fid, fid, August 2020. Version 0.3.0.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2017. URL https:tpss.. http///xxxivv///// abs//s//n 07120.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: instruction-foolooiing llama model. https:/ /github..../. com/ commtatsu-llaa htt.ualaaasttafforrreaa stanford_aaaaaaaocoooo 2023.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning, 2018. URL https: //arxiv. www.wiiwv.o..oo.o.oo.....oooo abs / 812. 05159.
12

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention, 2020. URL https: /arxiv.//rivvvvaeaa org/ abs /
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anme Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang. and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, pp. 12196-12205, 2022.
Kai Wang. Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang. and Yang You. Dim: Distilling dataset into generative model. arXiv, 2023.
Max Welling. Herding dynamical weights to learn. In ICMLg, pp. 1121-1128, 2009.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm, 2021.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In ICLR, 2023. URL https:/ wpwpeeeeeeww..eee///ee/.. net forum? (omummidd=00e d=wddddd
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding, 2018. URL https:://///. http://axvvvv...o orq/ abs /10///10...0 10221.
Beining Yang, Kai Wang, Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In NeurlPS, 2023a.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset prun- ing: Reducing training data by examining generalization influence. In ICLR, 2023b. URL https: /epes////eeeeeeeeeeee net
Yang You, Igor Gitman, and Boris Ginsburg, Large batch training of convolutional networks, 2017.
URL https: aariiiiiivv wrg.abs/////oooooo.o....o......e 100. 03888.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert 76 minutes, 2019. URL https: /ttx/aivvvv.////////a.////aaaeva org/ /994. 00962.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
Junhao Zhang, Heng Wang, Chuhui Xue, Rui Yan, Wenqing Zhang, Song Bai, and Mike Zheng Shou. Dataset condensation via generative model. http:://arxivorr//ls//330..... 076998 2023.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018
Tianle Zhang, Yuchen Zhang, Kun Wang, Kai Wang, Beining Yang, Kaipeng Zhang, Wenqi Shao, Ping Liu, Joey Tianyi Zhou, and Yang You. Two trades not baffled: Condensing graph via crafting rational gradient matching, 2024a.
Yuchen Zhang, Tianle Zhang, Kai Wang, Ziyao Guo, Yuxuan Liang, Xavier Bresson, Wei Jin, and Yang You. Navigating complexity: Toward lossless graph condensation via expanding window matching. arXiv preprint arXiv:2402.050/1 2024b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. arXiv. 1(2):3 2021.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. In NeurIPS Workshop, 2022.
13

---

Published as a conference paper at ICLR 2024
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 1(2):3, 2021. 2021.
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling, 2014. URL https://arxis: org/ hwtp///////aae/ abs/1401. 2753..
Haizhong Zheng. Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high prun- ing rates. In ICLR, 2023. URL https: opw.reveere.eeeeeeeeeee net forum? wttpmmmm1dd0wwwww0802...
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pp. 633-641, 2017.
Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization, 2023.
14

---

Published as a conference paper at ICLR 2024
A MORE DETAIL OF EXPERIMENTS
We further demonstrate the details of experiments here. The OncCycle Smith Topin (2017) scheduler (with cosine annealing) has a learning rate curve as Fig. 4
In Tab. 1, the CIFAR-10 experiment be reproduced with different optimizers: LARS use max learning rate 2.3 for the OneCycle scheduler under the batch size of 128, and a maximum learning rate of 5.62 for a batch size of 256. SGD using a maximum learning rate of 0.2 under the batch size of 128 achieves similar result. For CIFAR- 100 experiment with R-18, SGD used with
OneCycle LR Schedule
2.0
learning rate
1.5
max learning rate of 0.03 and batch size of 128 for baseline; due to reduced steps, InfoBatch uses a learning rate of 0.05 in this setting.
1.0
0.5
Tab. is implemented based on Pytorch/cxamples. Lars optimizer and maximum learning rate of 6.4 is used for batch-size 1024 on ImageNet- 1K experiments.
0.0-
20000
40000
60000
Figure 4: The OneCycle learning rate schedule.
In Table. 3/4/5, CIFAR100 R-50 trained with LARS uses batch size 256 and its corresponding maximum learning rate is 5.2.
Tab. 7 uses the following hyperparameters: 150 epoch, batch size 128. SGD and LARS use the aforementioned maximum learning rate; Adam, LAMB use the learning rate of 0.0006, and 0.01, respectively.
A.1 DATASETS
CIFAR-10/100, Two CIFAR datasets consist of 32 32 size colored natural images divided into 10 and 100 categories, respectively. Both use 50,000 images for training and 10,000 images for testing.
ImageNet-1K is the subset of the ImageNet-21k dataset with 1,000 categories. It contains 1,281,167 training images and 50,000 validation images.
ADE20K is popular semantic segmentation benchmark. It contains more than 20,000 images with pixel-level annotations of 150 semantic categories.
FFHQ is a high-quality image dataset of human faces, contains 70,000 images.
A.2 DETAIL OF SEMANTIC SEGMENTATION EXPERIMENT
In the semantic segmentation task, there are two types of pixel-level Cross Entropy loss, one for segmentation and one for classification. In backward propagation, the two types losses are averaged separately at the batch level, then weighted and summed. InfoBatch uses a single score for choosing samples to prune, so we take a different strategy. We use the weighted sum of the two losses the sample level as the corresponding score. Thereby one can obtain the relative order of sample contribution to training.
The implementation is based on mmsegmentation. As the official code uses step number to configure training length, we reduce step number from 80k 48k and set 0.85 to approximate a similar annealing quantile in other experiments.
A.3 TINYIMAGENET EXPERIMENT
The Tiny-ImageNe (Le & Yang, 2015) experiment uses batch size 128, image resize 64, SGD the optimizer, and a learning rate of 0.1 at the start of cosine annealing. InfoBatch also achieves a lossless result on Tiny- ImageNet with ResNet-50. Since most previous works didn' 't report Iny-lmageNet, and Yang et al. (2023b) use a baseline of 51.1 which we cannot reproduce, we report our results here in Tab 10. infoBatch achieves lossless performance saving ratio 41.3%.
15

---

Published as a conference paper at ICLR 2024
A.4 TIMM EXPERIMENTS DETAIL
Timm is package including advanced training techniques of deep learning. State-of-the-art results can be obtained by training neural networks with it. To verify the compatibility of our method and advanced training techniques, we apply InfoBatch together with Timm train ResNet-50 and Swin- Tiny on ImageNet-1K. K. We further contribute to method to score samples with strong augmentations like MixUp/CutMMxx.
Table 10: Tiny-ImageNet Result
Acc. %
Save %
Baseline InfoBatch
63.5
63.4
41.3
For Swwn--Tin training, we use the following hyperparameters: total epoch 350, linear warmup 10 epoch from le-6 20e-4, optimizer AdamW, auto augmentation, weight decay 0.05, label smoothing 0.1, image size 224, drop-path 0.2, cutmix alpha 1.0, random erase with pixel model and probability 0.3, cosine annealing learning rate le-6. We also warmup InfoBatch for epochs, only recording scores without pruning.
MixUp/CutMiix are two augmentation methods that are not compatible with existing scoring methods: the two augmentation methods would mix the labels and content of images so that the direct loss or other metrics don't 't reflect a "per-sample" score but a mixture two sample's 's score. To reconstruct the per-sample score, utilize modified score and estimate the learning progress of sample as follows:
1. In MixUp/CutMMix, with batch-level implementation, images and labels are mixed as Yi-mii = atl, + (1 set for each batch. We take Pi[yy] as the class-wise score where P. is the probability vector predicted by the model for mixed sample
= az, + (1 a a). + j= batchsize - 1 and a is
2. To reconstruct score estimate for each sample, we utilize score; Pi-miz + Pj-miz [ys] and similar for scorej.
The idea is that MixUp/CutMixx creates linear interpolation hetween samples, therefore when samples from two classes are mixed, well-trained classifier should predict the mixed class just the mixed label, and the probability error from the source class can be traced back. When two samples are from the same class, one can distinguish the source of entropy by linear algebra if * 0.5, while is only about 0. % chance to mix two samples from the same class for ImageNet-IK.
A.5 VIT-BASE(MAE) EXPERIMENT DETAIL
We train ViT-Base on ImageNet-IK, based on the implementation of He et al. (2021). We train for 300 epochs with 40 epoch warmup, per GPU batch size 128 with GPU, base linear rate 1.5e-4, mast ratio 0.75, and weight decay 0.05.
A.6 DIFFUSION MODEL EXPERIMENT DETAIL
Our implementation is based on the public GitHub repo of Rombach et al. (2021). We use V100 for this experiment, due to memory limit, use half batch size and thus half learning rate according to McCandlish et al. (2018). On FFHQ. InfoBatch uses = 0.825 and all other settings as default. Public code didn't provide the implementation of FID calculation so refer Seitzer (2020) at link.
A.7 LLAMA INSTRUCTION-FIEETUSS EXPERIMENT DETAIL
Our implementation is based on Killamsetty et al. (2021a); Taori et al. (2023); Zhou et al. (2023).
The training config is the same provided by Zhou al. (2023).
16

---

Published as a conference paper at ICLR 2024
B FURTHER DISCUSSION AND ANALYSIS
B.1 EXPANDED PROOF OF 2.3
For Eqn. proof is:
arg min
E [Te())((2,0)
SEB
2E5.
[1 (P(2)))((22((22)))pp)
arg min lee
f2( Pr(z))((=)d=
J. [(z, ())(z)
arg min
[_(1 Pp(2)))(z)dzz
1 / [(z, ((=))d)
arg min
9eB Ct
For step expansion, one can refer to the Importance Sampling for sampling on a distribution.
B.2 ON A HARD PRUNE
Hard pruming does not provide enough guarantee on gradient expectation. There is contradiction about it. From the gradient expectation perspective, only if the hard- samples formulate the same gradient expectation direction as the full dataset will the gradient expectation after pruning be unbiased. It is formulated as follows:
E [V[(2) = E [V((2]]
ZED
E VC(=)] E.EDYS VL(2)
|D] - -S
zçS
(10)
D VL
c1 [| ESED VL(2)]
[D] |S|
02 E [[(()]
EDD
C1, C2 positive values. Therefore in hard pruning setting, if assuming the gradient expectation direction is unbiased, the pruned part should also be a valid compression of the original dataset. However, this could contradict some adopted assumptions about pre-defined scores used in previous hard pruning methods. For example, lowest-score samples are often supposed to contribute less to training; but if the resulted gradient expectation is unbiased, pruning them will result the same result as only using them for training. In contrast, our proposed soft pruning avoids this bias through introducing randomness.
B.3 FURTHER DISCUSSION ON ANNEALING
Only considering the gradient expectation direction, InfoBatch is unbiased. However, pruning is conducted before each epoch. After pruning, the L(S 0) could still be biased within one epoch (not expectation anymore), even though gradients number is low, expectation itself cannot guarantee an unbiased result. That the reason to introduce the annealing. This introduced 'bias' is not always bad, but annealing gives better stability.
rescaled. Thereby when the remaining epoch
B.4 FURTHER ANALYSIS ABOUT INFOBATCH'S INFLCENCE VARIANCE
We claimed in the main text that InfoBatch can achieve a similar result as training on the original dataset with fewer steps. Explicitly, the assumption is, during the training, after warm-up, the gradient will be relatively stable so that
Σ ZDD
[L(220))]
Σ
[V((2,0))
(11)
FES.
17

---

Published as a conference paper at ICLR 2024
A similar case is large-batch training(Goyya et al., 2017), which scales the learning rate to catch up with the training progress. Besides the gradient expectation, another change is in the gradient variance. We use G = VL to denote the expectation of original gradients.
Cov[Gs,] = El G,, G3 E|Gs, G
1
Σ
-L(-)(()))
G_G.
[D]2
GGT
S
(1-P((=))
(12)
EED
1
G_G² P((=))
|D|2 GGT.
=
S
|
ZED
The diagonal terms of the matrix are the actual variance of Gs,,, which is given by
G
DD
Σ
(13)
S
Px(2)
S
ZED
If Px(2) is uniform across all samples (random prune), then Eq. 13 becomes
[D] S. [ G2)
DD
Var [ar[Gs,]
S 2 Var[Gp].
(14)
The
ratio, which is the noise-to-signal ratio, is unchanged, since both standard deviation and mean is rescaled by [D]/|s,].
When using policy (z)=)) < Hr, assuming generally lower loss samples has smaller gradient, then the rescaled gradients are smaller, and we make the following derivation:
G²
G
Σ [1 - Px(2))
(D] G
ΣG EDD
(1- P-())
(15)
(=)=
= - + Σ
ED (1 - Pi(=))
As we already set P2(2) = 0 for larger GN (high loss samples), therefore when assuming (e)) (G -P([2)]] S [G for rescaling,
Σ G3
D S
S Σ P((z) = (ID) - [
<
(
S. G
(16)
RED (1 - (1))
=ED
where the equity only takes effect at [D] = [S)).
Substituting Eq. 16 back to Eq. 15, we get
G²
[D]
Σ ZED (1-P(())
_(-P(())
< D
R
G +
( - (. [ =
B
(17)
D
S
S
and then substitute back to Eq. 13,
D 2
2 S.
2 |
Var
G
G²
Var(Gn),
(18)
S,
ratio is likely to be decreased in our pruning condition. The analysis above discussed InfoBatch's 's influence on variance and signal-to-noise ratio; it also provides insight into the possible future design of dynamic pruning methods with rescaling, that for rescaled samples, Erescaled G22 P.())) < ED EDD.' [ could be important condition for rescaling.
thereby the standarr-deviaaioo----xxxaaaeddd
B.5 RESCALING LIMIT
Besides the variance discussed above, another key factor to be discussed is the step size. Linear scaling (Goyal et al., 2017) has been proposed in previous research on large batch training, which is observed to have upper limit when scaling batch size up. The curvature of the loss surface and the noise scale are the two factors affecting the quality of the update. As InfoBatch using a rescaled gradient, its influence updates should be considered.
18

---

Published as a conference paper at ICLR 2024
Suppose all samples 2 drawn from distribution p. Denoting the per-sample covariance matrix of original gradient G: = (o) as Σ(0) = p and = Es Taking a second order Taylor expansion at update using SGD optimizer. the loss changes as
=² G Hs, tr(HEs.)
EIL(0 eGs,)] L( =EIG's G]
2º²
(19)
D Stl
D
tr(HE)
L(0)
G
HG
(20)
B
In SGD update, this taking an effect as using learning rate of [2] down, using the original gradient we should have
instead of For the loss to go
[s)
G
E<
(21)
HH + ((()E)
B
and now we need
2|SSS
<< HGT
(22)
HG + ((( D B
Note that in the above section B.4 we have analyzed that the pruned variance is likely to be smaller than exact I [ times of original, which means the diagonal of the covariance matrix, namely the variance vector, is smaller than in equation 20. This could relax this bound when the Hessian is suitable (main weight on diagonal). Still, the curvature drreedereddepeddent G HG term affected, and one should take care when using a large learning rate where the learning rate is already very close to this bound.
C OTHER RELATED WORKS.
Large batch training fully exploits the accelerator's parallelism power. Nonetheless, it cannot reduce the overall cost. With fewer iterations, a large learning rate is needed (Goyal et al., 2017;
Hoffer et al., 2017) keep the update size but make the training unstable. LARS (You al., 2017)
and LAMB (You et al., 2019) stabilize the training process by normalizing layer-wise gradients. Stochastic optimization with importance sampling (Zhao & Zhang, 2014; Csiba Richtárik, 2016: 6;
Johnson & Guestrin, 2018) tries to accelerate the optimization convergence speed by sampling certain samples more frequently. drawback is that the speed-up is sensitive to the model and dataset(Zhao & Zhang, 2014). It cannot directly fit into training scheme without measurement.
Dataset distillation. Our work is also related to dataset distillation (DD), a method that aims to condense the large original datasets into small but informative ones. DD can be mainly divided into the following types: gradient matching methods (Zhao et al., 2021; Liu et al., 2023a; Cui et al., 2023; Yang et al., 2023a; Liu al., 2023b), distribution matching methods (Zhao & Bilen, 2021; Wang et al., 2022; Sajedi et al., 2023), trajectory matching methods (Cazenavette et al., 2022; Du et al., 2023; Guo al., 2024; Zhang al., 2024a;b), and generative prior methods (Zhao & Bilen, 2022; Wang et al., 2023; Cazenavette et al., 2023; Zhang et al., 2023; Gu et al., 2024). However, the main drawbacks of the DD method are its poor scalability and efficiency. DD hard to apply large-scale datasets, such as ImageNet- 1K and ImageNet-22K.
D SPEED OF DIFFERENT OPERATIONS
Table 11: Time cost same hardware of different operations different dataset sizes in ms.
sort
percentile
mean
ImageNet-Ike (1.28M)
123.4
13.0
1.2
ImageNet-22k (14M)
1630.1
232.6
6.4
Theoretically, for an array of length N, getting the mean or certain percentile of it cost O(N) time complexity, and sorting cost ONNlogN). We can from 11 that the ImageNet dataset scale, the
19

---

Published as a conference paper at ICLR 2024
time cost of different operations shows magnitude difference. Using expensive operations for even larger datasets like JFT-300M could incur much more overhead than cheaper operations due to the theoretical log(N) complexity difference and possible degree of parallelism.
E VISUALIZATIONS
E.1 INFOBATCH ON ENTROPY-BASED OBJECTIVE
In Fig. 5, we visualize the equivalent loss distribution after pruning and rescaling. The claim of "equivalent" is formulated as follows: In entropy-basee optimization, the original objective is the maximum likelihood
arg max P))
(23)
And after applying Bayesian Theorem, it becomes
arg max P(D arg max II sED
P(=10).
(24)
Take the negative log, the above equation becomes
arg min
Σ Log EED
(25)
9
which is the entropy-based optimization objective for training. Therefore, multiplying one sample's loss value by a factor is equivalent to duplicating it by the same ratio, from the maximum likelihood perspective.
The loss distribution of CIFAR-10 10 trained with ResNet50 is shown in the first sub-figure of Fig. 5. Soft pruning randomly prunes samples with loss smaller than the mean as in the sec- ond sub-figure in Fig. 5, This leads to insufficient and biased updates, re- sulting in performance degradation. The rescaling operation of InfoBatch shifts the distribution rightwards. In classification tasks, the Cross Cross-Entropy loss is actually the sum of negative ligliikeihoood. Therefore, rescaling a sample's 's loss by 1/(1 - r) times is basically equivalent to duplicating it into
InfoBatch
Original Distribution
Step Soft Praning
Step2: Roscaling
Count
Count
Count
10"
10
10 Loss
10"
10"
10'
Loss Value
Value
Loss
Value
Figure 5: Loss distribution visualizations before and after applying InfoBatch. Best viewed in color. 1/(1 r) copies. By rescaling, InfoBatch approximates loss distribution similar to the original one in the first sub-figure. We show the equivalent distribution
(the red dashed line) and the actual scaled distribution in the third sub-figure of Fig.5.
In Tab 12, we show the result of training ResNet- 50 with full dataset using the same computation as InfoBatch. By reducing the epoch number accordingly, observe performance degrada- tion of training on the full dataset. This indicates InfoBatch has better data efficiency.
Table 12: InfoBatch has higher sample efficiency.
CIFAR-10
CIFAR-100
InfoBatch
95.6
80.6
Full Dataset
95.2
79.4
In Fig. 6, we plot the pruning fraction for each epoch during the training of ResNet-50 on CIFAR-10 with InfoBatch. increases with training adaptively. The corresponding loss distribution shift is visualized in Fig 7. The loss is initially left-screwed and finally right-skewed with a long tail. This is a common tendency Cross-Entroph loss.
20

---

Published as a conference paper at ICLR 2024
CIFAR-10 Ro Pauding Ratie agains Spect
198
94
280
10
(a) Epoch
(b) Epoch 20
(c) Epoch
100-
03
X00
600-
200
2000
400
200-
Figure 6: Pruning fraction of mean threshold during training ResNet-50 on CIFAR-10 with InfoBatch. It in- creases with training adaptively.
(d) Epoch 50
(c) Epoch 100
(1) Epoch 150
Figure 7: CIFAR-10 10 ResNet-50 loss distribution over epochs. has a long tail tendency.
E.2 EFFECT OF DIFFERENT COMPONENTS
In Fig. 8, we visualize the training results ResNet- 50 on CIFAR-I00 under multiple runs with different components, We can see that with only annealing, the performance is slightly improved over the pruning baseline; with only rescaling (controlling the same pruning ratio with percentile). the performance increased with larger variance; with both rescaling and annealing applied, InfoBatch can surpass the full data baseline with stable performance.
Influence of different components
only only rescale anneal roccale
E.3 VISUALIZATION OF DIFFUSION RESULT
Full baseline
operation
In Fig. 9, we visualize some diffusion-model- generated images trained with InfoBatch on the FFHQ dataset. The image quality is as good as trained with uniform random sampling, while the training cost is saved by 27%.
80.0
80.2
30.4
performance
Figure 8: The effect of different components
21

---

Published as a conference paper at ICLR 2024
Figure 9: Latent-Diffusee Diffusion model generated samples trained with InfoBatch on FFHQ. All images are synthesized. We follow the privacy terms FFHQ dataset.
22


Reviews:
[{"review_id": "WX3m8IJD0h", "reviewer": "Reviewer_Nce2", "summary": "This work proposes a dynamic data pruning approach that can obtain lossless performances with less training cost.\nIt achieved the unbiased gradient update by randomly pruning a portion of less informative samples and rescaling the gradient of the remaining samples. The proposed approach consistently obtains lossless training results on various ML tasks.", "questions": "- How to illustrate the gradient trajectory with landscape in Fig 1? Is it an illustration or a real plot on some dataset?\n- Which dataset is used for Table 4? maybe ImageNet?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Clear presentation and easy-to-follow writing.\n- The proposed method is theoretically-supported and, more importantly, very efficient and easy to implement.\n- The evaluation, together with the analysis, is extensive and convincing.", "weaknesses": "The paper conducts a complete study on dynamic data pruning, and the following weakness is relatively minor.\n- Missing recent works: 1) static data pruning [a,b,c], 2) dynamic data pruning [d]\n\n---\n[a] Active learning is a strong baseline for data subset selection. NeurIPS workshop, 2022\n\n[b] Moderate: Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. ICLR, 2023\n\n[c] CCS: Coverage-centric Coreset Selection for High Pruning Rates. ICLR, 2023\n\n[d] Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML, 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cWu01OjXs6", "reviewer": "Reviewer_nCAL", "summary": "This paper presents InfoBatch, a novel data pruning approach which dynamically determines pruning probability over the course of training. InfoBatch soft prunes data with small loss value leading to negligible training cost overhead, and rescales remaining data so as to achieve unbiased gradient expectation. By conducting experiments across a wide range of tasks, the paper demonstrates the effectiveness and robustness of InfoBatch as a state-of-the-art data pruning technique in terms of tradeoff between performance and computational cost.", "questions": "- How many random seeds are used throughout the experiments? \n \n\n[1] Fort et al., “Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel.” 2020.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles a practically-relevant problem supported by a fair amount of experiments conducted across various tasks in the image domain. \n- InfoBatch is simple yet has a distinctive benefit over existing dynamic data pruning approaches: (i) replacing sorting operation with mean-thresholding significantly reduces the overhead cost, and (ii) gradient expectation bias is well addressed supported by theoretical analysis. \n- The proposed method demonstrates superior performance compared to the baselines, and the paper provides a comprehensive review of relevant previous works.\n- Overall, the paper is well-written and easy to follow.", "weaknesses": "- InfoBatch improves over UCB via throwing away the dataset sorting operation. However, in Table 2, the practical overhead cost saving seems negligible compared to the wall clock time and the total node hour. Also, how was the overall saved cost calculated in Tables 6 and 7?   \n- To my understanding, annealing utilizes the whole dataset without pruning for the last 0.125% of total training epochs. This raises several concerns: (i) As the wall clock time of UCB and InfoBatch in Table 2 are both 10.5h, is this value taking the annealing process into account? (ii) Regarding Table 1, all the baselines and InfoBatch are compared under the same dataset pruning ratio. I wonder whether this is a fair comparison when annealing is involved in InfoBatch. (iii) Why did the authors utilize annealing only as a means of stabilizing the optimization, rather than leveraging the full dataset at the very beginning of optimization when we know that the early epochs of training can heavily influence the convergence of the loss landscape [1]?  \n- The authors may need to provide further clarification regarding how annealing contributes to the stabilization of the rescaling process, especially if it does not seem to significantly impact the variance of the results in Table 4.\n- The authors claim that the use of loss values in pruning conditions serves two purposes: (i) it reflects the learning status of samples, and (ii) it theoretically ensures unbiased gradient expectations. However, in Table 5, it is observed that even a random pruning criterion achieves nearly the same performance as the original pruning condition. This result raises questions about the necessity and effectiveness of using loss values as a pruning criterion and may require further discussion or clarification in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRfCHbiEEc", "reviewer": "Reviewer_8u8Q", "summary": "The paper proposes a dynamic data pruning scheme to reduce the cost of stochastic gradient training. It does so by stochastically removing lower loss data points and correspondingly rescaling their gradients to maintain an unbiased overall estimates. In conjunction with training on all data towards the end of training, the scheme is able to reduce training cost by about 20-40% on a range of datasets and architectures, including some large scale ones such as ImageNet and a LLaMA model.\n\nOverall, this is a simple but highly pragmatic and practical approach. The scheme solely relies on quantities that are computed during training anyway, so incurs minimal overhead. Unfortunately, I believe that the comparison with the baselines is not entirely applies-to-apples and there are some minor issues with the write-up, so that all things considered I would lean towards rejecting the paper. Nevertheless, I hope these will be addressed over the course of the rebuttal and am open to increasing my score.\n\nEDIT: the extensive additional results for the rebuttal have addressed my concerns and I would now recommend acceptance.", "questions": "* I would like to see results for Random*, $\\epsilon$-greedy and UCB with annealing and gradient re-scaling (for Random*; for the other two as applicable). As much as possible for Table 1 and ideally Table 2 (ResNet-18 instead of 50 is perfectly fine if that makes it more realistic). My hypothesis here would be that all baselines will match InfoBatch in discriminative performance, which would necessitate the main claims in the paper being updated (again, I don't think this affects the value of the contribution). If all the baselines are already using annealing and rescaling, this point is of course void.\n* Add a full data baseline with reduced epochs as proposed in the weaknesses.\n* Is there anything more to section 2.3 than showing unbiasedness?\n* I would be curious what fraction of the data are soft-pruned throughout training? With the losses being bound below by 0, I would expect the distribution to become quite asymmetric as training proceeds. Could e.g. the median or some percentile be preferable? The median (not sure about arbitrary percentiles) can be computed in linear time, although I don't know if it is possible to update it online as for the mean.\n* Do you have any thoughts on how to set $r$ and $\\delta$ on a new dataset/architecture? The benchmarks in the paper are of course well-studied, but I think it would be a nice addition to the paper to discuss some heuristics for finding close-to-optimal values without needing a full training run (although suboptimal values already present a cost saving of course).\n\nTypos:\n- p1, §3: \"constrainT computation resources\" -> \"constrainED computation resources\"\n- p1, §4: \"cubersome\" -> \"cumbersome\"\n- Section 3.3, §1: \"see appendix\" -> missing specific reference\n- B4. last §: \"bond\" -> \"bound\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The approach is simple but pragmatic, I appreciate the care that is taken to not incur substantial overheads for additional computation such as thresholding by the mean score. This could be a broadly applicable technique for speeding up training, both for researchers and practitioners.\n* The method is described well, I think it would be straight-forward to implement this even without code being provided.\n* There is a broad range of experiments, including some larger scale setting involving ImageNet and language models, emphasizing the potential relevance of the approach.", "weaknesses": "* The comparison with the baselines does not seem entirely apples-to-apples to me due to the \"annealing\" period on the whole dataset. I suspect (intuitively and based on the ablations in Table 4 plus the pruning rule seemingly being irrelevant in Table 5) that ensuring the total length of the optimization trajectory remains comparable to that on the full dataset (by rescaling the gradients) in conjunction with the fine-tuning on all data towards the end of training is the \"secret sauce\" to making a dynamic pruning method perform without a degradation in test accuracy. I'm not familiar with the (Raju et al., 2021) paper, but would expect that at least the annealing period could be incorporated into this method without further issue. At the moment the paper presents its selection rule leading to performance matching that of full-data training as a core contribution, however if this can be achieved relatively easily with other selection techniques as well, I think it becomes more about the re-scaling/tuning as general purpose techniques and the cost comparison between different selection approaches being featured more prominently. I don't think this would worsen the paper at all, although it would change the key takeaways a fair bit and I think it is important that the latter accurately reflect the empirical results.\n* On a related note, I am a little bit concerned that the hyperparameters on e.g. ResNets for CIFAR10 are not tuned for achieving the final test performance as quickly as possible. I think it would be worth adding a baseline that trains on all data with a reduced number of epochs/learning rate decay milestones but increased learning rate corresponding to the computation saved by pruning (so hypothetically for 20% saved computation, train for 80 instead of 100 epochs but with learning rate 1.25 instead of 1 and halve it after 40 instead of 80 epochs). This is to ensure that pruning approaches meaningfully speed up training rather than benefitting from slack in the canonical hyperparameter choices for benchmark problems.   \n* I don't entirely follow what the theoretical analysis is trying to achieve in 2.3. Isn't this just showing that the soft-pruned and rescaled gradient is unbiased? Isn't this completely obvious from having independent Bernoullis multiplied onto the terms of a sum (the total gradient over the dataset) and the expectation of a Bernoulli being its probability (so that if we divide by the probability, we get an expectation of 1 and the sum remains unchanged)?\n* I found the paper to be fairly different to what the title made me expect. \"Info\" and \"lossless\" imply a connection with information theory and lossless compression to me, which is of course not present in the method. I appreciate that this is entirely subjective, but would suggest reconsidering the title of the paper. In particular, I would argue that the \"lossless\" part is a bit misleading since this is not theoretically guaranteed by the method, but merely and empirical observation in the experiments. Of course matching performance to the full dataset can always be achieved by letting $r \\rightarrow 0, \\delta \\rightarrow 1$, but this would remove any cost savings.\n* Similarly, I think the paper overstates its relationship with coreset/data selection methods a little bit. These are generally not for speeding up an initial training run, but subsequent ones e.g. for continual learning or hyperparameter tuning and typically incur an upfront cost. On the contrary, the proposed method speeds up a given training run without producing any artefacts (a coreset) that are useful downstream. So to me this seems more like a curriculum learning paper (although I am not particularly familiar with this branch of the literature, so this is a somewhat speculative statement).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZPomEw18ew", "reviewer": "Reviewer_DemB", "summary": "The authors propose a novel dynamic data pruning, aiming to remove (with a predefined probability) the samples with lower loss score. The algorithm does not require to sort the losses, but it does require to train the model with all samples in the final epochs. The experimental results show a significant speedup in the training procedure, with minimal (or zero) performance drop.", "questions": "- Why there is no discussion regarding to the threshold selection procedure? Different solutions like taking the mean plus a factor of the standard deviation can led to interesting results.\n\n- Did the authors experiment with a pruning probability that depends on the sample score?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**originality**: Although the idea of dynamic pruning is not novel (as clearly stated by the authors), but the solution provided is original enough.\n\n**quality**: The experimental results show the algorithm is able to obtain a clear speedup training, and also remarkable performance results, with almost no accuracy drop.\n\n**clarity**: The idea is simple and easy to implement. The ablation study also provides a solid explanation about how each novel idea affects the overall result.\n\n**significance**: Speeding up training procedures is of huge interest, as it can save a lot of time and energy.", "weaknesses": "**originality**: The idea is somehow similar to other dynamic pruning approaches. It does not provide a different point of view in the matter.\n\n**quality**: In the ablation study, I would like to see if different threshold selections (apart from the mean value) can affect the algorithm. I find it a little bit odd that there is no discussion regarding to this point.\n\n**clarity**: The text is too dense. The figures are too small to be read in a paper. I suggest the authors to increase the figures, while removing the text surrounding them. Some sections, like 2.3, can be summarized to make room for the adjustment.\n\n**significance**: the authors claim their threshold value can be established in constant time, whereas the state-of-the-art methods require a sorting part (the complexity should be $\\mathcal{O}(N \\log N)$. However, I think this improvement is no significant, as the speedup produced is residual compared to the time needed to train the network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)