Progressive Data Dropout: An Embarrassingly Simple Approach to Train Faster
Authors: Shriram M S
Xinyue Hao
Shihao Hou
Yang Lu
Laura Sevilla-Lara
Anurag Arnab
Shreyank N Gowda
Venue: NeurIPS 2025 poster ()
Keywords: Dropout
Regularization
Efficient
Primary area: infrastructure
TL;DR: We make neural network training cheaper and more accurate by progressively dropping parts of the data after each epoch.

Abstract: The success of the machine learning field has reliably depended on training on large datasets. While effective, this trend comes at an extraordinary cost. This is due to two deeply intertwined factors: the size of models and the size of datasets. While promising research efforts focus on reducing the size of models, the other half of the equation remains fairly mysterious. Indeed, it is surprising that the standard approach to training remains to iterate over and over, uniformly sampling the training dataset. In this paper we explore a series of alternative training paradigms that leverage insights from hard-data-mining and dropout, simple enough to implement and use that can become the new training standard. The proposed Progressive Data Dropout reduces the number of effective epochs to as little as 12.4\% of the baseline. This savings actually do not come at any cost for accuracy. Surprisingly, the proposed method improves accuracy by up to 4.82\%. Our approach requires no changes to model architecture or optimizer, and can be applied across standard training pipelines, thus posing an excellent opportunity for wide adoption. Code can be found here: \url{https://github.com/bazyagami/LearningWithRevision}.

Progressive Data Dropout: An Embarrassingly Simple Approach to Train Faster
Shriram M S", Xinyue Hao Shihao Hou', Yang Lu Laura Sevilla-Lara',
Anurag Arnab, Shreyank N Gowda*
Department of Computer Science, University of Manchester
2 School of Informatics, University of Edinburgh
Schooo of Informatics, Xiamen University
School of Computer Science, University of Nottingham
Abstract
The success of the machine learning field has reliably depended on training on large datasets. While effective, this trend comes an extraordinary cost. This is due to two deeply intertwined factors: the size of models and the size of datasets. While promising research efforts focus on reducing the size of models, the other half of the equation remains fairly mysterious. Indeed, it is surprising that the standard approach to training remains to iterate over and over, uniformly sampling explore series of alternative training
the training dataset. In this paper paradigms that leverage insights from hard-data mand-daaa-mining and dropout, simple enough implement and use that can become the new training standard. The proposed Progressive Data Dropout reduces the number of effective epochs to as little 12.4% of the baseline. This savings actually do not come at any cost for accuracy. Surprisingly, the proposed method improves accuracy by up to 4.82%. Our approach requires no changes model architecture or optimizer, and can be applied across standard training pipelines, thus posing an excellent opportunity for wide adoption. Code can be found here: hittps://gite h/t:::/bt.tt..... com/bazyagami/ LearningVithheviisioo
Introduction
Advances in deep learning over the last years have consistently come at the cost of substantial computational resources. Each year, the computational cost of the most notable models multiplies by over 4 times Epoch AIl [2023]. This poses challenges both in terms of real-world applications and sustainability Gowda al. [0233]
In response to these challenges, a significant body of research has emerged focused on developing efficient neural architectures. Models such as MobileNet Howard et al. [2017), Sandler al.) (2018). EfficientNet Tan and Le [2019]. and EfficientFormer [i et al. [002) have demonstrated that carefully designed architectures can achieve competitive performance with significantly reduced parameter counts and computational demands. These architectural innovations have been complemented by techniques such as pruning Han et al. (015), quantization Jacob al. 20188, and knowledge distillation Hinton et al. 2013]. all aimed at reducing the resource footprint of deep neural networks.
While most efficiency efforts focus the model, another underexplored direction is improving how we train. Standard training treats all samples equally across all epochs, but not all data points are equally useful throughout the learning process. Some examples consistently misclassified and carry high learning value, while others are confidently and correctly classified early on and may
* Equal Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Baseline
+4.69% Aco, 166 Epccha
200
Accurecy (9%)
(6.37)
82% Adc, 159 Epochs
Counn 10
66671
62
+4.85% Acc, 172 Epechi
60
0. 60.11)
58
20% Acc, 155 Coochs
56
2000. 33-911
Figure 2: A possible reasoning for the effective- ness of our approach that random dropout main- tains representative coverage and introduces bene- ficial stochasticity. Here, we see an example of the number of times each sample goes through back- propagation. Traditionally, each sample would go through this 'total epochs' number of times (red line).
Effective Epochs
EfficientNet-BO, MobileNNetv,, ResNet-50, EfficienfForment
Figure I: Using the proposed variants of Progres- sive Data Dropout, we can not only outperform the traditional baseline, but we can reduce the number of effective epochs significantly.Thi find- ing is consistent across range of models and datasets.
contribute little to further model improvement Katharopoulos and Fleuret [2018) computation to both types may be wasteful. Human learning strategies offer valuable insights for improving machine learning systems. When studying for examinations, humans rarely allocate equal attention to all material. Instead, they often adopt strategies that prioritize challenging concepts while ensuring comprehensive knowledge through periodic revision Dunlosky al. [2033] Bjork et al. (201T). This selective focus on difficult content, followed by holistic review, allows for efficient allocation of cognitive resources while maintaining broad coverage of the subject matter. A parallel in deep learning is dropout Srivastava et al. [2014). where neurons are randomly deactivated during training to prevent co-adaptation and improve generalization. However, dropout operates the model's is activations rather than the data. In contrast, data dropout Wang et al. [2018), the idea of selectively dropping training samples during learning, remains relatively unexplored.
Allocating equal
In this paper, we introduce Progressive Data Dropout (PDD), a simple yet surprisingly effective family of strategies that progressively drops subsets of training data across epochs. Our methods extremely easy to implement and grounded in both cognitive principles and practical efficiency goals. We explore three variants: difficulty-bar based approach that initially trains only on hard examples; a scalar-bases approach that randomly drops a fixed proportion data each epoch; and a hybrid strategy that drops random samples using the same schedule as the difficulty-basee method.
Beyond the cognitive inspiration, our approach offers significant practical benefits. By initially focusing difficult examples, the model processes substantially fewer data points during early epochs, leading to reduced computational requirements. This selective focus not only accelerates training but also appears guide the model toward more generalizable representations, evidenced by improved performance at test time. Our experimental results demonstrate that models trained with our approach achieve comparable superior accuracy while requiring only 0.124x the number of effective epochs compared to standard training procedures. This efficiency gain is particularly notable for dtready-efficien architectures like MobileNet and EfficientFormer, where further reducing training costs can significantly impact deployment scenarios with constrained resources. We plot some of these results on CIFAR-100 Krizhevsky and Hinton 2009 in Figure
Our work makes the following key contributions:
We propose Progressive Data Dropout, simple and general method to reduce training cost by progressively dropping subsets of data across epochs, inspired by cognitive learning strategics and model regularization techniques.
We design and investigate three dropout variants: (1) difficulty-based schedule that retains only hard examples early in training. (2) a scalar-based random sampling method, and (3) schedule-matched random dropout strategy that surprisingly yields the best accuracy.
Our approach significantly reduces effective epochs by up to 89.5% fewer epochs compared to standard training, whilst still improving or matching accuracy, demonstrating effectiveness

---

on multiple architectures including EfficientNet, MobileNet- V2, ResNet, EfficientFormer and ViT-B-MAE. We also show the potential cost saving in self-supervised learning.
We show that Progressive Data Dropout is compatible with standard supervised training pipelines and can be easily integrated without architectural or optimization changes.
2 Related Work
Several learning paradigms share a similar underlying goal with our proposed method, namely, improving training efficiency and generalization, though they differ sigmificantly in implementation and motivation. Curriculum learning, active learning, and data pruning are all relevant lines of research that intersect with our objectives. While we briefly discuss these in the below to contextualize our contributions, we focus our detailed comparisons on methods that explicitly implement forms of data dropout, given the page constraints.
Model Compression and Training Efficiency. Significant progress has been made in making neural networks more efficient at inference time through model compression techniques. Pruming
Han et al.] [2015]. quantization Jacob et al. [2018]. and knowledge distillation [inntn al.) 2015
reduce the size or precision of model weights, allowing faster inference with minimal accuracy loss. Parallel efforts in efficient architecture design, such as MobileNet Howard et al. [2017. EfficientNet
Tan and Le 2019), and EfficientFormer/ [i et al. (2022). focus on reducing the number of parameters and FLOPs during inference. In contrast, our work targets the training phase rather than inference, aiming to reduce effective epochs and data usage dynamically, independent of the underlying model architecture.
Curriculum Learning and Sample Importance. Curriculum learning Bengio et al. (2099 and hard example mining Shhiivstaaaa et al. (20T6) propose organizing training data to emphasize samples based their difficulty or relevance. These methods aim to present samples in a structured order to accelerate convergence or improve performance. More recent work KKtharopoullss and Fleuret (2018) studies the informativeness of data during training and suggests adaptive data selection to optimize learning. Our work differs that we progressively drop samples rather than reordering or reweighting them. Furthermore, introduce both tifficulty-bsse and random dropping strategies, showing that even unstructured data dropout can lead to improved generalization.
Training on Subsets and Data Pruning. Several recent works investigate training reduced subset of the data to minimize computational overhead. Data Diet Toneva et al. proposes filtering data based on metrics like gradient norm and error L2-norm identify examples that are likely to be forgotten during training. Dataset pruning methods, such as those proposed by Yang et al. Yang et al. (2022). aim to identify a minimal subset of training data that closely preserves the generalization performance of the full dataset, using optimization techniques with theoretical guarantees. Coleman al. Coleman et al. introduce "selection via proxy" (SVP) approach that accelerates data selection by training small proxy models instead of the full-scale target network, enabling faster core-set selection active learning. While related in goal, our method differs in important ways. Many of these approaches require substantial pre-processing or side-training before model training begins, for example, Data Diet requires tracking forgetting events early training Toneva et al.) and SVP uses proxy models trained in advance Coleman et al.) Influence-based selection methods such as Koh and Liang 2 Feldman and Zhang 2020 estimate the effect of removing samples on model predictions, which can be computationally expensive. Score-based heuristics (e.g., confidence scores or gradient magnitudes) also require repeated forward or backward passes over the entire training set Mirzasoleiman et al. [2020). Minderman et al. (2022). J. In contrast, our approach is integrated directly into the training loop, progressively dropping examples over epochs without requiring separate scoring, proxy models, or influence estimation.
Active Learning. Active learning Settles 2209 aims to reduce labeling and training costs by selectively querying the most informative data points from an unlabeled pool. Common strategies include uncertainty sampling Ngguyen et al. [2022] where examples with the lowest prediction confidence are chosen; margin sampling Balcan et al. [2007]. which selects samples near the decision boundary; and muery-by-committer Kee et al. [2018] which leverages disagreement among multiple
models to identify uncertain data. These methods are typically iterative and assume access to large
3

---

unlabeled datasets alongside labeling budget. In contrast, our approach operates in the fully supervised regime and does not require querying new data, rather, we progressively drop portions of the labeled dataset during training to reduce computation and potentially improve generalization.
Data Dropout and Data-Centric Regularization. Dropout Srivastava et al. [2014] is widely- used regularization method that randomly drops activations during training to prevent co-adaptation and overfitting. This concept has inspired architectural variants such as DropConnect Wan al. [2013] and stochastic depth Huang et al. [2016]. While these methods manipulate the network topology, relatively fewer studies have explored analogs at the data level. Most closely related to our method are approaches that drop data samples during training in order to improve efficiency or model generalization. These data dropout techniques differ in motivation and implementation. ranging from data augmentation to importance-based filtering. Some methods focus on data-level perturbations for augmentation purposes. For instance, Generalized Dropout Rahhmani and Atia (2018) applies dropout directly to image pixels, generating additional variations of the same training sample to improve generalization. While effective as a form of regularization, these approaches do not aim to reduce the number of training iterations epochs. Other works emphasize importance-based data pruning. Katharopoulos and Fleuret Katharopoulos and Fleuret (2018) propose a sampling strategy that prioritizes high-gradient or high-loss examples, dropping low-impact samples to accelerate convergence. Wang et al. Waan et al. (0188) introduce a two-round training scheme in which training samples are evaluated after initial round and permanently dropped if they are deemed negatively influence generalization. Similarly, DropSample| Yang et al. [2016] and Greedy DropSample Yang al. [2220] propose sample selection mechanisms that adjust data retention dynamically, particularly in the context of Chinese character classification. Zhong et al. Zhong et al. [2021] propose Dynamic Training Data Dropout (DTDD), which filters noisy samples over several epochs to improve robustness in deep face recognition. In contrast to all these methods, our approach is integrated directly into the
training loop and operates in a progressive manner. Rather than relying on pretraining, expensive influence function estimation (which require second-order gradients), or permanently dropping samples after single evaluation, iteratively modify the training set across epochs using cognitively inspired and stochastic strategies. This allows us to balance efficiency and accuracy without the need for auxiliary models, validation sets, or expensive influence computation.
3 Method
We introduce Progressive Data Dropout, an embarrassingly simple yet effective method to accelerate neural network training by progressively reducing the training across epochs. Inspired by cognitive science and dropout regularization, our approach incrementally drops training samples based on either difficulty randomized criteria, while ensuring full-data coverage in the final epoch. In this section, we describe three variants of our method. A full algorithm explaining the approach can be found in the supplementary material.
3.1 Overview
Let Du [(x,,,,)) be the original training dataset. At each epoch t, we construct subset D. S Dn such that fewer samples are used as training progresses. This reduction can be based either on model confidence (difficulty-basee uses the full dataset Do, mimicking the "revision" phase in human learning where comprehensive review reinforces generalization.
or through random sampling. all variants, the final epoch
3.2 Variant 1: Difficulty-First Training
This cognitively motivated strategy Bjork et al. 2011] focuses on "hard" examples early on and uses all examples at the last epoch as form of 'revision'. After each epoch, evaluate the model's confidence or correctness on each sample:
Let (2,) be the model's 's predicted probability for the correct class of sample x, at epoch t.
A sample is considered "easy" if (x.) 2 Tr, where J E [0, 1] is a fixed threshold.
The dataset for the each epoch is:
D. ( w:) E D pe(xs) < <T) =t= O,..

---

In the final epoch, we reset to Dr- Du-
This approach mimics human study habits: initially focusing on difficult material, then performing complete review. We refer to this method as Difficulty-Basee Progressive Dropout (DBPD).
3.3 Variant 2: Scalar-Based Random Dropout
This variant eschews difficulty estimation and instead uses scalar decay factor (0,1) to progressively reduce the dataset:
The number of samples in epoch is [D,] a
Dj created by randomly sampling (without replacement) from Do-
The full dataset is used in the final epoch: DT Do-
This method, which we call Scalar Random Dropout (SRD), is the simplest to implement and re- quires no per-sample evaluation. Despite its simplicity, it yields surprisingly competitive performance with substantial computational savings.
3.4 Is Difficulty Scoring Necessary? Variant 3: Schedule-Matchen Random Dropout
To test whether explicit difficulty-based scoring is essential for effective data dropout, we design variant that uses the same per-epoch data reduction schedule as Variant samples at random rather than based on confidence.
but drops
Let [D(*) be the dataset size at epoch t under the difficulty- based method.
this variant, we randomly subsample |D(") points from Do at each epoch t, without regard to difficulty.
As the other methods, the final epoch uses the full dataset: Dy = Do.
We refer to this Schedule-Matche Random Dropout (SMRD). While this approach is less practical to implement real-time (as it relies on the difficulty- schedule computed in separate run), it serves valuable ablation to probe the importance of difficulty estimation. Surprisingly, SMRD achieves the highest test accuracy across our experiments, indicating that structured data reduction alone, without any notion of sample difficulty, can yield strong generalization benefits.
To approximate the schedule without explicitly scoring sample difficulty, we also explore mathe- matical approximation of the number of samples dropped per epoch. Results of this approximation are included in the supplementary material.
3.5 Effective Epochs
To provide a hardwarreinndeeendent measure of training efficiency, we introduce the notion of Effective Epochs (EE). While traditional epoch counts assume that every training sample undergoes both forward and backward passes, our method performs forward pass on all samples but selectively applies backpropagation only subset. This design reduces computational cost while retaining compatibility with standard data pipelines.
We define the number of Effective Epochs as:
Effective Epochs Number of Samples through Backpropogation over entire run
Total Number of Training Samples
This quantity reflects how many full passes over the dataset (with backpropagation) were effectively executed. It captures the true training burden independent of batch size, hardware, or epoch length. Throughout the paper, we report effective epochs as our standard efficiency metric quantify the computational savings achieved by dropout strategies.

---

4 Discussion
Progressive Data Dropout (PDD) improves generalization by gradually reducing the training dataset during learning. At each epoch, a portion of data is dropped, randomly or based on difficulty, forcing the model to focus on more informative examples and avoid overfitting. This acts as a form of data- level regularization, analogous to dropout on model activations (Srivastava al. (2014). Surprisingly. random dropout outperforms difficulty-based dropout, suggesting that the benefit comes more from data reduction than selective removal. However, just randomly dropping a percentage of data does not work best (see SRD). However, having a scheduled based random dropout works best. Whilst, we try to mathematically approximate this and test, we not obtain our best results, leaving this to future work. Difficulty-based methods may skew the training distribution or overfit to noisy samples Yang al. [0166] 2020]. while random dropout maintains representative coverage and introduces beneficial stochasticity. PDD builds on insights from dataset pruning and curriculum learning. Prior work shows that large portions of training data can be discarded with minimal or even positive effects on accuracy Toneva et al. Yang et al. [2022]. Unlike methods requiring retraining, proxies, or selection phases Wang et al. [0188. 1. Coleman et al., PDD dynamically reduces data within a single run. Unlike curriculum learning that adds or reweights samples Bengio et al. (2009). SShriastaaa et al. (2016). PDD follows an anti-curriculum, reducing data as training progresses. Its success, especially in the random variant, highlights that training with less but diverse data can improve robustness and generalization.
5 Experimental Analysis
5.1 Datasets
We conduct all classification experiments using three standard image classification benchmarks: CIFAR- 10 Krizhevsky and Hinton (2009), CIFAR- 100 Krizhevsky and Hinton [2009], and Ima- [eng et al. [ The CIFAR-IO and CIFAR- datasets each consist of 60,000 32 x 32 color images, split into 50,000 training and 10,000 test images. CIFAR-10 contains 10 object classes, while CIFAR-100 spans more fine-graine taxonomy of 100 classes. Both datasets widely used to benchmark small and mid-sized neural architectures. We also use the ImageNet (ILSVRC2012)
dataset, which contains over 1.28 million training images and 50,000 validation images across 1, 1,000 object categories. The images are of higher resolution and greater diversity compared to CIFAR datasets, making ImageNet robust benchmark for evaluating both model capacity and training efficiency. In addition to supervised classification, we also evaluate our method in a self-supervised learning setup, using ImageNet as the benchmark dataset. This allows us assess the generality of Progressive Data Dropout beyond supervised learning scenarios.
5.2 Implementation Details
All variants are implemented with minimal overhead on top of standard training loops. To avoid introducing instability, batch normalization statistics and learning rates are kept consistent with standard training procedures. For variants with data reduction, ensure that the data loader is reset at each epoch to reflect the newly selected subset. Importantly, we do not alter model architectures, loss functions, or optimizers. Our method is architecture-annotticc and seamlessly integrates with any supervised training pipeline. We use popular models such as MobileNetv2 (andler et al. (0018) EfficientNet Tan and Le [2019], Efficiientformer/ Li et al. 2022). ResNet et al. 2016] and VIT- B Dosovitskiy et al. [2200] for the image classification tasks. We also consider MAE He et al. [2022) and perform masked pre-training using the Scalar Dropout idea and fine-tune the model using standard fine-tuning. We follow the official implementation details for ImageNet experiments, so for example EfficientNet is run for 350 epochs, while ResNet-50 is run for 100 epochs. However, for all CIFAR- 100 experiments we run all models for 200 epochs and for all CIFAR-10 10 experiments we run them for 30 epochs. For our CIFAR experiments, we use an Adam W optimizer with a StepLR scheduler with a step size of 1. Our initial learning rate is 0.0003. For the self sulf-supe learning experiments with MAE He et al. 20022) we follow standard practice He et al. (2016) and run the pre-training phase for 800 epochs. We use a fixed batch size of 32 for all our experiments. We use 8 A100 GPUs with 40GB memory. For ImageNet fine-tuning we use A A00 GPU with 40GB memory. For CIFARIO and CIFAR 100, we use 4060RTX GPU with 8GB memory.

---

Table 1: Accuracy (%) and effective epochs reported as accuracyleffective epochs. All methods besides ViT-MAE are trained from scratch unless otherwise mentioned. ViT-MAE uses the publicly available checkpoint He et al. [2022] All results have been conducted following official implemen- tations on timm Wightman 2019 Best results in blue and second best in green. The last column shows the gains in accuracy and drop in effective epochs of the SMRD variant.
Baseline
DBPD
SRD
SMRD
Acc. gain (%)
BE saved
CIFARIO
EfficientNet-B
90,41
88.22
92.07
90.55/20
0.14
MobileNet-VV V2
88.46 /
89.55 22.3
90.73 84
90.91 /4.00
2.45
8.33
ResNet-50
87.94 200
88.42 23
89.81 84
89.12 22.1
1.18
9.05
EfficientFormer-Ll
82.38 200
84.47 / 28.2
84.38 84
85.20/31
2.82
6,45 x
CIFARI00
EfficientNet-B0
66.32 200
67.15 24.8
66.75 24.88
71.01 35
4.69
5.71
MobileNet-N V2
61,25 200
62.85 29.6
62.80 /4.8
66.07 /41.4
4.85
4.83
ResNet-50
59.81 200
60.13/ 16.26
60.83/ 22.83/24.8
64.67 28.6
4.86
6.99
EificientFormer-Ll
55.91
57.62/
56.95 24.8
60.11 45
4.20
4.44
CIFARI00 finetuned from ImageNet
Efficien(Ne--B BO
83,31 200
83.95
83.58 24.8
84,45 / 33.8
1.14
5.92 x
MobileNet-V2 V2
74.10 200
74.35
74.22/244
74.73 / 36.2
0.63
5.52
ResNet-50
78.59 200
79.92/21
79.63 / 24.8
81.54 33.2
2.95
6.02
EfficientFormer-Ll
85.30 / 200
86.74/36.6
84.78 / 24.8
87.51 / 51.2
2.21
3.91
ViT-B-MAE
86.81 200
/ 52.2
82,90 24.8
88.10/67. 67.2
1.29
2.98
ImageNet
#fficientNet-BOO
77.10/350 350
71.4//11..2
64.71 20
79.51 / 142.4
2.41
2.46 x
MobileNet-V2
71.60 250
68.42/ 54.5
63.85 20
73.25 / 98.0
1.75
2.55 x
ResNet-50
75.04/ 100
76.21 33.00
71.54/20
79.18 46.4
4.14
2.15 x
ElficienFormme-J/
79.11 /30
79.25/91.5
69.67 20
80,25 106.1
1.14
2.83 x
ViT-B-MAE
83.10/ 100
83.31/35.4
79.51
84.13 41.7
1.03
2.40 x
5.3 Supervised Image Classification
( Sandler ct al. (2018). ResNet-50 et al. (016). EfficientFormer// Li et al. [2022]. and ViT-B-MAE He et al. [2022] across CIFAR- 10, CIFAR 100, and ImageNet. We report results for each model under the baseline training regime as well as the three main variants of our proposed method: DBPD 0.3, SMRD 0.3, and SRD 0.95. For completeness, results for DBPD 0.7 and SMRD 0.7 are provided in the supplementary material. Among all configurations, SMRD 0.3 consistently achieves the highest accuracy across most settings, while the 0.3 threshold variants also offer the greatest reduction in effective epochs. Across datasets and architectures, our approach yields improvements in accuracy of up 4.82% and reduces the effective epochs by much as 12.4%.
In Table I we compare the performance of EfficientNet-B0 Tan and Le 2019). MobileNet-V2
5.4 Self-Supervised Training
Self-supervised learning is among the most computationally expensive training paradigms, often accounting for over 95% the total training cost when including fine-tuning across multiple down- stream tasks. Masked Autoencoders (MAE) et al. (2022) have shown strong promise in learning generalizable representations and improving performance on a wide range of tasks. In this work, focus the VIT-MAE-B model. While the original implementation pre-trains this model for 1600 epochs, we adopt the 800-epoch variant due to memory and compute limitations. Notably, the drop in accuracy for the 800-epoch model on ImageNet is less than 0.3%, making it reasonable compromise.
To apply Progressive Data Dropout in this setting, we use the SRD 0.98 variant, which is the simplest to adapt to a self-supervised objective. Additional results using other variants are included in the supplementary material. Following pretraining, we fine-tune the model using the official MAE procedure on ImageNet and CIFAR-100. and also evaluate it on the COCO object detection benchmark to test cross-tas generalizability.
7

---

Table 2: Comparison of effective epochs and accu- racy for ViT-MAE-B under standard MAE (base- line) and SRD 0.98 configurations across various tasks. Aceuracy reported as Top-1 (%) for clas- sification tasks and mAP (%) for object detection.
Table 3: Baseline trained to match the effective
number of epochs (EE) used by dropout variants.
Variant
EE
Baseline (%)
Accuracy (%)
DBPD 0.3
25
63.32
67.15
DBPI 0.7
33
63.57
68.80
Model
Pretraining EE
ImageNet Accuracy
CIFAR-100
Accuracy
COCO
Ap
SRD 0.95
63.32
66.75
SMRD
63.52
Baseline
83.10%
86.81%
49.9
SMRD 0.7
63.88
70.75
SRD 0.98
49.95
77.89%
83.81%
47.2
Despite its simplicity, SRD reduces the total number of effective epochs from 800 to just 50, 16x reduction in training steps. This reduction especially impactful given the extreme time, memory, and energy requirements of MAE pretraining. Moreover, across all downstream tasks, method remains highly competitive with the baseline, with only negligible loss in performance. We report these results in Table
5.5 Comparison to State-of-the-Art
We conducted comparisons against a suite of existing methods, including DataDiet (ELN and Forget) Paul et al. [202I]. IES Yuan al. [2025]. Early Stopping, and InfoBatch Qin al. We followed the official implementations for all baselines and used EE to evaluate efficiency. Our method outperforms all compared approaches in both accuracy and EE, with higher values indicating better performance for both metrics. The results, reported as Accuracy / EE, are obtained by fine-tuning models on CIFAR- 100 (pretrained on ImageNet) for 200 epochs.
Table 4: Comparison with strong active learning baselines CIFAR-100.
Model
Baseline
DataDiet ELN
DataDiet Forget
IES
Early Stopping
InfoBatch
SMRD (ours)
ResNet-50
78.5/ / 200
78.6/ 141
79.1 142
79.5/ 157
74.3/ 121
80.4/ 145
81.5/ 33
EfficientNet
/ 200
82.7
83.0/
83.7 153
76.6/ 132
84.0/ 141
84.4
ElliientFormer-LI
8 200
84.8 142
85.0/ 140
85.8/ 155
78.2/125
86.6/ 143
8
5.6 Generalization in Language Tasks
We used GPT-2 pre-training with SRD on the FineWeb-Edu dataset (Penedo [2024). The results in the Table show that SRD achieved substantial computational savings, reducing both the number of training steps and wall-clock time by more than half.
/ataset/Penedo
Table 5: NLP task performance of SRD for small GPT-2 pretraining experiments.
Model
PPL val
PPL train
Num__step
Wall clock
GPT-2 (Baseline)
29.37 32.81
28.50 31.78
2, 2,560, 000
13.34 hrs
SRD (Threshold 0.99)
1, 1,066, 720
5.91 hrs
5.7
Ablation Study
To better understand the behavior of Progressive Data Dropout, we conduct an ablation study along three dimensions: dropout threshold, training duration, and matched-step baseline comparisons. For all our experiments, we use EfficientNet-BB trained from scratch on CIFAR-
Are our efficiency gains just due to fewer effective epochs? To ensure improvements not simply due to reduced training time, we train a baseline EfficientNet-BD for the same number of epochs the equivalent effective epoch for each dropout variant. This lets us isolate whether performance improvements come from the dropout schedule itself rather than step count alone. We report these in Table
How sensitive is difficulty- dropout to the choice of threshold? We vary the confidence threshold used for dropping samples in the difficulty-baas based variant (DBPD), ranging from 0. to 0.9

---

in increments of 0.1 and report these results in T Table 7 and Table 8 Higher thresholds lead to more conservative early-stage dropout, while lower values remove more data per epoch. Hence, lower thresholds tend to have much lower step count. To ensure balance between accuracy and epochs, we use thresholds of 0.3 and 0.7 in all experiments. Experiments in Table are conducted using EfficientNet-Bi BO on CIFAR-100 for 200 epochs with the final epoch being full-dataset revision. For SRD, we applied in the most straightforward form of data dropping. Table | presents the results on CIFAR-100 across different thresholds for reference.
Table 6: Varying threshold on EfficientNet-BO (CIFAR-100, 200 epochs). Threshold 0.0 = baseline.
Threshold
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Accuracy (%)
66.32
66.61
66.92
67.15
68.07
67.53
68.80
68.89
69.03
68.93
Table 7: Analysis for different dropout thresholds on CIFAR-100 dataset with training from scratch.
Threshold
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
EfficientFormer 55.91
56.37
56.88
57.52
57.85
98
58.12
58.08
58.17
58.22
MobileNet-v2
61.25
61.49
62.17
62.85
62.98
63.21
63.27
63.31
63.86
63.79
Table 8: Analysis for different dropout thresholds on ImageNet with training from scratch.
Threshold
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
ResNet-55
75.04
75.55
75.80
76.21
76.34
76.42
76.79
77.14
77.20
77.19
Table 9: SRD Threshold sensitivity on CIFAR-100 dataset.
Threshold
0.0
0.2
0.4
0.6
0.8
0.9
0.92
0.94
0.95
0.96
EfficientFormer
55.91
17.84
27.92
38.85
48.83
56.15
56.74
56.91
56.95
56.97
MobileNet-v2
61.25
24.41
35.22
48.63
57.79
62.18
62.59
62.77
62.80
62.82
Can we shorten training and still retain performance? To assess training-time time sensitivity, we repeat all dropout variants under a 50-epoch training schedule (including final revision). We compare them to a 50-epoch baseline. Despite shorter training. all variants outperform the baseline (and the baseline trained to 200 epochs as well). Further, with just 50 epochs we are only 1.75% worse off than our variant with 200 epochs. However, the baseline drops by 2.67%. This shows that the proposed data dropout can improve generalization even under constrained budgets. The results can be seen in Table 10
What happens when we use more epochs for revision? We observe that test performance often peaks shortly after the first revision epoch, followed by gradual decline. As shown in Figure 3 the model experiences significant boost in accuracy during the first revision but this subsequently followed by a steady drop, eventually converging toward the performance of the baseline model. We hypothesize that this pattern arises because the model, after the initial generalization gains, begins to overfit to the full dataset, thereby losing some of its earlier generalization capability.
How many samples we dropping over time? Figure illustrates the number of training samples retained at each epoch under the DBPD (Difficult--asee Progressive Dropout) strategy for two different threshold settings: 0.3 (darker line) and 0.7 (lighter line). In both cases, training begins with the full dataset. As the model improves and classifies more samples with high confidence, the number of retained examples drops sharply, particularly in the early epochs. The higher threshold (0.7) causes a more aggressive pruning of data, dropping larger fraction of confident samples per epoch. As a result, it reaches a low sample count much earlier, within the first 15-20 20 epochs, leaving fewer than 10,000 samples for much of training. In contrast, the 0.3 threshold maintains more samples for a longer duration, leading to a more gradual decay. Both strategies conclude with a final epoch in which the full dataset is reintroduced for revision, producing sharp spike back to 50,000 samples. This U-shaped trajectory reflects the design principle of DBPD: prioritize hard examples early, reduce redundant computation mid- bod-trrining, and consolidate knowledge through a final full-dataset pass.

---

Table 10: Accuracy (%) and effective epochs reported as accuracy/efecctiv epochs for CIFAR-100 with EfficientNet-Bee BO, trained for 25 and 50 epochs from seratch.
Model
Baseline
DBPD 0.3
DBPD 0.7
SRD 0.95
SMRD 0.3
SMRD 0.7
(50 epochs)
EfficientNet-OO (C100)
63.65 5050
67.65 14.48
67.62 / 18.64
68.32 / 24.8
69.26 / 16.76
67.44 / 21.8
(25 epochs)
EfficientNet-Bl (C100)
63.32/
67.22 / 10.8
66.94/ 14.02
17.36 / 17.36
67.17 11.56
66.18 14.62
afcoetree 31.03
100
rotilsne
Number of Samples used
400
Accuracy (96)
40
5STERRS 15:03
22 20
75
150
125
150
1.75
200
1000
2000
Time (seconds)
3000
4000 5000
5000 7000
Figure 4: Here, notice an exponential de- cay in the number of samples being picked at each epoch for backpropagation. The last epoch includes all samples again. The lighter plot denotes threshold 0.7 while the darker one represents 0.3.
Figure 3: We evaluate EfficientNet and Mo- bileNet by adding more than I epoch worth of revision. We see that after an initial boost, there is drop in performance.
Limitations
While Progressive Data Dropout demonstrates significant improvements in training efficiency and generalization, it has certain limitations. The method is most effective when training models from scratch; in fine-tuning or transfer learning scenarios, the relative gains are slightly lower. Whilst there are still some gains, pre-training helps to reduce that. Our current evaluation is limited to classification tasks, with preliminary extensions. However, broader applicability to other domains such segmentation, language modeling, multi-moda learning remains to be investigated. Additionally, we observe that larger datasets generally require more effective epochs and show lower gains, which may limit some of the training savings. Finally, the method introduces hyperparameters such as thresholds or decay rates, that can be sensitive to the dataset and model. While our default settings perform well across multiple benchmarks, optimal tuning may be necessary in specialized contexts. Future work would be coming up with mathematical approximations to avoid the use of difficulty based scheduling and adapting the approach to different modalities and tasks.
7 Conclusion
We introduced Progressive Data Dropout, a simple yet effective approach for improving training efficiency by progressively reducing the amount of data used in each epoch. Through three vari- ants, difficulty- scalar-based, and schedule-matched random dropout, we demonstrated that substantial computational savings can be achieved without sacrificing, and in many cases even im- proving, generalization performance. Notably, the schedule-matchen matched random variant consistently outperformed more structured approaches, highlighting the surprising strength of stochastic data reduction. Our method reduces effective epochs to as little as 12.4% of standard training, with accuracy gains of up to 4. 4.82%. These results suggest that thoughtful data dropout scheduling can serve powerful tool for accelerating training in resource-constratt for architectural changes or complex model tuning. We hope this work inspires further exploration of data-centric regularization strategies.
settings, without the need
10

---

References
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory, pages 35-50. Springer, 2007.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
Elizabeth L Bjork, Robert A Bjork, et al. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society, 2(59-68), 2011.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-awaar margin loss. Advances in neural information processing systems, 32, 2019.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009. doi: 10.1 (009///VR.20099526666
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprini arXiv:2010.11999, 2020.
John Dunlosky, Katherine A Rawson, Elizabeth J Marsh, Mitchell J Nathan, and Daniel T Willing- ham. Improving students' learning with effective learning techniques; Promising directions from cognitive and educational psychology. Psychological Science in the Public interest, 14(1):4-55 2013.
Epoch AI. Key trends and figures in machine learning, 2023. URL httpsttss///::tc.h... ai trends Accessed: 2025-10-19.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Shreyank N Gowda, Xinyue Hao, Gen Li, Shashank Narayana Gowda, Xiaobo Jin, and Laura Sevilla- Lara. Watt for what: Rethinking deep learning's energy Seergy-performacce relationship. arXiv preprint arXiv:2310.06522 2023.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503. 02531, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
2017.
11

---

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECC ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661 Springer, 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetti-onlly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pages 2525 25344. PMLR, 2018.
Scho Kee, Enrique Del Castillo, and George Runger. query-by-commiter improvement with diversity and density in batch active learning. Information Sciences, 454:401-411 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894 PMLR, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL httpttttt///ww...... cs. torooooooeddu. edu/"kriz/
featuree-eeaturr---eaaeeeeeee TR.
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35:12934-12948 2022.
Tsung-Yo Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 2980-2988, 2017.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630-15649, PMLR, 2022.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950-6960. PMLR, 2020.
Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1):89-11) 122, 2022.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances neural information processing systems, 34: 20596-20607, 2021.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale, 2024. URL https:///// caaexivv.or//aee org /20...246...0 17557
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, Yang You, et al. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations.
Mostafa Rahmani and George K Atia. Data dropout in arbitrary basis for deep network regularization. In 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pages 66-70. IEEE, 2018
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiee -Chgh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 45 4510-4520, 2018.
Burr Settles. Active learning literature survey. 2009.
12

---

Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 761-769, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (111):199-1995))) 2014.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058-1066. PMLR, 2013.
Tianyang Wang, Jun Huan, and Bo Li. Data dropout: Optimizing training data for convolutional neural networks. In 2018 IEEE 30th international conference on tools with artificial intelligence (ICTAI), pages 39-46, IEEE, 2018.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-works human annotations. arXiv preprint arXiv:2110.12083 2021.
Ross Wightman. Pytorch image models.
htttps://githabb.com////gg/
pytorch- image-modelk 2019.
Naisen Yang. Hong Tang, Jianwei Yue, Xin Yang, and Zhihua Xu. Accelerating the training process of convolutional neural networks for image classification by dropping training samples out. JEEE Access, 8:
2020.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint Sitiv:2200.099329 2022.
Weixin Yang, Lianwen Jin, Dacheng Tao, Zecheng Xie, and Ziyong Feng. Dropsample: A new training method to enhance deep convolutional neural networks for large-scale unconstrained handwritten chinese character recognition. Pattern Recognition, 58:190-200 203, 2016.
Suqin Yuan, Runqi Lin, Lei Feng. Bo Han, and Tongliang Liu. Instance-dependent early stopping. In The Thirteenth International Conference on Learning Representations, 2025.
Yaoyao Zhong, Weihong Deng, Han Fang, Jiani Hu, Dongyue Zhao, Xian Li, and Dongchao Wen. Dynamic training data dropout for robust deep face recognition. IEEE Transactions on Multimedia, 24:1 18111861119 197, 2021.
Broader Impact
This work proposes simple and general-purpose strategy for improving the training efficiency of deep neural networks through selective data dropout. The proposed methods, difficulty-basek random, and sehedule-matched dropout, reduce the number of training examples that undergo backpropagation while maintaining or improving final model accuracy. This can lead to significant reductions in computational cost and energy consumption during training.
Positive societal impact. By reducing training cost without modifying model architectures or optimizers, our approach lowers the barrier to entry for researchers and practitioners with limited access to high-performance computing resources. This has the potential to democratize access to deep learning research and make machine learning more environmentally sustainable. Furthermore, the method is easy to integrate with existing training pipelines and is applicable to a wide range of model families and datasets, promoting broader adoption of energy-efficient training practices.
13

---

Potential risks. Our experiments are currently focused on image classification tasks using benchmark datasets. In more sensitive imbalanced domains (e.g., healthcare selectively dropping data, even temporarily, could unintentionally bias model learning or overlook rare but important examples. Although our dropout methods reintroduce the full dataset at the end of training, further work is needed to ensure that performance is equitable across subgroups in real-world settings, Additionally, while our method offers efficiency gains, it does not guarantee robustness against adversarial data distributions.
criness-crrtiaal upplications),
Environmental impact. The method designed to reduce the number of effective epochs required to train model to convergence. We report gains of up to 10x reduction in backpropagation steps, which translate real-world energy savings, especially in large-scale training scenarios. This aligns with growing interest in making machine learning more environmentally responsible.
In summary, work offers promising and practical step toward efficient and accessible deep learning. We encourage future work to investigate its impact on underrepresented data groups and its integration with fairness-aware
peeeccypreseriing training pipelines.
A Results for 0.7 variants
In the main paper, report results on the 0.3 variants since that is our best performing variant in most cases along with being more efficient terms of effective epochs. For completeness, we also report results for the 0.7 threshold in Table Again, report results as Accuracy/Effective Epochs.
Table Accuracy (%) and effective epochs for threshold 0.7 variants. refers the model being pre-trained on ImageNet, else the model is trained from scratch.
Model
DBPD 0.7
SMRD 0.7
EfficientNet-BO (C10)
90.01 /9.16
89.87 9.72
EfficientNet-BO (C100)
68.80 / 33
70.75 46.4
"EfficientNet-BO (C100)
84.42 / 41
84.67 51
MobileNet-V2 (C10)
87.44 / 9.08
87.62 10.42
MobileNet-V22(1100 (C100)
63.31 40.8
65.73 / 55.2
"MobileNett-22(C100
74.58 40.6
75.23 47.2
ResNet-50 (C10)
88.20 / 7.12
87.01 /9.8
ResNet-50 (C100)
60.22 / 21
62.62 37.6
*ResNet-50 (C100)
80.51 35.6
81.45 42.8
EfficientFormer-LI (C10)
80.66 / 15.52
80.81 15.62
EfficientFormer-..11 (C100)
58.08 / 39
59.85 62.4
#EfficientFormer-LLI (C100)
86.89 / 59
87.78 76.8
-VI---MMAE (C100)
86.94 / 71.8
87.45 82.2
B
Algorithm
We provide unified training algorithm (Algorithm [ that captures all three variants of Progressive Data Dropout: Difficulty-Bass Progressive Dropout (DBPD), Scalar Random Dropout (SRD), and Schedule-Mattched Random Dropout (SMRD). The shared logic between DBPD and SMRD is highlighted in blue, the SRD-specific logic violet, and SMRDDss additional stochasticity in red.
C Mathematical Approximation for SMRD
Among the three variants of Progressive Data Dropout we explore, the Schedule-Matchin Random Dropout (SMRD) variant achieves the strongest performance across datasets and architectures. SMRD matches the per-epoch data retention schedule of the difficulty-based method (DBPD), but applies dropout stochastically rather than using sample confidence. This yields both generalization benefits
14

---

Algorithm 1: Unified algorithm for all 3 variants, the common steps between DBPD and SMRD are blue, specific steps for SRD is in violet and steps for SMRD is in red
Input: Model fe with parameters 8, Training data Duain test data Diest Number of epochs E, revision start epoch Exev, Confidence threshold r 0 S S 1, Loss function DDcaa
factor: 0 ?
for epoch- = to do
if epoch < Frev then
if SRD then
r + ep
end
foreach batch yX E Dirain do
SRD then
number of samples in X
k < r.]
if k 0 then
continue
end
Randomly sample k indices: M C 11, 1, n
end
if DBPD or SMRD then
y fo (X)
if 0 then
M ii arg max ii + yr)
else
= (51)
M i p: <
end
end
M of 9 then if SMRD then
Randomly sample [M] indices from O,... X
end
0 + optimizer(n, 0, ((f(((M))
end end
else
foreach batch (X,y) E Dimin do
Standard training: 0 0 (VVo(((((X)))))
end
end
end
and practical simplicity, but also introduces a dependency on an external schedule derived from separate DBPD run.
To eliminate this dependency and make SMRD easier to use and evaluate independently. we propose family of simple mathematical functions that approximate the number of training samples used at each epoch. These approximations allow us to:
Generate a complete dropout schedule without running DBPD.
Integrate SMRD seamlessly into standard data loaders.
Enable reproducible and lightweight evaluation of training efficiency.
We approximate the SMRD schedule using a function [a; a), where æ is the epoch number and controls the decay rate. To maintain full-dataset training in the final epoch, we define:
15

---

Table 12: Accuracy (%) and effective epochs reported as accuracyeefecttiv epochs for CIFAR-100 with EfficientNet-BO, epochs from scratch on various scheduler functions.
Model
Accuracy
Effective Epochs
Baseline
63.65
50
Inverse Linear Function
63.51
9.4
Logarithmic Decay
68.62
19.12
SMRD 0.3
69.26
16.76
SMRD 0.7
67.44
21.8
SRD 0.95
68.32
24.8
a(z;)), if < 50,000, x T
f())
Here, is the total number of training epochs (e.g.. T = 200), and f(x; a:) is one of the following parameterized decay functions, each normalized to output maximum of 50,000 (Using CIFAR-100 as our example) samples:
(1) Power-Law Decay:
f(x;a) =
(2) Exponential Decay:
(1; 00
(3) Logarithmic Decay:
f(z;a) = log( x + +a)
(4) Inverse Linear Decay: f(x;a) x + a
(5) Sigmoid-Complenen Complement Decay: f(x;a) = 1 am
Each function captures different decay behavior, and the choice of allows for close approximation of SMRD thresholds (e.g., 0.3 and 0.7) using single parameter. Lower a values lead to gentler decay (slower dropout), while higher values correspond more aggressive early-stage data reduction.
These approximations not only reduce the reliance on a difficulty-basee schedule but also make SMRD deployment fully self-contained and differentiable, facilitating future theoretical analysis and integration into automated training pipelines.
D Evaluating the approximations on CIFAR-10 and CIFAR-100
We evaluate these approximations epochs. Barring the Inverse Linear Function, we notice improvements over the baseline in all cases. However, SMRD 0.3 still gives our best results. We do note the performance of Logarithmic Decay and consider more detailed ablations on this for future work. The results can be seen in Table [2
CIFAR- 100 and train EfficientNet-BB BD from scratch for 50
Do we still have consistent improvements if we are having longer schedule? We conducted additional experiments using longer training runs (200 epochs) on CIFAR-10 with EfficientNet-BOl ResNet-50, and MobileNet-v2 architectures. To ensure fairness, used uniform optimization parameters across all models. Table 13 show that our method achieves consistent improvements under the longer training schedule.
What happens we use strong regularization or augmentation method? Using stronger regu- larization or augmentation reduces the impact of the current threshold settings in our method. As mentioned in the paper, we report results with threshold of 0.3 achieve balance between speed and accuracy. However, if accuracy is prioritized, higher thresholds can be used, demonstrated in the table. These results are based on CIFAR-10 training for 200 epochs, using CutMix style augmentation with the parameter set to I.
16

---

Table 13: Fine-tuning models pretrained on ImageNet (200 epochs).
Model
Baseline
DBPD (0.3)
SRD (0.99)
SMRD (0.3)
EfficientNet-BO
95.55 200
96.11 25
96,40 / 84
96.83 / 31
ResNet-50
96.41 200
96.73 27
96.83 84
97.71 / 33
MobileNet-v2
93.91 200
94.41 / 26
95.12 / 84
95.55 / 33
Table 14: Performance under CutMix augmentation.
Model
Baseline
DBPD(0.3)
DBPD(0.8)
SRD(0.99)
SMRD(0.3)
SMRD(0.8)
MobileNet v2
90.41/200
90.96/22
91.45/32
93 23.29/88 84
91.41/22
93.24/42
How does increasing revision epochs extending total epochs affect? As shown in Figure adding more revision epochs causes overfitting and can degrade test accuracy. To validate this further, tested SMRD with 25, 50, and 75 revision epochs (compared to the standard setting of 1). While these settings brought the accuracy closer to the baseline, they did not yield any improvements and risked overfitting as shown in Table Additionally, we increased the overall number of training epochs to bring the overall computation closer to the baseline. As shown in Table T6. this did not really improve our performance much.
What happens if the dataset has class imbalance or noisy labels? is important to emphasize that PDD fundamentally focuses computational efficiency rather than robustness to class imbalance or noise. Table 17 presents results for Long- Tail Classification on CIFAR-10 10 with & = 100, where denotes the ratio between the sample sizes of the most frequent and least frequent classes. The dataset generated following the same procedure as LDAM-DRW Cao et al.. [2019]. We use Focal Loss Lin al. [217]] consistent with the baseline, ensure a fair comparison. From the table, all PDD variants show substantial gains over the baseline, with SMRD (0.3) emerging as the best-performing method under class imbalance, achieving the highest accuracy. However, compared the standard setting, this configuration requires more EE. We evaluated our method on noisy label classification using the Noisy CIFAR- 10 dataset, following the setup from Wei et al. [202]). with a noise rate of 17.3% and random noise type, training for 50 epochs. Table 18 shows that all three PDD variants outperform the baseline both accuracy and efficiency. Notably, SRD (0.95) achieves the highest accuracy, though with slightly higher EE. SMRD and DBPD provide better trade-off, offering much lower EE while maintaining strong performance, demonstrating their robustness under label noise.
What is the ideal dataset characteristics for PDD? PDD is particularly beneficial for mid-sized to smaller
large datasets, where the computational savings are most significant. We also evaluated fine-grained datasets, such as CUB and Flowers, and observed consistent computational savings without any loss accuracy as shown in Table 19 These results highlight PDD's 's suitability across different dataset scales and complexities.
How do the three variants perform given the same fixed compute budget? We conducted preliminary experiment on CIFAR- 100 using ResNet-50, aligning the EE across all methods. All models were fine-tuned from ImageNet pre- me-training. Table 20 shows that under equal EE, all PDD variants outperform the baseline, with SMRD remaining the best-performing variant.
Table 16: Results of extending total training epochs.
Table 15: Results of SMRD by increasing revision epochs.
Method
Total Epochs
Accurucy (%)
Effective Epochs (EE)
DBPD
200
67.1 07.15
25
Method
Revision Epochs
Accuracy (%)
67.09
DBPD
400
66.
33
SMRD (standard)
71.01
DHPD
500
66 (d)
36
SMRD
68.55
SMRD
SMRD
50
66.35
SMRD)
SMRD
300
100
70.75
70.82
44
49
SMRD
75
66.27
51
Baseline
66.32
Baseline
200
68.32
200
Bascline
300
66
200
Baseline
500
08.35
200
17

---

Table 17: Long-tail classification of ResNet-34
Table 18: Results of noisy label classification on
on CIFAR-10.
Noisy CIFAR-10
Model
Accuracy (%)
Effective Epochs (EE)
Baseline
46.04
200
Model
Accuracy (%)
Effective Epochs (EE)
DBPD (0.3)
58.09
64
Baseline
71.91
50
DEPD (0.2)
$4.82
55
DBPD (0.3)
74.26
DBPD (0.1)
24.28
65
SRD (0.95)
75.23
18
SRD (0.95)
54.28
SMRD (0.3)
73.76
SMRD (0.3)
Table 19; Model performance comparison on smaller fine-grained datasets.
Model
Dataset
SMRD (0.3)
Baseline
EfficientNet-BB
Airplane
80.07 / 9
78.06 100
EfficientNet-BO
CUB
71.03 9
69.83 100
MobileNet-v2
Airplane
76.92 / 10
73.87 100
MobileNet-v2
CUB 68.94 / 10
67.45 100
What is the practical usability of SMRD? To demonstrate the practical usability of SMRD, derived simple mathematical approximations of the dropout schedule that closely replicate the oracle SMRD performance (Appendix Section C.. Among these, the logarithmic decay approximation provides the closest match. Empirical results in Table 21 show that these approximations still clearly outperform the baselines, confirming their practicality. Moreover, SRD and DBPD also outperform the baselines in most scenarios, further validating their effectiveness and applicability.
What is the wall-clock time using PDD? While primary metric, Effective Epochs (EE), enables hardware-independenn comparison, we also provide several wall-clock time savings estimates in Table 22 for reference. These experiments conducted single NVIDIA RTX 4060 GPU with 8 GB of RAM.
E
Number of samples required for SRD
In our setting, This could be formalized as
constant decay y that samples at the same rate from the dataset iteratively.
f(e) gf N
(1)
where e given epoch and is the number of samples the dataset. Since we include all samples as part of our last (revision) epoch E. this would be piecewise function:
f(e) = 7 x N N
if SeSF-1
ife Te=b E
F
Number of samples required for SMRD
For our SMRD setting, the number of samples chosen is equal to the number of samples that are above the threshold for epochs 1 < E where É is the total number of epochs:
f(e) ( D (x:)) 7 Vi 1.....N
We know that the number of epochs progresses, the number of samples gradually reduce. Assuming the model's prediction probabilities pe (z) follows a beta distribution Pe(x) Beta(ae, (aea((a,))))) 3). So, at each epoch the number of training samples would be:
f(e) N (r(p.()) <) N Fe(T; ac,
Table 20: Performance of aligning Effective Epochs (EE) across methods on 100 (ResNet-50)
Model
DBPD
SRD
SMRD
Re
79.92 / 21/ 79.11 / 20.8 8 0.93
80.61 / 20.3 0.23
18

---

Table 21: Practical Usability of SMRD with Logarithmic Decay Approximation.
Model
Variant
Accuracy
Effective Epochs
ResNet-50
SMRD (oracle)
81.54
33
ResNet-50
SMRD (log decay)
81.11
37
EfficientNet-B0
SMRD (oracle)
84.45
34
EfficientNet-BO
SMRD (log decay)
83.92
36
MobileNet-v2
SMRD (oracle) SMRD (log decay)
74.73
36
MobileNet-v2
74.36
38
Table 22: Wall clock time savings estimates.
Model
Baseline
DBPD (0.3)
SRD (0.95)
SMRD (0.3)
EfficientNet-BO CIIFR--100)
15.74 hr
7.89 hr
6.98 hr
9.95 hr
MobileNet-v2 (CIFAR-100)
12.62 hr
7.11 hr
6.67 hr
9.51 hr
where (T aee B) is the cumulative distribution function (CDF) of the Beta distribution.
G MAE pre-training
In the main paper, we report results by pre-training MAE using SRD 0.98 and this effectively resulted in 16 cheaper pre-training cost whilst showing extremely competitive fine-tuning performance. We also consider a variant of the DBPD by setting threshold to the MSE loss of each sample. Specifically, we consider two extremes: a threshold of 0.95 which results in most samples being dropped and threshold of 0.05 which would drop sample if the MSE is less than 0.05, which helps keep more samples in the pre-training loop. We varying amount of revision epochs. For this case, we only perform downstream image classification on ImageNet and only for 50 epochs. Results can be seen Table 23 'No MAE' refers to using a plain ViT-B backbone from scratch (these typically need fine-tuning for over 300 epochs). "DBPD 0.95 ends up stopping pre-training as early as 600 epochs with no samples remaining for threshold. This is expected as an MSE of 0.95 is very high. Setting such high threshold naturally results in very low EE of just 0.69 as most samples are dropped. This also results in poor representations. The result with revision epoch is reflecting that. Matching results of training VIT-B from scratch. Using more revision epochs in this case basically means using MAE for said number of epochs. Using a threshold of 0.05 however is much lower and this results in much higher EE, However, this also gives much higher accuracy, even outperforming the full MAE,
H
Estimation of Effective Epochs (EE) for Comparative Methods
Several prior methods in dataset-level optimization employ data dropout resampling strategies but do not directly report the number of times the network effectively sees the data during training. To allow a meaningful comparison, we estimate the Effective Epochs (EE) defined as the cumulative number data sample passes relative to the full dataset size-for both CIFAR-10 and C00.
H.1 DataDropout on CIFAR-10 and CIFAR-100
The DataDropout method reports removing 1,283 "unfavourable" samples after the first epoch. Assuming 30 training iterations for CIFAR-10 10 and 200 for CIFAR-100, the EE is calculated as follows:
CIFAR-10:
1 50,000 + (30 - 1) (50,000- 1,283)
50,000 + 29 48,717
EECIFAR-10 =
R3 29.26
50,000
50,000
19

---

Table 23: Accuracy (%) for ImageNet fine-tuning. 'PT EE' corresponds to Pre-training Effective Epochs.
Model
PT EE
Accuracy
No MAE
58.61
Full MAE
800
80.26
SRD 0.98
49.95
77.89
DBPD 0.95
Revision 1
1.69
58.05
DBPD 0.95
Revision 200
200.69
70.90
DBPD 0,05
Revision 1
743.02
80.56
DBPD 0.05
Revision 50
792.02
80.65
CIFAR-100:
EECIFAR-100 1. 1,50,00 + (200 1) (50,000 1,283)
50,000 + 199
48,717
195.61
50,000
50,000
H.2 Importance Sampling on CIFAR-10 and CIFAR-100
The Importance Sampling (IS) method dynamically resamples the entire dataset at each iteration. Thus, the Effective Epochs are equal to the total number of training epochs used.
CIFAR-10:
EECHAR-10 30
CIFAR-100:
EECHFAR-100 200
Table 24 summarizes the reported accuracy and estimated Effective Epochs for each method and dataset.
Table 24:
Reported estimated Effective Epochs (EE) for DataDropout and Importance Sampling (IS) on CIFAR-10 and CIFAR-100.
Method
Dataset
EE (Estimated)
Baseline
CIFAR-10
30
DataDropout
CIFAR-10
29.2
IS
CIFAR-10
30
Ours
CIFAR-10
6.98
Baseline
CIFAR-100
200
DataDropout
CIFAR-I00
195.61
IS
CIFAR-100
200
Ours
CIFAR-100
28.6
Directly comparing accuracy is hard as DataDropOut Wang et al. [2018) and IS Katharopoulos and Fleuret (2018) use different backbones, but our estimations show that the effective epochs saved are extremely marginal whilst their reported improvement over the baseline used marginal compared to
I Multiple Runs for testing robustness to random results
To verify the robustness of our results, we check the standard deviation over five trials as reported in table 25 From the results, accuracy of 88.056 08.%. We run EfficientNet-BO on the CIFAR 10 dataset. We notice similar results across different models and datasets showing the robustness of the approach.
notice a minimal standard deviation of 0.2515 with mean
20

---

Table 25; Accuracy (%) and Effective Epochs (EE) for five trials done using ElficientNet-BO with
SMRD 0.3 variant on CIFARIO dataset.
Trial
EE
Accuracy
Trial
6.65
88.44
Trial 2
6.71
87.74
Trial 3
6.69
88.00
Trial 4
6.67
88.02
Trial 5
6.74
88.09
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims about efficiency improvements, simplicity of the method, and accuracy gains supported by the experimental results and clearly scoped.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations section (Sec. laddresses key boundaries: the method works best from scratch, current focus is on classification tasks, and hyperparameter tuning may be needed for datasets and tasks.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
21

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
This paper is entirely empirical in nature and does not include any formal theoretical results proofs. Instead, we focus on demonstrating the effectiveness of our proposed data dropout strategies through extensive experimental evaluation across multiple datasets and architectures.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides full details of all experiments including model architec- tures (EfficientNet-BOO. MobileNet-V2. ResNet-50, etc.), datasets (CIFAR-10, CIFAR- 100, ImageNet), training schedules, dropout thresholds, and effective epoch calcula- tions. The methods are implemented using standard training pipelines from timm, and all hyperparameters and variants (DBPD, SRD, SMRD) are described in the methods and ablations. We also introduce hardware-independee Epochs) and report this consistently across models and datasets. Code is available at https://github. www.coom//a//.e.e howp://aazaaaaaaaaaaaaannniitttr
efficiency metric (Effective
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
22

---

instructions for how to replicate the results, access to a hosted model (e.g., g. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Source code
available at:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.cc/publle wttpcccccppccccceeeeee. http://gguiiee///oeeubiiieeeee wwabbbisssionPa.. ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We use publicly available models and datasets. We follow splits provided by the datasets. The same set of hyperparameters is used for baseline and our variants for fair comparison. We list the CIFAR hyperparameters the implementation details. For ImageNet results, use the hyperparameters of the official papers of these models.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: While the main paper reports point estimates for clarity and space, the sup- plementary material includes average and standard deviation over multiple runs for key experiments on different models and datasets. This provides insight into the stability and statistical robustness of the proposed dropout strategies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the GPUs used for different experiments along with the memory of these GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp///paarrr....coooooo//iaaaa
Answer: [Yes]
Justification: Our research proposes a general-purpose, efficiency-orientedd training strategy that modifies how labeled data is used during training. It does not involve human subjects, personally identifiable data, surveillance, or generation of potentially harmful content. All datasets used (CIFAR-10, CIFAR-100, and ImageNet) are publicly available and widely used benchmarks the machine learning community. We do not use deprecated datasets, nor do we release any new data models requiring licensing access control. Our method aims to reduce computational and environmental cost through training efficiency, aligning with sustainability goals outlined in the NeurIPS Code of Ethics, No known risks harms (societal, environmental, or security-related) are associated with this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the primary positive societal impact of our work in terms of reducing the computational and environmental cost of training deep neural networks. By introducing a simple data dropout strategy that leads to large efficiency gains without sacrificing accuracy, our approach can make model training more accessible researchers and practitioners with limited compute resources. We also highlight that our method is model-agnostic, easy to implement, and compatible with standard training pipelines, increasing its practical utility and democratizing access to efficient ML training.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymennn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
25

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Our work does not involve the release of pretrained language models, gen- erative models, or newly scraped datasets that pose a high risk for misuse. We evaluate widely used public classification datasets (CIFAR-10, CIFAR-100, and ImageNet) and do not propose or release any models intended for sensitive applications or open-ended generation. Our contributions focus training efficiency through data selection strategies and do not introduce any new assets with dual-use concerns.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Used datasets and models are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that are re se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assests are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
26

---

Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing and human experiments conducted.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No research with human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Not used or core component.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpaa://///ii..../ Cct.oooeeeeeeccce/222555. for what should or should not be described.
27


Reviews:
[{"review_id": "CcESaS5Q3S", "reviewer": "Reviewer_bHp2", "summary": "This paper proposes Progressive Data Dropout (PDD), a simple method to accelerate training by progressively dropping parts of the training data across epochs. Three variants are explored: difficulty-based, random scalar-based, and a hybrid that follows the difficulty-based schedule with random sampling. The method shows significant reductions in backpropagation steps (effective epochs), yielding good performance with less compute. It can be seamlessly integrated to any training procedure and is demonstrated across CIFAR, ImageNet, and MAE self-supervised learning.", "questions": "I am currently not in favor of acceptance (as reflected in my “Reject” score), but I would be happy to revise my rating (potentially up to an Accept) if the authors are able to address the following concerns. I understand the rebuttal period is short, so I do not expect full experimental results, but even partial answers or indications that the paper could be strengthened would be very much appreciated.\n\n1. Could you extend experiments to standard-length training runs (e.g., 200 epochs on CIFAR-10) to test if gains persist?\n\n2. How does PDD perform under strong regularization or data augmentation (Mixup, CutMix, etc.)?\n\n3. Would it be possible to make the dropout schedule adaptive based on training signals?\n\n4. Could you test your method on pretraining a small NLP model (eg. a small GPT-2 on Open Web Text).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- S1 : The idea is extremely simple and general and can be seamlessly integrated in standard pipelines.\n- S2 : The experimental setup is sound, with various tasks and architectures tested (although see W1).\n- S3 : The \"effective epochs\" metric is clear and really insightful. Significant computational savings are demonstrated which is a very good point. \n- S4 : The code is open-sourced and reproducibility seems good. Given the fact that the method can be integrated to any training pipeline, this can be very useful for the community.\n\n---\n**Weaknesses**\n\n- W1 : The main weakness I find is that the training hyperparameters used are quite odd. Specifically, the training schedules are very limited for some baselines (e.g., only 30 epochs on CIFAR-10 ResNet). This significantly weakens the impact of reported gains. The simplified training schedules or short runs in some cases, possibly mask practical drawbacks.\nAs standard baselines are often 200+ epochs for ResNets, the reported gains on these benchmarks are difficult to interpret or trust in a high-performance regime.\nThe effectiveness in SOTA training regimes is yet to be tested as it is for now unclear how the method performs compared to highly tuned baselines. \n\n- W2 : No comparisons to strong active learning baselines like Data Diet [1] or coreset selection [2] are provided which makes it hard to evaluate.\n\n- W3 : The evaluation remains narrow in task diversity (mostly image classification). While the method is well-motivated, the strongest gains may be dataset- or setup-specific. As the method is claimed to be seamlessly integrated to any training procedure, adding NLP experiments (eg. pretraining on OpenWebText and finetuning pretrained modelson standard tasks ) would greatly strengthen the paper given the number of tokens used to train SoTA models nowadays (trillions).\n\n\n\n---\nOther minor weaknesses: \n- Some hyperparameters (e.g., dropout thresholds) may be sensitive, but robustness to them is only lightly tested.\n- The overall writing style is very GPT-like (lots of itemization for example in Section 3, many sentences with bits in dash parenthesis). While this may be a matter of personal taste, it occasionally makes the text harder to follow.\n\n\n\n[1] Paul, M., Ganguli, S., & Dziugaite, G. K. (2021). Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34, 20596-20607.\n\n[2] Lee, H., Kim, S., Lee, J., Yoo, J., & Kwak, N. (2024). Coreset selection for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7682-7691).", "was_revised": true, "final_justification": "See the discussion period. \nWhile I originally gave this paper a 2, I think the added experiments and interpretation greatly improved the contribution, which is why I know give it a 5.", "rebuttal": "We sincerely thank **Reviewer bHp2** for their valuable and constructive feedback. Below, we address each comment and question individually. Due to the limited time, we only report results on some models but will add the complete results to the final version.\n\n### **Short training schedules (30 epochs on CIFAR-10 ResNet) limit interpretability of results.**\n\nWe acknowledge that CIFAR-10 had a short schedule. We report results below for a longer schedule. However, CIFAR-100 and ImageNet were run for standard settings and we see consistent improvements across models for these datasets as well. We initially employed shorter schedules to efficiently explore multiple architectures and standardized comparison models. However, in response, we conducted additional experiments using **standard-length training runs (200 epochs)** on CIFAR-10 with EfficientNet, ResNet, and MobileNet v2 architectures.\n\n_Preliminary results_ indicate that **Progressive Data Dropout (PDD)** persists in providing sustained computational savings while **improving accuracy**. We used uniform optimization parameters across models for fairness, although this could have slightly affected overall accuracy.\n\n**Accuracy / Effective Epochs (EE) Table (CIFAR-10 with 200 epochs):**\n\n| Model           | Baseline   | DBPD (0.3) | SRD (0.99) | SMRD (0.3) |\n|-----------------|------------|------------|------------|-------------|\n| EfficientNet    | 90.41/200  | 88.22/18   | **92.07/84** | 90.55/20    |\n| MobileNet v2    | 88.46/200  |  89.55/22         | 90.73/84   | **90.91/24** |\n\n**Fine-tuning models pretrained on ImageNet (200 epochs):**\n\n| Model           | Baseline   | DBPD (0.3) | SRD (0.99) | SMRD (0.3) |\n|-----------------|------------|------------|------------|------------|\n| EfficientNet    | 95.55/200  | 96.11/25   | 96.40/84   | **96.83/31** |\n| ResNet-50       | 96.41/200  | 96.73/27   | 96.83/84   | **97.71/33** |\n| MobileNet v2    | 93.91/200  | 94.41/26   | 95.12/84   | **95.55/33** |\n\n---\n\n### **Missing comparisons to strong active learning baselines (Data Diet, coreset selection).**\n\nWe appreciate this suggestion. While **PDD** is complementary yet distinct from active learning and pruning methods, comparisons are insightful. Based on your suggestion (and Reviewers FSoK and 3Tz5), we conducted comparisons by comparisons against a suite of methods. We compare against DataDiet both ELN and Forget [1], IES [2], Early Stopping and InfoBatch [3]. We follow the official implementations and again use EE to compare efficiency. Our method **outperforms all compared baselines** in both accuracy and EE. These are listed below as Accuracy / EE. Higher the better for both. For all methods, we report fine-tuning a model on CIFAR-100 (pretrained on ImageNet) for 200 epochs.\n\n| Model              | Baseline  | DataDiet ELN [1] | DataDiet Forget [1] | IES  [2] | Early Stopping | InfoBatch [3] | **SMRD (ours)** |\n|--------------------|-----------|--------------|-----------------|-------|----------------|-----------|-----------------|\n| ResNet-50          | 78.5/200   | 78.6/141    | 79.1/142       | 79.5/157 | 74.3/121     | 80.4/145 | **81.5/33**  |\n| EfficientNet       | 83.3/200   | 82.7/139    | 83.0/139       | 83.7/153 | 76.6/132     | 84.0/141 | **84.4/34**  |\n| EfficientFormer-L1 | 85.3/200  | 84.8/142    | 85.0/140       | 85.8/155 | 78.2/125      | 86.6/143 | **87.5/51**  |\n\n---\n\n### **Limited task diversity; NLP tasks (pretraining small GPT-2).**\n\nWhile our focus remains on image tasks for clarity and practicality, we recognize the reviewer's suggestion that NLP would be helpful. We also used PDD for MAE self-supervised pre-training. Following the reviewer's suggestion, we used GPT-2 pre-training with SRD on the **Fine-Web Edu dataset [4]** (comparable in nature yet much larger than OpenWebText). Since the task is different and modifying DBPD and SMRD for this particular task would need more time, we used our most simple approach of SRD. We set a really high threshold due to the amount of data that we would have to go through and could not commit to more experiments due to compute limitations. Limited time prevented us from trying even more thresholds. Despite the simplified implementation, SRD provided substantial computational savings, more than halving the number of steps and wall-clock time:\n\n| Model                | **PPL_val** | **PPL_train** | **Num_steps** | **Wall clock** |\n|----------------------|-------------|---------------|---------------|----------------|\n| GPT-2 (Baseline)     | 29.37       | 28.50         | 2,560,000     | 13.34 hrs      |\n| GPT-2 SRD (0.99)     | 32.81       | 31.78         | **1,066,720** | **5.91 hrs**   |\n\nWe emphasize again our main experiments remain in image classification, and will revise the claims in our paper to reflect this.\n\n---\n\n### **Hyperparameter sensitivity (dropout thresholds).**\n\nWe deliberately minimized hyperparameter tuning for generalizability. Below we report additional results for CIFAR-100 and ImageNet (both from scratch), in addition to the existing results in Table 4 of the main paper:\n\n**Threshold sensitivity on CIFAR-100**\n\n| Threshold        | 0.0  | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | 0.8  | 0.9  |\n|------------------|------|------|------|------|------|------|------|------|------|------|\n| EfficientFormer  | 55.91| 56.37| 56.88| 57.62| 57.85| 57.98| 58.12| 58.08| 58.17| 58.22|\n| MobileNet v2     | 61.25| 61.49| 62.17| 62.85| 62.98| 63.21| 63.27| 63.31| 63.86| 63.79|\n\n---\n\n**Threshold sensitivity on ImageNet (ResNet-50)**\n\n| Threshold | 0.0  | 0.1  | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  | 0.8  | 0.9  |\n|-----------|------|------|------|------|------|------|------|------|------|------|\n| ResNet-50 | 75.04| 75.55| 75.80| 76.21| 76.34| 76.42| 76.79| 77.14| 77.20| 77.19|\n\nFor SRD, we only used it as the most trivial way of data dropping and hence did not conduct an extensive study on this. However, based on the reviewer’s suggestion, we test SRD on CIFAR-100 for varying thresholds and report them below.\n\n**SRD Threshold sensitivity (CIFAR-100):**\n\n| Threshold        | 0.0   | 0.2   | 0.4   | 0.6   | 0.8   | 0.9   | 0.92  | 0.94  | 0.95  | 0.96  |\n|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| EfficientFormer  | 55.91 | 17.84 | 27.92 | 38.85 | 49.83 | 56.15 | 56.74 | 56.91 | 56.95 | 56.97 |\n| MobileNet v2     | 61.25 | 24.41 | 35.22 | 48.63 | 57.79 | 62.18 | 62.59 | 62.77 | 62.80 | 62.82 |\n---\n\n### **Writing style (GPT-like, itemized lists).**\n\nThank you for highlighting this stylistic issue. This was a personal choice, but we will revise the manuscript to reduce excessive itemization and improve readability, ensuring greater clarity throughout.\n\n---\n\n### **Performance under strong regularization or augmentation (Mixup, CutMix)**\n\nThis is an insightful direction. Using stronger regularization or augmentation reduces the effect of the current set thresholds for our methods. However, as mentioned in the paper, we report on 0.3 threshold for a balance of speed and accuracy. However, if the focus is on accuracy, we can obtain that with a higher threshold as shown below. These results are on CIFAR-10 for 200 epochs. We use CutMix style augmentation with $\\alpha$ set to 1.\n\n| Model         | Baseline   | DBPD(0.3) | DBPD(0.8) | SRD(0.99) | SMRD(0.3) | SMRD(0.8) |\n|---------------|------------|-----------|-----------|-----------|-----------|-----------|\n| MobileNet v2 | 90.41/200  | 90.96/22  | 91.45/32  | **93.29/84** | 91.41/26  | 93.24/42  |\n\n---\n\n### **Could dropout schedules become adaptive based on training signals?**\n\nWe acknowledge this as a potential improvement. However, as the core of the paper talks about an _\\\"embarrassingly simple\\\"_ to train faster and adding training signals to make this adaptive increases the complexity of the method. We show results of 2 adaptive ideas below whilst maintaining the idea of being _\\\"embarrassingly simple\\\"_. For 200 epochs, we start with a threshold of 0.9 for the first 20 epochs and progressively reduce the threshold after every 20 epochs. This would be akin to how a learning rate scheduler works with a different decay rate. We also do the opposite, where we start with a threshold of 0.3 and increment this by 0.1 every 60 epochs.\n\nWe tested two simple adaptive schedules, preserving our method's _\\\"embarrassingly simple\\\"_ nature, yielding strong performance:\n\n- **ResNet-50 (CIFAR-100) 0.3 starting threshold with 0.1 increment every 60 epochs**: 79.96 % accuracy, 23 EE.\n- **ResNet-50 (CIFAR-100) 0.9 starting threshold with 0.1 decrease every 20 epochs**: 70.77% accuracy,  15 EE.\n- **ResNet-50 (CIFAR-100) static 0.3 threshold**: 79.92% accuracy,  21 EE.\n\nWe notice similar performances between the first dynamic one and the currently used static one. However, we acknowledge there is potential for improvement in the dynamic one in terms of scheduling.\n\n---\n\nWe sincerely appreciate the insightful feedback, greatly helping us strengthen our manuscript.\n\n[1] Deep learning on a data diet: Finding important examples early in training. Neurips 2021.\n[2] Instance-dependent Early Stopping, ICLR 2025.\n[3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n[4] https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu"}, {"review_id": "rjsST4s1Ei", "reviewer": "Reviewer_bgJ1", "summary": "Progressive Data Dropout is a technique to speed up neural network training by progressively discarding subsets of data across epochs. Three variants are explored in this paper: a difficulty score based dropout, a scalar-based random decay method, and a hybrid approach. The paper demonstrates that PDD significantly reduces \"effective epochs\" while often improving or matching accuracy across architechtures.", "questions": "> Can you provide theoretical reasoning or empirical evidence explaining why random dropout is more effective than selecting easy or hard examples?\n\n> How does PDD affect performance on datasets with significant class imbalance, which is common in real-world scenarios? It would be valuable to evaluate all three variants under such conditions. You could simulate class imbalance using power-law distributions. Additionally, please consider conducting experiments with varying levels of label noise (e.g., 20%, 30%) to test robustness.\n\n> The revision phase in PDD intuitively performs a form of relearning after intentional forgetting via data dropout to improve generalization. Can the authors draw parallels between PDD and prior work on forgetting and relearning (e.g., neuron- or connection-level forgetting vs. data-level forgetting)? It would be helpful to understand how such forgetting dynamics influence generalization. (See below for related papers)\n\n> Empirical results on ImageNet suggest that larger datasets require more effective training epochs and exhibit smaller relative gains. Could the authors elaborate on why efficiency improvements diminish with increasing dataset size (e.g., web-scale datasets used in foundation models)? This is especially important since computational savings are most critical at that scale.\n\n> Following the previous point, what dataset scale or characteristics is PDD best suited for? It is unclear when to use this method (small datasets, large datasets)? It would be insightful to include results on smaller fine-grained datasets such as CUB, Flowers, or Aircraft.\n\n> Are the reported improvements statistically significant? How many random seeds were used for the experiments?\n\n> Could the authors analyze model performance on specific subsets of the data before and after dropout, and prior to the final revision phase? This would help reveal transient \"forgetting\" effects and better characterize the behavior of each method.\n\n> While the paper observes that random data dropout outperforms difficulty-based strategies, it lacks a theoretical or intuitive explanation for this counterintuitive result. Providing such insights would significantly strengthen the contribution.\n\nRelated works:\n> [1] Ramkumar, V.R.T., Arani, E. and Zonooz, B., 2023. Learn, unlearn and relearn: An online learning paradigm for deep neural networks. arXiv preprint arXiv:2303.10455.\n> [2] Ramkumar, V.R., Arani, E. and Zonooz, B., 2024. Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets. Advances in Neural Information Processing Systems, 37, pp.65048-65071.\n> [3] Zhou, H., Vani, A., Larochelle, H. and Courville, A., 2022. Fortuitous forgetting in connectionist networks. arXiv preprint arXiv:2202.00155.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n> The concept of progressively dropping data, particularly the effectiveness of random dropping, is novel and simple.\n> The use of \"Effective Epochs\" as a hardware-independent proxy for backpropagation effort is a valuable contribution.\n> Extensive experiments across architectures and ablation studies.\n\nWeakness:\n> The primary empirical evaluation is limited to image classification tasks. Although the authors explicitly acknowledge that \"broader applicability to other domains such as segmentation, language modeling, or multi-modal learning remains to be investigated,\" this limitation reduces immediate confidence in the method's generalizability across the broader spectrum of deep learning applications. Empirical evidence demonstrating PDD’s effectiveness in non-vision domains, such as NLP, would strengthen the paper.\n> Real-world datasets often exhibit long-tailed class distributions, significant class imbalance, and varying degrees of label noise. However, the paper does not evaluate how PDD performs under these realistic conditions.", "was_revised": true, "final_justification": "The authors have provided a comprehensive rebuttal that effectively addresses the primary concerns raised in my review. The new experiments on class imbalance and label noise are particularly impactful, demonstrating that the Progressive Data Dropout method, even in its random variants, provides significant performance gains in these challenging scenarios.\n\nThis evidence, combined with the authors' insightful explanations of the underlying mechanisms, greatly strengthens the paper's claims about both computational efficiency and practical robustness. The work is a valuable contribution to the community, with a core idea that is both simple and applicable across many domains. However, the experiments are limited to computer vision. The authors' limited demonstration of its effectiveness in the NLP domain during rebuttal further solidifies its broad utility.", "rebuttal": "We sincerely thank **Reviewer bHp2** for their thoughtful and constructive feedback. Below, we respond to each comment in detail. While time constraints limited us to reporting results on a subset of models, we will include the full set of results in the final version of the paper.\n\n### **Empirical evaluation limited to image classification.**\n\nWe agree that our method lacks sufficient diversity. However, we clearly demonstrate its *generalizability* across a variety of architectures and datasets (**CIFAR-10**, **CIFAR-100**, **ImageNet**), as well as self-supervised learning (with **MAE**, which is *extremely* time-consuming). While our focus remains on image tasks for clarity and practicality, we recognize the reviewer's suggestion that increasing diversity would be helpful. Following the Reviewer **bHp2’s** suggestion, we used GPT-2 pre-training with SRD on the **Fine-Web Edu dataset [8]** (comparable in nature yet much larger than OpenWebText). Since the task is different and modifying DBPD and SMRD for this particular task would need more time, we used our most simple approach of SRD. We set a really high threshold due to the amount of data that we would have to go through and could not commit to more experiments due to compute limitations. Limited time prevented us from trying even more thresholds. Despite the simplified implementation, SRD provided substantial computational savings, more than halving the number of steps and wall-clock time:\n\n**GPT-2 Results:**\n\n| Model           | PPL_val | PPL_train | Num_steps | Wall clock |\n|----------------|---------|-----------|-----------|------------|\n| GPT-2 (baseline) | 29.37  | 28.50     | 2,560,000 | 13.34 hr   |\n| SRD (0.99)      | 32.81  | 31.78     | **1,066,720** | **5.91 hr** |\n\n---\n\n### **No evaluation on long-tailed class distribution or label noise.**\n\nWe appreciate the suggestion but would like to emphasize that **PDD fundamentally focuses on computational efficiency**, rather than robustness to class imbalance or noise. Nevertheless, following reviewer suggestions, we performed the following experiments.\n\n**Long-Tail Classification CIFAR-10, ρ = 100 where ρ is used to denote the ratio between sample sizes of the most\nfrequent and least frequent class**  \nDataset generated using the same recipe used by LDAM-DRW [1]. The baseline below corresponds to using Focal Loss [9]. We use Focal Loss [9], for it's ease to work with our current codebase. Results reported as * Accuracy / Effective Epochs (EE)*.\n\n| Model     | DBPD-0.3 | DBPD-0.2 | DBPD-0.1 | Baseline |\n|-----------|----------|----------|----------|----------|\n| ResNet-34 | 58.09/64 | 54.82/55 | 24.28/65 | 46.04/200 |\n\nWe see *improvements of over 12%* while also *reducing the effective epochs (EE) by more than a third*. We are happy to add these results, but note that a full-scope comparison of long-tailed experiments is beyond the scope of this paper.\n\n---\n\n### **Label Noise:**\n\nWe evaluated our method on **noisy label classification** using the **Noisy CIFAR-10** dataset, following the setup from [2]. Again, we use [2] for the ease of implementation with our current codebase.\n\n- **Noise rate:** 17.3%  \n- **Noise type:** Random  \n- **Epochs:** 50\n\n**Results:**\n\n| Model     | Accuracy (%) | Effective Epochs (EE) | \n|-----------|--------------|------------------------|\n| Baseline  | 71.91       |  50               | \n| DBPD 0.3  | **74.26**     | **9**                  |\n\nWe observe that DBPD achieves strong accuracy with significantly fewer effective epochs.\n\n\n---\n\n### **Theoretical reasoning for random dropout effectiveness over difficulty-based methods.**\n\nWe appreciate this insightful question. While it may seem counterintuitive, **random dropout can outperform difficulty-based dropout due to its regularizing properties and improved sample diversity during training**. Difficulty-based dropout, though principled, can prematurely eliminate low-loss (potentially clean or informative) samples, leading to overfitting on harder or noisy examples and reducing generalization.\n\nIn contrast, **random dropout introduces stochasticity into the data stream**, which aligns with principles from *curriculum learning* and *stochastic optimization*, where varied or randomized sample exposure helps models escape narrow local minima and enhances generalization [3, 4, 5]. From a theoretical perspective, this stochastic exposure acts as a form of *implicit regularization*, similar to what has been observed in small-batch SGD training and randomized curricula [6, 7].\n\nEmpirically, we find that random dropout consistently performs robustly across architectures and datasets, validating the intuition that **simpler, less biased sampling often leads to better generalization**, especially when revisited data is used in the final phase of training.\n\n---\n\n### **Parallels between PDD and forgetting–relearning dynamics.**\n\nWe appreciate this connection and agree that **PDD embodies a form of transient data-level forgetting**, analogous to neuron- or connection-level forgetting in prior literature (Ramkumar et al., 2023; Zhou et al., 2022). Our method implicitly leverages such dynamics by **periodically revisiting previously dropped data**, thereby enhancing generalization. We provide theoretical justification to this above.\n\n---\n\n### **Efficiency gains diminishing with increasing dataset size.**\n\nWe acknowledge that **efficiency gains relatively decrease** with larger datasets since the redundancy per epoch reduces. However, *absolute computational savings remain significant and practically meaningful*. We would like to point out two things:\n\n1. The results in Table 1 use thresholds optimized for both **speed and accuracy**. For *accuracy alone*, higher thresholds would give us even bigger improvements. For example, our **ImageNet experiments** show that setting thresholds to 0.8 could give us an *additional accuracy boost of over 2%*.\n\n2. As demonstrated in the **self-supervised training results on MAE** and the **GPT-2 results** above, experiments with massive data and epochs, despite marginal performance drop, compute saved is significant. Even modest percentage improvements at scale translate into *substantial absolute savings*.\n\n---\n\n### **Ideal dataset characteristics for PDD.**\n\n**PDD benefits mid-sized to large datasets**, where computational savings become significant. We also tested on smaller fine-grained datasets, following the reviewer's suggestion like **CUB** and **Flowers**, observing continued computational savings *without accuracy degradation*. This clarifies its *suitability across scales and complexities*. These results will be explicitly included in our revision. Results reported as *Accuracy / EE*.\n\n| Model         | Dataset   | SMRD-0.3 | Baseline |\n|---------------|-----------|----------|----------|\n| EfficientNet  | Airplane  | **80.07 / 9** | 78.06 / 100 |\n| EfficientNet  | CUB       | **71.03 / 9** | 69.83 / 100 |\n| MobileNet v2  | Airplane  | **76.92 / 10** | 73.87 / 100|\n| MobileNet v2  | CUB       | **68.94 / 10** | 67.45 / 100 |\n\n---\n\n### **Statistical significance and number of random seeds.**\n\nWe report one such study in **Section I** of the supplementary material. We saw *similar results across 3 different models on all three classification tasks*. Our conclusion was that **there is no significant statistical anomaly to worry about**.\n\n---\n\nWe appreciate the reviewer’s detailed comments and questions, as addressing these **clarifies misconceptions** and **strengthens our contribution significantly**.\n\n### **References**\n[1] Cao, K., Wei, C., Gaidon, A., Arechiga, N. and Ma, T. (2019) *Learning imbalanced datasets with label-distribution-aware margin loss.* Neurips.\n[2] Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., & Liu, Y. (2022) *Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations*. ICLR.\n[3] Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). *Curriculum learning*. ICML.  \n[4] Hacohen, G., & Weinshall, D. (2019). *On the power of curriculum learning in training deep networks*. ICML.  \n[5] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2017). *On large-batch training for deep learning: Generalization gap and sharp minima*. ICLR.  \n[6] Hardt, M., Recht, B., & Singer, Y. (2016). *Train faster, generalize better: Stability of stochastic gradient descent*. ICML.  \n[7] Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021). *Understanding deep learning (still) requires rethinking generalization*. Communications of the ACM.\n[8] https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu\n[9] Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. CVPR."}, {"review_id": "YhsPXnMMix", "reviewer": "Reviewer_FSoK", "summary": "The authors propose three methods which reduce the number of training examples as the model progresses during training, and find that their data dropout method is able to improve accuracy while significantly reducing the total training cost over standard training. The authors demonstrate results on multiple architectures for CIFAR-10, CIFAR-100, and ImageNet.", "questions": "- I'm not sure I understand the argument in line 209, where data dropout \"[forces] the model to focus on more informative examples and avoid overfitting\". Wouldn't data dropout increase the risk of overfitting, since the model sees fewer examples every epoch?\n- Can you describe the model training procedure and learning rate scheduler? If the number of examples per epoch is different for each method, is the learning rate changing once per epoch or once per X number of training samples? Would that be another difference between SRD and SMRD?\n- How does the total number of available training samples impact the method? In many modern settings, the total number of training samples is extremely large, so the model may not even iterate through multiple epochs. Conversely, if the total number of training samples is very small, then perhaps dropout is unnecessary and could lead to overfitting.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The method proposed by the author is simple, requires minimal overhead, and leads to higher accuracy compared to standard training across many architectures and datasets. \n- The schedule-matched random dropout is interesting and allows us to disentangle the effects of difficulty vs the dropout schedule. This is an interesting ablation which allows us to better understand the effects of data dropout.\n- The methods are clearly described, and the experiments are very reproducible. The code is also included. \n\n**Weaknesses**\n- The impacts of the hyperparameters for the method are not thoroughly explored, even though I assume they have a significant impact on the final results. The authors compare DBPD/SMRD 0.3 and 0.7, which feels somewhat arbitrary to me, and there is no ablation for the alpha in SRD. \n- This method is not compared to any other baselines, including very simple methods like early stopping. Therefore, it is unclear to me if focusing on a subset of the data is beneficial compared to the full dataset, which is the main claim of the paper.\n- Table 1 is difficult to interpret, since the number of effective epochs varies across all of the methods, and the results only consist of final accuracies rather than showing the full training curves. For example, if SMRD is more efficient with 6.7 EE but has lower accuracy than SRD with 15.71 SRD, I can't draw any conclusions about the results because maybe SMRD would outperform SRD if it had 8 EE instead. It would be helpful to have a table of results where the number of effective epochs is held constant (maybe by changing the hyperparameters for each method), so I can better understand which method I should use if I have a specific amount of compute. Or, it would also be helpful to show the training curves (accuracy vs number of train examples) so I can better compare the performance.\n- The method that performs the best, SMRD, is not practically usable since it depends on DBPD. The authors explore different data dropout schedulers in the supplementary material, and I think that this analysis would be key to making this a practical method. Again, a comparison of the training curves (accuracy vs number of train examples) could be really helpful here to understand the differences between schedules.", "was_revised": true, "final_justification": "During the rebuttal period, the authors provided many more baselines and showed that their method was competitive across many settings. They also thoroughly answered my questions regarding the model training procedure.", "rebuttal": "We thank **Reviewer FSoK** for their insightful comments and valuable suggestions. Below, we directly address the reviewer’s points, providing clarifications and empirical evidence to resolve misconceptions and reinforce our position.\n\n---\n\n### **Hyperparameter impacts not thoroughly explored.**\n\nSince we wanted to ensure minimal hyperparameter tweaks for each task, model, or dataset, we only performed the tweaks for one model on one dataset and consistently stuck to those same values. We report the same for CIFAR-100 on two new models as well as on ImageNet for one model showing similar trends to Table 4 in the paper.\n\n**Threshold sensitivity (CIFAR-100):**\n\n| Threshold        | 0.0   | 0.1   | 0.2   | 0.3   | 0.4   | 0.5   | 0.6   | 0.7   | 0.8   | 0.9   |\n|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| EfficientFormer  | 55.91 | 56.37 | 56.88 | 57.62 | 57.85 | 57.98 | 58.12 | 58.08 | 58.17 | 58.22 |\n| MobileNet v2     | 61.25 | 61.49 | 62.17 | 62.85 | 62.98 | 63.21 | 63.27 | 63.31 | 63.86 | 63.79 |\n\n**Threshold sensitivity (ImageNet - ResNet-50):**\n\n| Threshold | 0.0   | 0.1   | 0.2   | 0.3   | 0.4   | 0.5   | 0.6   | 0.7   | 0.8   | 0.9   |\n|-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| ResNet-50 | 75.04 | 75.55 | 75.80 | 76.21 | 76.34 | 76.42 | 76.79 | 77.14 | 77.20 | 77.19 |\n\nFor SRD, we only used it as the most trivial way of data dropping and hence did not conduct an extensive study on this. However, based on the reviewer’s suggestion, we test SRD on CIFAR-100 for varying thresholds and report them below.\n\n**SRD Threshold sensitivity (CIFAR-100):**\n\n| Threshold        | 0.0   | 0.2   | 0.4   | 0.6   | 0.8   | 0.9   | 0.92  | 0.94  | 0.95  | 0.96  |\n|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| EfficientFormer  | 55.91 | 17.84 | 27.92 | 38.85 | 49.83 | 56.15 | 56.74 | 56.91 | 56.95 | 56.97 |\n| MobileNet v2     | 61.25 | 24.41 | 35.22 | 48.63 | 57.79 | 62.18 | 62.59 | 62.77 | 62.80 | 62.82 |\n\n---\n\n### **Lack of comparison to baselines like early stopping.**\n\nWe appreciate this suggestion. Early stopping terminates training prematurely to avoid overfitting but does not strategically alter data utilization. Based on this suggestion and Reviewers **bHp2** and **3Tz5**, we compare it to a suite of methods (active learning, data pruning, early stopping) listed below. We follow the official implementations and again use EE to compare efficiency. Our method **outperforms all compared baselines** in both accuracy and EE. These are listed below as Accuracy / EE. Higher the better for both. For all methods, we report fine-tuning a model on CIFAR-100 (pretrained on ImageNet) for 200 epochs.\n\n| Model              | Baseline  | DataDiet ELN [1] | DataDiet Forget [1] | IES  [2] | Early Stopping | InfoBatch [3] | **SMRD (ours)** |\n|--------------------|-----------|--------------|-----------------|-------|----------------|-----------|-----------------|\n| ResNet-50          | 78.5/200   | 78.6/141    | 79.1/142       | 79.5/157 | 74.3/121     | 80.4/145 | **81.5/33**  |\n| EfficientNet       | 83.3/200   | 82.7/139    | 83.0/139       | 83.7/153 | 76.6/132     | 84.0/141 | **84.4/34**  |\n| EfficientFormer-L1 | 85.3/200  | 84.8/142    | 85.0/140       | 85.8/155 | 78.2/125      | 86.6/143 | **87.5/51**  |\n\n---\n\n### **Interpretation of Table 1 (varying effective epochs).**\n\nWe thank the reviewer for this excellent suggestion. Comparing methods at a fixed compute budget (equal EE) is indeed informative. However, this is a **computationally intensive experiment**, and due to the wide range of additional results included in the rebuttal (e.g., full-length runs, wall-clock time, NLP, and SMRD approximations), we were **unable to complete it in time**.\n\nMatching EE across methods also requires **tuning thresholds** for two methods while keeping one fixed, a **non-trivial process**, as each method responds differently to threshold changes.\n\nThat said, we ran a preliminary experiment aligning EE across CIFAR-100 for ResNet-50. All models are fine-tuned from ImageNet pre-training. Results reported as Accuracy / EE / Threshold value:\n\n| Model     | DBPD | SRD | SMRD | \n|-----------|-------------------|------------------|-------------------|\n| ResNet-50 | 79.92 / 21 / 0.3       | 79.11 / 20.8  / 0.93     | 80.61 / 20.3 / 0.23     | \n\nEven under equal EE, **PDD variants outperform the baseline**. We still see that **SMRD** is the best performing variant. We are extending this study to more models and will include full results in the final version. Additionally, **training curves** (accuracy vs. number of samples) will be added to the supplementary materials for deeper comparison.\n\n\n---\n\n### **Practical usability of SMRD due to dependency on DBPD.**\n\nWe agree that SMRD’s practicality is inherently limited. Although initially informed by DBPD’s schedule, we have derived **simple mathematical approximations** of the dropout schedule which closely replicate the oracle SMRD performance (Appendix Section I). In particular, we consider log decay as our closest approximation.\n**Logarithmic Decay:**  \n$$\n\\hat{f}(x; \\alpha) = \\frac{1}{\\log(x + \\alpha)}\n$$\n\n\nEmpirical experiments demonstrate that our approximations still clearly outperform baselines, validating practical usability. More importantly, **SRD and DBPD still outperform the baselines in most scenarios** and are practical.\n\n| Model          | Variant           | Accuracy | Effective Epochs |\n|----------------|-------------------|----------|------------------|\n| ResNet-50      | SMRD (oracle)     | 81.54    | 33              |\n| ResNet-50      | SMRD (log decay)  | 81.11 | 37           |\n| EfficientNet   | SMRD (oracle)     | 84.45   | 34              |\n| EfficientNet   | SMRD (log decay)  | 83.92 | 36           |\n| MobileNet v2   | SMRD (oracle)     | 74.73   | 36               |\n| MobileNet v2   | SMRD (log decay)  | 74. 36 | 38          |\n---\n\n### **Data dropout potentially increasing risk of overfitting?**\n\nWe appreciate the reviewer’s concern and are happy to clarify. While it may seem that seeing fewer examples per epoch could increase overfitting, **our data dropout method is designed to reduce redundancy, not diversity**. By selectively removing well-learned examples, we encourage the model to focus on underfit or more informative samples.\n\nThis is similar to **pruning unnecessary updates** during training. Importantly, **our final revision phase reintroduces the full dataset**, ensuring no information is permanently lost and mitigating any risk of forgetting.\n\nEmpirically, we observe that **data dropout consistently improves generalization and accuracy** across datasets and models, despite reduced per-epoch exposure. This suggests it serves as a form of **regularization**, injecting stochastic variation and helping prevent overfitting to a fixed subset.\n\nIn short, **data dropout enhances generalization by improving efficiency, without sacrificing full data coverage over training.**\n\n\n---\n\n### **Learning rate scheduler and training procedure.**\n\nWe use a standard **StepLR scheduler** across all experiments, with a decay factor of **0.98 applied every epoch** (step size = 1). The learning rate schedule is synchronized with the number of training epochs, not the number of samples seen. So, it remains consistent across all methods, regardless of how much data is dropped in a given epoch. As for the second part of the question: the difference between **SRD** and **SMRD** is **entirely at the data level**. These techniques determine which samples are included per epoch but do **not interact with the model's parameters, gradients, or learning rate**. So while the number of samples per epoch may vary, the learning rate schedule is unaffected and consistent across variants.\n\n---\n\n### **Impact of total number of training samples.**\n\nWe conducted experiments across a wide range of datasets. For example, **CIFAR-10** and **CIFAR-100** consist of 50,000 training samples, whereas **ImageNet** has 1.2 million. We also tested on a **simulated long-tailed version of CIFAR** as well as **fine-grained datasets** such as *Airplane* and *CUB* based on Reviewer **bgJ1’s** suggestion. The simulated long-tailed CIFAR-10 has 12,440 samples, while Airplane and CUB have 10,000 and 5,994 training samples respectively.\n\nResults reported as *Accuracy / EE*.\n\n| Model         | Dataset   | SMRD-0.3 | Baseline |\n|---------------|-----------|----------|----------|\n| EfficientNet  | Airplane  | **80.07 / 9** | 78.06 / 100 |\n| EfficientNet  | CUB       | **71.03 / 9** | 69.83 / 100 |\n| MobileNet v2  | Airplane  | **76.92 / 10** | 73.87 / 100|\n| MobileNet v2  | CUB       | **68.94 / 10** | 67.45 / 100 |\n\n**Long-Tail Classification, CIFAR-10, ρ = 100**  \nDataset generated using the same recipe used by LDAM-DRW [4] and all results use Focal Loss [5] for ease of implementation to our code base.\n\n| Model     | DBPD-0.3 | DBPD-0.2 | DBPD-0.1 | Baseline  |\n|-----------|----------|----------|----------|-----------|\n| ResNet-34 | **58.09**/64 | 54.82/**55** | 24.28/65 | 46.04/200 |\n\nWe actually see **big improvements in the long-tailed scenario**, but most importantly, see similar improvements on all datasets regardless of size. Thus, **PDD remains beneficial across varying dataset sizes**.\n\n---\n\n[1] Deep learning on a data diet: Finding important examples early in training. Neurips 2021.\n[2] Instance-dependent Early Stopping, ICLR 2025.\n[3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n[4] Learning imbalanced datasets with label-distribution-aware margin loss. Neurips 2019. \n[5] Focal loss for dense object detection. CVPR 2017.\n\nWe deeply appreciate Reviewer **FSoK’s** thoughtful questions and comments, as addressing these points clarifies misconceptions, strengthens our argument, and significantly improves the manuscript."}, {"review_id": "FUWrVo3JrE", "reviewer": "Reviewer_3Tz5", "summary": "This paper introduces \"Progressive Data Dropout\" (PDD), a family of methods aimed at accelerating neural network training by progressively reducing the amount of training data used across epochs. The core idea is to selectively drop data based on difficulty or random chance during intermediate epochs, with a final epoch utilizing the full dataset for \"revision.\" The authors propose three variants: Difficulty-Based Progressive Dropout (DBPD), Scalar Random Dropout (SRD), and Schedule-Matched Random Dropout (SMRD). The paper claims that PDD can significantly reduce the number of \"effective epochs\" required for training (by as much as 87.6%, achieving this with as little as 12.4% of baseline effective epochs) and can sometimes improve model accuracy (reportedly by up to 4.82%) on various image classification tasks and architectures.", "questions": "The SMRD variant, often reported as the best performing, requires knowledge of the data reduction schedule from DBPD. The paper mentions exploring mathematical approximations for this schedule, but the practical performance and robustness of SMRD with such approximations (compared to the oracle SMRD and other PDD variants) are not sufficiently clear.\nCould the authors provide detailed results on how SMRD performs when using a practically derived (approximated) schedule? How does this compare to the oracle SMRD and other PDD variants in terms of both accuracy and effective epochs?\n\nA substantial improvement in my evaluation score would require:\n\nDemonstrating significant benefits of PDD over baselines trained under standard, competitive settings (e.g., standard epoch counts for CIFAR/ImageNet).\nComprehensive discussion and empirical comparison against relevant and recent (post-2022) state-of-the-art data selection/pruning methods.\nDo not over-claim the method’s originality.", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper presents initial results across several datasets and architectures suggesting a reduction in effective epochs while sometimes maintaining or even improving accuracy.\n\n**Weaknesses:**\n\n1.   Originality and Significance Over-claimed\n\nThe related work section appears to conclude its survey around 2022. This is a significant concern as the field of data pruning, curriculum learning, and coreset selection is rapidly evolving. Crucially, the paper fails to compare PDD against more recent (post-2022) and highly relevant dynamic data pruning [1] or sample selection strategies.\nFor example, methods like Instance-dependent Early Stopping [2] (which share exactly same concepts with PDD) and InfoBatch [3] are not discussed or used as comparative baselines. \n\n2. Questionable Settings\n\n- The setup for baseline comparisons. For instance, on CIFAR-10, baseline models are trained for only 30 epochs. This is substantially shorter than standard training regimens for CIFAR-10, which typically yield much higher baseline accuracies (e.g., >97% for ResNet variants). Training baselines for such a short duration likely underestimates their true potential and unfairly inflates the perceived relative improvements of PDD. \n- While the authors define \"Effective Epochs\" (EE), the improvements in wall-clock time, which is a critical practical measure, are not extensively detailed or benchmarked against other speedup techniques. While \"Effective Epochs\" measures backpropagation load, it doesn't fully capture the overhead of the PDD methods themselves (e.g., difficulty scoring in DBPD, which requires a forward pass on potentially more data than is used for backpropagation in early epochs, or managing dynamically changing data subsets).\n\n[1] Accelerating deep learning by focusing on the biggest losers, arXiv preprint 2019.\n\n[2] Instance-dependent Early Stopping, ICLR 2025.\n\n[3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.", "was_revised": true, "final_justification": "The authors' revisions partially address my concerns.", "rebuttal": "### Reviewer 3Tz5\n\nWe thank **Reviewer 3Tz5** for the detailed feedback and valuable suggestions. Below, we directly address each point, clarifying misconceptions and reinforcing our contributions through additional empirical evidence.\n\n---\n\n#### **Originality concerns and omission of recent related works.**\n\nWe respectfully disagree that our work lacks originality or adequate distinction from recent literature. While we acknowledge the rapidly evolving field, our approach fundamentally differs from dynamic pruning methods like **InfoBatch** or **Instance-dependent Early Stopping (IES)**. Unlike methods focused on pruning difficult examples or samples, our **Progressive Data Dropout (PDD)** explicitly targets computational efficiency through progressive reduction of data.\n\nTo address the reviewer's concern, we have empirically compared PDD directly against these recent methods, along with those suggested by Reviewers **bHp2** and **FSoK**, clearly demonstrating our method’s distinctiveness and superior performance. We compare against DataDiet both ELN and Forget [1], IES [2], Early Stopping and InfoBatch [3]. We follow the official implementations and again use EE to compare efficiency. Our method **outperforms all compared baselines** in both accuracy and EE. These are listed below as Accuracy / EE. Higher the better for both. For all methods, we report fine-tuning a model on CIFAR-100 (pretrained on ImageNet) for 200 epochs.\n\n| Model              | Baseline  | DataDiet ELN [1] | DataDiet Forget [1] | IES  [2] | Early Stopping | InfoBatch [3] | **SMRD (ours)** |\n|--------------------|-----------|--------------|-----------------|-------|----------------|-----------|-----------------|\n| ResNet-50          | 78.5/200   | 78.6/141    | 79.1/142       | 79.5/157 | 74.3/121     | 80.4/145 | **81.5/33**  |\n| EfficientNet       | 83.3/200   | 82.7/139    | 83.0/139       | 83.7/153 | 76.6/132     | 84.0/141 | **84.4/34**  |\n| EfficientFormer-L1 | 85.3/200  | 84.8/142    | 85.0/140       | 85.8/155 | 78.2/125      | 86.6/143 | **87.5/51**  |\n---\n\n#### **Questionable baseline settings (short training schedules).**\n\nWe acknowledge this limitation on CIFAR-10. **We ran our experiments on CIFAR-100 and ImageNet experiments for the extended schedule**. We initially employed shorter schedules to efficiently explore multiple architectures and standardized comparison models. However, in response, we conducted additional experiments using **standard-length training runs (200 epochs)** on CIFAR-10 with EfficientNet, ResNet, and MobileNet v2 architectures.\n\n_Preliminary results_ indicate that **Progressive Data Dropout (PDD)** persists in providing sustained computational savings while **improving accuracy**. We used uniform optimization parameters across models for fairness, although this could have slightly affected overall accuracy.\n\n**Accuracy / Effective Epoch Table (CIFAR-10):**\n\n\n| Model           | Baseline   | DBPD (0.3) | SRD (0.99) | SMRD (0.3) |\n|-----------------|------------|------------|------------|-------------|\n| EfficientNet    | 90.41/200  | 88.22/18   | **92.07/84** | 90.55/20    |\n| MobileNet v2    | 88.46/200  |  89.55/22         | 90.73/84   | **90.91/24** |\n\n**Fine-tuning pretrained models (200 epochs):**\n\n| Model           | Baseline   | DBPD (0.3) | SRD (0.99) | SMRD (0.3) |\n|-----------------|------------|------------|------------|------------|\n| EfficientNet    | 95.55/200  | 96.11/25   | 96.40/84   | **96.83/31** |\n| ResNet-50       | 96.41/200  | 96.73/27   | 96.83/84   | **97.71/33** |\n| MobileNet v2    | 93.91/200  | 94.41/26   | 95.12/84   | **95.55/33** |\n\n\n---\n\n#### **Lack of detailed wall-clock time evaluation.**\n\nWhile our primary metric (Effective Epochs, EE) provides hardware-independent comparability, we list a few wall clock savings estimates below. These experiments were run on a NVIDIA 4060 RTX GPU with 8GB RAM.\n\n**Wall Clock Time:**\n\n| Model                | Baseline  | DBPD 0.3 | SRD (0.95) | SMRD 0.3  |\n|---------------------|-----------|----------|------------------|-----------|\n| EfficientNet (CIFAR100) | 15.74 hr | 7.89 hr | 6.98 hr        | 9.95 hr  |\n| MobileNet (CIFAR100)   | 12.62 hr | 7.11 hr | 6.67 hr        | 9.51 hr  |\n\nDespite minimal optimization efforts in our current implementation, these clear reductions confirm practical efficiency. Further code optimization could lead to even greater wall-clock time savings. We also note that in the **GPT-2** experiment (below), **SRD reduced wall-clock time by more than half** while maintaining competitive performance. The GPT-2 experiment was run on 2 x NVIDIA A-100 GPU with 80 GB RAM.\n\n| Model                | **PPL_val** | **PPL_train** | **Num_steps** | **Wall clock** |\n|----------------------|-------------|---------------|---------------|----------------|\n| GPT-2 (Baseline)     | 29.37       | 28.50         | 2,560,000     | 13.34 hrs      |\n| GPT-2 SRD (0.99)     | 32.81       | 31.78         | **1,066,720** | **5.91 hrs**   |\n\nWe will add wall-clock times to the paper, but want to point out that **EE is a hardware-independent** way of reporting efficiency. As we can see between these two experiments using different GPUs results in *wildly different wall clock times*.\n---\n\n#### **Q: Practical performance and robustness of SMRD with approximated schedules**\n\nWe thank the reviewer for raising this important concern regarding the practicality of **SMRD** when using an *approximated* data reduction schedule rather than the oracle version derived from DBPD. We have some approximations listed in Section I of the Appendix.\n\nTo address this, we conducted additional experiments using a **logarithmic decay schedule** for SMRD, which is entirely derived without referencing DBPD. This approximation is simple, easy to implement, and **does not require any task- or model-specific tuning**. \n\n**Logarithmic Decay:**  \n$$\n\\hat{f}(x; \\alpha) = \\frac{1}{\\log(x + \\alpha)}\n$$\n\nCritically, we observe that:\n\n- **Logarithmic SMRD is competitive with the oracle SMRD** in both **accuracy** and **effective epochs**.\n- In the appendix, we listed this in comparison to EfficientNet from scratch on CIFAR-100\n- Similar trends hold across other architectures such as EfficientNet, MobileNet v2 and ResNet, showing that our approximation generalizes well. All models are pre-trained on ImageNet.\n\n| Model          | Variant           | Accuracy | Effective Epochs |\n|----------------|-------------------|----------|------------------|\n| ResNet-50      | SMRD (oracle)     | 81.54    | 33              |\n| ResNet-50      | SMRD (log decay)  | 81.11 | 37           |\n| EfficientNet   | SMRD (oracle)     | 84.45   | 34              |\n| EfficientNet   | SMRD (log decay)  | 83.92 | 36           |\n| MobileNet v2   | SMRD (oracle)     | 74.73   | 36               |\n| MobileNet v2   | SMRD (log decay)  | 74. 36 | 38          |\n\n\nMost importantly, **SRD and DBPD already perform strongly on their own**. In particular, **SRD remains a simple yet highly efficient method**, showing consistent gains across image classification, self-supervised learning (MAE), and language modeling (GPT-2, shown above in previous weakness).\n\n---\n\nWe thank the reviewer again for the detailed feedback, which has helped clarify our contributions and strengthen the final paper."}]

Author rebuttal:

Meta-review: This paper introduces Progressive Data Dropout (PDD), which drops parts of the training data across epochs. Reviewers had concerns about the details of the method and some of the experimentation. Through the rebuttal process, all reviewers increased their scores, some significantly. All reviewers appreciated the simplicity of the method and the results achieved. Please consider including some of your additional experiments and results in the final version of the paper.
Decision: Accept (poster)