Title: Progressive Data Dropout: An Embarrassingly Simple Approach to Train Faster
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dropout

Regularization

Efficient
Abstract: The success of the machine learning field has reliably depended on training on large datasets. While effective, this trend comes at an extraordinary cost. This is due to two deeply intertwined factors: the size of models and the size of datasets. While promising research efforts focus on reducing the size of models, the other half of the equation remains fairly mysterious. Indeed, it is surprising that the standard approach to training remains to iterate over and over, uniformly sampling the training dataset. In this paper we explore a series of alternative training paradigms that leverage insights from hard-data-mining and dropout, simple enough to implement and use that can become the new training standard. The proposed Progressive Data Dropout reduces the number of effective epochs to as little as 12.4\% of the baseline. This savings actually do not come at any cost for accuracy. Surprisingly, the proposed method improves accuracy by up to 4.82\%. Our approach requires no changes to model architecture or optimizer, and can be applied across standard training pipelines, thus posing an excellent opportunity for wide adoption. Code can be found here: \url{https://github.com/bazyagami/LearningWithRevision}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"Progressive Data Dropout\" (PDD), a family of methods aimed at accelerating neural network training by progressively reducing the amount of training data used across epochs. The core idea is to selectively drop data based on difficulty or random chance during intermediate epochs, with a final epoch utilizing the full dataset for \"revision.\" The authors propose three variants: Difficulty-Based Progressive Dropout (DBPD), Scalar Random Dropout (SRD), and Schedule-Matched Random Dropout (SMRD). The paper claims that PDD can significantly reduce the number of \"effective epochs\" required for training (by as much as 87.6%, achieving this with as little as 12.4% of baseline effective epochs) and can sometimes improve model accuracy (reportedly by up to 4.82%) on various image classification tasks and architectures.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper presents initial results across several datasets and architectures suggesting a reduction in effective epochs while sometimes maintaining or even improving accuracy.\n\n**Weaknesses:**\n\n1.   Originality and Significance Over-claimed\n\nThe related work section appears to conclude its survey around 2022. This is a significant concern as the field of data pruning, curriculum learning, and coreset selection is rapidly evolving. Crucially, the paper fails to compare PDD against more recent (post-2022) and highly relevant dynamic data pruning [1] or sample selection strategies.\nFor example, methods like Instance-dependent Early Stopping [2] (which share exactly same concepts with PDD) and InfoBatch [3] are not discussed or used as comparative baselines. \n\n2. Questionable Settings\n\n- The setup for baseline comparisons. For instance, on CIFAR-10, baseline models are trained for only 30 epochs. This is substantially shorter than standard training regimens for CIFAR-10, which typically yield much higher baseline accuracies (e.g., >97% for ResNet variants). Training baselines for such a short duration likely underestimates their true potential and unfairly inflates the perceived relative improvements of PDD. \n- While the authors define \"Effective Epochs\" (EE), the improvements in wall-clock time, which is a critical practical measure, are not extensively detailed or benchmarked against other speedup techniques. While \"Effective Epochs\" measures backpropagation load, it doesn't fully capture the overhead of the PDD methods themselves (e.g., difficulty scoring in DBPD, which requires a forward pass on potentially more data than is used for backpropagation in early epochs, or managing dynamically changing data subsets).\n\n[1] Accelerating deep learning by focusing on the biggest losers, arXiv preprint 2019.\n\n[2] Instance-dependent Early Stopping, ICLR 2025.\n\n[3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.\n\nQUESTIONS:\nThe SMRD variant, often reported as the best performing, requires knowledge of the data reduction schedule from DBPD. The paper mentions exploring mathematical approximations for this schedule, but the practical performance and robustness of SMRD with such approximations (compared to the oracle SMRD and other PDD variants) are not sufficiently clear.\nCould the authors provide detailed results on how SMRD performs when using a practically derived (approximated) schedule? How does this compare to the oracle SMRD and other PDD variants in terms of both accuracy and effective epochs?\n\nA substantial improvement in my evaluation score would require:\n\nDemonstrating significant benefits of PDD over baselines trained under standard, competitive settings (e.g., standard epoch counts for CIFAR/ImageNet).\nComprehensive discussion and empirical comparison against relevant and recent (post-2022) state-of-the-art data selection/pruning methods.\nDo not over-claim the method’s originality.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose three methods which reduce the number of training examples as the model progresses during training, and find that their data dropout method is able to improve accuracy while significantly reducing the total training cost over standard training. The authors demonstrate results on multiple architectures for CIFAR-10, CIFAR-100, and ImageNet.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The method proposed by the author is simple, requires minimal overhead, and leads to higher accuracy compared to standard training across many architectures and datasets. \n- The schedule-matched random dropout is interesting and allows us to disentangle the effects of difficulty vs the dropout schedule. This is an interesting ablation which allows us to better understand the effects of data dropout.\n- The methods are clearly described, and the experiments are very reproducible. The code is also included. \n\n**Weaknesses**\n- The impacts of the hyperparameters for the method are not thoroughly explored, even though I assume they have a significant impact on the final results. The authors compare DBPD/SMRD 0.3 and 0.7, which feels somewhat arbitrary to me, and there is no ablation for the alpha in SRD. \n- This method is not compared to any other baselines, including very simple methods like early stopping. Therefore, it is unclear to me if focusing on a subset of the data is beneficial compared to the full dataset, which is the main claim of the paper.\n- Table 1 is difficult to interpret, since the number of effective epochs varies across all of the methods, and the results only consist of final accuracies rather than showing the full training curves. For example, if SMRD is more efficient with 6.7 EE but has lower accuracy than SRD with 15.71 SRD, I can't draw any conclusions about the results because maybe SMRD would outperform SRD if it had 8 EE instead. It would be helpful to have a table of results where the number of effective epochs is held constant (maybe by changing the hyperparameters for each method), so I can better understand which method I should use if I have a specific amount of compute. Or, it would also be helpful to show the training curves (accuracy vs number of train examples) so I can better compare the performance.\n- The method that performs the best, SMRD, is not practically usable since it depends on DBPD. The authors explore different data dropout schedulers in the supplementary material, and I think that this analysis would be key to making this a practical method. Again, a comparison of the training curves (accuracy vs number of train examples) could be really helpful here to understand the differences between schedules.\n\nQUESTIONS:\n- I'm not sure I understand the argument in line 209, where data dropout \"[forces] the model to focus on more informative examples and avoid overfitting\". Wouldn't data dropout increase the risk of overfitting, since the model sees fewer examples every epoch?\n- Can you describe the model training procedure and learning rate scheduler? If the number of examples per epoch is different for each method, is the learning rate changing once per epoch or once per X number of training samples? Would that be another difference between SRD and SMRD?\n- How does the total number of available training samples impact the method? In many modern settings, the total number of training samples is extremely large, so the model may not even iterate through multiple epochs. Conversely, if the total number of training samples is very small, then perhaps dropout is unnecessary and could lead to overfitting.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProgressive Data Dropout is a technique to speed up neural network training by progressively discarding subsets of data across epochs. Three variants are explored in this paper: a difficulty score based dropout, a scalar-based random decay method, and a hybrid approach. The paper demonstrates that PDD significantly reduces \"effective epochs\" while often improving or matching accuracy across architechtures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n> The concept of progressively dropping data, particularly the effectiveness of random dropping, is novel and simple.\n> The use of \"Effective Epochs\" as a hardware-independent proxy for backpropagation effort is a valuable contribution.\n> Extensive experiments across architectures and ablation studies.\n\nWeakness:\n> The primary empirical evaluation is limited to image classification tasks. Although the authors explicitly acknowledge that \"broader applicability to other domains such as segmentation, language modeling, or multi-modal learning remains to be investigated,\" this limitation reduces immediate confidence in the method's generalizability across the broader spectrum of deep learning applications. Empirical evidence demonstrating PDD’s effectiveness in non-vision domains, such as NLP, would strengthen the paper.\n> Real-world datasets often exhibit long-tailed class distributions, significant class imbalance, and varying degrees of label noise. However, the paper does not evaluate how PDD performs under these realistic conditions.\n\nQUESTIONS:\n> Can you provide theoretical reasoning or empirical evidence explaining why random dropout is more effective than selecting easy or hard examples?\n\n> How does PDD affect performance on datasets with significant class imbalance, which is common in real-world scenarios? It would be valuable to evaluate all three variants under such conditions. You could simulate class imbalance using power-law distributions. Additionally, please consider conducting experiments with varying levels of label noise (e.g., 20%, 30%) to test robustness.\n\n> The revision phase in PDD intuitively performs a form of relearning after intentional forgetting via data dropout to improve generalization. Can the authors draw parallels between PDD and prior work on forgetting and relearning (e.g., neuron- or connection-level forgetting vs. data-level forgetting)? It would be helpful to understand how such forgetting dynamics influence generalization. (See below for related papers)\n\n> Empirical results on ImageNet suggest that larger datasets require more effective training epochs and exhibit smaller relative gains. Could the authors elaborate on why efficiency improvements diminish with increasing dataset size (e.g., web-scale datasets used in foundation models)? This is especially important since computational savings are most critical at that scale.\n\n> Following the previous point, what dataset scale or characteristics is PDD best suited for? It is unclear when to use this method (small datasets, large datasets)? It would be insightful to include results on smaller fine-grained datasets such as CUB, Flowers, or Aircraft.\n\n> Are the reported improvements statistically significant? How many random seeds were used for the experiments?\n\n> Could the authors analyze model performance on specific subsets of the data before and after dropout, and prior to the final revision phase? This would help reveal transient \"forgetting\" effects and better characterize the behavior of each method.\n\n> While the paper observes that random data dropout outperforms difficulty-based strategies, it lacks a theoretical or intuitive explanation for this counterintuitive result. Providing such insights would significantly strengthen the contribution.\n\nRelated works:\n> [1] Ramkumar, V.R.T., Arani, E. and Zonooz, B., 2023. Learn, unlearn and relearn: An online learning paradigm for deep neural networks. arXiv preprint arXiv:2303.10455.\n> [2] Ramkumar, V.R., Arani, E. and Zonooz, B., 2024. Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets. Advances in Neural Information Processing Systems, 37, pp.65048-65071.\n> [3] Zhou, H., Vani, A., Larochelle, H. and Courville, A., 2022. Fortuitous forgetting in connectionist networks. arXiv preprint arXiv:2202.00155.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Progressive Data Dropout (PDD), a simple method to accelerate training by progressively dropping parts of the training data across epochs. Three variants are explored: difficulty-based, random scalar-based, and a hybrid that follows the difficulty-based schedule with random sampling. The method shows significant reductions in backpropagation steps (effective epochs), yielding good performance with less compute. It can be seamlessly integrated to any training procedure and is demonstrated across CIFAR, ImageNet, and MAE self-supervised learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- S1 : The idea is extremely simple and general and can be seamlessly integrated in standard pipelines.\n- S2 : The experimental setup is sound, with various tasks and architectures tested (although see W1).\n- S3 : The \"effective epochs\" metric is clear and really insightful. Significant computational savings are demonstrated which is a very good point. \n- S4 : The code is open-sourced and reproducibility seems good. Given the fact that the method can be integrated to any training pipeline, this can be very useful for the community.\n\n---\n**Weaknesses**\n\n- W1 : The main weakness I find is that the training hyperparameters used are quite odd. Specifically, the training schedules are very limited for some baselines (e.g., only 30 epochs on CIFAR-10 ResNet). This significantly weakens the impact of reported gains. The simplified training schedules or short runs in some cases, possibly mask practical drawbacks.\nAs standard baselines are often 200+ epochs for ResNets, the reported gains on these benchmarks are difficult to interpret or trust in a high-performance regime.\nThe effectiveness in SOTA training regimes is yet to be tested as it is for now unclear how the method performs compared to highly tuned baselines. \n\n- W2 : No comparisons to strong active learning baselines like Data Diet [1] or coreset selection [2] are provided which makes it hard to evaluate.\n\n- W3 : The evaluation remains narrow in task diversity (mostly image classification). While the method is well-motivated, the strongest gains may be dataset- or setup-specific. As the method is claimed to be seamlessly integrated to any training procedure, adding NLP experiments (eg. pretraining on OpenWebText and finetuning pretrained modelson standard tasks ) would greatly strengthen the paper given the number of tokens used to train SoTA models nowadays (trillions).\n\n\n\n---\nOther minor weaknesses: \n- Some hyperparameters (e.g., dropout thresholds) may be sensitive, but robustness to them is only lightly tested.\n- The overall writing style is very GPT-like (lots of itemization for example in Section 3, many sentences with bits in dash parenthesis). While this may be a matter of personal taste, it occasionally makes the text harder to follow.\n\n\n\n[1] Paul, M., Ganguli, S., & Dziugaite, G. K. (2021). Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34, 20596-20607.\n\n[2] Lee, H., Kim, S., Lee, J., Yoo, J., & Kwak, N. (2024). Coreset selection for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7682-7691).\n\nQUESTIONS:\nI am currently not in favor of acceptance (as reflected in my “Reject” score), but I would be happy to revise my rating (potentially up to an Accept) if the authors are able to address the following concerns. I understand the rebuttal period is short, so I do not expect full experimental results, but even partial answers or indications that the paper could be strengthened would be very much appreciated.\n\n1. Could you extend experiments to standard-length training runs (e.g., 200 epochs on CIFAR-10) to test if gains persist?\n\n2. How does PDD perform under strong regularization or data augmentation (Mixup, CutMix, etc.)?\n\n3. Would it be possible to make the dropout schedule adaptive based on training signals?\n\n4. Could you test your method on pretraining a small NLP model (eg. a small GPT-2 on Open Web Text).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}