Progressive Data Dropout: An Adaptive Training Strategy for Large-Scale Supervised Learning
Authors: David Lee Patrick
Zachary Seligman
Amanda S Fernandez
Venue: Submitted to ICLR 2023 (2023)
Keywords: data dropout
training optimization
adaptive training
classification
large-scale

Abstract: Common training strategies for deep neural networks are computationally expensive, continuing to redundantly train and evaluate on classes already well-understood by the model. A common strategy to diminish this cost is to reduce data used in training, however this often comes at the expense of the model's accuracy or an additional computational cost in training. We propose progressive data dropout (PDD), an adaptive training strategy which performs class-level data dropout from the training set as the network develops an understanding for each class. Our experiments on large-scale image classification demonstrate PDD reduces the total number of datapoints needed to train the network by a factor of 10, reducing the overall training time without significantly impacting accuracy or modifying the model architecture. We additionally demonstrate improvements via experiments and ablations on computer vision benchmarks, including MNIST, Fashion-MNIST, SVHN, CIFAR, and ImageNet datasets.

Under review as a conference paper at ICLR 2023
PROGRESSIVE DATA DROPOUT: AN ADAPTIVE TRAINING STRATEGY FOR LARGE-SCALE SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Common training strategies for deep neural networks are computationally ex- pensive, continuing to redundantly train and evaluate on classes already well- understood by the model. A common strategy to diminish this cost to reduce data used in training, however this often comes at the expense of the model's ac- curacy or an additional computational cost in training. We propose progressive data dropout (PDD), an adaptive training strategy which performs class-level data dropout from the training set as the network develops an understanding for each class. Our experiments duces the total number of datapoints needed to train the network by a factor 10, reducing the overall training time without significantly impacting accuracy or modifying the model architecture. We additionally demonstrate improvements via experiments and ablations on computer vision benchmarks, including MNIST. Fashion- Fashion-MNIST. MNIST, SVHN, CIFAR, and ImageNet datasets.
large-scale image classification demonstrate PDD re-
INTRODUCTION
Deep neural networks have made significant impact on broad range of applications over the last decade. However, these networks are notoriously data-intensive, often requiring significant computational power and large datasets in order to properly train for optimal performance. This can become problematic for many real-world applications as the computational expense of training these networks will often prevent them from being adopted.
Many optimization techniques have arisen to address this problem in training from reducing neu- subsampling data. In this work, we focus on reducing the computational time and cost needed to fully train any deep network, without modifying the model and while utilizing the entire training dataset.
In order to properly set the discussion, we need to define some terms that are used throughout this paper. First, define datapoint as a single data sample that sent through the network during the training process. For example, if you were to train a network using dataset of 10 samples for epochs, you would have used 50 datapoints to train that network. So our datapoint calculations are a collection of the number of samples that were sent to the network during training rather than the number of unique data samples was provided. This because networks often need see examples of class multiple times before understanding them, Secondly, since our proposed method modifies the training process, the term epoches no longer applicable as we will not iterate over the entire dataset after dropping data. Instead, we will use the term training rounds to indicate how many times we iterated over the remaining training set when training the network.
Now, consider one simple research question: Can data be dropped during training once it is well- understood by a deep leaning model? We propose Progressive Data Dropout (PDD), an adaptive training process which leverages the network's understanding of the data to determine when data should be dropped from the training process. In comparison with existing data dropout techniques which focus on identifying "important" samples, we instead evaluate the simple of full and partial class-removal.
The main contributions of our simple adaptive training strategy, PDD, include:

---

Under review as a conference paper at ICLR 2023
Reduces time and computational resources required for training.
Model-agnostic implementation which works for any supervised single-label w classification task.
Data-agnostic implementation which does not preprocess or examine the data for sample quality, balance, etc.
Provides inherent stopping criterion for training models sufficiently large datasets.
Further sections motivate the of PDD for large-scale image classification tasks and describe the design of this simple strategy. We propose PDD to be complimentary of existing regularization tech- niques, well recent learning/training strategies such as continual learning, curriculum learning, and others.
2 RELATED WORK
There are a breadth of optimization techniques for the training deep neural networks, motivated by the traditionally large data requirements of deep models. Given that we are proposing a gener- alized strategy for optimizing training, this section discusses several types of solutions learning techniques, dropout for network regularization, and dropout of data during training.
2.1 CURRICULUM & CONTINUAL. LEARNING
Several learning approaches are fundamentally similar objective to our proposed method, though they take a drastically different approach. Curriculum learning, for example, aims slowly intro- duce difficult aspects of the problem by training the network from the easiest samples to the hardest samples. Lyu & T Tsang (0199) introduce curriculum loss for learning with noisy labels. By troducing loss function that is bounded and able to select samples during network training, the authors are able to combine continual and robust learning.
Meta-learning similarly aims to quickly adapt to new tasks by improving the learning algorithm itself, improving computational bottlenecks as well as generalization. [Finn et al. (2017) However, both meta-learning and curriculum learning suffer from occasional catastrophic forgetting of the original learned task. Continual learning aims to prevent the catastrophic forgetting of the original learned task Veniat et al. (2021). More recently, Co2L (Cha et al., 2021) contrastive continual learn- ing proposed a contrastive learning objective which learned and preserved representation through distillation. These curriculum and continual learning strategies complimentary to PDD in that our proposed method can be leveraged toward their respective objectives.
2.2 DROPOUT FOR NETWORK OPTIMIZATION & REGULARIZATION
While our proposed approach is focused on dropping data, the term dropoul more often applies to the dropping of neurons during the training process for regularization of neural network (Srivastaval
ct al. 2014). Network dropout regularization techniques K C ct al. (202I) and pruning approaches
(Tanaka al. 2020) reduce the size of network during training response to input stimuli.
Adaptive dropout for training deep neural networks (Ba & Frey 2013) overlaid binary belief net- work on top of neural network allowing the network to adapatively regularize the network by selectively setting parts to zero. Dropconnect (Wan et al. 2013) regularized networks by dropping randomly selected weights instead of randomly selected activations. Curriculum dropout (Moreni et al. 2017) showed that the fixed neuron dropout probability was sub-optimal and instead imple- energy- dropout proposed by EDropout (Salchinejad & Valace 2021 ) used energy based loss to find the best pruning ap- ply to the original neural network. In application to different network architectures, Dropout-GAN dropped connections between generator and multiple discriminators in GAN in order to ensure diversity of generated samples, avoiding mode collapse (Mordido et al. 2018).
mented time scheduler for updating the dropout probability.
While dropout is effective for regularization, these methods often require significant modification of the network itself in order to be adopted. In comparison, our proposed approached is entirely model-agnostic, modifying only the number of samples used in training instead of the network or
2

---

Under review as a conference paper at ICLR 2023
data itself. We therefore do not compare with these types of approaches in our experiments, they are complementary to PDD rather than comparative.
2.3 DATA DROPOUT
Most similar to our proposed method are approaches which drop data samples during the process of training a neural network. Some such approaches focus on data augmentation, in an effort to improve the quality of the data fed to the network and model accuracy. Generalized Dropout (Rahmani & Atia 2018) for example is method of data augmentation, dropping random pixels from an image in order to generate additional samples for training.
Other methods for data dropout emphasize importance sampling, aiming to first identify quality samples and then drop the lower quality samples for remaining training epochs (Katharopoulos & Fleuret, 2018). Data Dropout (Wang et al.) 2018)) removes samples from training after they's re deemed unfavorable in the first epoch, saving time in further training rounds. Similarly, DropSe (Yang et al. 2016) applied this concept for Chinese character classification. Subsequently Greedy DropSample (Yang et al. 2020) generalizes the method even further by using a greedy algorithm, temporarily dropping data needed for training acceleration, but retaining it for use if the model requires. In a similar fashion, dataset summarization techniques which strategically subsample the dataset have been demonstrated outperform full-set models (Wang et al. 2020).
Recently Han et al.] (2020) dropped highly negative predictions iteratively through the utilization of influence functions. This helped remove highly noisy or of date labels from the training set. Similarly, Dynamic Training Data Dropout (DTDD) (Zhong et al. 2022) drops samples deemed
noisy after several epochs, with specific application to noise-robust deep face recognition.
In contrast to these methods, our progressive data dropout takes cues from the model while training to determine when drop majority of a class, While each of these existing approaches similar to our proposed PDD, we do not compare with those which are so Noomann/apllicationsseee not be applicable standard image classification benchmarks, including DTDD, DropSample for character classification, and those removing noisy labels. Such methods are complimentary to PDD rather than comparative as they can be used combination for their application-specifi tasks.
as
PROGRESSIVE DATA DROPOUT
In this work, we present Progressive Data Dropout (PDD), a novel training strategy which combines data dropout with a residue component to train networks. Like most current data dropout techniques, PDD can paired easily with most networks since requires no network modifications. However, unlike most other data dropout techniques, PDD progressively removes data from the training set as the network develops an understanding for classes. By removing data from the training set, allows us to create subsets of the data for network to train on, speeding up the training process. To do this, we utilize dropout score in conjunction with a residue component to control which subset of the dataset is provided to the network.
Figure D and further demonstrated through our extensive experiments, displays a general view of the data used in training neural network. Baseline training strategies have constant number of data- points since all data is used in every training round, whereas comparative data dropout approaches drop datapoints during the training process. This figure demonstrates the significant reduction in data feasible in training network. The following subsections detail PDD components and training strategy more detail.
3.1 DROPOUT SCORE
An important component to PDD's strategy is selecting a dropout score since it is responsible for determining when data should be dropped from the training examples. When determining PDD's dropout score, there are two important factors consider. First, the dropout score should be a metric that accurately reflects the performance of the individual classes in the network. For example, in our experiments we used fl-score when evaluating classification models since it depicts how well the network is performing on a per-class basis. Secondly, once a metric has been selected as the dropout score, we then need to select appropriate threshold. Determining the threshold should be treated
3

---

Under review as a conference paper at ICLR 2023
1.2
Datapoints millions)
1.1
1.0
Baseline
0.9
Data Dropout
PDD 0.80
PDD 0.85
0.8
0.90
60
75
Rounds of Training
Figure 1: Visualization of the amount of training datapoints used training ResNet34 (e et al. 2016
on ImageNet (Deng et al. 2009). Best viewed in color.
as extremely sensitive hyper-parameter which can have significant effects on the network training process. If the threshold value is set too low, the network will often not have firm grasp of the data being dropped, resulting in the eventual forgetting of the data. If the threshold value is set too high, many of the training data points may never be dropped, severely reducing the benefit provided by PDD. Unfortunately, like many other hyper-parameters, there is universally applicable threshold value as is highly dependant on your both the metric and the dataset. To help determine an appropriate metric and its threshold, recommend experimenting with a smaller data subset first.
3.2 RESIDUE
Intuitively, and reinforced in our experiments, fully removing a class from the training dataset causes the network suffer catastrophic forgetting due the loss only penalizing the network for predict- ing the removed class. Rather than modifying the loss, which would be considered network mod- ification, we instead introduce a residue component into PDD to prevent the catastrophic forgetting problem. This residue component controls what percentage of randomly-selected training examples associated with a class are left behind when that class crosses the dropout score threshold. Unlike the dropout score threshold, empirically show through our experiments that leaving behind ten percent of a class usually enough to encourage the network to remember it. important to note that this residue percentage be increased; however this will come at the cost of training time.
3.3 RESIDUE SWAPOUT
When leaving a small percentage of data behind as residue, alleviating the catastrophic forgetting problem, it presents a new problem where networks will often easily over-fit on the residue exam- ples. To address this problem, we also introduce residue swapout component to determine how often residue training examples are swapped out for newer ones. While this does add additional computational costs, it also helps prevent the network from over-fitting on the residue examples, especially over long periods of training.
3.4 WARM-UP PERIOD
We also introduce warm-up period component into PDD. This parameter specifies how long the network is allowed to train before our training strategy is allowed to begin. This parameter is to ensure that the network's feature space has been well established before we begin removing data. Typically, this parameter is mostly used for lower-dimensionally data where a network can converge on dropout score too quickly.
4

---

Under review as a conference paper at ICLR 2023
3.5 TRAINING STRATEGY
Once the components of PDD have been established, we begin PDD by creating mask of the entire datasets' labels before beginning network training. This mask is used throughout the training process to determine which subset of the dataset provide to the network during a single training round. Once the mask has been created, PDD begins the network training process with the entire mask on, providing the network with the entire dataset. At the end of a training round, dropout score is calculated for each individual class in the dataset. If a class has a dropout score that is above the provided threshold, and assuming the warm-up period has also been met, assume the network has developed an understanding for that class and its presence can be safely reduced in the training data. In order reduce the class' presence, we update the mask exclude all but randomly selected residue examples of the class, creating a new data subset to use in the next round of training. PDD concludes the network training process when either every class has exceed the provided dropout score threshold or when the maximum rounds of training has been reached. Overall, PDD reduces the number of datapoints that are needed to train the network, which in turn reduces the overall training time of a network. In addition, PDD can provide a clear point to when your network should be stopped with training and help determine classes the network has difficulty converging on.
4 EXPERIMENTS
In order to demonstrate the effectiveness of PDD, we first conduct image classification experiments in this section and then explore ablation studies for PDD's components in the subsequent section. For the purposes of all of our experiments, we selected fl-score as our dropout score metric, Also, unless otherwise specified, used the following values for PDD's components: residue of .10, residue swapout training round, and a warmup period of 5 training rounds.
4.1 IMAGE CLASSIHCATION BENCHMARKS
We first conducted experiments on lower dimensionality datasets with well-established classification
models to show the efficacy of PDD. We trained Resnet-18 on MNIST (LeCun et al., 2010). Fash-
ionMNIST (Xiao al. 2017). SVHN (Netzer al. (011), and CIFARIÓ (Krizhevsky (2009) using
a single RTX 8000 GPU. During training, we used the following typer-paramelte
batch sizes of
32, a standard categorical cross-entropy loss, standard SGD optimizer with learning rate of 0.01,
and max number of training rounds of 20. Finally, we report the accuracy of the model in relation
to the dataset's ' test set. Table shows the results of our experiments in comparison with other data dropout techniques.
4.2 DROPOUT SCORE THRESHOLD EXPERIMENT
After conducting our benchmark experiments, we then demonstrate the effect the dropout score threshold has PDD. Table 2 shows the results of this experiment. Besides MNIST, our experi- ments show that a higher droput score yields a better network performance at the cost of training time. Our experiments also demonstrate faster training time due PDD concluding the network training process early. This is because every class has exceed the provided dropout score threshold thus ending the training of the network.
4.3 LARGE-SCALE IMAGE CLASSIFICATION
We extend our experiments to a large-scale image classification problem. Our experiments show the result of our training method when applied to the ImageNet (Deng et al.) 2009) dataset on a well- established classification model. To do this, we trained ResNet-34 on the ImageNet dataset using V100 GPUs, applying a different training strategy to each run. During training, we followed the standard PyTorch training procedure TorThr 2022) which included: batch sizes of 32, a standard categorical cross-entropy loss and a SGD optimizer with a momentum 0.9, weight decay of Ie- 4, and learning rate of 0.01. As the network training progressed, reduced the learning rate by a factor of 10 for every 30 rounds of training that occurred. The results of this experiment are shown on Table which evaluated the performance of the network training methods on the ImageNet
5

---

Under review as a conference paper at ICLR 2023
Method
Dataset
Model
Acc.
Time (mins)
RT
# Datapoints
Baseline
MNIST
ResNet18
0.99
0:13:45
20
1,080,000
GD
MNIST
custom
0.99
250
DropSample
MNIST
custom
0.99
0:25:00
800
PDD
MNIST
ResNer18
0.99
0:04:12
6
324,000
Baseline
FashionMNIST
ResNet18
0.91
0:14:03
1, 1,080,000
PDD
FashionMNIST
ResNet18
0.88
0:04:25
334,265 7
Baseline
SVHN
ResNet18
0.92
0:17:08
20
1,318,640
DataDropout
SVHN
ResNet152
0.99
0:28:20
50
2,107,811
PDD
SVHN
ResNer18
0.91
0:05:04
6
395,592
Baseline
CIFAR
ResNet18
0.77
0:11:59
20
900,000
IS
CIFAR-10
ResNer28
0.95
5:30:00
50,000
DropSample
CIFAR-
custom
0.92
0:18:20
24
DataDropout
CIFAR-10
ResNet1
0.95
4:10:00
500
21, 21,901,200
PDD
CIFAR-IO
ResNet18
0.71
0:04:27
324,011
Table 1: Comparison of metrics of various data dropout techniques. Compares a baseline training strategy, GeneralizedDropou (GD) (Rahmani & Atial 2018). ImportanceSampling (IS) Katharopou- los & Fleuret (2018), Greedy DropSample (Yang et al. 2020), DataDropout Wang et al. (2018) 8). and and PDD (our proposed Progressive Data Dropout method, with 0.85 FI dropout score). Estimated values are italicized, and clarified in the appendix. Missing values are noted as '-', and all other values reported in the original works.
Method
Dataset
Dropout Score
Accuracy
Time (mins)
RT
# Datapoints
Baseline
MNIST
0.99
0:13:45
20
1,080,000
PDD
MNIST
0.85
0.99
0:04:12
324,000
PDD
MNIST
0.95
0.99
0:04:20
324,000
Baseline
MaSSI
0.91
0:14:03
20
1,080,000
PDD
FashionMNISTT
0.85
0.88
0:04:25
334,265 17
PDD
FashionMNIST
0.95
0.90
0:08:24
597,705
Baseline
SVHN
0.92
0:17:08
1,318,640 20
PDD
SVHN
0.85
0.91
0:05:04
395,592
PDD
SVHN
0.95
0.91
0:06:21
477,939 20
Baseline
CIFAR-IO
0.77
0:11:59
900,000
PDD
CIFAR-10
0.85
0.71
0:04:27
9
324,011
PDD
CIFAR-10
0.95
0.75
0:07:25
544,913
Table Fi-score PDD experiment on various datasets using ResNet18 (He et al. 2016). RT stands for rounds of training. PDD settings shared across runs: Warmup period of 5 rounds, residue of .10, and swapout period of round. Dashed values are not applicable.
validation dataset. We additionally provide visual which shows the number of datapoints the network's 's training in Figure
Fl Dropout Score
Method
Validation Accuracy
Time (hours)
RT
# Datapoints
Baseline
0.73
8:30:12
90
109,544,850
Data
dropout
0.79*
60*
108.375,212*
(Wang al. 2018)
PDD
0.80
0.70
7:31:38
90
94,957,921
PDD
0.85
0.72
8:08:12
90
103,996,877
PDD
0,90
0.73
8:26:22
90
108,570,798
Table 3: Comparison of run times on ImageNet (Deng et al. 2009). RT stands for rounds of training Denotes the value taken calculated from the original paper.

---

Under review as a conference paper at ICLR 2023
5 ABLATION STUDIES
To assess the effectiveness of each component of our proposed method, the following studies demon- strate the effect of removing our residue, swapout, and warm-up features from PDD. We conduct these studies on the lower-dimensionaliy data using the same training specifications that were men- tioned Section 4.1 In addition, for these studies, PDD components were dropout score of 0.85, a residue of .10, residue swapout of 1 training round and warmup of training rounds unless that component was turned off, in which case its value was set O.
the following values:
5.1 RESIDUE COMPONENT
In this study, we looked at the effects the residue component had on PDD network training. Table 4 shows the results of these studies. As shown in the table, the networks trained on FashionMNIST and CIFAR-I0 performed significantly worse there no residue component. For the MNIST and SVHN networks, PDD ended the training process immediately after the warm-up period. This meant that all of the classes exceed the dropout score after the warmup period ended, resulting in the residue and no residue experiments being treated exactly the same.
Method
Dataset
Accuracy
Time (mins)
RT
# Datapoints
Residue
MNIST
0.99
0:04:12
6
324,000
No Residue
MNIST
0.99
0:04:15
6
324,000
Residue
FashionMNISST
0.88
0:04:25
7
334,265
No Residue
MNSSI
0.10
0:04:23
7
329,410
Residue
SVHN
0.91
0:05:04
395,592 6
No Residue
SVHN
0.92
0:05:05
6
395,592
Residue
CIFAR-I0
0.71
0:04:27
9
324,011
No Residue
CIFAR-10
0.15
0:04:25
9
319,497
Table 4: Residue PDD experiment on various datasets using ResNet18. RT stands for rounds of training. PDD settings: Dropout score of 0.85, residue of 0.10 where applicable, swapout period of training round, and a warmup period of 5 rounds.
5.2 SWAPOUT COMPONENT
In this study, we looked at the effects the residue swapout component had on PDD network training. Table shows the results of these studies. Similar to the last study, networks trained on Fashion- MNIST and CIFAR-I0 suffered from performance drops when removing the swapout component, although significantly less than when compared to the residue component. Again, PDD ended the training process of the MNIST and SVHN networks immediately after the warm-up period, resulting in the swapout experiments being treated the exact same.
Method
Dataset
Accuracy
Time (mins)
RT
# Datapoints
Swapout
MNIST
0.99
0:04:12
324,000
No Swapout
MNIST
0.98
0:04:14
324,000
Swapout
FaShionMNIST
0.88
0:04:25
7
334,265
No Swapout
FashionMNIST
0.86
0:04:29
334,265
Swapout
SVHN
0.91
0:05:04
395,592
No Swapout
SVHN
0.93
0:05:03
395,592
Swapout
CIFAR-
0.71
0:04:27
324,011
No Swapout
CIFAR-IB 10
0.70
0:04:26
319,950 9
Table 5: Swapout PDD experiment various datasets using ResNet18. RT stands for rounds of training. PDD settings: Dropout score 0.85, residue 0.10, swapout period of training round where applicable, and warmup period of 5 rounds.

---

Under review as a conference paper at ICLR 2023
5.3 WARMUP COMPONENT
Finally, we studied the effects that the warmup component had on PDD network training. Table shows the results of these studies. Unlike the previous studies, all networks but ones trained on CIFAR-10 suffered from performance drops when removing the warmup component. However, it important to note that having no warmup component did improve training time significantly. As for the networks trained CIFAR-10, since it is unlikely to drop class within the first 5 rounds of training due to the complexity of the data, the warmup period component did not matter.
Method
Dataset MNIST
Accuracy
Time (mins)
RT
# Datapoints
Warmup
0.99
0:04:12
6
324,000
No Warmup
MNIST
0.97
0:00:42
54,000
Warmup
PashionMNIST
0.88
0:04:25
7
334,265
No Warmup
FashionMNIST
0.87
0:02:01
138,886 5
Warmup
SVHN
0.91
0:05:04
395.592
No Warmup
SVHN
0.87
0:01:58
3
146.314
Warmup
CIFAR-IO 10
0.71
0:04:27
9
324,011
No Warmup
CIFAR-I0
0.72
0:04:18
9
311,847
Table 6: Warmup PDD experiment various datasets using ResNet18. RT stands for rounds of training. PDD settings: Dropout score of 0.85, residue of 0.10, swapout period training round, and warmup period of 5 rounds where applicable.
CONCLUSIONS
In this work, we propose Progressive Data Dropout (PDD), new training optimization strategy for deep learning networks. Through extensive experimentation on well-established classification networks and benchmarks, we demonstrate an effective training strategy which reduces data over network training on both large-scale and small-scale image classification tasks. By reducing the data over network training, we are able to reduce the overall time needed to train a network while also showing which classes network is having difficulty learning.
6.1 LIMITATIONS
Although we demonstrate the effectiveness of PDD, there are limitations that need to be addressed. One major limitation of proposed training method is that it is designed with classification net- works in mind. This is mainly due to our dropout score working inherently with a classification problem rather than other tasks such regression. Another major limitation of training strat- egy is that we assume that the training dataset is relatively large and balanced, which means our strategy would likely cause performance issues for problems such few-shot or anomaly detection. Finally, our current implementation creates copy of dataset labels to use as the mask, which could be expensive for problems with high label cost such segmentation and multi-label tasks.
6.2 FUTURE WORK
This proposed PDD implementation randomly selects the residue examples that are left behind when class crosses the dropout score threshold. In future works, this randomized residue component could be replaced with a more deterministic component, such method which finds low confi- dence samples, in order to improve the overall classification performance of the network. However, when adding the deterministic component, is important to ensure that the overhead of the new method maintains a lower computational cost than just training with the entire dataset.
In conjunction with the deterministic residue component, interesting avenue of exploration is more dynamic residue component. With a dynamic residue component, the network could have different levels of residue for each class depending their performance. However, there some roadblocks to implementing this, such as determining a proper metric tracking and computational overhead.
8

---

Under review as a conference paper at ICLR 2023
Though PDD focuses data-level techniques, it easily be paired with other data dropout or network modification techniques in order to further improve the overall performance network, including those identified in section 2 Since PDD tracks how well network is performing a per-class basis, that information could be leveraged with other techniques to assist with class understanding and retention.
Finally, our proposed training method could be extended to other tasks in computer vision, language processing, and general classification. While PDD could be extended to other single label prob- lems with relatively minimal changes, multi-label problems such as semantic segmentation or even multi-label classification would require some extension order identify when a sample could be appropriately dropped. In large enough dataset, policy could be established for justifying the removal of data samples containing only well-understood classes, for example.
REFERENCES
Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26, 2013.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 9516-9525, October 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135, PMLR, 2017.
Hojae Han, Seung- won Hwang, Young-In Song, and Siyeon Kim. Training data optimization for pairwise learning to rank. In Proceedings of the 2020 ACM SIGIR on International Conference Theory of Information Retrieval, pp. 13-20, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kishan K C. Rui Li, and MohammadMahdi Gilany. Joint inference for neural network depth and dropout regularization. In Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, neurips. Pactsaacdd-apppe---ee pdff
pp. 26622-26634. Curran Associates, Inc., 2021,
Angelos Katharopoulos and François Flcuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning (ICML), pp. 2525-2533 PMLR, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yannllecu commexsb/nnnit, 2. 2010.
Yueming Lyu and Ivor W Tsang. Curriculum loss:
Robust learning and generalization against label corruption. International Conference on Learning Representations (ICLR), 2019.
Gonçalo Mordido, Haojin Yang, and Christoph Meinel. Dropout-gan: Learning from a dynamic ensemble of discriminators. ACM SIGKDD CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING, 24, 2018.
Pietro Morerio, Jacopo Cavazza, Riccardo Volpi. Rene Vidal, and Vittorio Murino. Curriculum dropout. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.

---

Under review as a conference paper at ICLR 2023
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing digits in natural images with unsupervised feature learning. In NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
PyTorch. Pytorch muccvisionn.oodlss 2022.
URL http:://pptorreh.org//viii
stable /mmoemll.stt.
Mostafa Rahmani and George K. Atia. Data dropout in arbitrary basis for deep network regulariza- tion. In 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pp. 66-70, 2018. doi: 1AL109//CSSC.2288885555000
Hojjat Salehinejad and Shahrokh Valace. Edropout: Energy-based dropout and pruming of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2021. doi: N0.1009/TNNSS.221...0000
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems, 33:6377-6389, 2020.
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. In 9th International Conference on Learning Representations (ICLR), 2021.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning (ICML), pp. 1058- 1066. PMLR, 2013.
Tianyang Wang. Jun Huan, and Bo Li. Data dropout: Optimizing training data for convolutional neural networks. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 39-46, 2018 doi: 10.1 10.100//CAA122110000000
Zifeng Wang, Hong Zhu, Zhenhua Dong. Xiuqiang He, and Shao-Lun Huang. Less is better: Un- weighted data subsampling via influence function. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6340-6347, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Naisen Yang, Hong Tang, Jianwei Yue, Xin Yang, and Zhihua Xu. Accelerating the training process of convolutional neural networks for image classification by dropping training samples out. IEEE Access, 8:142393-142403 2020.
Weixin Yang, Lianwen Jin, Dacheng Tao, Zecheng Xie, and Ziyong Feng. Dropsample: new train- ing method to enhance deep convolutional neural networks for large-scale unconstrained hand- written chinese character recognition. Pattern Recognition, 58: 190-203, 2016. ISSN 0031-3203. doi: https://doi.org//11 wtw.//////....././.ooooooo.ooo patcoo/ 2016.04.000. URL httpt:::..... : wwww sci enced rect.. com/sci
//0001 32031 6300401
Yaoyao Zhong, Weihong Deng, Han Fang, Jiani Hu, Dongyue Zhao, Xian Li, and Dongehao Wen. Dynamic training data dropout for robust deep face recognition. IEEE Transactions on Multime- dia, 24:1186-1197, 2022. doi: 1.L.100/MM..22..1122252..
A APPENDIX
A.1 CALCULATION OF ESTIMATED VALUES
In our paper we compare our proposed PDD approach with related data-dropping methods on image classification benchmark datasets. However, not all of these cited works provided results
10

---

The DataDropout method proposed by Wang et al. (2018) reported 1220 "unfavorable" training samples in CIFAR-10 found after the first epoch. Therefore the calculated number of datapoints for the 500 training rounds noted in the paper for ResNet-110 was (1 epoch of the full CIFAR-10 training set) + (full CIFAR-10 training set 1220) * (500-1 training rounds) = (45,000) + (45,000- 1,200) 499 21,901,200. If it took our baseline ResNet-18 18 model 0.6 minutes per training round on CIFAR-IO, we can conservatively estimate training this ResNet-1 10 model on fewer datapoints to be 0.5 (number of training rounds 0.5 500 250 minutes hours and 10 minutes our Under review as a conference paper at ICLR 2023
on all datasets, nor code replicate their results. Instead, were able to estimate of their missing information via their described algorithm. In an effort to be as transparent as possible, we include these calculations here, organized by tublc/experiment.
A.1.1 TABAA ESTIMATIONS
single GPU.
Similarly, for SVHN, DataDt reported 24,261 "unfavorable" training samples found after the first epoch. Therefore the calculated number of datapoints for the 50 training rounds in the paper for ResNet-152 was epoch of the full SVHN training set) (full SVHN training 24,261) (50-1 training rounds) = conservatively 65,932 + 155,922-24,2611 + 2,107,811, Their reported accuracy was 98.53. If it took our baseline ResNet-18 model 51 seconds per training round on SVHN, we can conservatively estimate training rounds = (51 51/3) 50 rounds of training = 34 50 1,700 seconds 28 minutes and 20 seconds.
A.1.2 TABLE IMPORTANCE SAMPLING ESTIMATIONS
The Importance Sampling method proposed by Katharopoulos Fleuret (2018) reported sampling the CIFAR-10 dataset in variable fashion, where each iteration resamples the dataset. For this reason, and because it was not reported, the number of datapoints not calculable. Their graph in their figure 3 approximates the error close to 0.95. This graph reports run time of conservatively 20,000 seconds, which 5 hours and 30 minutes.
A.1.3 TABLE DROPSAMPLE ESTIMATIONS
The Greedy DropSample method proposed by Yang al. (2020) reported stopping the training of a neural network when the "accuracy stays at one for 10 epochs." This work used custom CNNs, rather than out-of-the-box models for the experiments we compare with. For both the MNIST and CIFAR-10 10 experiments, graphs are reported to show values for total number of dropped samples for each round of training a model. Due to the number of training rounds, we are unable to fully estimate number of datapoints for this related work.
For MNIST, their CNN model shown in their figure 6 as a 16-layer convoluational network, consisting of: [convx2 + pool dropout] 2 [conv pool] [dropout + dense] x 3. They trained on MNIST for approximately 800 epochs, according to their figure 7 graph, taking approximately 1,500 seconds, which is 25 minutes. They report an rate at 0.4%, and therefore report 0.995 accuracy for their model.
For CIFAR-10, the model used the DAWN Benchmark winning entry. which consisted of 3-layer CNN with residual connections. They ran this model for 24 epochs, taking conservatively 1,100 seconds based upon their graph in figure 11 this is 18 minutes and 20 seconds. Approximated from the same figure, we estimate error rate at 8%, and therefore report estimated 0.92 accuracy for their model.
A.1.4 TABLE GENERALIZEDD ESTIMATIONS
The GeneralizedDropouu method was proposed by Rahmani & Atia (0018). They ran custom CNN model consisting of 5-five layers: 3 convolutional and fully-connected This model is trained on 15,000 images randomly sampled from MNIST. It is trained for 50 epochs without data dropout, then for 200 epochs "using different basis matrices for the random data dropout," Due to the random data dropping in their proposed method, without reported run time or count of data samples dropped per round, we are unable to estimate these values.


Reviews:
[{"review_id": "zcx1dGSP65", "reviewer": "Reviewer_9Ung", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on the problem of progressive data pruning, in which you attempt to reduce the number of images (in absolute quantity) during training. They do this by subsampling according to class using a dropout score, which is related to the performance of the network on a particular class in the training set. The introduce a hyperparameter (training threshold), which determines when data should be dropped wrt the dropout score. ", "summary_of_the_review": "I vote to reject.\n\nTheir primary experiments are flawed. As is, we cannot to properly assess the efficacy of their method (see Weaknesses for suggestions to improve).\n", "strength_and_weaknesses": "\n##### Strengths\n\n- Method is intuitive and easy to understand.\n- Problem relevance is apparent, reducing training time would have immediate impact.\n\n##### Weaknesses\n\n- Table 3 shows that the author's reimplementation of their baseline was flawed. The results of the existing data dropout paper are reproducible.\n- It's very unclear what Tables 1 & 2 are supposed to convey. There are no comparable comparisons in any of these experiments. The authors should fix an architecture and see how many data points it takes to train to the _same accuracy_. As it is, with variations in architecture, final model performance, and # of data points, we can't draw any conclusions from these tables.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NeTuwMZEau2", "reviewer": "Reviewer_YwiM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel, adaptive strategy to drop classes from the dataset during training based on the f1 score of the model. They show that their approach, Progressive Data Dropout (PDD), is able to achieve similar test/val accuracies while using significantly lower data and time to train on image classification tasks.", "summary_of_the_review": "While the approach considered in the paper is interesting and novel to my knowledge, the results leave much to be desired. The baseline approach considered seems considerably worse than expected, especially compared to the alternatives from the literature. There also seems to be a significant cost to using PDD in the final test/val accuracy of the model.", "strength_and_weaknesses": "# Strengths\n- The paper is very well written and easy to comprehend.\n- The problem tackled by the paper is very interesting and important to the community as it could speed up training significantly.\n- The approach presented is novel to my knowledge and explores an interesting direction that I haven't seen before.\n\n# Weaknesses\n- The results are difficult to parse. It's really hard to compare the different approaches since they are trained for a different number of rounds using potentially different models. \n- There also seems to be a significant cost in accuracy to using PDD.\n- Why do the alternative data-dropping strategies achieve higher test accuracy than even the baseline? It seems like the baseline should be the best approach since it has access to the most data.\n- There should be more experiments with different model architectures.\n- There are no standard errors in the results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Wjzqso_Km", "reviewer": "Reviewer_De4X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a progressive data dropout (PDD) framework that gradually discards the majority of the samples for already-accurate classes, in order to speedup the training. There are other design choices including warmup, residue and swapout. Experiments on MNIST, CIFAR, SVHN and ImageNet are performed.", "summary_of_the_review": "Given the insignificance of the results I recommend rejection. ", "strength_and_weaknesses": "Strengths:\n-- The data dropout idea is well-motivated - time or \"datapoints\" spent on different classes does not have to be equal. Some classes may add little value once they are learned and it makes sense to discard them and save time.\n\n-- The experiments are performed on multiple datasets of different sizes and compared with various previous methods.\n\nWeaknesses:\n-- The results on MNIST, CIFAR and SVHN do not convincingly demonstrate the effectiveness. The time spent is drastically reduced but the accuracy is also hurt, sometimes by large margins. The comparison are done usually without controlling the time or accuracy so it is hard to draw useful conclusions. \n\n-- The baseline ResNet-18 accuracy of CIFAR-10 is at 77%. This should be around 88% instead for ResNet-18 with a reasonable recipe. It is thus hard to judge the reliability of the results given the baseline may be too suboptimal.\n\n-- On the larger scale ImageNet experiments, the saved time is too minimal (less than 1%) when the accuracy is maintained.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-jaBiZfEJss", "reviewer": "Reviewer_tLVt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a technique for removing data points for standard supervised learning problems. Their best performing method partially drops data from a class with a sufficient F1 score, and shows that they can train on ~6M total less data points while only reducing ImageNet accuracy by 1 percentage point. ", "summary_of_the_review": "The paper is well motivated and performs careful ablations, however, the comparison to prior methods was extremely hard to follow and discussion of key related work was missing.", "strength_and_weaknesses": "Strengths: Method is well motivated and an ablation is performed on each of their design decisions.\n\nWeaknesses:\n- I found the comparisons to the baselines to be very unclear. For instance, in Table 1 there are various different architectures, values of epochs, and # of datapoints, such that it is not really clear how to compare properly to baselines. So, while the method is sound, it would be extremely useful to see a comparison where the task, architecture, RT, and datapoints are matched to make a comparison.\n\nComments & Questions:\n- May be good to discuss related work such as [1,2].\n- How would the method be extended to large datasets which are often unlabelled?\n\n[1] https://arxiv.org/abs/2107.07075\n[2] https://arxiv.org/abs/2206.14486", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject