AUTOSAMPLING: SEARCH FOR EFFECTIVE DATA SAMPLING SCHEDULES
Authors: Ming Sun
Haoxuan Dou
Baopu Li
Junjie Yan
Wanli Ouyang
Venue:  (2021)
Keywords: Hyper-parameter Learning
AutoML
Computer Vision

Abstract: Data sampling acts as a pivotal role in training deep learning models. However, an effective sampling schedule is difficult to learn due to its inherent high-dimension as a hyper-parameter. In this paper, we propose the AutoSampling method to automatically learn sampling schedules for model training, which consists of the multi-exploitation step aiming for optimal local sampling schedules and the exploration step for the ideal sampling distribution. More specifically, we achieve sampling schedule search with shortened exploitation cycle to provide enough supervision. In addition, we periodically estimate the sampling distribution from the learned sampling schedules and perturb it to search in the distribution space. The combination of two searches allows us to learn a robust sampling schedule. We apply our AutoSampling method to a variety of image classification tasks illustrating the effectiveness of the proposed method.

Under review as a conference paper at ICLR 2021
SEARCH FOR EFFECTIVE DATA SAM- PLING SCHEDULES
Anonymous authors Paper under double-blind review
ABSTRACT
Data sampling acts as pivotal role in training deep learning models. However, an effective sampling schedule difficult learn due its inherent high- high-dimension as hyper-parameter. In this paper, we propose the AutoSampling method to automatically learn sampling schedules for model training, which consists of the culti-exploitation step aiming for optimal local sampling schedules and the exploration step for the ideal sampling distribution. More specifically, we achieve sampling schedule search with shortened exploitation cycle to provide enough supervision. In addition, we periodically estimate the sampling distribution from the learned sampling schedules and perturb it to search in the distribution space. The combination of two searches allows us to learn a robust sampling schedule. We apply our AutoSampling method to a variety of image classification tasks illustrating the effectiveness of the proposed method.
INTRODUCTION
Data sampling policies can greatly influence the performance of model training in computer vision tasks, and therefore finding robust sampling policies can be important. Handcrafted rules, e.g. data resampling, reweighting, and importance sampling, promote better model performance by adjusting the training data frequency and order (Estabrooks et al., 2004 Weiss al.) 20077 Bengio et al. 2009
Jesson Johnson & Guestrin 2018: Katharopoulos & Fleuret 2018 Shrivastava et al., 2 et al.
2017). Handcrafted rules heavily rely on the assumption over the dataset and cannot adapt well to datasets with their own characteristics. To handle this issue, barning-based methods (Li et al. 2019 Jiang et al. 2017 7 Fan et al. 2017) were designed to automatically reweight or select training data utilizing meta-learning techniques or policy network.
However existing learning-basse sampling methods still rely on human priors as proxies to optimize sampling policies, which may fail in practice. Such priors often include assumptions policy network design for data selection (Fan et al. 2017). or dataset conditions like noisiness (Li et al. 2019)] Loshchilov & Hutter 2015) imbalance (Wang al.) 2019). These approaches take images features, losses, importance or their representations as inputs and use the policy network or other learning approaches with small amount of parameters for estimating the sampling probability. However, for example, images with similar visual features can be redundant in training, but their losses or features fed into the policy network are more likely to be close, causing the same probability to be sampled for redundant samples if we rely on aforementioned priors. Therefore, we propose to directly optimize the sampling schedule itself so that no prior knowledge is required for the dataset. Specifically, the sampling schedule refers to order by which data are selected for the entire training course. In this way, we only rely on data themselves determine the optimal sampling schedule without any prior.
Directly optimizing sampling schedule is challenging due to its inherent high dimension. For example, for the ImageNet classification dataset (Deng et al. 2009) with around one million samples, the dimension of parameters would be in the same order. While popular approaches such as deep reinforcement learning (Cubuk et al. 2018 Zhang et al. 2020), Bayesian optimization (Snock et al. 2015), population- based training (Jaderberg et al. 2017) simple random search (Bergstra & Bengio, like augmentation sched- ules, their upplications in directly finding good sampling schedules remain unexploited. For instance, the dimension a data augmentation policy is generally only in dozens, and it needs thousands of training runs (Cubuk et al. 2018) to sample enough rewards to find an optimal augmentation
2012) have already been utilized to tune low-dimensional

---

Under review as a conference paper at ICLR 2021
policy because high-quality rewards require many epochs of training to obtain. As such, optimizing sampling schedule may require orders of magnitude more rewards than data augmentation to gather and hence training runs, which result in prohibitively slow convergence.
To overcome the aforementioned challenge, we propose a data sampling policy search framework, named AutoSampling, to sufficiently learn an optimal sampling schedule in a ippulation-based training fashion (Jderberg et al.) 2017). Unlike previous methods, which focus on collecting long- term rewards and updating hyper-parameters or agents offline, our AutoSampling method collects rewards online with shortened collection cycle but without priors. Specifically, the AutoSampling collects rewards within several training iterations, tens or hundred times shorter than that in existing works (Ho et al.] [2119] Cubuk et al. (2018). In this manner, we provide the search process with much more frequent feedhack to ensure sufficient optimization of the sampling schedule. Each time when a few training iterations pass, we collect the reward from the previous several iterations, accumulate them and later update the sampling distribution using the rewards. Then we perturb the sampling distribution to search in distribution space, and use it to generate new mini-batches for later iterations, which are recorded into the output sampling schedule. As illustrated in Sec. [4.]] shortened collection eycles with less interference also can better reflect the training value of each data.
Our contributions are as follows:
To our best knowledge, we are the first to propose to directly learn a robust sampling schedule from the data themselves without any human prior or condition on the dataset.
We propose the AutoSampling method to handle the optimization difficulty due to the high dimension of sampling schedules, and efficiently learn a robust sampling schedule through shortened reward collection eycle and online update of the sampling schedule.
Comprehensive experiments CIFAR- 10/100 and ImageNet datasets (Krizhevsky 2009 Deng et al. 2009) with different networks show that the Autosampling can increase the top-1 accuracy by up to 2.85% on CIFAR-10, 2.19% on CIFAR-100, and 2.83% on ImageNet.
BACKGROUND
2.1 RELATED WORK
Data sampling of great significance to deep learning, and has been extensively studied. Approaches with human-designed rules take pre-defined heuristic rules to modify the frequency and order by which training data presented. In particular, one intuitive method is to resample or reweight data according to their frequencies, difficulties or importance in training (Estabrooks et al. 22004]
Weiss et al.) 2007 [Drummond et al., 2203, |Beniio et al.) 2009; Lin et al., 2017; 2017; Shrivastava et al. 2016 Loshchilov & Hutter] 2015 Wang et 9 & Guestrin 2018 Katharopoulos &
Fleuret 2018] |
& Lipton| 2018]Jesson et al. 2017). These methods have been widely used in
imbalanced training or hard mining problems. However, they are often restricted to certain tasks and datasets based on which they are proposed, and their ability to generalize to broader range of tasks with different data distribution may be limited. In another word, these methods often implicitly assume certain conditions on the dataset, such as cleanness or imbalance. In addition, searning-basee methods have been proposed for finding suitable sampling schemes automatically. Methods using meta-learning or reinforcement learning are also utilized to automatically select or reweight data during training (Li et al. 2019 Jiang al. [017 Ren et al. 2018 Fan al. 2017). but they are only tested on small-scale or noisy datasets. Whether or not they can generalize over tasks of other datasets still remain untested. In this work, we directly study the data sampling without any prior, and we also investigate its wide generalization ability across different datasets such as CIFAR- 10, CIFAR- 100 and ImageNet using many typical networks.
As for let al., 2, /Zhng et al. 2020), Bayesian optimization (Snoek et al., 2015) or simply random search (Bergstra& & Bengio, (012) have already been utilized to tune low-dimensional proven to be effective. Nevertheless, they have not been adopted to find good sampling schedule due to its inherent high dimensiona. Some recent works tackle the challenge of optimizing high- dimensional hyper-paramene parameter. MMaKKy et al. (2019) uses structured best-response functions and Jonathan Lorraine (2019) achieve this goal through the combinations of the implicit function theorem and efficient inverse Hessian approximations. However, they have not been tested on the task of optimizing sampling schedules, which is the major focus of our work in this paper.
tuning, popular approaches such as deep reinforcement learning (Cubuk)
paper-parrmeters and
2

---

Under review as a conference paper at ICLR 2021
Malli-Exphitaiin (7 Intervab)
Smooth
Miccure
Warar.
Lufice Stricheto
WakerJ
by
Samplng
Replett
Regiet
Expker
Kalinate
Samplng schedule next multi-sspettation
Recarded Sciidaak
Exploration
Figure 1: Overview of AutoSampling illustrated through one multi-xpploitation and -ndeexrloontion cycle. The multi-exploitation step, illustrated by the left half. the process of learning optimal sampling schedule locally. The same color of model for each worker indicates that the same model weight is cloned into it. Also for simplicity, exploration step, shown by the right half, is to search in the sampling distribution space. Specifically, we estimate the sampling distribution from the schedules collected in the nulti-exploitation step and perturb it to generate new sampling schedules for all workers.
this figure we adopt the exploitation interval of length 1. b) The
2.2 POPULATION BASED TRAINING
Hyper-parameter tuning task can be framed bi-level optimization problem with the following objective function,
min C("", h) NEW
(1)
subject to 0" = arg max eval (f, h) AE
where 8 represents the model weight and h (h1, h2. hr)) is the paper-parameter schedule for T training intervals. Population based training (PBT) (Jaderberg et al. (017)) solves the bi- level optimization problem by training population P of child models in parallel with different hyper-parameter schedules initialized:
== [(0,, 1)
(2)
where 0g, hú respectively represents the child model weight, the corresponding byper-parameter schedule for the training interval t on worker and N, the number of workers. PBT proceeds in intervals, which usually consists of several epochs of training. During the interval, the population of models are trained in parallel to finish the lower-level optimization of weights A,
Between intervals, an exploit-une aadloittandeeaee and-explore procedure is adopted to conduct the upper-level optimiza- tion of the hyper-parameter schedule. In particular for interval exploit we evaluate child models on a held-out validation dataset:
h4 02
arg max eval(0,, h4)
0" - -,, 0,, Np
We record the best performing pppee--rrreeter setting h; and broadcast the top-performing model 0r all workers. To explore, we initialize new hyper-parameter schedules for interval + 1 with different random seeds on all workers, which can be viewed as a search in the hyper-parameter space. The next exploit-and-explore cycle will then be continued. In the end, the top- por-perrorning hyper-parameter schedule h" = (hi, h2j ... hj can be obtained.
PBT is applied to tune low-dimenisal et al. 2019) Jaderberg et al., 2017). However, it cannot be directly used for finding sampling strategies due the high dimension. Unlike PBT, our AutoSampling adopts a multi-exploitation-add exploration structure, leading to much shorter reward collection cycles that contribute to much more and effective rewards for sufficient optimization within a practical computational budget.
papee--arameters such as data augmentation schedules (Ho
AUTOSAMPLING WITH SEARCHING
The overview of our AutoSun is illustrated Fig AutoSampling alternately runs multi- exploitation step and exploration step. In the exploration step, update the sampling distribution
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1: The Multi-Exploisee Step
Input: Training dataset D. population P {(0,, hi, 1) M number of workers N,, number of exploitation intervals T. exploitation interval length N,
Initialize H 5- Q
for = 1 to T do
for j = to N2, do
for (0;, n4, P do
0; + V 17 ha.s)
> update the weight of child model
end for
hū 02 = arg maxp ev, h:)
H' < H + h;
> update the sampling for child model
for i = 1 to Np do
+ 8
> clone the optimal weight
end for end for end for Return H P
using the rewards collected from the multi-xxploitaiion step (the sampling distribution is uniform distribution initially);: 2) perturb the updated sampling distribution for child models so that different child models have different sampling distributions; 3) use the corresponding perturbed sampling distribution for each child model to sample mini-batches of training data. In the multi-xxploitation step, we 1) train multiple child models using the mini-batches sampled from the exploration step; 2) collect short-term rewards from the child models. AutoSampling finishes with recorded top- performing sampling schedule, which can be transferred to other models.
3.1 MULTI-EXPLOITATIIO BY SEARCHING THE DATA SPACE In the multi-exploitation step, we aim to search locally in the data space by collecting short-term rewards and sub-schedules. Specifically, we wish to learn a sampling schedule for T exploitation intervals. In each interval, there are a population P of N, child models. Denote he,; as the training data sub-schedule in the (th interval for the jth child model. When all of the exploitation intervals for the 7th child model are considered, we have H, (hr,e/ l I,..., T) N is the number of training data for the culti--xplooitaiion step. Each interval consists of N, training iterations that is also equivalent N₂ training AutoSamp is expected to produce a sequence of training samples, denoted by H so that a given model is optimally trained. The population (H,) forms the local search space, from which aim to search for an optimal sampling schedule H
[-1, , .NN where
where N. is the length of the interval.
Given the population P, we train them in parallel on N, workers. Once interval of data h containing N₂ training batches have been used for training, we evaluate all child models and use the top evaluation performance as the reward. According to the reward, we record the top-performing weight and sub-schedule for the current interval t, in particular,
hū 8 = arg max eval he,,)) A,,)
(4)
On the other hand, we update all child model weights of P by cloning into them with the top- performing weight 07 so we can continue searching based on the more promising child. We will continue the exploit steps through the whole training process, and output the recorded optimal sampling schedule H = {hi h2g h. By using exploitation interval of mini-batches rather than epochs or even entire training runs adopted by earlier methods, AutoSampling may yield a better and more robust sampling schedule. It should be pointed out that even though in AutoSampling rewards are collected within a much shorter interval, they remain effective. As we directly optimize the sampling schedule, we are concerned with only the data themselves. The short-term rewards reflect the training value of data from the exploitation interval they are collected. But for global hyper- parameters such as augmentation schedules, short-tene term rewards may lead to inferior performance as these hyper- parameters are concerned with the overall training outcome. We describe the multi- exploitation with details in Alg [
4

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Search based AutoSampling
Input: Training dataset D. population size N, Initialize H <- O, P(D) <- uniform(D) and initialize child models 0,, ON, while not end of training do
for 1 to N do
Sample h, from Mixture(log(P(D) + 9), No x uniform(D)
end for
Initialize p = (0,, h4, 1)
H", P &- [
Estimate P(D) according to Equation 5
Update P(D) according Equation
(6)
H &- H + H
end while
Return H", P(D)
3.2 EXPLORATION BY SEARCHING IN SAMPLING DISTRIBUTION SPACE
In the exploration, search in sampling distribution space by updating and perturbing the sampling distribution. We first estimate the underlying sampling distribution P(D) from the top sampling schedule h" produced in the multi-exploitation, that is, for x: E D,
count(x E (H))
P(x) E EVED count(æ H+)
(5)
where count(x H)) denotes the number of a's appearances in H We further perturb the P(D) and generate the sampling schedules on each worker for the later multi-exploitation. We introduce perturbations into the generated schedules by simply sampling from the multinomial distribution P(D) using different random seeds. However, in our experiments, we observe that the distribution produced by PDD) tends to be extremely skewed and a majority of the data actually have zero frequencies. Such skewness causes highly imbalanced training mini-batcher and therefore destabilizes subsequent model training.
Distribution Smoothing To tackle the above issue, we first smooth P(D) through the logarithmic function, and then apply probability mixture with uniform distributions. particular for the dataset D,
PD) Mixturr((og(P((D) + 3), Nr x uniform(D))
(6)
where 3 > 1 is the smoothing factor and N₂ x uniform(D) D) denotes N₂ uniform multinomial distributions on the dataset D. The smoothing through the log function can greatly reduce the skewness, however, log( (DD) + 3) may still contain zero probabilities for some training data, resulting unstable training. Therefore, we further smooth it through a probability mixture with N, uniform distribution uniform(D) () to ensure presence of all data. This is equivalent to combining N, epochs of training data to the training batches sampled from P(D), and shuffling the union. Once we have new diverse sampling schedules for the population, we proceed to the next multi-exploitation step.
We continue this alternation between multi- multi-exploitation and exploration steps until the end of training. Note that to generate sampling schedule for the first multi-cxplootation run, we initialize P(D) to be an uniform multinomial distribution. In the end, output a sequence of optimal sampling schedules H" = (H), H)) for n alternations. The entire process is illustrated details in Alg
4 EXPERIMENTS
In this section, we present comprehensive experiments on various datasets to illustrate the perfor- mance of AutoSampling, and also demonstrate the process of progressively learning better sampling distribution.
4.1 ABLATION STUDY
For this part, gradually build up and test components of AutoSampling CIFAR- 100, and then examine their performances on CIFAR-10 10 and ImageNet datasets. The training implementation details and computational complexity can be found in Appendix
A.1 |
5

---

Under review as a conference paper at ICLR 2021
Table 1: Performance CIFAR-100 using different configurations of AutoSampling and baselines.
Worker is the number of workers used and Interval the exploitation interval in terms of batches.
NETWORK
WORKER
INTERVAL
EXPLORATION TYPE
TOPI(%)
RESNET18
ZHANG ET AL.
2019)
7..34=0.05
RESNET18
1
UNIFORM
78.46=00035
RESNE118
20
80 BATCHES
RANDOM
78.76=0.000
RESNET18
20
20 BATCHES
RANDOM
78.99:10.003
RESNET18
80
20 BATCHES
RANDOM
79.09=10.017
RESNET18
20
20 BATCHES
MIXTURE
79.44-0.020
RESNET5O (JIN ET AL. 2019
79.34
RESNET50
1
UNIFORM
79.70+t0.022
RESNET50
20
80 BATCHES
RANDOM
80.55+0.122
RESNET50
20
20 BATCHES
RANDOM
81.05
RESNET50
80
20 BATCHES
RANDOM
81.19-0.077
RESNET50
20
20 BATCHES
MIXTURE
81.53 0.88
DENSENET121
UNIFORM
80.13+0.002
DENSENET121
20
BATCHES
RANDOM
80.62:10.694
DENSENET121
20
20 BATCHES
RANDOM
81 11.1110.117
DENSENET121
80
20 BATCHES
RANDOM
81.0810.00
DENSENET121
20
20 BATCHES
MIXTURE
80 80.97=0.006
Table 2: Experiments on CIFAR-10.
Table 3: Experiments on ImageNet.
NETWORK
EXPLORATION TYPE
ToP1(%)
NETWORK
EXPLORATION TYPE
Topl (%)
RESNET 18
UNIFORM
93.01100009
RESNET 88
UNIFORM
70.38
RESNET 18
RANDOM
95.86_10.003
RESNET1
RANDOM
72.07
RESNET18
MIXTURE
95.80==0.018
RESNET18
MIXTURE
72.91
RESNET5O
UNIFORM
93.60110.004
RESNET34
UNIFORM
74.09
RESNET50
RANDOM
96.1011.0002
RESNET34
RANDOM
76.
RESNETSO
MIXTURE
96.09110.070
RESNET34
MIXTURE
76.92
Adding Workers To look into the influence of the worker numbers, we conduct experiments using worker numbers of 1, 20, 80 respectively with the same setting (N, = 20 with random exploration). With the worker number of I, the experiment is simply the normal model training using stochastic gradient descent. To show the competitiveness of our baselines, we also include state-of of the-art results on CIFAR-100 with ResNet-18 18 and ResNet-50 (Zhang et al.) 2019: Jin et al. 2019). We notice significant performance gain using the worker number of 20 ResNet- 18, ResNet-50 and note that
DenseNet- 121 (He et al. 2015 Huang et al., (017), as illustrated in Table I However, increasing worker number from 20 to 80 only brings marginal performance gains across various model structures, shown in Table 1 Therefore, set the worker number to be 20 for the rest of the experiments.
Shortening Exploitation Intervals To study the effects of the shortened exploitation interval, run experiments using different exploitation intervals of 20 and 80 batches(iteration)) respectively. As shown in Ta models with the shorter exploitation interval of 20 batches(iterations) perform better than the one with the longer exploitation interval across all three network structures, conforming to our assumptions that the reward collected reflects value of each data used in the exploitation interval. This result adheres to our intuition that shorter exploitation interval can encourage the sampler to accumulate more rewards to learn better sampling schedules. For the rest of this section we keep the exploitation interval of 20.
Adding Exploration Type We further add mixture as the exploration type to see the effects of learning the underlying sampling distribution, and completing the proposed method. As shown in Table I with ResNet-18 and ResNet-50 we push performance higher with the mixture exploration, and outperform the baseline method by about and 1.8 percentage CIFAR-100 respectively. However, we found that it is not true in the case of DenseNet-121 and this case may be attributed to the bigger capacity of DenseNet-12 121.
Generalization Over Datasets In addition, we experiment on other datasets. We report the results on CIFAR10 in Table and the results of ResNet-18. ResNet-34 on ImageNet in Table 3 For CIFAR-10. we notice that the mixture and random exploration methods are comparable while both outperforming the uniform baseline, and we believe is due the simplicity of the dataset. In the more challenging

---

Under review as a conference paper at ICLR 2021
Spcch-05
220
330
20068
Figure 2: The comparison between histograms estimated from the sampling schedules of Epoch 80, 160 and 240 from CIFAR- 100 with 18. We divide the 50000 training images into 500 segments of 100 images, and calculate the histograms of total data counts of all segments. We reorder the a-axis based on the ranking of data counts for epoch 240 for easier comparison.
Table 4: Static vs dynamic sampling schedule on CIFAR-100 (%)
NETWORK
SAMPLING TYPE
UNIFORM
STATIC
DYNAMIC
RESNET18 78.46110.035
78.8010.0077
79.44=0.020
RESNET50
79.70110.023
80.21 81.53
ImageNet, the mixture exploration outperforms the random exploration by clear margin. We also compare our AutoSampling with some recent non-uniform sampling methods on CIFAR-100, which can be found in Appendix A.2
4.2 STATIC VS DYNAMIC SCHEDULES
We aim to see the final sampling distribution estimated by our AutoSampling is sufficient produce robust sampling schedules. In another word, we wish to know training with the AutoSampling is either process of learning robust sampling distribution, or a process of dynamically adjusting the sampling schedule for optimal training. To this end, we conduct training using different sampling schedules. First, we calculate the sampling distribution estimated throughout the learning steps of AutoSampling, and use it to generate the sampling schedule of full training process, which denote as STATIC. Moreover, we denote the sampling schedule learned using AutoSampling as DYNAMIC, since AutoSampling dynamically adjust the sampling schedule alongside the training process. Finally, we denote the baseline method as UNIFORM, which uses the sampling schedule generated from uniform distribution.
We report results on CIFAR-100 with ResNet-18 and ResNet-50 Table Model trained with STATIC sampling schedules exceeds the baseline UNIFORM significantly, indicating the superiority of the learned sampling distribution over the uniform distribution. shows the ability of AutoSampling learn good sampling distribution. Nonetheless, note that models trained with DYNAMIC sampling schedules outperform models trained with STATIC, by margin bigger than the one between STATIC and UNIFORM. This result shows the fact that despite the AutoSampling's capability of learning good sampling distribution, its flexibility during training matters even more. Moreover, this phenomenon also indicates that models at different stages of learning process may require different sampling distributions to achieve optimal training. One single sampling distribution, using AutoSampling, seems incapable of covering the needs from different learning stages. We plot the histograms of data counts in training estimated from schedules of different learning stages with ResNet-18 18 on CIFAR-100 in Fig showing the great differences between optimized sampling distributions from different epochs.
gradually estimated
4.3 ANALYZING SAMPLING SCHEDULES LEARNED BY AUTOSAMPLING
To further investigate the sampling schedule learned by AutoSampling, review the images the tail and head part the sampling spectrum. In particular, given sampling schedule learned we rank all images based on their appearances in training. Training images at the top and bottom of the order are extracted, corresponding to high and low probabilities of being sampled respectively. Fig B we show 4 classes of exemplary images. The images of low probability tend to have clearer imagery
7

---

Under review as a conference paper at ICLR 2021
Low Probability
High Probability
Baby
Bike
Chimp
Camal
Example images the head and tail of the sampling spectrum. The images the left the ones with low sampling probability, while the images on the right more likely to be sampled. We obtain these images using AutoSampling with the ResNet-18 model on CIFAR-100.
Figure 3:
Table 5: Transfer of sampling distributions learned by three model structures to ResNet-50 on CIFAR-100 (%). UNIFORM denotes the baseline result using uniform sampling distribution.
NETWORK
SAMPLING SCHEDULE SOURCE
UNIFORM
RESNETI8
RESNET30
DENSENETI21
RESNET50
79.70:10.023
80.27100014
80.2110.014
80.47:10.14
features enabling easy recognition, while the images of high probability tend to be more obscure, indicating that the sampling schedule may show hard samples mining effects. However, as shown in A.3]and and Fig. [ the loss values and probabilities of being sampled seem to not highly correlated, which indicates more potential of AutoSampling beyond visually hard example mining. In addition, we notice the images of low probability also contain low quality images. For instance, in Fig 3 the leftmost image of CAMAL class contains only legs. This shows that AutoSampling may potentially rule out problematic training data for better training.
A.3
Furthermore, examine the transfer ability of sampling distributions learned by AutoSampling to other network structures. Specifically, we run training on ResNer-50 (He et al. 2015) using STATIC sampling schedule generated by three distributions learned by AutoSampling
3 different models. As shown in Table [ using sampling schedules learned by AutoSampling from other models, we demonstrate similar improvements over the UNIFORM baseline. This result, combined with the above observations on images of different sampling probability, indicates that there may exist common optimal sampling schedule determined by the intrinsic property of the data rather than the model being optimized. Our AutoSamp is an effort to gradually converge to such an optimal schedule.
4.4 DISCUSSIONS
The experimental results and observations from Section 4.2 and 4.3 shed light the possible existence of an optimal sampling schedule, which relies only on the intrinsic property of the data and the learning stage of the model, regardless of the specific model structure or any prior knowledge. The learned sampling schedule may provide enough rewards in the searching process, leading to sufficient convergence compared other related works. Once obtained, the optimal sampling schedule may also be generalized over other model structures for robust training. Although AutoSampling requires relatively large amount of computing resources find robust sampler, want to point out that the efficiency of our method can be improved through better training techniques. Moreover, the possibility of an optimal sampling schedule relying solely the data themselves may indicate more efficient sampling policy search algorithms, if one can quickly and effectively determine data value based on its property.
CONCLUSIONS
In this paper, we introduce a new search based AutoSampling scheme to overcome the issue of insufficient rewards for optimizing high-dimensional sampling hyper-parameter by utilizing a shorter period of reward collection. We use a shortened exploitation interval to search the local data space and provide sufficient rewards. For the exploration step, we estimate sampling distribution from the searched sampling schedule and perturb it to search the distribution space. We test our method and it consistently outperforms the baseline methods across different benchmarks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Anmual International Conference on Machine Learning, ICML '09, pp. 41-48, New York. NY, USA, 2009, Association for Computing Machinery, ISBN 166568555.. doi: 10.1145/15533741555333
URL.ttts: /ttp:://ee..orr/10... 1ee.ee111511113331 1553380
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Mach. Learn. Res., 13(mull):2113300 February 2012. ISSN 1532-4435.
Jonathon Byrd and Zachary C. Lipton. Weighted risk minimization & deep learning. CoRR, abs/1812.033 2018. URL /
03372
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation policies from data. CoRR,
2018. URL http://arxx
orq/ 1805. 09501
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Chris Drummond, Robert C Holte, al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-sampling. In Workshop on learning from imbalanced datasets II, volume 11, pp. 1-8. Citeseer, 2003.
Andrew Estabrooks, Taeho Jo, and Nathalie Japkowiez. A multiple resampling method for learning from imbalanced data sets. Computational intelligence, 20(1):18-36, 2004.
Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Tie-Yan Liu. Learning what data to learn. CoRR, abs/1702.08635, 2017. URL
org/ abs aww.a////00 08635
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URLpttt:///axix wttp://wwrrr..aa........
Daniel Ho, Eric Liang, lon Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. CoRR, abs/1905.05393, 2019. http:
//arxiv. org/ abs 1905 05393
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, 0bs///111.99444 2017.
URL httttt//p//xxxi org/ abs/ / 711. 09846
Andrew Jesson, Nicolas Guizard, Sina Hamidi Ghalehjegh, Damien Goblot, Florian Soudan, and Nicolas Chapados. Cased: Curriculum adaptive sampling for extreme data imbalance. In Maxime Descoteaux, Lena Maier-Hein, Alfred Franz, Pierre Jannin, D.. Louis Collins, and Simon Duchesne (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2017, pp. 639-646, Cham, 2017. Springer International Publishing. ISBN 978-3-319-66179.1
Lu Jiang, Zhengyuan Zhou, Thomas Leung. Li-Jia Li, and Li Fei-Fei. Mentornet: Regularizing very deep neural networks on corrupted labels. CoRR, abs/ 151.005555 2017. URL http:
7/arxiv.org) abss
abs /tobss/1111
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, approximate im- portance sampling. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems pp. 7265-7275. Curran Associates, Inc., 2018. URLpttte /ttt::t//ppprrre nips httpps.cccppppe/799555

---

Under review as a conference paper at ICLR 2021
David Duvenaud Jonathan Lorraine, Paul Vicol. Optimizing millions of hyperparameters by implicit differentiation. proceedings of AISATS, 11, 2019.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. CoRR, abs//803. 000.00803,, 2018. URL http::/axxi....g///ab////////
00942
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Zhenmao Li, Yichao Wu, Ken Chen, Yudong Wu, Shunfeng Zhou, Jiaheng Liu, and Junjie Yan. LAW: learning to auto weight. CoRR, 1ts/1995.11058, 2019. URL
11058
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Ilya Loshchilov and Frank Hutter. Online batch selection faster training neural networks. CoRR, abs/151 11.511.06443 2015. URL htttht/p:///aiiii http:aawwwwooogaaaaaaoaaaaaaaaas abs/asssl111 006343
Matthew MacKay, Paul Vicol, Jon Lorraine, and Roger Grosse David Duvenaud. Self-tuming networks: Bilevel optimization of hyperparameters using structured restonee functions. proceedings of ICLR, abs/1903.03088, 2019. /ttp://////a http://www..aaab/////.aaa 03088
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. CoRR, abs/1803.09050, 2018. URL http::/arxiv..r /ww.a////rrgaa//oaoooo/oooaa// 1803
09C50
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2171-2180, Lille, France, 07-09 Jul 2015. PMLR. URL http::/ www.press..v......
ml
snoek15. html]
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Gary M Weiss, Kate McCarthy, and Bibi Zabar. Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs? Dmin, 7(35-41):24, 2007.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146 2016. URL
Michael R. Zhang, James Lucas, Geoffrey F. Hinton, and Jimmy Ba. Lookahead optimierr: k steps forward, step back. CoRR, abs/1907.08610, 2019. URL / 08610
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. In International Conference on Learning Representations, 2020. URL
ew. www.:rrwwwwww..eee.......e..
id By-ByxUUSSkvS)
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Experiments on CIFAR We use the same training configuration for both CIFAR-I00 and CIFAR-IO 10 datascts, which both consist of 50000 training images. In particular, for model training use the base learning rate of 0.1 and a step decay learning rate schedule where the learning rate is divided by 10 after each 60 epochs. We run the experiments for 240 epochs. In addition, set the training batch size to be 128 per worker, and each worker is for one Nvidia V100 GPU card.
We run the explore step for each N₂ + epochs with N, 3, but note that take the first explore step after the initial 20 epochs to better accumulate enough rewards. The experiments require 4800 epochs of training for 20 workers, and roughly 14 hours of training time.
Experiments on ImageNet For ImageNet which consists of 1.28 million training images, we adopted the base learning rate of 0.2 and a cosine decay learning rate schedule. We run the experiments with 100 epochs of training. For each worker we utilize eight Nvidia V100 GPU cards and a total batch size of 512. Eight workers are used for all ImageNet experiments, and the rest of the setting adheres to that of CIFAR experiments. In addition, we utilize FP16 computation to achieve faster training, which has almost no drop in accuracy in practice. The experiments require 800 epochs of training for 8 workers, and roughly 4 days of training time.
A.2 COMPARISON WITH EXISTING SAMPLING METHODS
To better illustrate the effectiveness of our AutoSampling method, we conduct experiments in comparison with recent non-uniform sampling methods DLIS (Johnson & Guestrin 2018) and RAIS (Katharopollos & Fleuret 2018). DLIS (Johnson & Guestrin 20188) achieves faster conver- gence by selecting data reducing gradient norm variance, while RAIS (Katharopoulos & Fleuret 2018) does through approximating the ideal sampling distribution using robust optimization. The comparison is recorded in Table 6
First, run AutoSamp using Wide Resnet-28-2 (Zagoruyko & Komodakis 2016 CIFAR-100
with the training setting aligned roughly to (Katharopoulos & Fleuret 2018). AutoSamp achievs
improvement of roughly 3 percentage points (73.37+1.09% 76.224 1.0%%. while Katharopoulos
& Fleuret shows improvement of 2 percentage points (66.0% - 68.0 %). Second, we report the comparison between AutoSampling and RAIS on CIFAR-100, Johnson & Guestrin shows no improvement (76.4%- + 76.4 %) on accuracy and 0.027 (0.989 + 0.962 decrease in validation loss, while our method shows improvement of 0.008 (78.6% - 79.4%) on accuracy and 0.014 (0.886 - 0.872 ) decrease in validation loss. As such, our method demonstrates significant improvements over existing non-uniform sampling methods.
Table 6: Comparisons between AutoSampling and existing sampling methods on CIFAR-100
Methods
Network
Baseline (%)
With method (%)
Improvement (%)
DLIS
WRN-28-2
66.0
68.0
2.0
AutoSampling (ours)
WRN-28-2
73.37+1.09
76.24+ 1.02
2.87
RAIS
ResNet18
76.4
76.4
0.0
AutoSampling (ours)
ResNet18
78.46-t0.035
79.44:10.000
0.98
A.3 COMPARISON BETWEEN LEARNED SAMPLING SCHEDULES AND DATA LOSS VALUES
To further interpret the learned sampling schedules, we compare the sampling frequency of each training image and its loss values in different epochs during training of C1F with ResNet- 18. We draw the comparison for randomly selected 500 training images in Fig. for epoch 80, 160, and 240. As shown in the figure, across different learning stages, the correlation between loss values and sampling frequencies of training data is not obvious. The high chance of being sampled by AutoSampling does not necessarily lead to high loss values, which demonstrates that AutoSampling
11

---

Under review as a conference paper at ICLR 2021
Epoch 80
284
Epoch 160
144
23
Epoch 240
290
300
Figure 4: The comparison between the sampling frequency of each training image and its loss values of Epoch 80, 160 and 240 from CIFAR-100 with ResNet-18. We randomly selected 500 training images, and calculate their sampling frequency and loss values. The x-axis is the indexes of 500 training images, while the left y-axis denotes loss values and the right y-axis denotes the sampling frequency. The blue line represents the sampling frequencies and the red lines represents the loss values of all 500 images. As we can see from the figure, the two lines are not obviously correlated.
merely over-sampling diffieult samples pointed by the loss. The resulting sampling schedule learned by AutoSampling would be significantly different from the one guided by loss. Moreover as the training progresses the loss values of data reduced, which is expected.
12


Reviews:
[{"review_id": "SHzPcO5KVp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents an interesting exploration of learning optimal data sampling probability. It is formulated with a population based training strategy. To this end, authors propose to record the data frequencies through all previous steps and thereby generate the sampling policy for the next step.\n\nIt has been a recent trend to make almost all key factors of deep networks learnable, such as differentiable data augmentation. This work is taking a novel aspect for effectively training neural networks, since rare effort has been devoted to make data sampling learnable. Authors frame the task in PBT and take incremental modification to PBT to adapt to the new problem. I have no complaint regarding the task per se. However, the empirical validation seems finished in rush and not sufficient. \n\nThe chosen baselines are essentially standard sampling scheme or variants of the proposed method. Authors should compare with a few state-of-the-art data-sampling or data-reweighting methods, such as Focal Loss proposed by He et al. In addition, Tables 2 and 3 should be clarified in the rebuttal. It seems that very small margins are observed in comparison with random exploration. The follow-up analysis attributes the success to the emphasis of hard examples during training, which is consistent to previous common practice. However, authors are encouraged to go deeper for the analysis, beyond the selected illustrative samples in Figure 3. For example, it may be meaningful to check the alignment of data weights and classification scores.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HcUj9JCXDX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors mainly concentrate on data sampling. To address the issue of optimizing high-dimensional sampling hyper-parameter in data sampling and release the requirement of prior knowledge from current methods, the authors introduce a searching-based method named AutoSampling. This method is comprised of exploration step and exploitation step which are conducted alternatively. The exploitation step train multi child models with current sampling strategy and save the best model for next iteration. while the exploration step estimates the sampling distribution according to the sampled data in exploitation step and rectifies it to sample all data possibly. The authors have conducted sufficient experiments to verify the superior of their method, especially for the effectiveness and generalizability. \n\nAdvantages:\nl\tThe exploitation step and exploration step in AutoSampling is interesting, it is straightforward that this method can work well as the sampling strategy is updated dynamically according to the current state of model.\nl\tThe proposed AutoSampling is simple and effective, one can implement it without much effort.\nl\tThis method owns great generalizability and does not require any knowledge prior.\nl\tThis paper is well organized and written.\n\nDisadvantages:\nl\tIn Table 1, the number of workers does have an influence on performance, and this influence is positively correlated in my opinion, however, we can see a performance degradation for DenseNet121. The authors did not explain this.\nl\tThe transferability of the gained optimal sampling schedule is discussed in Section 4.4, a simple experiment is recommended. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-uekHB7HwvC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses an approach to perform importance sampling by reviewing performance over past batches and suggesting future batches.\n\nFirst I will comment on the listed contributions:\n\n> • To our best knowledge, we are the first to propose to directly learn a robust sampling\nschedule from the data themselves without any human prior or condition on the dataset.\n\nFrom this description I believe this paper has done it before: \"CASED: Curriculum Adaptive Sampling for Extreme Data Imbalance\". \n\nThe paper seems to go beyond this and focuses on multi node training so this may be a way to refocus the paper in contrast to the existing work.\n\n> • We propose the AutoSampling method to handle the optimization difficulty due to the high\ndimension of sampling schedules, and efficiently learn a robust sampling schedule through\nshortened reward collection cycle and online update of the sampling schedule.\n\nIt is not clear to me what the intuition of P(x) is. If H* is the \"optimal sampling schedule\" and then P is defined to be the ratio of x in H* over all x in H* then it is not clear to me what you are converging to. It would be better for the reader to make this as easy to understand as possible. Also, I think this would be the place where this approach is different from the CASED approach.\n\nOn the experiments to demonstrate that this method is better: The replicate runs seem to almost have no variance. It is the case that only the model init was varied between runs with fixed train/valid/test sets? If so this does not seem sufficient to confirm that this method works because the data you sample from is always the same. This should be randomized at least for CIFAR.\n\nTypo: \"Algorithm 2: Search based AuoSampling\" \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work focuses on a new method for sampling hyper-parameter based on an "Population-Based Training" schedule that tend to sample more often configurations that gave good performances in the past. The authors have conducted experiments to verify the superior of their method, especially for the effectiveness and generalisability.

Pros:
- simple method that can be implemented without much effort,
- good empirical performances on Imagenet,
- paper well organised and written.

Cons:
- lack of explanation about the DensNet121 performance degradation [partially addressed in the rebuttal],
- additional simple experiments in Section 4.4 were recommended to evaluate the generality of the method [addressed in Table 5],
- empirical validation seems not sufficient [partially addressed in the rebuttal],
- similarity with respect to prior art, such as the focal loss [partially addressed in the rebuttal],
- clarification of the randomisation strategy in experiments [addressed in the rebuttal].

Despite most of the issues being addressed, the reviewers decided that this paper would benefit more work to be accepted for the conference this year.
Decision: Reject