Training Data Distribution Search with Ensemble Active Learning
Authors: Kashyap Chitta
Jose M. Alvarez
Elmar Haussmann
Clement Farabet
Venue:  (2020)

Abstract: Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, it is conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimization. Modifying the training distribution in a way that excludes such samples could provide an effective solution to both improve performance and reduce training time. In this paper, we propose to scale up ensemble Active Learning methods to perform acquisition at a large scale (10k to 500k samples at a time). We do this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us to automatically and efficiently perform a training data distribution search for large labeled datasets. We observe that our approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We perform an extensive experimental study of this phenomenon on three image classification benchmarks (CIFAR-10, CIFAR-100 and ImageNet), analyzing the impact of initialization schemes, acquisition functions and ensemble configurations. We demonstrate that data subsets identified with a lightweight ResNet-18 ensemble remain effective when used to train deep models like ResNet-101 and DenseNet-121. Our results provide strong empirical evidence that optimizing the training data distribution can provide significant benefits on large scale vision tasks.

Under review as a conference paper at ICLR 2020
TRAINING DATA SUBSET SEARCH WITH ENSEMBLE ACTIVE LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimiza- tion. Modifying the training distribution in a way that excludes such samples could provide an effective solution to both improve performance and reduce train- ing time. In this paper, we propose to scale up ensemble Active Learning methods large scale (10k 500k samples at a time). We do
perform acquisition this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us automatically and efficiently perform a training data subset search for large labeled datasets. We observe that approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We per- form an extensive experimental study of this phenomenon on three image classifi- cation benchmarks (CIFAR-10 10, CIFAR-100 and ImageNet), analyzing the impact of initialization schemes, acquisition functions and ensemble configurations. We demonstrate that data subsets identified with lightweight ResNet-18 18 ensemble remain effective when used to train deep models like ResNet-101 and DenseNet- 121. Our results provide strong empirical evidence that optimizing the training data distribution can provide significant benefits on large scale vision tasks.
INTRODUCTION
Deep Neural Networks (DNNs) have become the dominant approach for addressing supervised learning problems. They are trained using stochastic gradient methods, where (i) data is subsampled into mini-batches, and (ii) the network parameters are iteratively updated by the gradient of the pa- rameter weights, relative to a loss function for the given labeled mini-batch of data. The architecture of the DNN, hyper-parametes of the training process, and distribution of the dataset used are all crucial ingredients that impact the final performance of the DNN.
Given the remarkable success of this recipe for supervised learning, there is significant interest in automating the end-to-end process of applying DNNs to real-world problems (Wong et al.) 2018 He et al.) 2018 2019). While there has been a considerable effort towards methods and frameworks that automate DNN architecture search (Elsken et al. 2018 Luo al. 2018 Liu al. 2018 Xie ctt al. 2019 Dong & Yang, 2019) and training hyper- parameerreeer search Golovin et al. 2017 Mutny & Krause 2018. Kandasamy al. 2019); the process of searching for the right training data dis- tribution (also called dataset curation) is still performed by experts, requiring several heuristics and significant manual effort. With the rapid growth in the availability of labeled data for large-scale vi- sion tasks, to the order of billions of samples (Sun et al. 2017 Zeki Yalniz et al. 2019). automating the training data subsel search would make the application of DNNs much easier for non-experts, and potentially lead to datasets and models that outperform those that were curated by hand. With the cost of data storage also becoming increasingly important, the ability to automatically identify data to remove from the training distribution is appealing from practical perspective.
In this paper, we present a simple yet effective method to perform a training data subset search by using ensemble Active Learning (AL). The typical goal of AL is to select, from a large unlabeled dataset, the smallest possible training set to label in order to solve a specific task (Cohn et al.) 1994). We instead propose use AL to build data subxets of large labeled training dataset that give more

---

Under review as a conference paper at ICLR 2020
accurate DNNs in less training time. We tackle several key issues that have not been addressed so far by state-of-the- art AL methods. The first is the difficulty in scaling the mumber of models for ensemble AL. While it seems intuitive that more ensembles
improve performance, existing studies show no gains in AL performance beyond 10 models, and even recommend the use of only 5 models (Lakshminarayanan
al. 2017; Ovadia et al., 2019). Implicit ensemble methods for AL have been found similarly ineffective, but only evaluated in settings with with < 10 models (Beluch al., 2018). In this study, we propose the use of implicit ensembles with hundreds of training checkpoints from different experimental runs, and empirically demonstrate the effectiveness of this approach. Second, we switch a large-scale experimental setting compared to what is typically used for AL experiments. For example, [Beluch et al. (2018) and |iihaa et al. (2019) only acquire 40k and 64k samples at a time respectively on ImageNet, never use more than 30% the dataset, and do not compare to the full dataset performance. In contrast, for ResNet- 18 training on ImageNet, we acquire up to 500k samples at a time and use 80% of the dataset, improving top-1 accuracy by 0.5% over model trained with the entire dataset. Third, demonstrate the transferabiliiy of the subsets obtained with our approach between DNNs with very different network capacities, ranging from 10-layer ResNets (He et al., 2016) 121-layer DenseNets (Huang al. 2016). Given the rapid pace of model development, this shows that our data subsets remain valuable even after a particular model is surpassed by newer architectures. This has been difficult to achieve in prior AL studies (Lwelll et al. 2018). and not investigated in detail for large-scale vision tasks.
To summarize, our contributions are as follows: (i) we propose a simple approach scale up en- semble AL methods to hundreds of models with negligible computational overhead at train time; (ii) we evaluate several methods reduce the size of existing large datasets with AL; and (ii) we conduct a detailed empirical study on three popular image classification benchmarks, studying the impact of key design choices, and the robustness of the selected subsets to changes in model archi- tecture. Our study has significant practical implications for DNN training with stochastic gradient methods, as well as for storage of large training datasets.
2 BUILDING DATA SUBSETS WITH ACTIVE LEARNING
Consider a distribution p(x, y) over inputs x and labels y. In Bayesian framework, the predic- tive uncertainty of a particular input x* after training on dataset L is denoted as P(y k|x", L). The predictive uncertainty will result from data (aleatoric) uncertainty and model (epistemic) un- certainty (Kendall & Gal 2017). A model's estimates of data uncertainty are described by the posterior distribution over class labels given a set of model parameters 0. This is typically the soft- max output in classification DNN. Additionally, the model uncertainty is described by the posterior distribution over the parameters 0 given the training data L (Malinin & Gales 2018):
Ply = K(x,
Ply = k/xx 0) (O
Predictive
Data
Model
We see that, uncertainty in the model parameters L) induces a distribution over the softmax distributions P(y k|x", .0). The expectation is obtained by marginalizing out the parameters 0. Unfortunately, obtaining the full posterior T()]..) using Bayes' rule is intractable. If we train a single DNN, we only obtain single sample from the distribution L)) Ensemble uncertainty estimation techniques approximate the integral from Eq. by Monte Carlo estimation, generating multiple samples using different members of an ensemble (Lakshminarayanan et al. 2017):
P(y Ki=
R E
Σ P(y==ki',00 0g q()) EEE
(2)
where (0) represents approach used for building the ensemble. The strength of the ensemble approximation depends on both the number of samples drawn (E), and how the parameters for each model in the ensemble are sampled (i.e., how closely the distribution q()) matches T.)...)
2

---

Under review as a conference paper at ICLR 2020
Algorithm Building Data Subsets with Active Learning.
Initialize parameters of acquisition model (01")) Je=1 and subset model 0, Initialize data subset S
initialization
Compute acquisition function axx, MM) x L
b acquisition
Append N₂ samples with the maximum acquisition function to S Use to optimize parameters 0, of subset model
> optimization
Uncertainty estimation acts as the core component of our approach to build data subsets, for which the pseudo code is presented in Algorithm | The algorithm is a modification of the typical AL. setting, and involves the following:
1. A labeled dataset, consisting of N₁ labeled pairs, = ((x) 1) where each x X
is a data point and each y² E Y is its corresponding label.
2. An acquisition model, Ma X + Y. For our ensemble AL approach, the acquisition model M, takes the form of set of E different DNNs with parameters { Temt-
3. A data subset, S = {x2), 81) where is a subset of L selected using an acquisition function ((x.MM,).
4. A subset model, M₁ :X + Y, with parameters 0z. trained on S.
As detailed in Algorithm|] I we decouple the data selection and final optimization of AL into two dif- ferent models. The acquisition model selects a subset data using ensemble uncertainty estimation, which is then used optimize the parameters of the subset model.
2.1 INITIALIZATION SCHEMES
We consider three different initialization schemes for the parameters of the acquisition and subset models: pretrain, compress and build up. The pretrain scheme uses the entire dataset L for pretrain- ing both the acquisition and subset models. During optimization, the subset model is then finetuned on the data subset S. In the compress scheme, the acquisition model is pretrained on L but the subset model is randomly initialized and trained from scratch on S. The acquisition model therefore accesses all the data and then 'compresses' the dataset for the subset model. For the pretrain and compress schemes, the subset initialized with an empty set.
Finally, in the build up scheme, we follow an iterative AL loop. Specifically, we start by initializing with randomly selected subset of the data to train an acquisition model. After performing acquisition, instead of training single subset model, we optimize an ensemble of E networks. This ensemble used as an acquísition model for a subsequent iteration. Our goal is to finally reach a subset of N, samples. As observed by Chitta et al. (2018a), exponentially growing the dataset size offers practical benefits in an AL loop setting. We therefore follow this approach, by initializing with N. random samples, and iterating two further times at 4 and N, samples before obtaining 8
2
final subset of size N..
2.2 ACQUISITION FUNCTIONS
In our experiments, we empirically evaluate four acquisition functions of the form :x-xx RR for AI- entropy, mutual information, variation ratios and error count. We choose these well-known acquisition functions to maintain simplicity and scalability. For detailed theoretical analysis of these acquisition functions, we refer the reader to Gal (2016).
Bayesian AL approaches argue that the acquisition function must target samples with high model uncertainty rather than data uncertainty. Intuitively, the reasoning behind this that model uncer- tainty is result of not knowing the correct parameters. which can be reduced by adding the right data and retraining the model. However, data uncertainty exists even for a model with the most optimal parameters, and cannot be reduced by adding more data (Kendall & Gal 2017).
3

---

Under review as a conference paper at ICLR 2020
In the of classification, the predictive uncertainty for sample P(y k//x, L) from Eq. | is multinomial distribution, which can be represented as a vector of probabilities over each the K
classes. We can obtain the predictive uncertainty for sample as its entropy (Shannon 1948
H(P)
p" log p.
However, once we marginalize out 0 from Eq. D impossible to tell if this predictive uncertainty result model uncertainty data uncertainty. One acquisition function that explicitly looks for large disagreement between the models (i.e., model uncertainty), is mutual information (Houlsby) al. 2011 Smith & Gal 2018). also called BALD and -heenShnnnon Divergence:
J()
HP))
Σ H(D CEE
(4)
Model
Predictive
Data
where p(e) denotes the prediction of an individual member of the ensemble before marginalization. Since entropy is always positive, the maximum possible value for J(p) is H(p). However, when the models make similar predictions, 1 Σe= H(p,) + M(p), and J(p) - O, which is its minimum E value. This shows that J encourages samples with high disagreement to be selected during the data acquisition process. An alternate way to look at the metric is that from the predictive uncertainty, subtract away the expected data uncertainty, leaving approximate of the model uncertainty (Depeweg et al. 2017).
Variation ratios Gaa 2016) is another acquisition function that looks for disagreement. It is defined as the fraction of members in the ensemble that do not agree with the majority vote
M Mode(arg max Pk (c)
eEE KEK
V(P) = 1 JL
Σ CEE
(arg max Pk M),
F.
KEK
where K is the number of classes. This is the simplest quantitative measure of variation, and prior applications in literature show that works well in practice (Gal et al.) 2017 Beluch et al. 2018 Chitta et al.) (018b). Our work focuses on classification due to its prevalence in the large-scale setting, but we note that shares similarities to variance and standard deviation, which are used as acquisition functions in the Bayesian setting for regression Tsyymalov et al. 2018).
Our final acquisition function, error count, similar variation ratios, but checks for disagreement with the ground truth label rather than the mode of predictions M:
(p) =
Σ 6EE
(c) (arg max Pk = y).
KEK
This is typically not used in AL experiments it cannot be computed without the ground truth labels. In our setting, where the labels are available, we use this function as a baseline which, in principle, prioritizes mistakes made by the network when selecting data.
2.3 ENSEMBLE CONHIGURATIONS
State-of-the-ar ensemble-based AL approaches use different random seeds to construct ensembles (Beluch al. 2018). They recommend the number of samples drawn be the range E (5, 10) models (Lakshminarayann
et al.) (017). In theory, the error of a Monte Carlo estimator should decrease with more samples, which is evident in other BNN based uncertainty estimation techniques, that require the number of stochastic samples drawn to be increased to the range E E (50, 100) Gal & Ghahramani 2015).
The major limiting factor preventing the training of E E (50, 100) models with different random seeds for ensemble AL the computational burden at train time. Implicit ensembling approaches that are computationally inexpensive, such Dropout (Srivastava et al., 2014), suffer from mode
4

---

Under review as a conference paper at ICLR 2020
collapse, where the different members in the ensemble lack sufficient diversity for reliable uncer- tainty estimation (Pop& & Fulop 2018). An altemaae approach, called snapshot ensembles, that is less computationally expensive train time, uses cyclical learning rate to converge multiple local optima in a single training run (Huang et al. 2017]. However, this technique is also limited to ensembles in the range of F 6 members. In work, we present an implicit ensembling ap- proach that allows users to draw a large number of samples using the catastrophic forgetting property in DNNs (Toeva et al. 2018). Specificallyy, we exploit the disagreement between different check- points stored during successive training epochs to efficiently construct large and diverse ensembles. We collect several training checkpoints over multiple training runs with different random seeds.
Though we study the impact of the number of samples, our work does not attempt specifically to better match q()) from Eq. [ and [ from Eq. 2 which an open and active research area (Malinin & Gales, 2018). The overall algorithm is generic and can potentially benefit from more advanced techniques for uncertainty estimation.
EXPERIMENTS
In this section, we demonstrate the effectiveness of AL for building data subsets of three image classification benchmarks. We initially investigate the impact of the initialization schemes discussed in Section 2.1 and acquisition functions from Section 2.2 for the ResNet-18 18 architecture (He et al.] 2016). We then focus on scaling up the ensemble and evaluating the robustness of our subsets to architecture shifts. We experiment with three datasets: CIFAR-10 and CIFAR-100 100 (Krizhevsky) 2009), as well ImageNet (Deng al. 2009). The CIFAR datasets involve object classification tasks over natural images: CIFAR- 10 is coarse-grained over 10 classes, and CIFAR-100 is fine- grained over 100 classes. For both tasks, there are 50k training images and 10k validation images of resolution 32 x 32, which are balanced in terms of the number of training samples per class. ImageNet consists of 1000 object classes, with annotation available for 1.28 million training images and 50k validation images of resolution 256 x 256. This dataset has a slight class imbalance, with 732 to 1300 training images per class.
3.1 IMPLEMENTATION DETAILS
Unless otherwise specified, we use 8 models with the R8 (He et al. 2016) architecture to build the acquisition and subset models. For ImageNet, each ResNet-18 uses the standard kernel sizes and counts. For CIFAR-10 and CIFAR-100, we use variant of ResNet-18 as proposed in He let al. (2016). For all three tasks, we do mean-std pre- pre-processing and augment the labeled dataset on-line with random crops and horizontal flips. Optimization is done using Stochastic Gradient Descent with a learning rate of 0. and momentum of 0.9, and weight decay of 10 On CIFAR, we use a patience parameter (set to 25) for counting the number of epochs with no improvement in validation accuracy, in which case the learning rate is dropped by factor of 0.1. We end training when dropping the learning rate also gives no improvement in the validation accuracy after a number of epochs equal to twice the patience parameter. If the early stopping criterion is not met, we train for a maximum of 400 epochs. On ImageNet, we train for a total of 1 150 epochs, scaling the learning rate by factor of 0.1 after 70 and 130 epochs. Experiments are run on Tesla V100 GPUs.
3.2 RESULTS
Initialization schemes. In our first experiment, we compare the three initialization schemes intro- duced in Section 2.1 to a random subsampling baseline. For this experiment, we fix the number of ensemble members to 8 for CIFAR and 4 for ImageNet, each with a different random seed. We fix the acquisition function to Mutual Information (J), as defined in Eq. 4 For the pretrain scheme, we pretrain 8 (or 4) models with different random seeds on the entire dataset L, and finetune them start- ing with a learning rate of 10-3 3 on the chosen subset S. For the other schemes, we train the subset model from seratch on S. We report the top-1 validation accuracy of three independent ensembles, each from a different experimental trial, plotting the mean with one standard deviation as an error bar. These results are summarized in Fig.
We observe certain common trends for all three datasets: random subsampling (blue in Fig. I leads to steady drop-off in performance; and the pretrain scheme (orange in Fig. 1 does not significantly
5

---

Under review as a conference paper at ICLR 2020
CIFAR-10
CIFAR-100
Image
72
7.
02
voldsmoo
66
Ranzor Portuis Comprets Up
27
Randam
harrzh
Lundom
Compresu
Comprese Up
Bull by
Fall
Hull DRaset
hu Ditaact
62
23X
3X
4X.
108
4x
56t
DPM
Numbbr trantin
Number of Traiing images
Mamber of traiting imaga
Figure 1: Comparison of pretrain, compress and build up initialization schemes on CIFAR-IO, CIFAR- 100 and ImageNet. Results shown mean and std over three trials of the top-1 valida- tion accuracy of an ensemble of ResNet-18 18 models. While compress does not perform well on small subsets of data, build up provides strong results on all subset sizes for all three tasks.
impact the performance in comparison to training with the full dataset. Interestingly, the compress scheme (green in Fig. D performs extremely poorly when the subset chosen is very small, but performs well when the acquisition and subset models have a similar overall dataset size (eg.. IM samples on ImageNet). The poor performance of the compress scheme implies that the uncertainty estimates for this experiment are not robust to large changes in the dataset distribution between the acquisition and subset models. The build up scheme (red in Fig. [ consistently outperforms random subsampling by large margin. The performance is robust across all three tasks. Based on these observations, we fix the initialization scheme to build up for the next experiments. With sufficiently large subset of the data (eg., 25k on CIFAR-10, 1M on ImageNet), the build up scheme slightly outperforms a model trained on the full dataset.
Acquisition functions. In our next experiment, we compare random subsampling against the four acquisition functions from Section 2.2, Similar to the previous experiment, we use ensembles of 8 members for CIFAR and members for ImageNet for both the acquisition and subset models. We run three experimental trials, and report the mean validation accuracy of the subset model ensemble for each trial, in Table
For all four acquisition functions, the subset models significantly outperform the baseline (random) at the final iteration. Further, when using 50% of the data on CIFAR- and 80% the data on CIFAR-100 and ImageNet, we obtain subsets of data that improve performance compared to training on the full dataset (100%). Among the four functions, mutual information (J) and variation ratios (V) outperform entropy (?) and eтrоr count (E). Interestingly, though the E acquisition function is similar to V. and also has access to the ground truth labels, the data selected by in the first iteration leads poor performance, which is completely recovered in the subsequent iterations. This indicates that a sample for which all the ensemble members collectively make an error may be 'too difficult" and therefore not an ideal choice for the training dataset.
Among the two best acquisition functions, on the CIFAR-I0 dataset, J outperforms V. This is be- cause there are very few absolute disagreements in this setting due to the small number of classes and high performance, which leads to a very small number of samples with non-zero values for V. However, on the CIFAR-100 and ImageNet datasets, with more classes and lower overall perfor- mance, there is a greater number of disagreements, and V outperforms I by significant amount when using 80% of the dataset. Compared to training on the full dataset, we obtain a 0.5% abso- lute improvement in validation accuracy on both CIFAR- 100 and ImageNet when using the subset acquired by V. Additionally, this performance improvement is accompanied by a 20% reduction in training time both these tasks.
Ensemble configurations. We now explore the possibility of further gains in performance by scal- ing up the ensemble to increase the number of samples drawn in the Monte Carlo estimator as per Eq. 2 For the remaining experiments, we focus on the final AL iteration for the ImageNet dataset per the build up scheme. To do this, start by setting up 5 different training runs on 40% of the

---

Under review as a conference paper at ICLR 2020
Table I: Top-1 validation accuracy (in %) of ensembles trained subsets of CIFAR-10. CIFAR-100 and ImageNet (mean of 3 trials), comparing different acquisition functions. Each column indicates subset with size equal the specified percentage of the full dataset. Mutual information (J) and variation ratios (V) give the best performance.
Dataset
CIFAR-10
CIFAR-IOU 100
ImageNet
Acquisition F Function
12.5%
25%
50%
20%
40%
80%
20%
40%
80%
Random
87.65
91.88
94.30
63.96
74.18
80.65
63.51
68.54
71.66
Entropy (H)
89.67
94.29
96.08
65.83
76.35
81.94
64.11
69.64
72.00
Mutual Information (J)
89.46
94.34
96.30
66.11
76.29
82.02
64.65
70.18
72.36
Variation Ratios (V)
89.86
94.42
95.76
65.28
76.27
82.37
64.39
69.20
72.78
Error Count (E)
87.30
94.02
96.08
65.37
76.13
82.06
58.13
64.66
72.10
Full Dataset (100%)
96.18
81.86
72.33
Table 2: Effect of increasing the number of models in an acquisition ensemble trained with 40%
of ImageNet. Results shown are top-1 accuracy evaluated on the 40% selected data (512k samples used for training) and 60% unselected data (768k remaining samples). Gains in performance on the unselected data show that large scale checkpoint based ensembles exhibit high diversity.
Eval Set Single (1) Seeds (5) Checkpoints (5) Checkpoints (20) Combined (100)
Selected 58.10
74.98
72.59
79.60
83.78
Unselected
70.85
82.55
81.47
84.02
85.57
ImageNet dataset (512k samples) as selected by the best performing acquisition function in Table (T). For each of these 5 training runs, we store the 21 checkpoints obtained in the final stage of training (epochs 130-150, exact implementation details provided in appendix). We pick 4 ensemble configurations from these ResNet-18 training runs to utilize as the acquisition model for an ablation study: (i) random seeds, which uses a total of 5 models from the best performing epoch of each run; (ii) checkpoints, which uses the models from epochs 130:5:150 of a single run; (iii) 20 checkpoints, which uses the 20 models from epochs 131:1:150 of single run; and (iv) combined, which uses the '20 checkpoints' combined over all runs give 100 models.
We initially evaluate the performance of these four ensemble configurations on the data available for sampling. To this end, we report the top-1 accuracy of the four ensemble configurations, along with a baseline of a single model, when evaluated on the 40% selected data (i.e, the training set) and 60% unselected data of ImageNet. Our results are shown in Table 2 Results for the Single (1), Checkpoints (5) and Checkpoints (20) columns use the best seed of 5 runs. As the number of members in the ensemble is scaled up, we observe large and clear improvements in performance on both selected and unselected data. This shows that the checkpoints obtained with no additional computational cost at train time can be used to generate diverse ensembles. A detailed analysis of the diversity and agreement between checkpoints is provided in the appendix.
The significant gains (around 3%) in performance on unselected data in Table 2 as the number of models increased from 5 to 100 indicates the potential for better sampling by scaling up. is additionally worth noting that for all ensemble configurations, the performance is better on the larger unselected subset of data than on the selected training set. For single model, the gap in top-1 accuracy between these two subsets is nearly 13%. This demonstrates the huge amount of redundancy in the unselected part of the dataset as a result of our AL based selection.
Further, we are interested in how the ensemble configurations affect the acquisition function. To this end, we query for an additional 40% of the unselected data (512k samples) using the variation ratios (V) acquisition function, with each of the four ensemble configurations. We obtain four new subsets, each with 80% of the samples in ImageNet. The top-1 validation accuracy subset model trained using each of these new subsets is shown in Table 3 along with baseline of random sampling of 80% of the data. We observe a steady increase in performance as the number models in Ma during acquisition is increased, showing the benefits of scaling up ensemble AL. In particular, the combined configuration improves top- accuracy over random sampling by 1.1%. Note that the earlier results in Table I use ensemble of models for evaluation; but Table 3 always evaluates
7

---

Under review as a conference paper at ICLR 2020
Table 3: Effect the subset model of increasing the number of ensemble members in the acquisi- tion model using the V acquisition function. Results shown are top-1 validation accuracy of a single ResNet-18 model trained using 80% of the ImageNet dataset selected with different ensemble con- figurations, compared to a baseline of random acquisition. Best results are obtained by the combined configuration, which uses different random seeds and training checkpoints in the acquisition model. Random Seeds (5) Checkpoints (5) Checkpoints (20) Combined (100) 69.24
69.97
70.10
70.18
70.34
Table Transferring the 80% subset of ImageNet using ResNet-10 and ResNet-18 to new architec- tures for the subset model. We observe that these subsets provide near equivalent performance to training on the entire dataset (Full- across all new architectures.
Dataset
ResNet-18
ResNet-34
ResNet-50
ResNet-101
DenseNet-121
Random-80
69.24
73.00
75.15
76.72
74.59
AL-R10-88
70.31
73.49
75.91
77.87
75.28
AL-R18 18-80
70.34
73.61
76.18
77.75
75.42
Full-100
70.12
73.68
76.30
77.99
75.30
single model, to allow for fair comparison between the datasets acquired with 5 models vs. 100 models. Our results show that scaling up is key to exploiting implicit ensembling techniques for AL. Existing work that uses the related idea of snapshot ensembles for AL performs poorly (Beluch) et al. 2018). This is likely due to (i) the small number of models used, and (ii) the smaller-scale experimental setting (only 2k samples acquired each iteration).
Robustness to architecture shift. Finally, evaluate the robustness of the subsets to changes in model capacity. Specifically, we evaluate the robustness of the best performing subset selected with ResNet-18 18 acquisition model in Table ('combined" Additionally, we consider the subset obtained with an even more lightweight ResNet-10 model, where we remove 2 convolutional layers from each residual block of ResNet-18. We use these subsets (referred to as AL-R18-80 and AL-R 10-80) to train subset models with the ResNet-18, ResNet-34, ResNet-50, ResNet-101 (He) al. 2016 and DenseNet-121 (Huang et al.) 2016) architectures. For reference, we also evaluate randomly subsampled 80% of Image.Net (Random-80) and the full dataset (Full-100). Our results are summarized in Table As shown, on all 5 architectures, the subset obtained by AL with ResNet- 18 achieves similar performance to training on the full dataset, with 20% reduction in overall training time. The AL-R10-80 subset also strongly outperforms the random baseline. This ability transfer selected subsets to larger architectures has significant implications in domains where training time crucial, such as MLPerf (MLP 2019).
RELATED WORK
comprehensive review of classical approaches AL is presented in /Setles (2010). In these approaches, data samples for which the current model is uncertain are queried for labeling. It is occasionally observed in the classical AL literature that training on a subset of data can give better models than training on the full dataset (Schohn & Cohn 2000 Lapedriza et al. 2013 Gavves al. 2015: Konyushkova et al., 2013). However, this is not investigated with of-te-of-the-ar AL techniques for DNNs (Beluch et al. 2018 [Sinha et al.; (2019). These methods typically assume that performance obtained by training on the entire data pool is an upper bound. We show that the full dataset is not an upper bound, demonstrating distinct advantages of training on data subsets when they are sufficiently large.
Despite the widespread use of image classification datasets, there has only recently been an increased interest in understanding the properties of data subsets. Core-set selection (Sener & Savarese 2017) is one such attempt reduce dataset sizes, by finding representative subset of points based on relative distances in the DNN feature space. Vodrahalli et al. (2018) also aims to find representative subsets, using the magnitude of the gradient generated by each sample for the DNN as an impor-
8

---

Under review as a conference paper at ICLR 2020
tance measure. It is important to note that these techniques are unable match improve the performance of a model trained with all the data for DNNs.
More recent techniques able to successfully reduce dataset sizes by identifying redundant exam- ples, albeit to a much smaller extent than our approach. [Brookkar et al. (2019) uses clustering in the DNN feature space identify redundant samples, leading discovery of 10% redundancy in the CIFAR-10 and ImageNet datasets. Select via proxy (Coleman et al. (019) uses simple uncertainty metrics (confidence, margin and entropy) select subset of data for training, and removes 40% of the samples in CIFAR-10 without reducing the network performance. In comparison to these methods, our technique not only maintains, but also improves the performance of a DNN. Further, we empirically show that this can be done with much less data than these methods, for example, on CIFAR-10, we only require 50% of the training dataset.
A second category of works on data subsets is closely related to catastrophic forgetting in DNNs. Toneva et al. (2018) uses the number of instances in which a previously correctly classified sample 'Torgotten" and misclassified during training an importance measure. By doing so, this method able to remove 30% the samples that are "unforgettable" from CIFAR-10 without significantly reducing performance. [Chang et al. (2017) propose similar idea of emphasizing data points whose predictions have changed most over the previous epochs during training. Rather than directly sub- sampling. this variance in predictions is used to increase decrease the sampling weight during training, and therefore the approach has no significant impact on training time. In comparison, by training only specific subset, not only improve performance but also cut down training time by 20% to 50% for the datasets used in our study.
5 CONCLUSION
In this paper, we presented an approach to build data subsets for deep neural networks. Our method uses ensemble Active Learning to estimate the uncertainty of each sample in a dataset, and then chooses only the highest uncertainty samples for training. Our results demonstrate that this im- proves the performance of a DNN compared to training with the entire dataset on three different image classification benchmarks. Moreover, we propose simple technique scale up ensembles leading to additional accuracy gains with minimum computational overhead. Importantly, our re- sults demonstrate that datasets obtained using AL can be effectively reused for training new models with different capacity or network architectures.
REFERENCES
MIperf benchmark suite for measuring performance of ml software frameworks, ml hardware accel- erators, and ml cloud platforms. httt::://mmpprteee org/ 2019. Accessed: 2019-09-23.
William H. Beluch, Tim Genewein, Andreas Nurnberger, and Jan M. Kohler. The power of ensem- bles for active learning in image classification. In CVPR, 2018
Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio. Semantic Redundancies in Image- Classification Datasets: The 10% You Don't Need. arXiv e-prints, art. arXiv:1901.11409, Jan 2019.
Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples. arXiv e-prints, art. Apr 2017.
Kashyap Chitta, Jose M. Alvarez, and Adam Lesnikowski. Large-Scale Visual Active Learning with Deep Probabilistic Ensembles. arXiv e-prints, art. arXiv: 18..0355 Nov 2018a
Kashyap Chitta, Jianwei Feng, and Martial Hebert. Adaptive Semantic Segmentation with Strate- gic Curriculum of Proxy Labels. arXiv e-prints, art. arXiv:181 .03542, Nov 2018b.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Mach. Learn., 15(2):201-22) 2211, May 1994, ISSN 0885-6125. doi: 10.10233A:102226550021. URL
https: :/d/: .oriii.ogggoo 1023///111127773563

---

Under review as a conference paper at ICLR 2020
Cody Coleman, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Select via proxy: Efficient data selection for training deep net- works, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decom- position of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. arXiv e-prints, art. arXiv:1710.07283, Oct 2017.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR, June 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural Architecture Search: A Survey. arXiv e-prints, art. 10Xi::8088.0537,, Aug 2018.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model
Uncertainty in Deep Learning. arXiv e-prints, art. arXiv:1506.0214: Jun 2015.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian Active Learning with Image
Data, arXiv e-prints, art. arXiv: 703.02210, Mar 2017.
E. Gavves, T. Mensink, Tommasi, C. G. M. Snoek, and T. Tuytelaars. Active transfer learning with zero-shot priors: Reusing past datasets for future tasks. In ICCV, pp. 2731-2739, Dec 2015. doi: 10. 10.1109/CCV.2205.33333
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In SIGKDD, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. arXiv e-prints, art. arXiv: 10t.::16000002,,, Mar 2016.
Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML: A Survey of the e-prints, art. arXiv: 000.109000000000 Aug 2019.
arXiv
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for Model Compression and Acceleration Mobile Devices. arXiv e-prints, art. arXiv: 102.:0402,,3444, Feb 2018.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning. arXiv e-prints, art. arXiv:1112.5745, Dec 2011.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. arXiv e-prints, art. arXiv:1608.006
Aug 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q. Weinberger. Snapshot Ensembles: Train 1, get M for free. arXiv e-prints, art. 1rXiv:1704.00109, Mar 2017.
Kirthevasan Kandasamy, Karun Raju Vysyaraju. Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnahas Poczos, and Eric P. Xing. Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly. arXiv e-prints, art. arXiv: Mar 2019.
Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? arXiv e-prints, art. 103.v:1703.04977. Mar 2017.
K. Konyushkova, R. Sznitman, and P. Fua. Introducing geometry active learning for image seg- mentation. In ICCV, pp. 2974-2982, Dec 2015. doi: 10. 20.100/9
10

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
Agata Lapedriza, Hamed Pirsiavash, Zoya Bylinskii, and Antonio Torralba. Are all training exam- ples equally valuable? arXiv e-prints, Nov 2013.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. arXiv e-prints, art. arXiv: 1iii:1800.00000 1806.09055, Jun 2018.
David Lowell, Zachary C. Lipton, and Byron C. Wallace. How transferable are the datasets collected by active learners? arXiv e-prints, art. urXiv:1807.04801, Jul 2018.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimiza- tion. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), NIPS, pp. 7816-7827. Curran Associates, Inc., 2018. URL hattp://papers.nips cc/paper/800 - -ww.errallacccttcccreeeooo--- wat.:ptpiiiiiiiiai.ai.mmmamaaaet ion. zaniin.tioooooo. pdf
Andrey Malinin and Mark Gales. Predictive Uncertainty Estimation via Prior Networks. arXiv Feb 2018.
e-prints, art,
Mojmir Mutny and Andreas Krause. Efficient high dimensional bayesian optimization with additiv- ity and quadrature fourier features. In NIPS. 2018.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. arXiv e-prints, art. WtXiv:19060025300 Jun 2019.
Remus Pop and Patric Fulop. Deep Ensemble Bayesian Active Learning Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles. arXiv e-prints, art. A.1iv1881.00388 Nov 2018
Greg Schohn and David Cohn. Less is more: Active learning with support vector machines. In ICML, pp. 839-846, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1-55860-707-2. Utthttp:://dl) acm. org httg:/iiaaiiioooo www.oo....ffm.. cfmidd=mmdi-di 4tt.::at.ooooc.c...d44444555.... 657802]
Ozan Sener and Silvio Savarese. Active Learning for Convolutional Neural Networks: A Core-Set Approach. arXiv e-prints, art. arXiv:1708: 770.00489, Aug 2017.
Burr Settles, Active learning literature survey. Technical report, 2010.
Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-423 7 1948. doi: 10. 153002/..133-7300000 Leeexplore ieee org/
URL https:/7
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational Adversarial Active Learning. arXiv e-prints, art. arXiv:1904.000
Mar 2019.
Lewis Smith and Yarin Gal. Understanding Measures of Uncertainty for Adversarial Example De- tection. arXiv e-prints, art. arXiv:1803.08533. Mar 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. In ICML, 2014.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting Unreasonable Ef- fectiveness of Data in Deep Learning Era. arXiv e-prints, art. arXiv:1707 02068,, Jul 2017.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An Empirical Study of Example Forgetting during Deep Neural Network Learning. arXiv e-prints, art. MiXi::881..051555 Dec 2018.
Evgenii Tsymbalov, Maxim Panov, and Alexander Shapeev. Dropout-based Active Learning for Regression, arXiv e-prints, arXiv: 1801188066666 Jun 2018.

---

Under review as a conference paper at ICLR 2020
Kailas Vodrahalli, Ke Li, and Jitendra Malik. Are All Training Examples Created Equal? An Empirical Study. arXiv e-prints, art. 1.X.::8111.12666 Nov 2018.
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer learning with neural automl. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), NIPS, pp. 8356-8365, Curran Associates, Inc., 2018. URL with neural- aut pdf
nips.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference Learning Representations, 2019. URL https://
openreview net http://ormmmidryloooooolo
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification, arXiv e-prints, art. arXiv:1905.000 1905,0056,, May 2019.
12

---

Under review as a conference paper at ICLR 2020
ImageNet
Randorn-80
50
Pretrain-80
Compress-80
Bulld-Up-80
Full-100
Validation Error
40
35
100k
200K
300k
400k
500k
600K
700K
Training Iterations
Figure 2:
Comparing initialization schemes for ImageNet ResNet-18 training with variation ratios. All three initialization achieve similar performance. Compared to training on the full dataset, train- ing time is reduced by 20%.
Table 5: Effect of leaving out the highest uncertainty samples as outliers while sampling 50% the CIFAR-10 10 and CIFAR-100 datasets with the compress initialization scheme and mutual information acquisition function. Results shown are the mean top-1 validation accuracy over 3 trials. We observe that the highest uncertainty samples outliers on CIFAR- 100, but this is not the case CIFAR-10.
CIFAR-10
CIFAR-IOU
No Outliers 95.77
12.5% Outliers
25% Outliers
Outliers
12.5% Outliers
25% Outliers
94.85
93.81
75.89
76.76
76.41
A ADDITIONAL EXPERIMENTS
Validation curves for different initialization schemes. To analyze the runtime and convergence of different models, we plot the top-1 validation error of single ResNet-18 model for the three initialization schemes from Section 2.1 This experiment is using the variation ratios acquisition function on 80% of ImageNet three settings: Pretrain-80, Compress-80 80 and Build-Up-80. We compare the baselines 80% randomly sampled data (Random-80) and the full dataset (Full- Initially, the error for Pretrain-80 is lower than the other
100). Our results are shown in Fig. 2 approaches, but towards the end of training, Pretrain-80, Compress-80 and Build-Up-80 obtain similar validation accuracy. All three initialization schemes finish training in 20% less time than using the full dataset.
Outliers. It is possible that very small subset with high uncertainty is informative to the acqui- sition model, but too difficult for the subset model which is trained from scratch with just these samples. To check for this, we repeat the experiment for the compress scheme on CIFAR from Fig. while choosing subset of 25k samples (50%), but set aside a percentage of the highest uncertainty samples as outliers instead of adding them to S them during acquisition. For example, for 12.5% outliers, after sorting by the acquisition function, we select the samples in the range 37.5% to 87.5% the subset S instead of 50% to 100%. As shown in Table leaving outliers does not improve the performance of the compress scheme on CIFAR-10. Even in the case of CIFAR- 100, where there an improvement in performance after removing outliers, the obtained accuracy of 76.76% is well
13

---

Under review as a conference paper at ICLR 2020
Table 6: Effect of class-balanced random sampling the ImageNet dataset. Results shown the top-1 validation accuracy of a single trial for training a single ResNet-18 model. We observe that class balancing does not have significant impact on performance. Random
Random with Class Balance
10 %
20%
40%
10%
20%
40%
49.57
57.76
64.43
49.10
57.80
64.34
ImageNet
700k
Res/Net-18 Pairwise
Number of Samples in Conserisus
ResNei-18 All
ResNet-11 Pairwise
ResNet-10
Data
500k
400k
200k
100k
2
10
11
12
13
14
15
16
17
18
19
20
Checkpoints
Figure 3: Analyzing the prediction consensus between different checkpoints during ImageNet train- ing. We observe very little, but consistent agreement in predictions between any consecutive pair of checkpoints. The overall agreement among the group models decays rapidly. With 20 check- points, there is only consensus around 90k samples for ResNet-18 18 and 50k samples for ResNet- 10 out of the 768k samples used for this evaluation.
short of the build up scheme (red in Fig. D. which reaches 79.37%. This indicates that though the highest uncertainty samples sub-optimal for training when taken alone, they important when used as part of a larger set of samples.
Additionally, conduct experiment checking for outliers ImageNet with the build up scheme in the setting of Table 3 In this experiment, we sample 80% of the dataset while leaving out 50k samples (approximately 4%) outliers for the 'combined' ensemble configuration. Doing reduces subset model performance from 70.34% to 70.09%, indicating that the highest uncertainty samples are indeed crucial outperforming the model trained on the full dataset.
sampling on ImageNet. While CIFAR-10 10 and CIFAR- have a balanced number of samples per class, ImageNet has slight class imbalance. This allows potentially more refined baseline than pure random sampling. by additionally incorporating class balance. We implement this approach by collecting 128, 256 and 512 samples per class randomly for all 1000 classes. This leads to datasets of 10%, 20% and 40% of the full dataset. We compare this baseline to standard random sampling in terms of top-1 validation accuracy for ResNet- training. Our results are summarized in Table 6 We observe significant differences between standard random sampling and the new random sampling with class balance.
Checkpoint consensus. We extend our analysis on the 768k unselected samples of ImageNet from the setting in Table by checking the consensus between the group of n final training checkpoints using both the ResNet-18 and ResNet- 10 architectures for consensus in predictions for every consecutive pair of checkpoints as a reference. Our results are presented in Fig. We observe that for R8 training, any consecutive pair of checkpoints
to 20. In addition,
check the
14

---

Under review as a conference paper at ICLR 2020
only agree the top-1 prediction of around 450k samples (blue in Fig. 3 which is 58% of the data). All 20 checkpoints of a single run only agree on 90k samples (orange in Fig. 3 which 11.7% of the data). This is surprising, since the top-1 I accuracy of each model
the unselected data is
above 70%, indicating that though they are all from the same training run and have similar accuracy, each checkpoint makes different kinds of erros. These results also provide further support to the findings of Toneva et al. (2018), which show that a very small subset of samples are 'unforgettable" once learned by the network, and many samples are repeatedly relearned and forgotten. Since the number of disagreements provides valuable information about the uncertainty of each sample for the acquisition functions used our study, these results showease the benefits of using more models in ensemble for uncertainty estimation. The consensus trends for both pairwise and all models remain similar for the ResNet- 10 architecture, though the absolute values lower than ResNet- 18 due to the lower model accuracy (green and red in Fig.
15


Reviews:
[{"review_id": "Skl6MjRhKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work makes use of uncertainty estimation methods from active learning to select a subset of training data that produces models with similar (or better) performance compared to models trained on the full training set. It proposes a way to improve the Monte Carlo estimation of model uncertainty by including multiple checkpoints that are generated \"for free\" during a training run, thereby increasing the number of samples from 5-10 in previous work to 100 in this work. It compares several initialization schemes for the subset model using mutual information as the acquisition function, finds that a \"build-up\" approach (based on Chitta et. al 2018a) works best, and uses that for the rest of the studies. It then compares several acquisition functions, using the build-up approach, finds that variation ratio performs best, and uses that for the rest of the studies. Next, it compares the Top-1 accuracy on ImageNet obtained by evaluating the ensemble models produced by different ensembling schemes, and finds that ensembling 20 checkpoints from 5 training runs with different random seeds work best. Then, it uses acquisition models that use ensembles from each ensembling scheme to select subsets of the ImageNet data to be used for training the subset model, and then compares the performance of the subset models. Finally, it demonstrates this method of selecting a subset of the training data works even if the subset is used to train a model with a different architecture from the acquisition model.\n\nStrengths:\n- Algorithm is likely to be useful in practice. Training dataset can be \"compressed\" using a smaller architecture like ResNet-18, then used to train larger architectures like DenseNet-121, thus saving the amount of compute per training epoch. Using training checkpoints in the ensemble is very practical but not obvious (to me), since my first intuition would be that checkpoints from the same training run would not provide enough diversity to improve the acquisition function. I am glad that there were thorough experiments to address this concern and demonstrate that it works.\n- Experiments answer key questions about the method proposed, and the sequence of experiments have a clear logical flow. Good baselines. Clear notation and problem set-up.\n\nWeakness that affected the score:\n- Missing detail on the build-up initialization scheme. The work referred to Chitta et al., 2018a, but that algorithm requires selecting a growth parameter. This growth parameter determines the number of times the subset model needs to be retrained, which can affect the viability of this method in practice. I would like to see the build-up initialization scheme described in greater detail.\n\nClarifications:\n- In Table 2 and Table 3, are the results in the \"Single (1)\",  \"Checkpoints (5)\", and \"Checkpoints (20)\" columns obtained by averaging over the 5 random seeds?\n- In the last column of Table 2, Top-1 accuracy of ~84% from an ensemble of 100 ResNet-18s (Table 2) seem very high. In comparison, ResNet-50 and AmoebaNet-A (2019) obtained a Top-1 accuracy of 77.2% and 83.9% respectively. What do the authors think about this?\n- Why would one expect high accuracy of the ensemble of NNs in the acquisition model to indicate good sampling quality of the acquisition model? (caption for table 2)\n\nMinor issues:\n- Algorithm 1: first two steps should be kept un-italicized, like the rest of the steps.\n- Page 7, first paragraph: \"This shows that the checkpoints are obtained with no additional computational cost at train time can be used to generate diverse ensembles.\" The first \"are\" in this sentence is unnecessary.\n- Build-up was chosen as the initialization scheme for the rest of the studies, as it performed best when the acquisition function was fixed at *mutual information*. However, the acquisition function that was finally chosen for the rest of the studies is *variation ratio*, since it performed best when the initialization scheme was fixed at build-up. It would be more convincing if figure 1 also includes variation ratio.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HygyKZzzsB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper proposes a new method that uses uncertainty estimation to do ensemble active learning on image classification tasks. The proposed method mainly consists of two steps: \n1. select a subset of data based on the uncertainty estimation from an ensemble model. The ranking of data point is calculated via an acquisition function, which measures the model uncertainty.\n2. train a “subset model” on the selected subset.\nThe paper then considers 3 initialization methods, 4 acquisition functions and 4 ensemble configurations and makes an empirical study on different combinations of these three. The experiment is done progressively. The paper first fixes the acquisition function to Mutual Information and finds the build-up initialization most promising. It then fixes the build-up initialization and finds Variation Ratios and Mutual Information result in better performance. Finally, with build-up initialization and Mutual Information acquisition, the paper compares different ensemble configurations and shows checkpoints ensemble can scale up better (less computation burden) and result in better performance when the number of ensembles is large. In addition, the paper also shows the obtained subset can help models with richer capacity as well. Results on deeper architectures outperform their corresponding performance when trained on the entire training set.\n\nStrengths:\n- The idea of using checkpoints as free samples of models is new and practical.\n- Experiments are comprehensive and convincing in terms of the performance.\n\nConcerns:\n- Although saving checkpoints is “cheap” and shows empirical good performance, the models are somehow dependent on each other, particularly in experiments where consecutive checkpoints are saved. I am not sure whether this fits well in the bayesian framework of uncertainty estimation.\n- The build-up initialization method lacks details for reproducibility if the Algorithm 1 of Chitta et al., 2018a is used.\n- It is interesting that large number of ensembles increases the accuracy of acquisition model by a lot, but doesn’t boost the performance of the subset model too much (according to table 2 and 3). Does this imply that it is the ensemble rather than the active learning that helps?\n\nMinor issue:\n- It might be better to clarify which ensemble is used in the ensemble configuration experiment since previous experiments can have two different ensembles (acquisition and subset).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1l1WVc0FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review Summary\n--------------\nOverall, I'm not quite convinced this method would be worth the trouble to implement. On the more realistic benchmarks, they need to keep ~80% of the total dataset size and the claimed \"improvement\" is rather small (less than 0.6% absolute gain in accuracy, e.g. from 81.86% to 82.37% on CIFAR100 and from 72.33% to 72.78% on ImageNet). There is no runtime comparison, there are missing baselines, and most of the method development seems guided by trying out many options instead of taking a principled approach. Without these, the paper is just not ready for a top conference like ICLR.\n\nPaper Summary\n-------------\nThe paper considers a new take on active learning for image classification: given a large fully labeled dataset, identify a subset of the data that, when training on that subset alone, yields similar performance as training on the (much larger) full dataset. The paper focuses on \"ensembles\" of deep neural network classifiers as the prediction model, following Lakshminarayanan et al. (2017). \n\nThe presented method is summarized in Algorithm 1. Given a suitably initialized \"acquisition model\", the model makes predictions on each example in the full dataset, then ranks examples using an acquisition function to find the subset of size N_s (top N_s examples by rank) where there is most \"disagreement\" among the model ensemble. This subset is then used to train a \"subset\" model (again, an ensemble of DNNs).\n\nExperiments consider several possible initializations, acquisition functions, and ensemble sizes. Evaluation is done using the validation sets of three prominent image classification benchmarks: CIFAR10, CIFAR100, and ImageNet (1000 classes). \n\n\nSignificance\n------------\nI don't think a successful case has been made that the proposed solution would generate significant widespread interest, because the gains demonstrated here are too minimal. Looking at the primary results in Table 2, it's really only when using 80% of the total images of imagenet or cifar100 (the most realistic benchmarks) that there is a small (<1%) absolute gain in accuracy over the simpler approach of just using the full dataset. Thus, the approach is not going to significantly reduce computational burden but adds a lot of complexity.\n\nNovelty\n-----------\nThe method seems new to me.\n\n\nExperimental Concerns\n---------------------\n## E1: Need to consider runtime in evaluation\n\nNone of the figures/tables that I can see report elapsed runtimes for the different methods. To me this is the fundamental tradeoff: not how many fewer examples can I learn from, but how much faster is the method than the \"standard\" of using the full dataset? Showing curves of validation progress over wallclock time would be a better way to present results. \n\nThe important thing here is that even *full dataset* makes progress after each minibatch. You'd need to show progress at checkpoints for each epoch in  0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, .... \n\n## E2: Potential missing baseline: Random subset with balanced classes\n\nWhen you select a random subset, are you ensuring class balance? If not, that seems like a more refined baseline. Perhaps won't make too much difference for cifar10, but could be important for ensuring rarer classes in ImageNet are represented well. \n\n\nPresentation Concerns\n---------------------\n\n## P1: Initialization description confusing\n\nI didn't understand the \"build up\" method as described in Sec. 3. How large is the subset used at each phase? How do you know when to stop? This could use a rewrite to improve clarity.\n\n## P2: Missing some details for reproducibility\n\nHow was convergence assessed for all models? How were learning rates set? Many of these are crucial to understanding the runtime required for different models. (Sorry if these are in the appendix, but some short summary is needed in the main paper)\n\n## P3: Title Change Recommended\n\nI don't think the presented method is really doing a \"Distribution Search\"... I would suggest \"Training Data Subset Selection with Ensemble Active Learning\"\n\nMinor Method Concerns\n---------------------\n\n## M1: What about regression?\n\nAcquisition functions seem specialized to classification. What to do for regression or structure learning? Any general principles to recommend?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an ensemble-based active learning approach to select a subset of training data that yields the same or better performance. The proposed method is rather heuristic and lacks novel technical contribution that we expect for top ML conferences. No theoretical justification is provided to argue why the proposed method works. Additional studied are needed to fully convincingly demonstrate the benefit of the proposed method in terms computational cost. 
Decision: Reject