Selection via Proxy: Efficient Data Selection for Deep Learning
Authors: Cody Coleman
Christopher Yeh
Stephen Mussmann
Baharan Mirzasoleiman
Peter Bailis
Percy Liang
Jure Leskovec
Matei Zaharia
Venue:  (2020)
Keywords: data selection
active-learning
core-set selection
deep learning
uncertainty sampling

Abstract: Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10× faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1.6× end-to-end training time improvement.

Published as a conference paper at ICLR 2020
SELECTION VIA PROXY: EFFICIENT DATA SELECTION FOR DEEP LEARNING
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia Stanford University
ABSTRACT
Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need be learned. In this work, we show that we can greatly improve the computational efficiency by using small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach several data selection tasks across five datasets: CIFAR1O 10, CIFARIOS 100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1 0.1%). For core-set selection on CIFAR1O, proxies that are over 10x faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to 1.6x end-to-end training time improvement.
INTRODUCTION
Data selection methods, such as active learning and core-set selection, improve the data efficiency of machine learning by identifying the most informative training examples. To quantify informativeness, these methods depend on semantically meaningful features trained model to calculate uncertainty. Concretely, active learning selects points label from large pool of unlabeled data by repeatedly training a model on a small pool of labeled data and selecting additional examples to label based on the model's 'ss uncertainty (e.g.. the entropy of predicted class probabilities) or other heuristics (Lewis) & Gale 1994 Rosenberg et al. 2005 Settles 2011 2012). Conversely, core-set selection techniques start with a large labeled or unlabeled dataset and aim to find small subset that accurately approxi- mates the full dataset by selecting representative examples (Har-Peled & Kushal 2007 Tsang et al.) 2005 Huggins al. 2016 Campbell & Broderick] 2017
& Savarese 2018).
Unfortunately, classical data selection methods are often prohibitively expensive to apply in deep learning (Shen et al. 2017 Sener & Savarese 2018 Kirsch et al. 2019). Deep learning models learn complex internal semantic representations (hidden layers) from raw inputs (e.g., pixels or characters) that enable them achieve state-of-the-as performance but result in substantial training times. Many core-set selection and active learning techniques require some feature representation before they can accurately identify informative points either take diversity into account as part of trained model to quantify uncertainty. As a result, new deep active learning methods request labels in large batches to avoid retraining the model too many times (Shen et al.) 2017; [Sener & Savarese 2018 Kirsch al. 2019). However, batch active learning still requires training full deep model for every batch, which is costly for large models (He et al.) 2016b] Jozefowicz et al. 2016 Vaswani et al. 2017). Similarly, core-set selection applications mitigate the training time of deep learning models by using
"Correspondence: cody sidy@eessttaffon..ddu

---

Published as a conference paper at ICLR 2020
bespoke combinations of hand-engineered features and simple models (e.g.. hidden Markov models)
pretrained on auxiliary tasks (Wei et al. (201: 2014 Tschiatschek et al. 2014; Ni et al. 2013).
In this paper, we propose selection via proxy (SVP) more computationally efficient for deep learning. SVP uses the feature representation from a separate, less computationally intensive proxy model in place of the representation from the much larger and more accurate target model we aim to train. SVP builds on the idea of heterogeneous uncertainty sampling from Lewis & Catleu (1944), which showed that an inexpensive classifier (e.g., naive Bayes) can select points to label for a much more computationally expensive classifier (e.g., decision tree). In our work, we show that small deep learning models can similarly serve as an inexpensive proxy for data selection in deep learning. significantly accelerating both active learning and core-set selection across a range of datasets and selection methods. To create these cheap proxy models, we can scale down deep learning models by removing layers, using smaller model architectures, and training them for fewer epochs. While these scaled-down models achieve significantly lower accuracy than larger models, we surprisingly find that they still provide useful representations to rank and select points. Specifically, we observe high Spearman's is and Pearson's 's correlations between the rankings from small proxy models and the larger, more accurate target models metrics including uncertainty (Settles, 2012). forgetting events Tooeva et al.) 2019), and submodular algorithms such as greedy k-centers (Wolf, 2011). Because these proxy models are quick to train (often 10x faster), we can identify which points to select nearly as well as the larger target model but significantly faster.
way to make existing data selection methods
We empirically evaluated SVP for active learning and core-set selection five datasets: CIFAR 10, CIFARI00 (Krizhevsky& & Hinton| 2009), ImageNet (Russakovsky et al., 2015), Amazon Review Polarity, and Amazon Review Full (Zhang et al. 2015). For active learning, we considered both least confidence uncertainty sampling (Settles 2012: [Shen et al., [017]] Gal et al. (017) and the greedy k-centers approach from [Sener & Savarese (2018) with a variety of proxies. Across all datasets, we found that SVP matches the accuracy of the traditional approach of using the same large model for both selecting points and the final prediction task. Depending on the proxy, SVP yielded up to 7x speed-up on CIFAR10 and CIFAR 100, 41.9x speed-up on Amazon Review Polarity and Full, and 2.9x speed-up on ImageNet in data selection runtime (i.e., the time it takes to repeatedly train and select points). For core-set selection, we tried three methods to identify subset of points: max entropy uncertainty sampling (Settles) 2012). greedy k-centers as a submodular approach (Wolf 2011), and the recent approach of forgetting events (Toneva et al. 2019). For each method, we found that smaller proxy models have high Spearman's 's rank-order correlations with models that are 10x larger and performed well these large models at identifying subsets of points train on that yield high test accuracy. On CIFAR1O, SVP applied to forgetting events removed 50% of the data without impacting the accuracy of ResNet164 64 with pre-activation (He al., 2016b), using a 10x faster model than ResNet1 64 to make the selection. This substitution yielded an end-to-end training time improvement of about 1.6x for ResNet164 64 (including the time to train and use the proxy). Taken together, these results demonstrate that SVP is promising, yet simple approach to make data selection methods computationally feasible for deep learning. While we focus on active learning and core-set selection, SVP is widely applicable to methods that depend on learned representations.
METHODS
In this section, we describe SVP and show how it can be incorporated into active learning and core-set selection. Figure shows an overview of SVP: in active learning, we retrain proxy model Af in place of the target model A7 after each batch is selected, and in core-set selection, we train the proxy Afe rather than the target AT, over all the data to learn feature representation and select points.
2.1 ACTIVE LEARNING
Pool-based active learning starts with a large pool of unlabeled data U = (xxheef where [n] = (1...., n.. Each example is from the space X' with an unknown label from the label space y' and is sampled i..d. over the space 2 = X x )' as (x:, vt) pz. Initially, methods label small pool of points " = {o] E [*]]se[m] chosen uniformly at random. Given U, a loss function E, and the labels {uaz hemm for the initial random subset, the goal of active learning is to select up to a budget of b points s o U (83 E [m] \ \80) [e to label that produces model A, with low error.
2

---

Published as a conference paper at ICLR 2020
Traditional Appreach
Intaa Subeet (e))
Soboct
Soleot Subter
Trein (621
(1)
Suteel
Date Selection
Approach:
Prozy
Approach: Selection Via Proxy
Intaa Subroe (59)
Train Pieay (A5)
Thain Procy [Sc-a?
Soleet (-aref)
Train Thrgel (122
Train (4G
Train
Subool (e
Thrgel
Data Selection
Data Selection
(a) Active Learning
(b) Core-Set Selection
Figure 1: SVP applied to active learning (left) and core-set selection (right). In active learning, we followed the same iterative procedure of training and selecting points to label as traditional approaches but replaced the target model with a cheaper-to-come compppt proxy model. For core-set selection, we learned a feature representation over the data using proxy model and used it to select points train larger, more accurate model. In both cases, we found the proxy and target model have high rank-order correlation, leading to similar selections and downstream results.
Baseline. In this paper, we apply SVP to least confidence uncertainty sampling (Settles) 2012 Shen
al. 2017 Gal al., 2017) and the recent greedy k-centers approach from /Snner & Savarese 201 8). Like recent work for deep active learning (Shen et al. 2017 Sener & Savarese 2018 Kirsch et al.
2019), we consider a batch setting with K rounds where we select points in every round. Following Gal et al. (2017); Senner & Savarese (2018); Kirsch et al. (2019). we reinitialize the target model and retrain on all of the labeled data from the previous & rounds avoid any correlation between selections (Frankle & Carbin 2018: Kirsch et al. 2019). We denote this trained model as AT U.....ukk or just Ag for simplicity. Then using Ag. either calculate the model's 's confidence as:
f Feontidence (x; A
max P(ù x; A
and select the examples with the lowest confidence or extract feature representation from the model's final hidden layer and compute the distance between examples (i.e., A(x,, x;; A to select points according to the greedy k-centers method from Wolf 1011); Sener & Savarese (2018) (Algorithm I The same model is trained on the final b labeled points to yield the final model, Ag. which is then tested held-out set to evaluate error and quantify the quality of the selected data.
Although other selection approaches exist, least confidence uncertainty sampling and greedy k-centers cover the spectrum of uncertainty-bo -based and representativeenss--e based approaches for deep active learning. Other uncertainty metrics such as entropy or margin were highly correlated with confidence when using the same trained model (i.e., above a 0.96 Spearman's 's correlation our experiments on CIFAR).
(Seung et al.) 1992) can be prohibitively expensive in deep learning, where training a single model already costly. BALD (Houlsby al., 201T) has seen success in deep learning (Gal et 2017 /Shen et al. 2017) but is restricted to Bayesian neural networks or networks with dropout (Srivastava al. 2014) 4) as an approximation (Gal & Ghahramani 2016).
Algorithm 2 FORGETTING EVENTS
TONEVA ET AL. 2019)
Algorithm I GREEDY K-CENTERS
[WOLF] 2011; SENER & SAVARESE 2018
1: Initialize prev_acee 0, [n]
Input: data x,, existing pool s trained model AT, and a budget b
2: Initialize forgetting events; = 0, [n]
while training forgetting_events,
done do
4: Sample mini-batch B from L
1: Initialize s
5: for example i B do
2: repeat
6: Compute acc;
3: u = arg maxxe[]] minse. A (x,,);AA))
7: if prev pprev_acc; > acc; then Norgetting_event += = acc;
4: s s s= [u}
8:
5: until
|s
9:
6: return 6
10: Gradient update classifier on B
11: return forgetting_events
3

---

Published as a conference paper at ICLR 2020
2.2 CORE-SET SELECTION
Core-set selection can be broadly defined as techniques that find subset of data points that maintain similar level of quality (e.g., generalization error of trained model minimum enclosing ball) as the full dataset. Specifically, we start with a labeled dataset L = =xx;, ys hiejn] sampled i.i.d. from Z
with pz and want to find a subset of m Σ n. points = {8) E [r) jem that achieves comparable
quality to the full dataset:
min mm Ex,grp2 [[(x, A. Ex,y<p [xx Am])]
Baseline. To find s for given budget m, we implement three core-set selection techniques: greedy k- centers [Wolf] 2011 Sennr & Savarese, 2018), forgetting events (Toneva et 2019). and max entropy uncertainty sampling (Lewis Gale 194, (ettless/202] Greedy k-centers is described above and in Algorithm [ Forgetting events are defined as the number of times an example is incorrectly classified after having been correctly classified earlier during training a model, as described in Algorithm 2 To select points, we follow the same procedure as Toneva et al. (2019): we keep the points with the m highest number of forgetting events. Points that are never correctly classified are treated as having an infinite number of forgetting events. Similarly, we rank examples based on the entropy from trained target [n] as:
Afm)) - fcurroyy (x: Almu) = -
P(i)x
log P(jx APmn))
and keep the m examples with the highest entropy. To evaluate core-set quality, we compare the performance of training the large target model on the selected subset AT to training the target model on the entire dataset [n]
by measuring error on a held-out test set.
2.3 APPLYING SELECTION VIA PROXY
In general, SVP can be applied by replacing the models used to compute data selection metrics such as uncertainty with proxy models. In this paper, we applied SVP to the active learning and core-set selection methods described in Sections 2. 2.2 as follows:
For active learning, replaced the model trained at each batch (Af) with proxy (Af)) but then trained the same final model A% once the budget b was reached.
For core-set selection, we used proxy A(r) instead of A to compute metrics and select s.
We explored two main methods to create our proxy models:
Scaling down. For deep models with many layers, reducing the dimension or the number of hidden layers is an easy way to trade-off accuracy to reduce training time. For example, deep ResNet models come in variety of depths (He et al. 2016b a) and widths (Zaoorykko & Komodakis 2016) that represent many points on the accuracy and training time curve. As shown in Figure 4a in the Appendix, a ResNet20 model achieves top-1 error of 7.6% on CIFAR1O in 26 minutes, while larger 164 model takes 4 hours and reduces error by 2.5%. Similar results have been shown for many other tasks, including neural machine translation (Vaswani el al. 2017), text classification (Conneau ct al. 2016), and recommendation (He et al., 2017). Looking across architectures gives even more options to reduce computational complexity. We exploit the limitless model architectures in deep learning to trade-off between accuracy and complexity to scale down to proxy that can be trained quickly but still provides a good approximation of the target's decision boundary.
Training for fewer epochs. Similarly, significant amount of training is spent on a relatively small reduction in error. While training ResNet20, almost half of the training time (i.e., minutes out of 26 minutes) is spent 1.4% improvement in test error, shown in Figure 44] in the Appendix. Based on this observation, we also explored training proxy models for a smaller number of epochs to get good approximations of the decision boundary of the target model even faster.
3 RESULTS
We applied SVP to data selection methods from active learning and core-set selection five datasets. After brief description of the datasets and models in Section 3.I] Section 33.2 evaluates evalauusss SVP's impact on active learning and shows that across labeling budgets SVP achieved similar or higher accuracy
4

---

Published as a conference paper at ICLR 2020
and up 41.9x improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points). Next, we applied SVP to the core-set selection problem (Section (3.3). For all selection methods, the target model performed nearly as well as better with SVP than the oracle that trained the target model on all of the data before selecting examples. On CIFARIO, 10, a small proxy model trained for 50 epochs instead of 181 epochs took only 7 minutes compared to the 4 hours for training the target model for all 181 epochs, making SVP feasible for end-to-end training time speed-ups. Finally, Section 3.4 illustrates why proxy models performed so well by evaluating how varying models and methods rank examples.
3.1 EXPERIMENTAL SETUP
Datasets. We focused on classification as a well-studied task in the active learning literature (see Section A for more detail). Our experiments included three image classification datasets: CIFARIO, CIFAR 1O0 (Krizhevsky & Hinton 2009), and ImageNet (Russakovsky et al., 2013); and two text classification datasets: Amazon Review Polarity and Full (Zhang & LeCun [2015] Zhang et al. 2015). CIFARIO a wearseggraine classification task over 10 classes, and CIFAR is a linegrriied task with 100 classes. Both datasets contain 50,000 images for training and 10,000 images for testing. ImageNet has 1.28 million training images and 50,000 validation images that belong to 1 of 1,000) classes. Amazon Review Polarity has 3.6 million reviews split evenly between positive and negative ratings with an additional 400,000 reviews for testing. Amazon Review Full has million reviews split evenly between the 5 stars with additional 650,000 reviews for testing.
Models. For CIFARIO and CIFARIOO, we used ResNet164 with pre-activation from He et al. (2016b) our large target model. The smaller, proxy models are also ResNet architectures with pre- activation, but they use pairs of 3 x convolutional layers as their residual unit rather than bottlenecks. For ImageNet, we used the original ResNet architecture from He et al. (2016a) implemented in PyToorc|| (Pazkke et al.) 2017) with ResNet50 as the target and ResNet18 as the proxy. For Amazon Review Polarity and Amazon Review Full, we used VDCNN (Conneau et al. 2017 and fastText (Joulin et al. (016) with VDCNN29 as the target and fastText and VDCNN9 as proxies. In general, we followed the same training procedure as the original papers (more details in Section A.2
3.2 ACTIVE LEARNING
We explored the impact of SVP on two active learning techniques: least confidence uncertainty sampling and the greedy k-centers approach from (Sener& & Savarese (2018). Starting with an initial random subset of 2% of the data, we selected 8% of the remaining unlabeled data for the first round and 10% for subsequent rounds until the labeled data reached the budget b and retrained the models from scratch between rounds as described in Section 2.1 Across datasets, SVP sped up data selection without significantly impacting the final predictive performance of the target.
CIFAR10 and CIFAR100. For least confidence uncertainty sampling and greedy k-centers, SVP sped-up data selection by up to 7x and 3.8x respectively without impacting data efficiency (see Tables [ndd 3 despite the proxy achieving substantially higher top-1 error than the target 1es model (see Figure 6 in the Appendix). The speed-ups for least confidence were a direct reflection of the difference in training time between the proxy in the target models. As shown in Figures 4 and in the Appendix, ResNet20 was about 8x faster to train than ResNet164, taking 30 minutes to train rather than 4 hours. Larger budgets required more rounds of selection and, in turn, more training, which led to larger speed-ups as training became a more significant fraction of the total time. Training for fewer epochs provided a significant error reduction compared to random sampling but was not as good as training for the full schedule (see Table the Appendix). For greedy k-centers, the speed-ups increased more slowly because executing the selection algorithm added more overhead.
ImageNet. For least confidence uncertainty sampling, SVP sped-up data selection by up to 1.6x (Table [ despite ResNet18's higher error compared to ResNet50 (Figure 6g in the Appendix). Training for fewer epochs increased the speed-up to 2.9x without increasing the error rate of ResNet50 (Table 4). Greedy k-centers was too slow on ImageNet due to the quadratic complexity of
Algorithm
htm
5

---

Published as a conference paper at ICLR 2020
Table SVP performance on active learning. Average (# std.) data selection speed-ups from runs of active learning using least confidence uncertainty sampling with varying proxies and labeling budgets on four datasets. Bold speed-ups indicate settings that either achieve lower error or are within I std. of the mean top-1 error for the baseline approach of using the same model for selection and the final predictions. Across datasets, SVP sped up selection without significantly increasing the error of the final target. Additional results and details are in Table 3
Data Selection Speed-up
Budget (b/n)
10%
20%
30%
40%
50%
Dataset
Selection Model
CIFARTO
ResNet164 (Baseline)
1.0x
1.0x
1.0x
1.0x
1.0x
ResNetl10
1.8x
1.9x
1.9x
1.8x
1.8x
ResNet56
2.6x
2.9X
3.0%
3.1x
3.1x
ResNet20
3.8x
5.8x
6.7x
7.0x
7.2x
CIFARIO0
ResNet164 (Baseline)
1.0%
1.0x
1.0x
1.0%
1.0x
ResNet110
1.5x
1.6x
1.6x
1.6X
1.6x
ResNet56
2.4x
2.7x
3.0X
2.9X
3.1x
ResNet20
4.0x
5.8X
6.6X
7.0%
7.2X
ImageNet
ResNet50 (Baseline)
1.0x
1.0x
1.0x
1.0x
1.0X
ResNet18
1.2x
1.3x
1.4X
1.5X
1.6X
Amazon
VDCNN29 (Baseline)
1.0%
1.0%
1.0%
1.0%
1.0X
Review
VDCNN9
1.9%
1.8x
1.8x
1.8%
1.8x
Polarity
fastText
10.6x
20.6X
32.2X
41.9X
51.3x
Amazon Review Polarity and Amazon Review Full. On Amazon Review Polarity, SVP with a fastText proxy for VDCNN29 led to up to relative error reduction of 14% random sampling for large budgets 3). while being up to 41.9x faster at data selection than the baseline approach (Table D Despite fastText's architectural simplicity compared to VDCNN29 and higher error (Figure bc). the calculated confidences signaled which examples would be the most informative. For all budgets, VDCNN9 was within 0.1% top-1 error of VDCNN29, giving consistent 1.8 speed-up. On Amazon Review Full, neither the baseline least confidence uncertainty sampling approach nor the application of SVP outperformed random sampling (see Table in the Appendix), so the data selection speed-ups were uninteresting even though they were similar to Amazon Review Polarity, For both datasets, greedy k-centers was too slow as mentioned above in the ImageNet experiments.
3.3 CORE-SET SELECTION
CIFARIO and CIFARI00. For all methods on both CIFARIO and CIFARI00, SVP proxy models performed as well as or better than an oracle where ResNet164 itself used as the core-set selection model, shown in Figure 2 (and Figure 7 in the Appendix). Using forgetting events CIFAR10 10, SVP with ResNet20 as the proxy removed 50% of the data without a significant increase in error from ResNet164. The entire process of training ResNet20 all the data, selecting which examples to keep, and training ResNer164 on the subset only took hours and 20 minutes (see Table (6 in the Appendix), which was a 1.6' speed-up compared training 6esNet164 over all of the data. If stopped training ResNet56 early and removed 50% of the data based on forgetting events from the first 50 epochs, SVP achieved an end-to-end training time speed-up of 1.8x with only a slightly higher top-1 eгror from ResNet164 (5.4% vs. 5.1%) shown in Table 7 the Appendix. In general, training the proxy for fewer epochs also maintained the accuracy of the target model on CIFAR10 because the ranking quickly converged (Figure 11a and 12a in the Appendix). On CIFAR100 100, partial training did not work as well for proxies at large subset sizes because the ranking took longer to stabilize and were less correlated (Figure [Tb and Figure [2b in the Appendix). On small 30% subsets with forgetting events, partial training improved accuracy on CIFARIO0.
ImageNet. Neither the baseline approach nor SVP was able to remove significant percentage of the data without increasing the final error of ResNet50, as shown in Table 5 in the Appendix. However, the selected subsets from both ResNet18 and ResNet50 outperformed random sampling with up to 1 % drop in t error using forgetting events. Note, due to the quadratic computational complexity of Algorithm]] [ were unable run greedy k-centers in reasonable amount of time.

---

Published as a conference paper at ICLR 2020
Full Dotaset
Random
RccNet164 (Baseline)
ResNet56 (3.4x]
ResNat20 (8x)
Top-1 Error (9))
10.0
10.0
10.0
7.5
7.5
7.5
5.0
5.0
30
40
50
60
70
30
40
50
60
70
30
40
50
60
70
Subset of Data Used (%)
Subset of Data Used (%)
Subset of Data Used (%)
(a) CIFARIO forgetting events
(b) CTFAR 10 entropy
(c) CIFAR10 greedy k-centers
Figure 2: SVP performance on core-set selection. Average (#: std.) top-1 error of ResNet over
5 runs of core-set selection with different selection methods, proxies, and subset sizes on CIFARIO 10. We found subsets using forgetting events (left), entropy (middle), and greedy k-centers (right) from a proxy model trained over the entire dataset. Across datasets and selection methods, SVP performed as well as an oracle baseline but significantly faster (speed-up in parentheses).
1.0
1.0
R20
0.91
R20
0.67
R20
0.38
0.5
0.5
0.5
R56
0.91
0.91
R56
0.39
0.41
0.38
0.47
R110
0,90
R110
0.37
R110
-0.5
-0.5
-0.5
R164
0,90
0.90 0.90
0.91
R164
0.45
0.42
0.38
0.52
R164
0.40
0.46
0.45
0.50
R20 R56 R110
R164
R20
R56
R110
R164
R20
R56 R110 R164
(a) CIFARIO forgetting events
(b) CIFARIO entropy
(c) CIFARIO greedy k-centers
Figure Comparing selection across model sizes and methods on CIFAR10. Average Spear- man's is correlation between different runs of ResNet (R) models and at varying depths. We computed rankings based on forgetting events (left), entropy (middle), and greedy k-centers (right). We saw a similarly high correlation across model architectures (off coff-diagonal) as between different runs of the same architecture (on-diagonal), suggesting that small models are good proxies for data selection.
Amazon Review Polarity and Amazon Review Full. On Amazon Review Polarity, we were able to remove 20% of the dataset with only 0.1 % increase in VDCNN29's top-1 error using fastText as the proxy (see Table 5 In comparison to VDCNN29, which took 16 hours and 40 minutes to train over the entire dataset on Titan V GPU, fastText was two orders of magnitude faster, taking less than 10 minutes on a CPU to train over the same data and compute output probabilities. This difference allowed us train VDCNN29 to nearly the same error in 13 and a half hours. However, on Amazon Review Full, both the baseline approach and SVP failed to outperform random sampling. Similar to ImageNet, we were unable run greedy k-centers in reasonable amount of time, and additionally, Facebook's 's fastText implementation 2 did not allow us to compute forgetting events.
3.4 RANKING CORRELATION BETWEEN MODELS
Models with fewer layers. Figure 3 (and Figure 9 in the appendix) shows the Spearman's 's rank- order correlation between ResNets of varying depth for three selection methods on CIFARIO (and CIFAR 100). For greedy k-centers,
started with 1. 1,000 randomly selected points and ranked the remaining points based on the order they are added to set s in Algorithm[] Across models, there was a positive correlation similar the correlation between runs of the same model. For forgetting events and entropy, we ranked points in descending order based on the number of forgetting events and the entropy of the output predictions from the trained model, respectively. Both metrics had comparable positive correlations between different models and different runs of the same model. We also looked at the Pearson correlation coefficient for the number of forgetting events and entropy Figure [5 in the Appendix and found similar positive correlation. The consistent positive correlation between varying depths illustrates why small models are good proxies for larger models in data selection.

---

Published as a conference paper at ICLR 2020
Models with different architectures. We further investigated different model architectures by cal- culating the Spearman's 's correlation between pretrained ImageNet models and found that correlations were high across wide range of models (Figure g in the Appendix). For example. MobileNet V2's (Sandler et al. 2018) entropy-based rankings were highly correlated to ResNet50 (on par with ResNer18), even though the model had far fewer parameters (3.5M vs. 25.6M). In concert with our fastText and VDCNN results, the high correlations between different model architectures suggest that SVP might be widely applicable. While there are likely limits to how different architectures can be, there is wide range of offs between accuracy and computational complexity, even within a narrow spectrum of models.
4 RELATED WORK
Active learning. There are examples in the active learning literature that address the computational efficiency of active learning methods by using model select points for different, more expensive model. For instance, instance,Lewis & Catlett (194)) proposed heterogeneous uncertainty sampling and used Naive Bayes classifier to select points label for a more expensive decision tree target model. Tomanek et al. (2007) uses committee- based active learning algorithm for an NLP task and notes that the set of selected points are "reusable" across different models (maximum entropy. conditional random field, naive Bayes). In our work, we showed that this can be generalized to deep learning by either using smaller models or fewer training epochs, where it can significantly reduce the running time of uncertainty-baat based (Senlle) 2012, Shen et al.) 2017 Gal et al. 2017) and recent representativeness- based (Sener & Savarese 2018) methods.
Core-set selection. Core-set selection attempts to find a representative subset of points to speed up learning or clustering; such as k-means and k-medians Har-Peled & Kushal 2007), SVM (Tsang et al. 2005). Bayesian logistic regression (Huggins et al. 2016), and Bayesian inference (Campbell & Broderick 2017] 2018). However, these examples generally require ready-to-ur use features as input, and do not directly apply to deep neural networks unless feature representation is first learned, which usually requires training the full target model itself. There is also a body of work on data summarization based submodular maximization (Wei et al., 2013) 2014 TTchiatschhk et al.) 2014 Ni et al. 2015), but these techniques depend on a combination of hand-engineered features and simple models (e.g., hidden Markov models and Gaussian mixture models) pretrained on auxiliary tasks. In comparison, our work demonstrated that we can use the feature representations of smaller, faster-to-train proxy models as an effective way to select core-sets for deep learning tasks.
Recently. Toneva et al. (2019) showed that large number of "unforgettable" examples that are rarely incorrectly classified once learned (i.e., 30% on CIFAR1O) could be omitted without impacting generalization, which can be viewed as a core-set selection method. They also provide initial evidence that forgetting events are transferable across models and throughout training by using the forgetting events from ResNet18 to select subset for WideResNet (Zagoruyko & Komodakis 2016) and by computing the Spearman's 's correlation of forgetting events during training compared to their final values. In work, we evaluated similar idea of using proxy models to approximate various properties of a large model, and showed that proxy models closely match the rankings of large models in the entropy, greedy k-centers, and example forgetting metries.
5 CONCLUSION
In this work, we introduced selection via proxy (Sv improve the computational efficiency of active learning and core-set selection in deep learning by substituting a cheaper proxy model's representation for an expensive model's s during data selection. Applied to least confidence uncertainty sampling and Sener& & Savarese (2018) 's greedy k-centers approach, SVP achieved up to 41.9x and 3.8x improvement in runtime respectively with no significant increase in error (often within 0. 0.1%). For core-set selection, found that SVP can remove up to 50% of the data from CIFAR10 in 10x less time than it takes to train the target model, achieving 1.6x speed-up in end-to-end training.
Code available http:///whhhhaa...ee./eeeerrrrrr
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This research was supported in part by affiliate members and other supporters of the Stanford DAWN project- Ant Financial, Facebook, Google, Infosys, NEC, and VMware as well as Toyota Research Institute, Northrop Grumman, Amazon Web Services, Cisco, and the NSF under CAREER grant CNS-16515700 Jure Leskovec is a Chan Zuckerberg Biohub investigator. We also gratefully acknowledge the support of DARPA under Nos. 01865018C7880 (ASED), N66001 1924033 (MCS); 110F-1661-0344 1-0342 (MURI). 1991NNF16-1-0011 (DURIP); NSF under Nos. OAC- 1835598 (CINES), 0AC-1934578 (HDR), DGE-1656518 (GRFP), DGE-114770 (GRFP); SNSF under Nos. 1EE2P21772887 Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, JD.com, Amazon, Boeing, Docomo, Huawei, Hitachi, Observe, Siemens, UST Global. The U.S. Government is authorized reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of SNSF, NSF, DARPA, NIH, ARO, or the U.S. Government.
ARO under Nos.
16-
REFERENCES
Trevor Campbell and Tamara Broderick. Automated scalable bayesian inference via hilbert coresets. arXiv preprint
2017.
Trevor Camphell and Tamara Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. arXiv preprint arXiv:1802.0173: 2018.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional networks for text classification. arXiv preprint arXiv:1606..c 01781, 2016.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional networks for text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 1, Long Papers, pp. 1107-1116. Association for Computational Linguisties, 2017. URL http:: http:://awwweeh org/ wtt.://aaaaaoooogg//-------
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /803.0330,3 2018
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Eearning-Volume 70, pp. 1183-1192. JMLR. org, 2017.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 2017.
Sariel Har- Peled and Akash Kushal, Smaller coresets for k-median and k-means clustering. Discrete & Computational Geometry, 37(1):3-19, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016b.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, pp. 173-182, International World Wide Web Conferences Steering Committee, 2017.

---

Published as a conference paper at ICLR 2020
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, volume pp. 3, 2017.
Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. In Advances in Neural Information Processing Systems, pp. 4080-40455 2016.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07:30 2016.
Armand Joulin, Edouard Grave, Piotr Bojanowski. and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv: 2016.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint wrXiv:1602.024100 2016.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. arXiv preprint
2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine Learning Proceedings 1994, pp. 148-156. Elsevier, 1994.
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In Proceed- ings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 3-12. Springer- Verlag New York, Inc., 1994.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 2017.
Stephen Mussmann and Percy Liang. On the relationship between data efficiency and error for uncertainty sampling. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 3674-6682 Stockholmsmässsan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:://proceedingg mlm www.:pressseee.ee wwwps///wm/msssseeeeeeeee /tt////mmmmmmmmmmmmm/mmm éa. html
Chongjia Ni, Cheung-Chi Leung, Lei Wang, Nancy F Chen, and Bin Ma. Unsupervised data selection and word-morph mixed language model for tamil low-resource keyword search. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 4714-4718 IEEE, 2015.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Álvaro Peris and Francisco Casacuberta. Active learning for interactive neural machine translation of data streams. In Proceedings of the 22nd Conference Computational Natural Language Learn- ing, pp. 151-160, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653v11/K111 1015. URL httpt:tt////t....... www. a web. web. hoyhohoooyyyycooooooooo /K1 8- 8-
Charles Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self- self-training of object detection models, January 2005.
10

---

Published as a conference paper at ICLR 2020
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255)) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. In International Conference on Learning Representations, 2018. URL https://
openreview wwtpeeeiwwwwwweeeeeeeee forum Hiw.mmmmid-HHaauui RM
Burr Settles. From theories queries: Active learning in practice. In Isabelle Guyon, Gavin Cawley, Gideon Dror, Vincent Lemaire, and Alexander Statnikov (eds.), Active Learning and Experimental Design workshop conjunction with AISTATS 2010, volume 16 of Proceedings of Machine Learning Research, pp. 1-18, Sardinia, Italy, 16 May 2011. PMLR. URLhttp: l/proceedings.nne mlr press http://leeeeetttttle.eeem
Burr Settles. Active learning. Synthesis Lectures Artificial Intelligence and Machine Learning, 6 (1):1-114, 2012.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 287-294. ACM, 1992.
Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 252-256, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi:
URL httpst/s:// www ac acllww wwe.. org/ ant loty/
W17 W17-2633
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1958 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Katrin Tomanek, Joachim Wermter, and Udo Hahn. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning LEMNLP-CONLL), 2007.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference Learning Representations, 2019. URL htttttt://pppeeeet////
/net/forie 3tw.rrmmmmmd333mmmmmmmmmmm3m
Ivor W Tsang, James Kwok, and Pak-Ming Cheung. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research, 6(Apr):363-392, 2005.
Sebastian Tschiatschek, Rishabh K Iyer, Haochen Wei, and Jeff A Bilmes. Learning mixtures of submodular functions for image collection summarization. In Advances in neural information processing systems, pp. 1413-1421, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998 6008, 2017.
11

---

Published as a conference paper at ICLR 2020
Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff Bilmes. Using document summarization techniques for speech data subset selection. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 721-726, 2013.
Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. Submodular subset selection for large-scale speech training data. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pp. 3311-3315. IEEE. 2014.
Gent W Wolf. Facility location: concepts, models, algorithms and case studies., 2011.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pp. 5987-5995. IEEE, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint (rXXiv1605.071444 2016.
Xiang Zhang and Yann LeCun. Text understanding from scratch. arXiv preprint arXiv: 1502.01710. 2015.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657, 2015.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 CHOICE OF DATASETS
For our experimental evaluation in Section 3 we focused on classification because it is a widely
studied task in active learning Lewis & Gale 1994 Lewis & Catleti 1994 Settles 2012: Sener
& Savarese 2018, Kirsch et al. 2019: Mussmann & Liang 2018: Gal et al. Houlsyy et al.
2011). While there a few bespoke solutions for machine translation (Peris & Casacuberta 2018))
and named entity recognition (Shen et al. (017). we wanted to compare against the broader body of active learning research. Many popular active learning methods like uncertainty sampling (e.g.. entropy, least confidence, and max- max-margin) assume a single categorical probability distribution for each example, which makes it hard to adapt to other domains. Instead of tackling open challenges for active learning like machine translation, we applied SVP to many classification datasets; however, the simplicity of SVP means that the core ideas of this paper can be applied more broadly in future work.
Specifically, we performed experiments on three image classification datasets: CIFAR1O, 10, CI- FARIOO (Krizhevsky & Hinton 2009), and ImageNet (Russakovsky al. 2015;; and two text classification datasets: Amazon Review Polarity and Full (Zhang & LeCun 2013 Zhang et al. 2013). While multiple tasks on roughly the same data distribution may seem redundant, the data efficiency of active learning depends on error (Mussmann & Liang, 2018). We included both CIFAR10 (low-error) and CIFARIÓN 100 (high-error) to demonstrate that our approach performs as well as standard active learning at different points on the error and data efficiency curve. The same rationale is also valid for Amazon Review Polarity (low-error) and Full (high-error). However, the Amazon Review dataset adds medium (text) and a much larger scale (3.6M and 3M examples, respectively). Adding ImageNet further allows us to investigate scale in the number of examples, but also the number of classes and the dimension of the input. To the best of our knowledge, we the first active learning paper to present results on the full ImageNet classification task.
A.2 IMPLEMENTATION DETAILS
CIFARIO and CIFARIOO. We used R6s with pre-activation from He et al. (2016b) as our large target model for both CIFARIO and CIFAR100. Note that as originally proposed in He et al. (0016a), the smaller, proxy models are also ResNet architectures with pre-activation, but they use pairs of 3 x convolutional layers as their residual unit rather than bottlenecks and achieve lower accuracy as shown in Figure 4 As with He et al. (2016b), the ResNets we used were much narrower when applied to CIFAR rather than ImageNet (256 filters rather than 2048 in the final layer of the last bottleneck) and have fewer sections, which means far fewer weights despite the increased depth. For example, ResNet50 on ImageNet has -25M weights while ResNet164 on CIFAR has -1.7M (see Table 2 More recent networks such as Wide Residual Networks (Zagoruyko & Komodakis 2016). ResNeXt (Xie et al. 2017), and DenseNets (Huang et al., 2017) use models with more than 25M parameters on CIFARIO. 10, making ResNet164 relatively small in comparison. Core-set selection experiments used a single Nvidia P100 GPU, while the active learning experiments used a Titan GPU. We followed the same training procedure, initialization, and hyperparameters as He al. (2016b) with the exception of weight decay, which was set to 0,0005 and decreased the model's 's validation error in all conditions.
ImageNet. et. we used the original ResNet architecture from He et al. (21 implemented in Py- Torch Paszke et al. 2017) with ResNet50 as the target and ResNet18 as the proxy. For training, we used a custom machine with 4 Nvidia Titan V GPUs and followed Nvidia's 's optimized implemen- tation with a larger batch size, appropriately scaled learning rate (Goyal et al. 2017). a 5-epoch warm-up period, and mixed precision training (Micikevicius et al. 2017) with the apex library. For active learning, we used the same batch size of 768 images for both ResNet18 and ResNet50 for simplicity, which was the maximum batch size that could fit into memory for ResNet50. However, R8sNet18 with a batch size of 768 underutilized the GPU and yielded a lower speed-up. With separate batch sizes for R8 and ResNet50, we would have seen speed-ups closer to 2.7x.
13

---

Published as a conference paper at ICLR 2020
Table Number parameters in each model.
Dataset CIFARIO
Model
Number of Parameters (millions)
ResNet164
1.7
ResNet1
1.73
ResNet56
0.86
ResNet20
0.27
ResNet14
0.18
ResNet8
0.08
CIFARTOO
ResNet1
1.73
ResNet1 10
1.74
ResNet56
0.86
ResNet20
0.28
R
0.18
ResNet8
0.08
ImageNet
ResNet50
25.56
ResNet18
11.69
Amazon Review Polarity
VDCNN29
16.64
VDCNN9
14.17
Amazon Review Full
VDCNN29
16.64
VDCNN9
14.18
Amazon Review Polarity (2-classes) and Full (5-classes). For Amazon Review Polarity and Ama- zon Review Full, we used VDCNN (Conneau et al., 2017) and fastText (Joulin et al. 2016) with VDCNN29 as the target and fastText and VDCNN9 as proxies. For Amazon Review Polarity, core-set selection experiments used a single Nvidia P100 GPU, while the active learning experiments used Nvidia Titan V GPU to train VDCNN models. For Amazon Review Full, core-set selection and active learning experiments both used a Nvidia Titan GPU, In all settings, we used the same training procedure from Conneau al. (2017) for VDCNN9 and VDCNN29. For fast'Text, used Facebook's 's implementation and followed the same training procedure from Joulin et (2016).
A.3 MOTIVATION FOR CREATING PROXIES
60
Top-1 Test Error (96)
12
ResNet8
Top-1 Test Error (%)
50
10
40
ResNet14
30
Mardyy/
ResNet20
ResNet56 ResNet110
20
ResNet164
10
0
1
2
0
10 15
20
25
Training Time in Hours on P100 GPU
Training Time in Minutes on 1 P100 GPU
(a) Top-1 I test error and training time on CIFARIO for ResNet with pre-activation and varying number of layers. There is diminishing return in accuracy by increasing the number of layers.
(b) Top-1 test error during training of ResNet20 with pre-activation. In the first minutes, ResNet20 reaches 9.0% top-1 error, while the remaining 12 min- utes are spent on decreasing error to 7.6%
Figure Top-1 test error on CIFARIO for varying model sizes (left) and over the course of training single model (right), demonstrating large amount of time is spent on small changes in accuracy.
14

---

Published as a conference paper at ICLR 2020
Top-1 Test Error (%6
40
ResNet8
Top-1 Test Test Error (%)
80
ResNet14
ResNet20
60
ResNet56
ResNet110
25
40
ResNet164
0
1
10 15
25
Training Time in Hours on P100 GPU
Training Time in Minutes on 1 P100 GPU
(a) Top-1 test error and training time on CIFAR 100 for ResNet with pre-activation and a varying number
(h) Top-1 test during training of ResNet20 with pre-activation. In the first minutes, ResNet20 reaches 33.9% top-1 error. while the remaining 12 minutes are spent on decreasing error to 31.1%
of layers. There are diminishing returns in accuracy from increasing the number layers.
Figure Top-1 test error on CIFARTO0 for varying model sizes (left) and over the course training single model (right), demonstrating large amount of time is spent on small changes in accuracy.
15

---

Published as a conference paper at ICLR 2020
A.4 ADDITIONAL ACTIVE LEARNING RESULTS
25.0
65
22.5
60
20.0
Test Eree 194
55
17.5
(Tayer
RIB ("apet)
15.0
45
12.5
40
30.0
-55
7.5
30
5.0
25-
2.50
240
Budget Labeled Data (76)
Budget of Labeled Data
(a) CIFARIO greedy k-centers
(b) CIFARI00 greedy k-centers
65p
1ItA (Prm))
Test Errur 163
(Proo)
45
35
7.5
5.0
25
2.50
2%0
15
20
25
30
35
4.5
455
50
15
20
25
30
35
60
45
50
Budget Labeled Data 176)
Budget o of Labeled Data 1961
(c) CIFARIO least confidence
(d) CIFAR100 least confidence
7.0
44-
43-
('arett
FEACO (heran)
Error 80
442
400 ('srett
6.0
Ful: Salaie:
41
Bit
5.5
5.0
39
4.5
37
4.0
360
Labeled Data
Budget of Labeled Data
(e) Amazon Review Polarity least confidence
(f) Amazon Review Full least confidence
45
40
25
30
25
20
15
20
25
30
35
40
65
50
Bunget Labeled Dats (%)
(g) ImageNet least confidence
Figure 6: Quality of proxies compared to target models. Average (+ std.) top-1 error from 3 runs of active learning with varying proxies, selection methods, and budgets on five classification datasets. Dotted lines show the top-1 error of the proxy models, while solid lines show the top-1 error of the target models. CIFAR 10 and 100 experiments used varying depths of pre-activation ResNet (R) models as proxies and ResNet1 64 (R164) as the target model (e.g., R20-R1 164 is ResNer20 selecting for ResNet164) ImageNet used ResNet18 (R18) as the proxy and ResNet50 (R50) as the target. Amazon Review Polarity and Amazon Review Full used VDCNN9 (V9) and fastText (FT) as proxies and VDCNN29 (V29) as the target. Across datasets, proxies, methods, and budgets, smaller proxies had higher I error than the target model, but selecting points that were nearly as good as the points selected by the target that did not harm the final target model's 's predictive performance.
16

---

Published as a conference paper at ICLR 2020
Table 3: SVP performance on active learning, Average (#: std.) top-1 error and data selection speed- speed-ups ups from 3 runs of active learning with varying proxies, methods, and labeling budgets on five datasets. Bold speed-ups indicate settings that either achieve lower error or are within I std. of the mean top-1 crror for the baseline approach of using the same model for selection and the final predictions. Across datasets and methods, SVP sped up selection without significantly increasing the emor of the final target.
50%
1.0
1.8x
3.1x
7.2x
1.0x
1.6x
2.8x
5,5x
1.0x
1.6x
3.1x
7.2x
1.0x
1.6x
3.0x
6.2
L0x
1.6x
10X
1.8x
51.3x
100
1.8x
4
Data Sclection Speed-up
40%
1.X
1.8x
3.1x
7.0x
1.0x
1.7x
2.8x
533x
1.0%
1.6x
2.9%
7.0x
1.0%
1.7x
3.1x
6.1
10X
1.5x
LOX
1.8%
41.9x
1.0%
188%
35.1x
30%
1.x
1.9x
3.0x
6.7x
1.0x
1.7x
2.8x
5.0x
10x
1.6x
3.0%
6.6%
1.0%
1.8%
3.1x
5.9%
LOX
1.4X
1.0
1.8x
32.2x
1.X
188%
26.7x
20%
1.0x
1.9x
2.9x
5.8%
1.0x
1.8x
2.9x
4.6x
1.0x
1.6x
2.7x
5.8%
1.0%
1.9x
3.2x
5.5x
1.0%
1.3x
1.0x
1.8x
20.6x
1.0
1.9%
17.7x
10%
1.0
1.8 x
2.6
3 x
1.0%
2.1x
3.0%
388
100
1.5x
2.4
4.0%
1.0%
2.3
3.3
4.5
1.0%
1.2x
1.0
1.9%
10.6x
1.0 X
2.0
8.7x
50%
7.5 0.11
5.3 5. 0.06
5.3 5. 0.05
5.5 _ 0.08
5.4 0.41
0.41
5.6 0.04
5.7 5.7 = 0.13
5.7 5.7 0.19
6.1 6.1 0.18
29.3 1. 0.16
1 0.21
27.2 1 0.25
26.4000 0.13
26.6 0.14
= 0.21
27.4 0.21
27.8 0.23
28.5 0.48
27.8 1 0.13
1. 0.16
0.02
4.7 0.03
4.2 0.02
4.3 003
4.3 0.02
37.00.10 0.10
1 0.01
37.7 1 0.05
26.9
26.4
27.3
26.3
26.4
37.6
37.7 1.
40%
8.5 3 0.22
6.1 1 01.32
5.9 + 0.33
6.1 111 0.06
5.9 0.19
6.6 6.6 0.24
0.10
6.3 0.17
0.19
0.48
1.18
29.7 0.41
0.08
0.10
+ U.33
0.46
0.22
0.25
31.6 0.31
0.42
1 0.32
28.2 1013
1.91
0.02
4.3 4.30
4.3
38.4
38.2 1: 0.03
38.1 1 0.09
18.1 0.06
04+00.1
11.400.4
19.90044
Top-1 Error of Target Model (%)
64
31.9
29.9
29.7
29.8
30.2
30.3
30.8
29.9
28.3
4.3
38.1
30%
10.1 0.24
7.4=0.1
7.5 7 0.11
7.4 7.4 0.10
7.4 0.23
8. 0.22
8.1 8.1
0.16
8 0.211
8.40
36.0 0.42
0.33
34 + 0.38
33. 43.7 0.33
33 8 0.33
0.40
0.40
35.2 0.40
35.8
$2.5 1.11
31.0
0.10
31.110.12
5.2 0.07
4.5 1001
4.5 4.5+0.0
0.02
4.6
39.0
6 0.10
38. 7
0.05
1
33.9
34.5
34.9
32.5
38. 7
38.6
20%
12.9 + 0.37
1 0.38
10.5 0.06
10.3 = 0.28
10.5 10.5 1 0.42
0.40
11.6 = 0.16
11.6 0.16
12.1 1 0.44
0.55
0.67
42.3 1 0.95
42.0 0.17
41.4 114+02 0.25
42.4 = 0.57
42.2 0.76
= 0.47
12.900.0 0.52
1 0.34
0.2
1019
5.6 0.07
0.04
0.0
5.2 0.17
1.05
1 0.22
1 0.23
1 0.02
42.6
36.110.1
4.9+000
10.4
11.3
42.5
42.2
42.0
42.6
42.9
37.5
35.9
36.1
4.8
4.9
39.9
39.7
39.8
39.8
10%
20.3 + 0.51
18.7 I 0.31
18.1 + 0.41
18.2 1 0.73
18.1 18.1 0.28
20.1 0.39
19.4 0.55
19.8 0.49
19.5 19.5 0.76
60.7 0.81
61.2 1.09
60.2 + 0.84
61.5 61.5 0.93
62.4 + 1.07
60.1 = 1.30
59.6 0.78
60.9 = 1.08
60.2 + 1.27 2.
48.5 08.51000 0.04
48.2 0.37
48.3 48.3 0.31
6.5 6.5 0.03
0.08
5.8 5.8 0.11
6.9 0
41.7 41.7 1.199
41.9 1 0.54
42.0 0.44
42.7 42.7 0.77
5.8 1
Budget (b/vx)
ResNet 644 (Base lime)
ResNet164 (Basc (Ba) (mm)
ResNetl (Base lime) (ane
ResNetl 664 (Base (mn)) (Bae
VDCNN29 (Baseline)
VDCNN29 (Baseline)
ResNer50 (Baseline)
Selection Model
ResNet 10
ResNe156
ResNet20
ResNet1
ResNer56
ResNet20
ResNetl 10
ResNe156
ResNer20
ResNet 10
ResNer56
ResNet20
ResNer18
VDCNN9
VDCNN9
fast
fastText
Confidence
Confidence
Confidence
Confidence
Confidence
Method
Random
k-Centers
Random Least
k-Centers
Random
Random
Random
Groedy
Greedy
Least
Least
Least
Least
10
CIFARION 100
ImageNet
Dataset CIFARIO
Amazon Review
Polarity
Amazon
Review
Full
17

---

Published as a conference paper at ICLR 2020
Table 4: Performance of training for fewer epochs on active learning. Average (# I std.) top-1 crror and data selection speed-upp -up from 3 runs of a'activee learning with varying proxies trained for a varying number of epochs on CIFAR CIFAR 100, and ImageNet, Bold speed-ups ups indicate settings that either achieve lower error or are within I std. of the mean top- I enor for the baseline approach of using the same model for selection and the final predictions. Training for fewer epochs can provide a significant improvement over random sampling but is not quite as good as training for the full schedule.
50%%
L0x
1.8x
366
7.2
12.7x
25.6x
1.0%
1.5x
2.5x
15x
2,5x
4.7%
2.6
4.8x
888
52x
10.2
17.2
1.0%
1.8%
3.6%
7.2x
13.1x
25.1
1.0%
1.6x
2.6x
1.6x
2.3x
5.0x
3.1x
5.0%
9.2
6.1%
10.9
17.7x
1,x
1.7
1,6x
3.1x
Data Selection Speed-upg up
40%
1.0x
188x
366
7.0x
12.3x
24.7x
1.0%
1.5x
2.3x
1.x
266
4.7x
277x
488
8
5.0%
9.7x
13.6x
1.0%
188x
3.6x
7.0x
12.7%
24.3x
1.0x
1.5x
2.5x
1.7x
24x
5.0x
3.1x
4
9.1x
39x
10.3x
163x
1.0%
188
15X
29x
30%
36x
11.6
23.1x
1,0
222
16X
26x
4.7%
26X
4.8
811
488
8.9
13.7x
10x
18x
36x
6.6%
12.0
22.4x
10X
15x
23xx
18x
25x
5.0%
32x
4.7x
87x
58X
9.5x
14.3x
10X
18x
14X
2.7x
20%
1.8x
3.6x
5.8%
10.3x
19.4%
1,0X
1.4X
1.9%
188
288
4.8x
4.7x
7.5x
4.5%
7.6X
11.1X
100
188x
3.6X
588
10.4x
18.9
1.0X
1.4x
2.0x
1.x
2.7x
5.0x
3.2%
4.7x
8.1x
3.4X
8.2x
11.6x
1.0%
1,8x
1.3%
25x
10%
1.8x
3.4x
3,8%
6.8%
11.4x
1.0%
1.3x
2.1x
3.2x
4.7x
2.9%
4.6x
6.3x
3,7X
5.8%
7.7x
1.0%
1.8x
3.3x
4.0%
6.8%
11.5x
1,0x
1.3x
1.7x
2.3x
311x
5.1x
3.4%
4.6x
688
4,5
6.xx
8.1x
1.0%
1.7x
1.2%
211x
50%
7.5 0.15
5.3 0.00
5.3 1 0.13
5.9 5. 0.14
54 441041
5.3 0.16
5.9 0.31
5.0 0005
6.1 0.10
7.0 0.11
5.6 0.10
5.8 5.8 0.07
6.9 0.122212
5.7 0.21
6.2 0.17
7.0 0.15
6.3 0.06
0.11
7.1 7.110.13
0.200
26.0
0:21
26.9 26.9 0:24
27.5 1043
26.0 1011
27.0 0.26
27.9 27.9 0.84
27.4 0.00
27.8 0.43
29.2 0.23
27.5 0.17
0.12
29.1 0.48
0.07
0.06
0.53
0.54
28.8 0.23
30.2
0.18
1
10
26.5 0
26.4 0.00
0.08
26.6
20.4
26.0
27.5
26.0
27.0
27.4
27.8
27.5
27.7
29.1
27.0
27.8
29.6
28.7
30.2
27.8
26.3
40%
8.0 0010
6.1 0..32
6.0 1.
7.2 0.115
5.9 10.10.1
5.9 6.931.31
6.8 +:0.0
6.7 10.25
6.8 +0010
8.0
4..00
6.9 0.00
+.012
6.2 +0.10
7.0 0
8.1 0
7.2 1.19
7.4 0.33
833 1023
02.0
1018
29.8 10
30.8 1
29.8 10.10
29.7 10.00
30.9 + 23
00.30
30.3 00.00 46
32.2+ 00222
+0.65
30.6
20.38
20225
10.3
30.9 10557
32.6 10062
31.7 10 4.
0.338
32.9 10
49
10..2
10.32
28.3 19
28.2.0.13
10.11
Top- Error of Target Model (%)
29.9
29.8
30.9
30.4
32.2
30.3
30.6
32.1
30.8
31
32.9
29.9
28.3
30%
10.1 0.18
7.4 7.410. 0.16
7.3 0.08
844 0.15
1410023
7.4 1.13
8.5
8.2 $2
8.7
9.9 0.29
0.29
8. 0.19
8.0 0.14
9.7 0.31
83 0.18
8.7 0.23
0.6 0.26
8.7 0.722
9.1 0.08
10.4 1144000 0.04
0.04
362 0.36
$3.90033 033
33 1. 0.60
35.2 1 0.23
83 0.7
34.3 0.37
45.4+02.2
0.23
34.4 0:45
34.9 0.09
30.3+033
033
0.:00
35.0 0.20
0.97
0.55
35.1 03
36.5 0.32
35.8 066
35.6122
0.23
1.05
1.05
0012
11.10.100
31.3
3
31.3
36.4
35.3
37.3
33.9
35.4
34.
36.3
31.7
36.
35.3
36.5
35.6
32.5
20%
12.0 12.00..0 0.30
10.4 10.400..0
10.3 1. 0.13
11.7 0.55
0.
10.3 0.20
11.5 0.33
11.3
0.08
133 0.23
11.50.22
0.15
12.9 0.31
11.6
11.7
13.0
0.44
12.00.600
12610.119 19
0.22 22
42.3
1.67
1141022
27
1022 24
41.4022
42.20.000
0.61
4
0.58
0..4
0
43.0
42.2 1.06
0..47
43.8 0.51
42.7 0.47
43.2
0.33
44.40.50 0.36
42840.70
13.20.039
45.1 0.61
37. 1034
35.
1022
36.3 0.03
36 0.
3.07
11.7
10.5
11.9
11.9
12.9
13.7
44.1
43.5
42.1
42.7
43.0
42.7
42.7
43.2
44.4
42.8
43.2
35.9
10%
20.2 0.10
18.7 0.31
18.4 0.23
19.2 0.35
18 181 0.28
184 0.33
1888 188 D.81
20.1 1033
19.9 19.9 +
21.9 11558
193 103+1
19.6 0.74
21.0 21.0 1191
11.7 19.7 11.18
19.7 19.7 0.83
20.5 20.5 0.05
19.0 0.35
20.1 0.31
21.6 0.55
6100
612 61221000
1.00
61.1 61.1 1.57
1.57
62.5 2.49
624
61.9 61.91.112
1.12
62.7 1.40
60.55 60.5 01.00
60.4+1665 1.65
60.5 0.11
6I.1 10.60
12.4
60.7 60.7.00.71 0.71
61.9 61.9 1.07
60.1 1.27
60.0 60.0 0.50
61.9 61.9 1.01
61.6 0.65
48.5 0.04
48. 2 1.37
18.70022
18310..3
4.31
Hudget (b/fn)
Epochs
181
100
50
100
50
181
100
50
181
100
50
181
100
50
181
100
50
181
100
50
18]
100
50
181
100
50
181
100
50
181
100
50
181
100
50
90
45
90
45
ResNeil 64 (Baseline)
ResNerTo4 (Baseline)
Re (Baseline)
ResNer 64 (Baseline)
ResNersO (Baseline)
Selection Model
Re
ResNerl11 10
ReaNer20
ResNe156
ResNer20
Res
ResNer56
ResNer200
ResNer18
Confidence
Least Confidence
Confidence
Method
Random
Groody K-Centers
Random
Greedy k C-Centers
Random
Least
Least
CIFARIOK 100
Dataset
CIFARIO
ImageNet
18

---

Published as a conference paper at ICLR 2020
A.5
ADDITIONAL CORE-SET SELECTION RESULTS
(Baseline)
(1.7x)
(8x)
Top-1 Error (%)
40
30
40
50 Subset of Data Used (%)
Subset of Data Used (%)
Subset of Data Used (%)
(a) CIFAR100 forgetting events
(b) CIFAR 100 entropy
(c) CIFAR 100 greedy k-centers
Figure SVP performance on core-set selection. Average (#: std.) top-1 error of ResNet 64 over 5 runs of core-set selection with different selection methods, proxies, and subset sizes on CIFAR 100. We found subsets using forgetting events (left), entropy (middle), and greedy k-centers (right) from proxy model trained over the entire dataset. Across datasets and selection methods, SVP performed as well as an oracle baseline but significantly faster (speed-ups in parentheses).
Table 5: Average top-1 methods on ImageNet, Amazon Review Polarity, and Amazon Review Full.
(+ 1 std.) from of core-set selection with varying selection
Top-1 Error (%)
Subset Size
40%
60%
80%
100%
Datasct
Method
Selection Model
ImageNet
Random
32.2 + 0.12
28.0
0.15
25.8
0.06
23.3 0.11
Entropy
ResNet50 (Baseline)
34.9 0.08
28.5 2 0.03
25,9
0.04
Entropy
ResNet18
32.2 :1 0.04
27.0
0.01
25.1
0.07
Forgelting Events
NesNe (Baseline)
31.9400000
26.7
0.06
24.8 4: 0.03
Forgetting Events
ResNet18
3.06
27.1
Amazon Review Polarity
Random
4.9
4.5 0.05
4.3 0.01
0.04
Entropy
VDCNN19 (Buseline)
0.44
4.02
4.240..22
Entropy
VDCNN9
0.02
4.20000
0.20
Entrpy
0.02
Amazon Review Full
Random
4.04
37.6 0.03
37.0 0.05
36.6 0.06
Entropy
VDCNN29 (Baseline)
42.71111
39.10.11
37.640.10
Entroy
VDCNN9
41.14
38.8
0.03
Entropy
fastText
3..0
37.8
0.06
3.01
19

---

Published as a conference paper at ICLR 2020
Table 6: Average (#: I std.) top-1 I error and runtime in minutes from 5 runs of core- sere-set selection with varying proxies, selection methods, and subset sizes on CIFARIO and CIFARTOO.
Table 7: Average top-1 error (#:) std.) and runtime in minutes from 5 runs of core-set selection with varying selection methods calculated from ResNet20 models trained for a varying number of epochs on CIFARIO and CIFARI00.
70%
425 1 41.7
+ 5.7
5.1
382 - 4.6
1 5.0
2.8
3.1
1: 3.7
126-:35 3.8
4.6
3.9
11.6
4.7
6.1
5.2
: 7.3
7.1
3.6
230 5.9
201 +37
70%
382 382 4.6
185 5.0
178 2.5
169 1.0
164 1.5
08213.1
190
176
1.9
0.7
4.7
193.16.
6.1
177 1.0
177 8.5
164 1.0
7.1
433.6
176 1.5
170 1.6
1 1.3
Total Runtime in Minutes
193
230
382
185
226
382
190
226
400
400
198
230
303
381
193
228
203
380
193
Total Runtime in Minutes 30%
582
160
166
38
193
380
193
165
50
1
04.3
147
1.0
1.0
1
1 4.8
6.8
142
2.5
+ 2.5
4.6
22
145 1.3
1.3
182 4.0
158.
151 3.6
185
1.5
8.5
253+
342
5.5
3,6
183 38
4.9
340
7.3
148
186
3.9
3.9
247 3.5
50%
340 +
142
135
126
1.4
1
+ 1.5
338
22
145
145113
1.3
134
+ 1.2
126
= 1.6
123
1 1.2
342
35.
148 3.6
131
134
132 +
123
+ 1.2
340
2.6
108
135 0.7
128
+ 1.3
123
039
406
187
340
184
338
185
253
148
183
250
340
148
186
440
296 3.2
2.6
1.5
2.0
0.8
1.5
#2
921
1.6
81
81
+ 5.7
10413 3.3
92 0.7
87 $7 3.6
81 1.1
1 1.3
1.7
1.5
85 85
0.
30%
342 1 47.7
104 + 1.9
142 4.7
- 3.2
2.6
5.4
1.5
2.2
4.8
52.7
+
142 1.7
208 7.3
5.7
104 1.3
141 T.1
207 +
173
105 1.7
142 12211.
1.9
3.3
101
92
85
1:
80
1
296
103
92
208
208
104
81
297
105
91
296
101
141
290
103
141
339
105
298
297
204
Data D Selection Runtime in Minutes
70%
219 1.5
25 1.5
14 0.7
7
d 4.2
166
25 1.
14 14
744
4102
261
25
0.7
14 1.3
8 0
25 2.7
14
8
4#0.1
218
Data Selection Runtime in Minutes
70%
200 1 42.6
30 + 2.2
3.4
- 1.5
1.5
4,0
4.0
16
1.6
1.5
25
63
70.2
30 1.4
67 2.2
132
3.5
6.1
25 0.7
62 4 1.9
127 27
6.4
25 0.7
62 1.9
127 3.1
218
221
25
25
68
219
25
63
218
25
63
296
132
221
221
220
25
62
50%
218 1.6
21 1.4
14
7
4
21811
21
14
7404
4102
661
2.7
14 11
8
1
104
2.7
14
7
4#0.1
221
50%
01.6
28 1.4
# 3.8
+ 1.6
14
1.4
4.0
1.7
24 1.5
4.
+ 158.7
28 1
66 1.9
131 3.6
1 6.1
25 25 0.7
62 2:2:266
127 2
2.7
25 0.7
62# 1.8
127 3.7
218
221
25
25
230
280
67
218
21
63
218
221
221
220
30%
218 + 1.4
24 1.3
14
7100
4:0
28
21
14211
7408
4103
21
24 24107
14
8
4+02
6.4
21
14
7402
4#0.1
325
218
221
220
30%
205 + 48.0
27 + 1.1
65 3.9
218+14
1.4
1.4
1.3
# 1.3
63 4.3
218411
1.4
24 1.3
03 #-333
+ 52.2
27 +
61 1.7
129 3.7
221 +
24 1.7
62 244
127 2.7
220
6.4
24 2.6
62 + 1.7
127 3.0
70%
5.0 0.12
5.1 5.1 0.07
5.0 5.000. 0.17
5.1 5.15 0.15
5.1 0.12
5.6 5.61 0.19
5. 5.3+0.00 0.09
52 122001 0.14
5.2 0.23
5.3 0.15
23.00..11
0.19
2 0.16
21.7 0:33
1 0.28
1 0331
254003
0.30
212 021
2L110.4
21.7 0.11
0.50
218
21
218
24
263
220
127
Top- 1 Error of Nes (%)
5.2
23
21.7
24.
25.1
24
70%
5.4 14::000 0.09
5.5 0.21
5.3 0.07
5.0 5.0 0.12
5.1 4: 0.07
5.1 5.10000 0.09
0.10
5.3 0.09
0.17
+4.04..22
0.42
24.7 + 0.30
21.7 0,40
24.7 0.38
28.5 0.19
23.410 14441.0110 0.16
23.3+0.00 0.28
+ 0.37
16.4#0.3 0.39
24.2 + 0.21
+ 0.29
2 0.34
Top- Error of ResNet164 (%)
5.6 5.0
50%
5.2 0.11
5.240.11
5.4 22
5.4
5. 0.00
0.27
5.7
5.100.17
6.3
6.2 6.30
77.40.40
0.40
27.1 0.14
27.7 0.22
0.24
28.4 0.32
1.112
0..15
29.7..0.36
0.36
30.0
30.8
1.37
24.6
21.7
24.7
23.5
23
23.3
23.4
26.4
21.8
25.5
6.4
27.1
28.2
28
30.1
29.7
29.7
50%
6.3 0.3 0.23
6.4 + 0.13
6.1 0.21
5.2 0.11
0.11
5.2 0.11
5.2 0.00
0.27
5.7 0.23
66 6.00 0.09
0.29
28.2 + 0.23
29.6 + 11.28
29.5 19.544 0..33
27.1 0.40
0.14
27.0 0.33
0.27
0.12
29.7
+ 0.45
29.6 0.13
4.400.35
30%
7.6 0 18
7.1 - 16
7.2 0.18
18
7.3 1
9.6 06
8.9 109
84 0014
10.4 1.19
10.2 1.34
36.5 88.85. 0.36
37.2 0.29
35.8 10
40
36.3 0.25
38.3 0
48
30.8 0.43
46.5 1.77
46.5 0.52
43.3 1.83
43.9 4 2.33
7.7 +
644
29.5
27.1
27.0
26.9
30.1
29.7
30.4
30%
8.9 8. 0.29
9.1 1 0.33
8.9 01009
7.7 7 19
7.6 + 11
7.7 1 0.27
9.6 0.16
8.9 0.188
9.9 9.9 0.29
40.8 + 0.20
35.2 + 1.37
41.8 + 11.99
42.3 + 0144
36.8 4 0.35
37.2 0.29
36.7 7 1.2.3
36.6 0.51
39.6 39.6=0.24 0.43
46.5 46.5 0.74
12.6 + 11.63
40..2 + 11:28
Subset Size
Epochs
181
181
100
50
25
181
181
100
25
181
181
100
25
181
181
1000
50
25
Subset Size
ResNet 164 (Baseline)
ResNet164 (Baseline)
ResNet 164 (Baseline)
ResNet164 (Baseline)
ResNet164 (Baseline)
164 (Baseline)
Selection Model RosNet (Baveline)
Res (Baseline)
64 (Baseline)
ResNet (Baseline)
Selection Model
ResNer20
ResNer56
ResNer20
ResNet56
ResNer20
ResNer56
ResNer20
ResNet56
ResNet110
ResNer20
ResNer56
ResNer110 110
ResNet20
ResNer56
ResNet110 110
ResNet20
ResNer20
2esNer20
ResNet20
Facility Location
Forgetting Events
CIFAR100 Facility Location
Forgeting Livents
Forgetiing Events
Forgetiigg Evvnnts
Method
Entropy
Entropy
Method
Entropy
Entropy
CIFARIO0
Dataset CIFARIO
Dataset CIFARIO
20

---

Published as a conference paper at ICLR 2020
A.6 ADDITIONAL CORRELATION RESULTS
ResNext50-32xx4 (25.0M)
RcsNiex1101-326d (88.8M)
16
ResNet34 (21.8M)
ResNet5O (25.6M)
10
0.44
ResNet101
632
0/1
ResNet152 (60.2M)
DenseNet121 (8.0M)
km
030
0.94
3.64
399
079
DenseNet169
0.2
0.1
312 022
DenseNet201 (20.0M)
DenseNet161 (26.7M)
5.12
0.79
3.64
384 033
36. EA5
MobllaNet
8m
2.98
0.66
974
014
92H E.5
GoogleNet (13.0M)
VGG-11 11 w/ Betchhorm (132.9M)
6.6
031
872
076
CTL
975
VGG-13 13 DatchNorm (133.1M)
841
8.77
6.6
0.82
3.72
3.34
033
025
6.72
6/2
3.6 376
845
16
VGG-19 w/ BatchNorm (143.7M)
SqueezeNet 1.0 (1.2M)
5.5
040
2.5
847
045
$30
0.6
3.9
261
235
GAr
SqueszeNet
Alextit (61. (M)
Redtektio 3zxdd (25 OM)
Reshiet28 (31, 7M))
Rochiet34 (21 (M)
Resfeell 01 (44 SM)
Resheet1 (60 2M)
Densofvert 60 (14 (14.1M)
DerseNee200 (20.000 OM)
verreeeeel 61 (28 7M)
GoogleNeet (1.. (M)
VGG-11 Betchfform (132 9M]
VOG w/ Reschtomm (138 AM)
VGG- wi BatchNomm 0433. 7M)
Equecooliet 1.1 2M)
Aloxtet 1M)
Reste (88
ReshietS (25,
DenseNver22
Mobileheet V2
VGG w/ Batchtoom
Squeccesiee
Figure 8: Comparing selection across model architectures on ImageNet. Spearman's 's correlation
between max entropy rankings from Py PyToc (Paszke et al., (017) pretrained models on ImageNet.
Correlations are high across a wide range model architectures Xie al. 2017: : He et al. 2016a
Sandler et al. 2018 Huang et al. 2017; Szegedy al.) 2015 Simonyan & Zisserman 2014 landola 2016) Krizhevsky et 2012). For example, MobileNet V2's mntropy--base rankings were highly correlated to ResNet50, even though the model had far fewer parameters (3.5M vs. 25.6M). In concert with our fastText and VDCNN results from Section 3.2 the high correlations between different model architectures suggest that SVP might be widely applicable.
21

---

Published as a conference paper at ICLR 2020
1.0
1.0
0.78
0.71
0.45
0.5
0.5
0.5
R56
0.78
0.84
R56
0.53
0.49
R56
0.37
0.59
0.0
0.0
R110
0.78
0.85
0.85
R110
0.35
0.38
0.35
R110
0.33
0.60
0.63
-0.5
-0.5
R164
R164
R164
R20
R56 R110
R164
1.0
R20
R56 R110
R164
-1.0
R20
R56
R110
R164
11.0
(a) CIFARI00 forgetting
(b) CIFAR 100 entropy
CIFARI00 greedy k-centers
Figure 9: Comparing selection across model sizes and methods on CIFARI00. Average Spear- man's 's correlation between different runs of ResNet (R) models and varying depths. We computed rankings based forgetting events (left), entropy (middle), and greedy k-centers (right). We saw a similarly high correlation across model architectures (off-diagonal) as between runs of the same architecture (on-diagonal), suggesting that small models good proxies for data selection.
1.0
R20
0.38
R20
0.44
R56
0.36
0.0
0.0
R110
0.37
0.46
0.46
R110
0.33
0.59
0.63
-0.5
0.40
0.46
0.45
0.50
R164
R20 R56 R110
R164
R20
R56
R110
R164
-1.0
(a) CIFARIO facility location
(b) CIFAR100 facility location
Figure 10: Spearman's rank-order correlation between different runs of ResNet (R) with pre-activation and varying number of layers on CIFARIO (left) and CIFARI00 (right). For each combination, we compute the average from 20 pairs of runs. For each run, we compute rankings based on the order examples are added in facility location using the same initial subset of 1,000 randomly selected examples. The results are consistent with Figure 3c and Figure 99] demonstrating that most of the variation due to stochasticity training rather than the initial subset.
0.9
Correlatonn RerNectos
ResNe120
0.8
ResNet56
ResNec110
0.7
0.6
0.5
BankcCom
0.3
ResNe:20
0.2
ResNet55
ResNet110
0.1
25
50
75
100
125
250
175
25
100
125
150
175
Epach
Epach
(a) CIFAR10 forgetting events
(b) CIFAR 100 forgetting events
Figure 11: Average (# std.) Spearman's 's rank-order correlation with ResNet164 during 5 training runs of varying ResNet architectures on CIFARIO (left) and CIFAR100 (right), where rankings were based forgetting events.
22

---

Published as a conference paper at ICLR 2020
ResNet20 RecNet56
ResNe:20
Correction Rcftec.4
ResNe156
ResNer110
0.4
Rank Comeation
0.3
0.2
0.2
25
50
75
100
125
150
175
0
25
75
300
125
150
175
Epoch
Epach
(a) CIFARIO entropy
(b) CIFARIO0 entropy
Figure 12: Average (# std.) Spearman's 's rank-order correlation with ResNet164 during training runs of varying ResNet architectures on CIFARIO (left) and CIFAR 100 (right), where rankings were based on entropy.
1.0
corelation Between
0.8
Rank Correcatoon Between
0.7
0.6
0.
ResNet20
ResNett2
ResNent6
ResNet55
ResNet110
0.3
ResNet164
ResNet364
Epach
Epoch
(a) CIFAR10 forgetting events
(b) CIFAR 100 forgetting events
Figure 13: Average (# std.) Spearman's 's rank-order correlation between epochs during training runs of varying ResNet architectures CIFAR10 (left) and CIFAR 100 (right), where rankings were based on forgetting events.
ResNerz0
ResNet20
0.9
Resllet56
0.9
ResNet54
ResNec110
ResNet110
ResNe1164
0.8
0.7
0.6
0.6
0.5
04
0.3
25
50
75
100
125
150
175
25
50
100
125
150
175
Epach
Epach
(a) CIFARIO 10 entropy
(b) CIFAR100 entropy
Figure 14: Average (# std.) Spearman's 's rank-order correlation between epochs during 5 training runs of varying ResNet architectures on CIFARIO (left) and CIFAR (right), where rankings were based entropy.
23

---

Published as a conference paper at ICLR 2020
1.0
0.92
0.40
0.5
0.5
R56
0.92
0.93
R56
0.24
0.18
0.0
0.0
R110
0.91
0.93
0.93
R110
0.20
0.15
0.14
-0.5
R164
R164
R20
R56
R110
R164
1.0
R20
R56 R110 R164
-1.0
CIFARIO 10 forgetting events
(h) CIFARIO entropy
1.0
1.0
R20
0.49
R20
0.66
0.5
R56
0.33
0.73
0.41
0.36
0.0
0.0
R110
0.30
0.77
0.84
R110
0.27
0.27
0.23
-0.5
-0.5
R164
0.30
0.76
0.83
0.88
R164
0.25
0.24
0.21 0.21
-1.0
R20
R56 R110 R264
R20
R56 R110 R164
(c) CIFAR100 forgetting events
(d) CIFARIOR 100 entropy
Figure 15: Pearson correlation coefficient between different runs ResNet (R) with pre-activation and varying number of layers on CIFARIN 10 (top) and CIFAR 100 (bottom). For each combination, we compute the average from 20 pairs of runs. For each run, we compute rankings based on the similarly
number of forgetting events (left), and entropy of the final model (right). Generally, high correlation across model architectures -off-diagonal) as between runs of the same architecture (on-diagonal), providing further evidence that small models are good proxies for data selection.
(a) 64 forgetting events
(b) ResNet164 entropy
(c) ResNet1 64 greedy k-centers
(d) ResNer20 forgetting events
(e) ResNet20 entropy
(f) ResNet20 greedy k-centers
Figure 16: 2D I-SNE plots from the final hidden layer of fully trained 66 model on CIFAR10 and a 30% subset selected (black). The top row uses another run of ResNetl 64 to select the subset and the bottom row uses ResNet20. Rankings are computed using forgetting events (left), entropy (middle), and greedy k-centers (right).
24

---

Published as a conference paper at ICLR 2020
(a) FE at epoch
(b) FE at 25 epochs
(c) FE at 50 epochs
(d) FE at 100 epochs
(c) Entropy at epoch
(f) Entropy at 25 epochs
(g) Entropy at 50 epochs (h) Entropy at 100 epochs
Figure 17: 2D t-SNE plots from the final hidden layer of fully trained ResNet164 model on CIFARIO and 30% subset selected (black) with ResNet20 trained after varying number of epochs. Rankings are calculated with forgetting events (top) and entropy (bottom). Notably, the ranking from forgetting events is much more stable because the model's uncertainty is effectively averaged throughout training rather than a single snapshot at the end like entropy. For t-SNE plots of the entire training run, please see http: /bit.biio 1y/ svp- ci farle- -ssee entropy and http: wiwssssp--iiaaaaoa tanee-tteeer--aeeeeeee--e forget for entropy and forgetting events respectively.
25


Reviews:
[{"review_id": "rkgV59XrYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to speed up the data selection in active learning and core-set learning. The authors present a simple idea: instead of using the full model to select data points, they use a smaller model with fewer layers, potentially trained for fewer iterations. The authors show that this simple approach is able to speed up the data selection portion of both processes significantly with minimal loss in performance, and also results in significant speedup of the entire pipeline (data selection + training).\n\nThis paper is timely and important -- there has been a lot of emphasis lately on the environmental costs of training deep learning models (e.g., Strubell et al., ACL 2019; Schwartz et al., 2019 arxiv:1907.10597). This paper shows that simple, almost trivial techniques can lead to significant runtime benefits for active learning and core-set learning. The authors present an extensive set of experiments that validate their hypothesis, and the paper is overall clearly written.\n\nComments:\n\n1. The correlation values in Figure 3 are quite diverse. It seemed \"forgetting events\" is much more correlated that the other two approaches. \n\n2. Why do the authors think that SVP and their baselines failed to outperform random sampling on Amazon Review Full (towards the end of 3.3)?\n\n3. Table 1 is very hard to interpret. I would advise the authors to look for a more succinct way to present their main findings. While this might seem contradictory, Figure 2, which is more reader-friendly, is also hard to interpret without the runtime values (nobody said visualization is easy!).\n\n4. One thing that's missing from this paper is training the smaller networks end-to-end. What would be the effect of using the proxy network as the main network as well? this is likely to lead to very significant runtime savings, and I wonder at what costs.\n\nMinor:\n\n1. Towards the end of 3.2: I disagree that significant speedups are \"uninteresting\" if they lead to small reductions in performance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJxUiL7GiH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for selecting a subset of a large dataset to reduce the computational costs of deep neural netwoks. The main idea is to train a proxy model, a smaller version of the full neural network, to choose important data points for active learning or core-set selection. Experiments on standard classification tasks demonstrate that this approach can yield substantial computational savings with only a small drop in accuracy.  \n\nThis paper is well-written and was easy to follow, with a clear motivation. The paper does good job of demonstrating that the proposed algorithm is effective through a comprehensive set of experiments. Overall, I favor acceptance and would be willing to increase my score if the following are addressed:\n\n1) Training for a smaller number of epochs was mentioned as a possibility to save computation. In the experiments, this was only done for one of the settings. Is this because models trained for fewer epochs are ineffective for data selection? \nLooking at Fig. 5b, it seems like the error drops significantly around 14 min before plateauing. In practice, for a new dataset, it could be difficult (or impossible) to know when to stop training a proxy a priori so that it achieves good performance relative to a larger model. It could be interesting to look at the effectiveness of a proxy at various points during training to see if the benefits are sensitive to the training time.\n\n2) For the active learning experiments, how many data points were added to the training set in each round? I could not find this number in the paper. Also, how many rounds of data selection were there?\n\n3) Is there an explanation why in Fig. 2b and Fig. 7c, the resnet20 proxy performs better than the larger (and more accurate) models? In particular, it even outperforms the 'oracle' baseline, which I find surprising.\n\n4) On a similar note, in Fig. 7, for small subsets (30%), the random baseline outperforms all of the SVP settings. Why is SVP ineffective in this case?\n\nMinor comments/suggestions:\n- In the abstract: \"improvement in data selection runtime\" Is this \"data selection runtime\" different from the total runtime? If not, it could be clearer to simply state it as the \"total runtime (including the time to repeatedly train and select points)\". I was unsure if this was a different measure.\n\n- Did the authors try further reducing the model capacity? With the success of the smaller models, it seems natural to try pushing further in this direction.\n\n- Since one of the findings was that models with similar architectures were effective as proxies but not models with different ones, perhaps there could be even higher correlation if the proxies were initialized with the exact same weights as a subset of the full model. \n\n- The writing in Table 1 (and the ones in the appendix) is a bit small.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1ewdKNaYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the paper about ?\n*The paper proposes a simple method to speed up active learning/core-set selection in deep learning framework. The idea is that instead of using full scale deep model for data selection/uncertainty sampling, use a smaller/faster proxy model and this proxy model’s uncertainty estimates are good enough for choosing data points even though proxy model’s accuracy may not be good.\n*The paper experiments with 5 datasets with 2 active learning and 3 core set selection strategies and shows promising speed-up with little to no regression in performance in most settings.\n\n\nWhat I like about this paper ?\n*The idea is simple and the paper is well written and easy to follow with ample references.\n*Unlike a lot of active learning papers, this paper considers both image and text domains.\n*Results in both Table 1 and Figure  2 have been reported after multiple runs 3 and 5 respectively. Authors also report standard deviation which is important because the results can vary a lot from run to run in active learning.\n*Plenty of empirical results do make the paper attractive and this definitely makes for a nice contribution to the active learning community.\n\n\nWhat needs improvement ?\n*Novelty is lacking. While the authors may be the first ones to apply this idea in the context of deep learning, they themselves note that the idea of using a smaller proxy like naive bayes for larger models like decision trees is not new. Although I agree this should not be a criteria for rejection especially given the extensive experimentation performed in the paper and has not impacted my score.\n*The choice of datasets is not justified in the paper and could have been better. I don’t see the point of showing that method works well for both Amazon Review Polarity and Amazon Review Full. Same can be said for CIFAR10 and CIFAR100. Would have appreciated a more diverse set of datasets. I would have been happy to see results on WMT given data selection is an open challenge in machine translation.\n*Paper initially proposes two methods for proxy: one by scaling down the model and two by reducing the number of epochs. fastext falls under neither of those categories and the giant speed up claims made by the paper like 41.7x are only with fastext. It is also important to note that while the performance drop using the fastext or smaller version of VDCNN is not much as compare to VDCNN, the performance gain is also not a lot compared to Random. Due to this reason I am not convinced if I should use SVP if I want speedup or just randomly select points especially in text domain.\n*Would have liked to see the comparison of #parameters in your different proxy models instead of just names like VDCNN29 and VDCNN9.\n*A lot of experimental details are missing which would make it difficult to reproduce the results unless the code is released.\n*Why fasttext worked as a proxy for VDCNN and AlexNet did not work with ResNet is unclear from the paper.\n\n\nQuestions for Authors ?\n*Do you plan to release the full code that can reproduce the results in the paper ? Based on my experience, I have found that a lot of active learning results are hard to reproduce and therefore it is difficult to draw any conclusions based on them.\n*How sensitive are the results to hyper-parameter tuning and small things like choice of pre-trained word embeddings etc ? How did you select hyper-parameters at each stage during active learning ? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to perform sample selection for deep learning - which can be very computationally expensive - using a smaller and simpler proxy network. The paper shows that such proxies are faster to train and do not substantially harm the accuracy of the final network.

The reviewers were all in agreement that the problem is important, and that the paper is comprehensive and well executed. I therefore recommend it should be accepted.
Decision: Accept (Poster)