AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning
Authors: Krishnateja Killamsetty
Guttu Sai Abhishek
Aakriti Lnu
Ganesh Ramakrishnan
Alexandre V. Evfimievski
Lucian Popa
Rishabh K Iyer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Hyperparameter Optimization
Efficiency
Data Subset Selection
Submodularity

Abstract: Deep neural networks have seen great success in recent years; however, training a deep model is often challenging as its performance heavily depends on the hyper-parameters used. In addition, finding the optimal hyper-parameter configuration, even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms, can be time-consuming, requiring multiple training runs over the entire dataset
for different possible sets of hyper-parameters. Our central insight is that using an informative subset of the dataset for model training runs involved in hyper-parameter optimization, allows us to find the optimal hyper-parameter configuration significantly faster. In this work, we propose AUTOMATA, a gradient-based subset selection framework for hyper-parameter tuning. We empirically evaluate the effectiveness of AUTOMATA in hyper-parameter tuning through several experiments on real-world datasets in the text, vision, and tabular domains. Our experiments show that using gradient-based data subsets for hyper-parameter tuning achieves significantly faster turnaround times and speedups of 3×-30× while achieving comparable performance to the hyper-parameters found using the entire dataset.

AUTOMATA: : Gradient Based Data Subset Selection for Compute-Efficien Hyper-parameter Tuning
Krishnateja Killamsetty', Guttu Sai Abhishek?, Aakriti², Ganesh Ramakrishnan"
Alexandre V. Evfimievski", Lucian Popa', Rishabh Iyer'
The University of Texas at Dallas, USA
Indian Institute Technology Bombay, India
IBM Research, USA
Karrishnateja.killamsttttt rishabh. Outrlaut edu (gsaiabhishek, aakriti, ganesh)@cse. iiti.iiieeee ac. in {evfimi, 1popa}@us. it..iimm.comm com
Abstract
Deep neural networks have seen great success in recent years; however, training a deep model is often challenging as its performance heavily depends on the hyper- parameters used. In addition, finding the optimal hyper-parameter configuration, even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms, be time-consuming, requiring multiple training runs over the entire dataset for different possible sets of syper-parametery informative subset of the dataset for model training runs involved in hyper- parameter optimization, allows us to find the optimal hyper-parameter configuration significantly faster. In this work, propose AUTOMATA, gradient-based subset selection framework for hyper-parameter tuning. We empirically evaluate the effectiveness of AUTOMATA in typer-parameter tuning through several experiments on real-world datasets in the text, vision, and tabular domains. Our experiments show that using gradient-based data subsets for hyper-parameter tuning achieves significantly faster turnaround times and speedups of 3x-30 x while achieving comparable performance the hyper-parametto found using the entire dataset.
Our central insight is that using
Introduction
In recent years, deep learning systems have found great success in a wide range of tasks, such as object recognition [15), speech recognition [17], and machine translation [1], making people's lives easier on a daily basis. However, in the quest for near-human performance, more complex and deeper machine learning models trained increasingly large datasets being used at the expense of substantial computational costs. Furthermore, deep learning is associated with significantly large number of syper-parameter: such as the learning algorithm, batch size, learning rate, and model configuration parameters (e.g., depth, number of hidden layers, etc.) that need to be tuned. Hence, running extensive syper-parameter tuning and auto-ml pipelines is becoming increasingly necessary to achieve tate-of-the-as models. However, tuning the hyper-pane parameters requires multiple training runs over the entire datasets (which are significantly large nowadays), resulting in staggering compute costs, running times, and, more importantly, CO2 emissions.
To give idea of staggering compute costs, we consider image classification task relatively simple CIFAR-10 10 dataset where a single training run using relatively simple model class of Residual Networks [16] for 300 epochs on V100 GPU takes around 4 hours. If perform 1000 training runs (which is not uncommon today) naively using grid search for hyper-parameter tuning, will take 4000 GPU hours. The resulting CO2 emissions would be between 440 1000 kg of CO2
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

02
1.0
Tae Tabeniis leri
CRAIG RANDOM
Spooolap (e Bull)
24
Hett Dhe Eanoe oe I-ail
12
1.0
0.21
(n) Hyper- hyper-purmmeter Tuning Performance
(h) Hyper- parameter Ordering Retention Capahility
Figure 1: Sub-figure Figure la shows perfornance summary of AUTOMATA with speedups, relative test errors, and tuning times on SST2, glue-SST2, CIFARIO, CIFAR (00, and CONNECT-4 datasets. We observe that AUTOMATA achieves spocdupstand similar energy savings) 10x 30x with around 2% performance loss using Hyperband AUTOMATA achieves speedup of around 2x with a performance loss of US 2%. Sub figure Figure shoss the Spearman ranking Conrelation values of hyper-parameter ordering returned by AUTOMATA
schoduler. Similarly, even when using more efficient ASHA scheduler.
hyper-parameter ordering returned by training using Grid Search. Spearman ranking correlation values suggest that AUTOMATA was able to retain prper--rrameter ordering betler than other subset selection haselines even when using small data subsets individual model training.
emitted] which is equivalent to 1100 to 2500 miles of car travel in the US. Similarly, the costs of training sfate-of-the-eee art NLP models and vision models on larger datasets like ImageNet are even more staggering [48
Naive hyper-parameter tuning methods like grid search [ often fail to scale up with the dimension- ality of the search space and are computationally expensive. Hence, more efficient and sophisticated 46 27 have dominated the field of hyper- paremmerreeter op-
Bayesian optimization methods 4 19 5
timization in recent years. Bayesian optimization methods aim to identify good
configurations quickly by building a posterior distribution over the search space and by adaptively selecting configurations based on the probability distribution. More recent methods [49] [50] 10 227
try to speed up configuration evaluations for efficient syper-parameter search; these approaches speed up the configuration evaluation by adaptively allocating more resources to promising hyper-panne parameter configurations while climinating poor ones quickly.
Recent works like SHA 20) Hyperband (32], ASHA 34 use aggressive early-stopping strategies to stop poromiig configurations quickly while allocating more resources to the promising ones. Generally, these resources can be the size the training set, number of gradient descent iterations, training time, etc. Nickson al. 40) Krueger et al. [29 try to quickly evaluate a configuration's performance on a large dataset by evaluating the training runs on small, random subsets and they empirically show that small data subsets could suffice estimate a configuration's quality. Similarly, past works [40 29 show that very small data subsets can be effectively used to find the best hyper- parameters quickly. However, all these approaches have naively used random training data subsets and did not place much focus on selecting informative subsets instead. Our central insight is that using small informative data subsets allows us to find good parammet configurations more effectively than random data subsets.
An earlier work called GRADMATCH 23 demonstrated the effectiveness of gradient- gradient-based data subset selection for selecting informative data subsets by using it for efficient model training in a supervised learning setting. In this work, empirically investigate the advantage of using gradient- based subset selection algorithms, mainly GRADMATCH, for hyper-parameter tuning compared to using random subsets and the full dataset for hyper-parameter tuning. Fundamentally, we use GRADMATCH to select data subsets that can be used for efficiently tuming the byper-parametedr To this end, we propose AUTOMATA, an efficient hyper-parameter tuning framework that combines existing hyper-parameter search and scheduling algorithmss with intelligent subset selection. Further, our experimental results illustrate that very small subsets (1%, 5% subsets) can be used for effective hyper- pprer-prrameter tuning because we are primarily concerned with retaining the ordering of hyper- parameters rather than the final accuracy, thereby enabling us to achieve better speedups and energy efficiency (and more significant CO2 emissions reduction).
httper//hlcoegggttabllim/eeeeeep https///baayylll.om//fffeeee

---

1.1 Related Work
Hyper-parameter Tuning Approaches: A number of algorithms have been proposed for hyper- parameter tuning including grid searct Bayesian algorithms R random search [43, etc. Further- more, a number of scalable toolkits and platforms for hyper-parameter tuning exist like Ray-tune [66, H2O automl (II) etc. See A5 554 for a survey of current approaches and also tricks for hyper- parameter tuning for deep models. The biggest challenges of existing hyper-parameter tuning approaches
the large hyper-parameter search space and b) the increased training times of training models. Recently, Li et. al. [33] have proposed an efficient approach for parallelizing hyper-parameter tuning using Asynchronous Successive Halving Algorithm (ASHA). AUTOMATA is complementary to such approaches and can be be combined effectively with them.
Data Subset Selection Approaches: Several recent papers have used submodular functions F for data subset selection towards various applications like speech recognition [53 52 machine translation (26] and computer vision [21]. Other common approaches for subset selection include the usage of coresets. Coresets are weighted subsets of the data, which approximate certain desirable characteristics of the full data (e.g., the loss function) [12). Coreset algorithms have been used for several problems including k-means and k-median clustering (14). SVMs [8] and Bayesian inference (6). Recent coreset selection- efficient and robust training of deep models. CRAIG 38 tries to select a coreset summary of the
methods [38
24
23 25] have shown great promise for
training data that estimate the full training gradient closely. Whereas GLISTER (24 poses the coreset selection problem as a set loss. Similarly, RETRIEVE [25] also uses discrete bilevel coreset selection problem to select unlabeled data subsets for efficient semi-supervised learning. Another approach GRAD-MATCH [23 selects coreset summary that approximately matches the full training loss gradient using orthogonal matching pursuit.
bilevel optimization problem that minimizes the validation
The contributions of our work can be summarized follows: AUTOMATA Framework: We propose AUTOMATA a framework that combines intelligent gradient based subset selection with hyper-parrmeter search and scheduling algorithms to enable faster hyper-parameter tuning. Our work empirically studies the role of intelligent data subset selection for hyper-paramen parameter tuning. In particular, seek answer the following question: Is it possible to use small informative data subsets between 1% to 30% for faster configuration evaluations in hyper- parameeer tuning. thereby enabling faster tuning times while maintaining comparable accuracy tuning byper-parameters on the full dataset? Importance of Intelligent Subset Selection: Our empirical results in Figure Tb demonstrate that GRADMATCH ordering better than other baseline subset selection strategies including RANDOM and CRAIG. In addition, gradient-basee subset selection strategies such as GRADMATCH and CRAIG preserve the original
23 adopted by AUTOMATA preserves the original hyperparameter
ordering better compared to RANDOM, thereby showcasing the importance of intelligent subset selection approaches. Effectiveness of AUTOMATA We empirically demon- strate the effectiveness of AUTOMATA framework used in conjunction with existing hyper-paramen parameter search algorithms like TPE, Random Search, and hyper-parameter scheduling algorithms like Hyper- band. and ASHA through a set of extensive experiments on multiple real-world datasets. We give summary of the speedup vs. relative performance achieved by AUTOMATA compared to full data training in Figure Tal More specifically, AUTOMATA achieves speedup of 3x 30x with minimal performance loss for hyper-parameter tuning. Further, in Section B we show that the gradient-based subset selection approach of AUTOMATA outperforms the previously considered random subset selection for hyper-parameter tuning.
2 AUTOMATA Framework
In this section, we present AUTOMATA hyper--aramettrr tuning framework, and discuss its different components shown in Figure 2 The AUTOMATA framework consists of three components: hyper-parameter search algorithm that identifies which configuration sets need to be evaluated, gradient-based subset selection algorithm for training and evaluating each configuration efficiently,
htttps:/ttinurz// com/ 3hb2hans com//bb2hans
Let V 21, ... denote ground set of items. A set function f f22% - R. submodular 113 it
satisfies the diminishing returns property: for subsets 5 C T' V and VVT, fUIS) f(Suj) - -(()) 2 fOlZ'.
3

---

Promooad Contiguration
Component-22
CorEgundion-1
DS5 bawed model training loop usdin selected
Modal's Velication oor
Deppeaa unti the resgal numberr
Pypaer sorr coneorrro an www.rr/..aooooootttoooo ewcokusteed
Componentis
Choice
Corfguraton
D65 based model training loop usin selected configuaation
Model's Valcatoon ser porformnce
Comparison
Component-3
Paper purarrttar
Hypergarammaler
Cont puretams
tpper-peancclr
Corkgundionnn
DSS based model training loop using configuration
Mooe''s
based modeI wramiing
Iraie patameler
Minsbutthh
9,
Bear Selection (GS5)
8x. 4g
Minstusshh den
Saa, Pag
Sansol Seardin
Cyp
91.90
spocia
tiran
Figure 2: Pipeline figure of AUTOMATA including syper--arameter search, subset based configuration evaluation (where models are trained on subsets of data), and hyper apper-parametr scheduler.
and a hyper- parameter scheduling algorithm that provides early stopping by eliminating the poor configurations quickly. With AUTOMATA framework. one can use any of the existing hyper-parameter search and papper-arameter scheduling algorithms and still achieve significant speedups with minimal performance degradation due to faster configuration evaluation using gradient- based subset training.
Notation Denote by H. the set of configurations selected by the
search algorithm.
Let D {< yi ) denote the set of training examples, and V = ((x,, My y= the validation set. Let 0; denote the classifier model parameters trained using the configuration É Hl. Let S; be the subset used for training the jth configuration model 0, and w. be its associated weight vector i.e., each data sample in the subset has an associated weight that is used for computing the weighted loss. We superscript the changing variables like model parameters 9, subset with the timestep to denote their specific values at that timestep. Next, denote by (0,) = (2, yj. 0z),,), the training loss of the jth data sample in the dataset for 1th classifier model, and let (r.) = EEDD L (6.) be the loss over the entire training set for 7th configuration model. Let, I (S, 02) LkEx Lr(ak, yk. 0,) be the loss on a subset S S of the training examples timestep j. Let the validation loss be denoted by Ly.
Component-1: Hyper-parameter Search Algorithm Given a peper-parameter search space, hyper-parameter search algorithms provide a set of configurations that need to be evaluated. A naive way of performing the hyper-parameter search is Grid-Search, which defines the search space as a grid and exhaustively evaluates each grid configuration. However, Grid-Scarch is a time-consuming process, meaning that thousands to millions of configurations would need be evaluated if the hyper-parameter space is large. In order to find optimal hyper-paramette settings quickly, Bayesian optimization-baat hyper-pane pprammerrreeter search algorithms have been developed. To investigate the effectiveness of AUTOMATA across the spectrum of search algorithms, we used the Random Search method and the Bayesian optimization- based TPE method as representative hyper- peerrparaeeter search algorithms. We provide more details on Random Search and TPE in Appendix
Component-2: Subset based Configuration Evaluation Earlier, we discussed how hyper- parameter search algorithm presents a set of potential eyper-parameter configurations that need to be evaluated when tuning hyper-paramette trained the model on the entire dataset until the resource allocated by the typeer-arameter scheduler is exhausted. Rather than using the entire dataset for training. we propose using subsets of informative data selected based on gradients instead. As result, given any hyper- parameter search algorithm, we can use the data subset selection to speed up each training epoch by a significant factor (say 10x), thus improving the overall turnaround time of the hyper- reper-parameter tuning. We use GRADMATCH [23], gradient based subset selection strategy in AUTOMATA because GRADMATCH able to preserve the original ordering of hyper-parameten better than RANDOM and other baselines even when using small subset sizes as shown in Figure [b Hence, the critical advantage of AUTOMATA is that we can achieve speedups while still retaining the hyper-paramett tuning algorithm's 's performance in
Every time a configuration needs to be evaluated, prior work

---

finding the best experiments in Section 3 The fundamental feature of AUTOMATA that the subset selected by AUTOMATA changes adaptively over time, based on the classifier model training. Thus, instead of selecting a common subset among all configurations, AUTOMATA selects the subset that best suits each configuration. We present the gradient-based subset selection process of AUTOMATA below.
We provide more details on haper-parameter ordering retention
Gradient Based Subset Selection (GSS): AUTOMATA adopts GRADMATCH (23], gradient based subset-selection strategy, to select a subset S and its associated weight vector w such that the weighted subset loss gradient best approximates the entire training loss gradient. The subset selection of AUTOMATA for tt configuration at time step t is as follows:
W. S
argmin SiiSs: sk,wf20
w V V (0.))))) VoIr(( + + \|/00 |
(1)
LES,
The additional regularization term prevents the assignment of very large weight values to data samples, thereby reducing the possibility of overfitting on few data samples. Killamsetty et al.
23 proved that the optimization problem given in Equation (T) is approximately submodular. Therefore, the above optimization problem can be solved using greedy algorithms with approximation
guarantees 9 37 Therefore, like Killamsetty et al. 233, we also use a greedy algorithm called orthogonal matching pursuit (OMP) to solve the above optimization problem. The goal of AUTOMATA to accelerate the hyper-paramenn Efficiency is an essential factor that AUTOMATA considers even when selecting subsets. Due to this, we employ a faster per-batch subset selection introduced in the work [23] in our experiments, which described in the following section.
tuning algorithm while preserving its original performance.
Per-Bateh Subset Selection: Instead of selecting a subset of data points, one selects a subset of mini-batches by matching the weighted sum of mini-bater batch training gradients to the full training loss gradients. We visualize the difference between per-sample and per-batch subset selection in Figure 5) of the Appendix. Therefore, one will have subset of selected mini-batches and the associated mini-batch weights. One trains the model on the selected mini-batches by performing mini-batch gradient descent using the weighted mini-batch loss. Let us denote the batch size as B, and the total number of mini-batches as by = B
N. and the training set of mini-batches as DB-
Let us denote the number of mini-batches that needs to be selected as ba B the subset of mini-batches that needs to be selected as Sg, and denote the weights associated with mini-bate math-batches as WB; = {WBI1, WB2 ... WBik } for the j model configuration. Let us denote the mini-batch gradients as L. (0.). .... VoL. BaN (0.) be the mini-batch gradients for the ith model configuration. Let us denote (0.) [ke(1,b)] Be (0.) be the loss over the entire training set. The subset selection problem of mini-batches at time step / can be written as follows:
& Let us denote
w/s: S/:
argmin wh St. 1 s (e 20
Σ LES.
w'si VOLE VoLE(()) + A W 12
(2)
In the per-batch version, because the number of samples required for selection is by is less than k. the number of greedy iterations required for data subset selection in OMP is reduced, resulting in speedup of Bx. A critical trade-off in using larger batch sizes is that in order to get better speedups. we must also sacrifice data subset selection performance. Therefore, it is recommended to use smaller batch sizes for subset selection to get a optimal trade-off between speedups and performance. A solution to the batch size dependency is to use a smaller batch size only for subset selection and combine the smaller batches to produce larger batches for training. In order to demonstrate that the effectiveness of AUTOMATA is not dependent Images, we use a fixed batch size (instead of as a hyper--prane parammerrmmterr of B = 20, and on text datasets, we use the batch size hyper-parameter with B E [16, 32, 64). Apart from per-batch selection, we use model -arm--tarting to get more informative data subsets. Further, in our experiments, a regularization coefficient of X 0 after analyzing the performance of AUTOMATA with different values of X. We present the ablation study on X in Appendix G We give more details on warm-starting below.
the size of the training batch, in our experiments on
Warm-starting data selection: We warm-start each configuration model by training on the entire training dataset for a few epochs similar to [23). The warm-starting process enables the model to
5

---

have informative loss gradients used for subset selection. To be more specific, the classifier model is *Th epochs, where & is the coreset size, T is the total
trained on the entire training data for T.. = NN' number of epochs, K is the fraction of warm start, and N is the size of the training dataset. We use & value of 0 (i.e., no warm start) for experiments using Hyperband as scheduling algorithm, and a value of 0.35 for experiments using ASHA. We present the ablation studies for K in Appendix G.3.2
Component-3: Hyper-parammeer Scheduling Algorithm- - pareee- scheduling algorithms improve the overall efficiency of the syper-parameter tuning by terminating some of the poor configurations runs early. In our experiments, we consider Hyperband [32, and ASHA [4], which extensions of the Sequential Halving algorithm (SHA) (20) that uses aggressive early stopping to terminate poor configuration runs and allocates an increasingly exponential amount of resources to the better performing configurations. SHA starts with n. number of initial configurations, each assigned with minimum resource amount r. The SHA algorithm uses a reduction factor 7) to fraction of configurations while also increasing the resources allocated to these configurations by 1) times each round. We discuss Hyperband and ASHA and the issues within SHA that each of them addresses in more detail in Appendix [ Detailed pseudocode of the AUTOMATA algorithm is provided in Appendix C due to space constraints in the main paper. We use the popular deep learning framework [41] for implementation of AUTOMATA framework, Ray-tune (36 for hyper-paramen search and scheduling algorithms, and CORDS 22 for subset selection strategies.
reduce the number of configurations in each round by selecting the top 1 th
Experiments
In this section, we present the effectiveness and the efficiency of AUTOMATA framework for hyper- parameter tuning by evaluating AUTOMATA on datasets spanning text, image, and tabular domains. Further, assess AUTOMATAS effectiveness across the spectrum of existing hyper-parameter search and scheduling algorithms, we conduct experiments using combinations of different search and scheduling algorithms. As discussed earlier, we employ Random Search 43 TPE 4 as repre- sentative hyper-parametts parameter search algorithms. and Hyperband [32), ASHA 34 as representative hyper-parannn parameerr scheduling algorithms. should be noted that the combination of TPE and Hy- perBand is not the same as the BOHB algorithm [T]. In BOHB M. a multivariate kernel density estimator (KDE) was used, whereas the TPE 4 used hierarchy of one-dimensional KDEs. How- ever, we believe the takeaways would remain the same even with other approaches. We repeat each experiment five times on the text and tabular datasets, thrice on the image datasets, and report the mean accuracy and speedups in the plots. Below, we provide further details on datasets, baselines, models, and the hyper-parameter search space used for experiments.
Our experiments aim to demonstrate the consistency and efficiency of AUTOMATA more specifically, the effectiveness of AUTOMATA'S gradient- subset selection (GSS) for hyper- parameter tuning. As baselines, we replace the GSS subset selection strategy in AUTOMATA with different subset selection strategies, namely RANDOM (randomly sample a same sized subset as subset selection proposed for efficient supervised learning), and FULL (using the entire training data for model training during configuration evaluation). For ease of notation, we refer to baselines by the names of corresponding subset selection strategies. Note that by CRAIG baseline, we mean the faster per-batch version of CRAIG (38 for subset selection shown 2 to be more efficient than the original. In addition, for all methods, we do not use any warm-start for experiments with Hyperband and use a warm start of 0.35 for experiments with ASHA. We give more details on the reason for using warm-start with ASHA and no warm-start with Hyperband in Appendix G.3.2 We perform experiments with different subset size fractions of 1%, 5%, 10%, and 30%. In our experiments, we compare our approach's s accuracy and efficiency (time/energy) with Full training, Per Batch CRAIG, and Random selection.
Baselines:
AUTOMATA from the training data), CRAIG [8] (a gradient-
Datasets, Model Architecture, and Experimental Setup: To demonstrate the effectiveness of AUTOMATA for hyper parrmmear tuning, we performed experiments on datasets spanning text, image and tabular domains. Text datasets include SST2 (7). SST5 [7), glue-SST2 [5I], and TREC6 335 [8. Image datasets include CIFAR 10 (88). CIFAR100 28), and Street View House Numbers (SVHN) (39. Tabular datasets include DNA, SATIMAGE, LETTER, and CONNECT-4 from LIBSVM (a library for Support Vector Machines (SVMs)) Z We give more details on dataset sizes and splits in Appendix G.2 For the Text datasets, use the LSTM model (from PyTorch)

---

RANDOM
AUTOMATA
CRAIG
7.5
Rel. The Emur
10
Kmer
5.0
2.55
-5
2.5
10
15
2.5
5
10
15
1.5
2.5
3.5
2.5
3.5
Specel
Rponal Thp
Spore Opp
Speee Mp
$SST5(Rundonn.IBB
SSTS(TPE,IIB)
SST55Random.ASII
SSTS(TPE,ASIAA)
20
Ral Tret Treer
Took Emer
Net. Tes Bire
10
2.5
10
2.5
15
0
15
255
3.5
3.5
Spead
Spoud The
Ruesdl fhp
Speed Up
e) IRBC6(Randon.HBB
TRECGTTEEHBB
b) TRECCGTPE.ASHA)
CF
0
2.5
10
25
2.5
10
15
1.3
2.5
3.5
1.3
2.5
3.5
Spoed
Speed
Speed Up
Speed Up
1) CIFAR 10/Random, HB)
CIFARRIO(PPE.H))
k) LIFARR10RRmdemmAAHAA
1) CIFAR1OTTPE,ASHA)
5.0
5.0
Bet. Ten Emur
2.5
22.5
0
0.0
2.5
10
15
10
1.5
2.5
3.5
25
3.5
Spost
Speed
Speed Up
Speed Up
CIFAR 100X Randomm,,)
n) CIFAR TORLTP
o) CIFAR 100 ASHA)
CIFAR CIFARIOOXTPE ASHA)
0.5
1.0
Hel. Ton Erree
Tel. Ted Emire
00.5
0.55
0.0
1.5
2.5
1.2
1.25
1.5
2
4) CONNECT-4((amddee)
CRONNECCT4(TPEJJT 5) CONNECT CONNNEETTARRRiLAAASS ((RaAAA
( CONNECT CONNECT-CTPEASSIIA
Figure 3: Comparison of performance of AUTOMATA with baselines(RANDON, CRAIG, FULL) for Hyper- parameter tuning. In sub-figures (a-l), present speedup vs. relative test (in %), compared to Full data tuning for different methods. A relative test error subset selection based tuning and a test error obtained through full data tuning. On each seatter plot, smaller subsets appear on the right, and larger ones appear on the left. Results are shown for (a-d) SST5, (c-h) TREC6, (i-l) 10, (m-p) CIFAR100, and (q-t) CONNECT-4 datasets with different combinations of hyper- parameerr search and scheduling algorithms. The scatter plats show that AUTOMATA achieves the best speedup-accuracy tradeoff almost every case (bottom-rigg riggtt corner of each plot indicates best speedup-accuracy tradeaff region).
the difference between a error obtained through

---

embeddings of 300 dimension as input. For Image datasets, we use the ResNet18 [66] and ResNet50 [16] models. For Tabular datasets, we use a multi-layer perceptron with 2 hidden layers. Once the best hyper-parameter configuration is found, we perform one more final training of the model using the best configuration on the entire dataset and report the achieved test accuracy. We use final training for all methods except FULL since the models trained on small data subsets (especially with small subset fractions of 1%, 5%) during tuning do not achieve high test accuracies. We also include the final training times while calculating the tuning times for more fair comparisor For text datasets, we train the LSTM model for 20 epochs while choosing subsets (except for FULL) every 5 epochs. The hyper-paramese & number of layers of LSTM, batch size of training. Some experiments (with TPE as the search algorithm) use 27 configurations in the prper-parameter space, while others use 54. More details on hyper-paran parameterr search space for text datasets are given in Appendix G.4. For image datasets, train the ResNet [16 model for 300 epochs while choosing subsets (except for FUI.I.) every 20 epochs i.e., R = 20. We present an ablation study analyzing the effect of epoch interval (R) on the performance of AUTOMATA in Appendix G.3.3 Based on the results, we observe that for vision experiments, using a R value of 20 gives best efficiency vs performance tradeoff. We use a Stochastic Gradient Descent (SGD) optimizer with momentum set 0.9 and weight decay factor set 0.0005. The hyper-parameter search space consists of a choice between the Momentum method and Nesterov Accelerated Gradient method, choice of learning rate scheduler and their corresponding parameters, and four different group-wise learning rates. We use 27 configurations in the parammee space for Image datasets. More details on hyper--aramene Appendix G.4.2 For tabular datasets, we train a multi-layer perceptron with 2 hidden layers for 200 epochs while choosing subsets every 10 epochs. The hyper-pane syper-parameter search space consists of a choice between the SGD optimizer or Adam optimizer, choice of learning rate, choice of learning rate scheduler, the sizes of the two hidden layers and batch size for training. We use 27 configurations in the hyper-parameter space for Tabular datasets. More details on hyper-parameter search space for tabular datasets are provided in Appendix G.4.3
with trainable GloVe 42
space includes learning rate, hidden size
search space for image datasets are given in
Sxpper-parameter Ordering Retention Experiments: We evaluate the effectiveness of different subset selection strategies in preserving original hyper-parameter ordering by comparing the ordering obtained using AUTOMATA' 's gradient-bused subset selection (GSS), RANDOM, CRAIG strategies with that obtained using full data. Intuitively, we want to analyze whether the original hyper- parameter ordering is preserved even when using small subsets for model training. To examine this, we experiment on the CI 28 Trec6 35 18 datasets using ResNet18 [66] model and LSTM model respectively. Hyper- parpmeer search for CIFAR10 dataset includes a grid search over 144 configurations of four grsep-wise learning rates of ResNet18 model, optimizer, and training batch size. Hyper-parameter search for TREC6 dataset includes a grid search over 108 configurations of learning rate, optimizer, LSTM hidden size, training batch size, and number of final fully connected layers. Figure Db shows the Spearman rank correlation values between the hyper-parameterr ordering obtained using 1%, 5%, and 10% subsets selected by RANDOM, CRAIG, and AUTOMATA 's gradicnt-basee subset selection (GSS) and Full data hyper- parameter ordering CIFARIO 28 and TREC6 [35] [8] datasets. Figure Tb demonstrates that GSS is more effective than RANDOM and CRAIG in preserving hyper-parameteer ordering even when using small subsets.
Hyper-parameter Tuning Results: Results comparing the accuracy vs. efficiency tradeoff of differ- ent subset selection strategies for hyper-parameter tuning are shown in Figure 3 Performance is com- pared for different sizes of subsets of training data: 1 1%, 5%, 10%, and 30% along with four possible combinations of search algorithm (Random or TPE) and scheduling algorithm (ASHA or Hyperband). Text datasets results: fib-figures(3al 3b 3c 3d) show the plots of relative test error vs. speed ups, both w.n. full data tuning for SST5 dataset with different combinations of search and scheduling we present the plots of relative test error vs. speed ups for TREC6 dataset. From the results, we observe that AUTOMATA achieves best speed up vs. accuracy tradeoff and consistently gives better performance even with small subset sizes unlike other baselines like RANDOM, CRAIG. In particular, AUTOMATA achieves a speedup of 9.88x and x.35x with a performance loss of 2.8% and a performance gain of 0.9% respectively on the SST5 dataset with TPE and Hyperband. Additionally, AUTOMATA achieves speedup of around 3. 2.68x with a performance gain of 3.4%, 4.6% respectively for the TREC6 dataset with TPE and ASHA.
methods. Similarly, in Sub-figuresBee 3r 3g 3h
Note that with a 30% subset, final training is not required as the models trained with 30% subsets achieve similar accuracy to full data training. However, for the sake of consistency, we use final training with 30% subsets well.
8

---

Image datasets results: Sub-figures 3i 3j 3k 31p show the plots of relative test eтror vs. speed ups. both W.Fl full data tuning for CIFARIO dataset with different combinations of search and scheduling methods. Similarly, sub-figures (3ml 3n 3o 3p) show the plots of relative test error vs. speed ups on CIFARTOD, TOD. The results show that AUTOMATA achieves the best speed up vs. accuracy tradeoff consistently compared to other baselines. More specifically, AUTOMATA achieves speedup of around 15x, 8.7x with a performance loss of 0.65%, 0.14% respectively on the CIFARIO dataset with Random and Hyperband. Further, AUTOMATA achieves a speedup of around 3.7x, 2.3 x with a performance gain of 1%. 2% for CIFARI00 dataset with TPE and ASHA. In Figure 4 show the effectiveness of the AUTOMATAWith large configuration space by repeating the experiment on the CIFAR 10 dataset with 1 110 configurations using Random as the search algorithm and Hyperband as the scheduler.
AUTOMATA
Rel. Fore
15
Speed Up
Figure 4: Performance of Au-
TOMATA on CIFARIO dataset with 110 search configurations
show the plots of relative test error vs. speed ups for the CONNECT-4 dataset. In practice, larger subset sizes should lead to better performance. do not depict the same picture. This is due to the inherent class imbalance within the CONNECT-4 dataset. In this case, smaller subset sizes with gradient based subset selection mitigated the effeets of class imbalance and enabled better model performance by selecting a similar proportion of samples from each class. Furthermore, using larger subset sizes caused gradient-basee based subset selection to select more samples from overrepresentative classes, thus reducing the final model's accuracy. Finally, AUTOMATA consistently achieved better speedup vs. accuracy tradeoff compared to other baselines on CONNECT-4 as well.
Tabular datasets results:
Sub-figures 3q 3r 3s
However, the plots 3q 3r
3s
31
Despite the fact that CRAIG and GSS have similar optimization problems, the performance of CRAIG is poor compared to AUTOMATA since CRAIG suboptimally optimizes an upper bound of the gradient error term. In addition, CRAIG is slower from a computational complexity standpoint since it requires the construction of a similarity kemel. Owing to space constraints, we provide additional results showing the accuracy vs. efficiency tradeoff on additional text, image, and tabular datasets in the Appendix G.5 It is important to note that AUTOMATA obtains better speedups when used for
tuning on larger datasets and larger models (in terms of parameters). Apart from the speedups achieved by AUTOMATA reductions of energy consumption and CO2 emissions, thereby making it more environmentally friendly. AUTOMATA can be used in conjunction with any hyper-parameter search algorithm and scheduler. However, It should be noted that the speedups obtained using AUTOMATA are heavily dependent upon the scheduler. Some schedulers, such as ASHA, are highly efficient and can effectively discard poor hyper-parameter configurations early, thus reducing the advantages caused by using data subsets. Therefore, the speedup achieved by AUTOMATA approximately 3x when using ASHA as a scheduler and around 10x-15x when using Hyperband. Nevertheless, we observe that using AUTOMATA still reduces tuning time and thus power consumption.
show in Appendix G.6 that it also achieves similar
4 Conclusion, Limitations, and Broader Impact
We introduce AUTOMATA an efficient hyper-parameter tuning framework that uses intelligent subset selection for model training for faster configuration evaluations. Further, we perform extensive experiments showing the effectiveness of AUTOMATA for Hyper-parameter tuning. In particular, it achieves speedups of around 10x- - 15x using Hyperband as scheduler and speedups of around 3x even with more efficient ASHA scheduler. AUTOMATA significantly decreases CO2 emissions and energy- eeeegy-efficient, reducing the environmental impact of hyper-papaaae parameter tuning on society at large. We hope that the AUTOMATA as framework will encourage community to consider subset selection approaches for faster syper-paramedd tuning, helping us move closer to the goal of Green AI [44). Research like this will help substantially reduce the cost of training large models and AutoML, thereby help democratize machine learning among smaller companies, individuals and academic groups. One of the limitations of AUTOMATA that in scenarios in which no performance loss is desired, we do not know the minimum subset size that gives the best speed up and, therefore, need to use larger subset sizes such as 10%, 30% In the future, we consider adapting the subset size based on model performance for each configuration to remove the dependency on subset size.

---

Acknowledgments and Disclosure of Funding
Rishabh Iyer and Krishnateja Killamsetty would like acknowledge support from NSF Grant Number IIS-21 IIS-2106937, a gift from Google Research, and the Adobe Data Science Research award. Ganesh Ramakrishnan is grateful IBM Research, India (specifically the IBM AI Horizon Networks IIT Bombay initiative) as well as the IIT Bombay Institute Chair Professorship for their support and sponsorship. Abhishek Guttu and Aakriti are grateful to IBM AI Horizon Networks IIT Bombay initiative. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF, IBM Research, Google Research, Adobe Data Science or the U.S. government.
References
[1] L. Barrault, O. Bojar, M. R. Costa-jussà, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, P. Koehn, S. Malmasi, C. Monz, M. Müller, S. Pal, M. Post, and M. Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy, Aug. 2019, Association for Computational Linguisties. doi: 10. 18653553/v/ W19-5301. URL https: http:///aaaaaahhoocoooooo org/oogg//////ooooo..ooo 5301
[2] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization Journal of Machine Learning Research, 13(10):2813305, 2012. URL http://jmlr. http:://jnlr.org/pppeerre prtpp//er/ppppppp////.
bergstrai2a. html
[3] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parametts optimization. In 25th anmual conference neural information processing systems (NIPS 2011), volume 24. Neural Information Processing Systems Foundation, 2011.
[4] Bergstra. R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter op- timization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Wein- berger, editors, Advances Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https: https::// ppocceedigss neurips. cc/ wttp:ccpppppp..eeee pttppp//////////leee pdf
[5] Bergstra, D. Yamins, and D. Cox. Making science of model search: Hyperparameter opti- mization hundreds of dimensions for vision architectures. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference Proceedings of Machine Learning Research, pages 115-123, Atlanta, Georgia, USA, 17-19 Jun html
Machine Learning, volume 28 of
2013. PMLR. URL,https: httpttts//ppcceeeinngg mlr htw...ppeee//////rrrrtrr... /ttpp/////ppoeeeiiii..peee//////
[6] T. Campbell and T. Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Machine Learning, pages 698-706, 2018.
[7] C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011. Software available at http:
ntu. .tu.uduu edu. aedduttw/ccr tt/""cjlin/libsm
[8] K. L. Clarkson. Coresets, sparse greedy approximation, and the Trank-wolfh algorithm. ACM Transactions on Algorithms (TALG), 6(4):1-30, 2010.
[9] A. Das and D. Kempe. Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection. arXiv preprint arXiv: 102.339,, 2011 1.
[10] T. Domhan, J. T. Springenberg, and F. Hutter. Speeding up automatic hyperparameter optimiza- tion of deep neural networks by extrapolation of learning curves. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'55 page 3460-3468. AAAI Press, 2015. ISBN 9781577357384.
[11] S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization scale. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Reseurch, pages 1437- 1 1446. PMLR, 10-15 Jul 2018. BRRl/https://proeeeeee .aaeeeddiinnmm.........mmmm.m..e mlr. www.pprrsssssseeeee (a1880ffalkee fae.rerlkarrrrrr 18a html
10

---

[12] D. Feldman. Core-sets: Updated survey. In Sampling Techniques Supervised or Unsupervised Tasks, pages 23 23-44. Springer, 2020.
[13] S. Fujishige. Submodular finctiions and optimization. Elsevier, 2005.
[14] S. Har-Peled and S. Mazumdar. On coresets for k-means and k-median clustering. In Pro- ceedings of the thirty-sixth annual ACM symposium on Theory of computing, pages 291-300, 2004.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[17] J. R. Hershey, S. J. Rennie, P. A. Olsen, and T. T. Kristjansson. Super- humann multi-talker speech recognition: graphical modeling approach. Comput. Speech Lang. 66,1) Jan. 2010. ISSN 0885-2308. doi: 10. 2.1001/j.esl.20088. 10.000 11.001. URL. wttph:tt::////tt http:///ooo...g11.101011111
J. j.csl. 2008. 11.001
[18] E. Hovy, L. Gerber, U. Hermjakob, C.-Y. Lin, and D. Ravichandran. Toward semantics based answer pinpointing. In Proceedings of the First International Conference Human Language Technology Research, 2001. URIhhttps::/ //www. aclweb. org/ attg//nnooloogyHBBO
[19] F. Hutter, H. H. Hoos, and K. Soyton-Brown, Sequential model- based optimization for general algorithm configuration. In C. A. C. Coello, editor, Learning and Intelligent Optimization, pages 507-523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-25566-33
[20] K. Jamieson and A. Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. A. Gretton and C. C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 240-248, Cadiz, Spain, 09-11 May 2016. PMLR. URL https //proceedings. mlr wwwss/ http////1/jaaiisnoo/c. html
[21] V. Kaushal, R. Iyer, S. Kothawade, R. Mahadev, K. Doctor, and G. Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1289-1299. IEEE, 2019.
[22] K. Killamsetty, D. Bhat, G. Ramakrishnan, and R. Iyer. CORDS: COResets and Data Subset selection for Efficient Learning, March 2021. URL https:://gtthhu/ com/ cow/decile-eeeccemmommcccc...eee team/ cords
[23] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, A. De, and R. Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5464-5474. PMLR, 18-24 Jul 2021. URL http:tt/ppocceeddiiiiiee
mlr. wwa..ppeesvvvvvv wttess/v119// http///aaaaaeettaaaaaaooeaaa
[24] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer. Glister: Generalization based data subset selection for efficient and robust learning. Proceedingg of the AAAI Conference on Artificia Intelligennce, 35(9): 10-8118, May 2021. URL/https://oja aaai. org/indees
[25] K. Killamsetty, X. Zhao, F. Chen, and R. K. Iyer. RETRIEVE: Coreset selection for efficient and robust semi-supervised learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https
/ //openreview net/ (tgeenneeieee.netfoouododdde
[26] K. Kirchhoff and J. Bilmes. Submodularity for data selection in machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 131-141, 2014.
11

---

[27] A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. In A. Singh and Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 528-536 PMLR, 20-22 Apr 2017. URL. htttps:/pproceedingg mlr wwwpss/eeeeleeeeeeeeee v5/kklein1//kele .ttll
[28] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report O, University of Toronto, Toronto, Ontario, 2009.
[29] T. Krueger, D. Panknin, and M. Braun. Fast tross-validation via sequential testing. J. Mach. Learn. Res.,
jan 2015. ISSN 1532-4435.
[30] A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions of machine learning, arXiv preprint arXiv:1910.09700, 2019.
[31] E. LeDell and S. Poirier. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, volume 2020, 2020.
[32] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 1161666765-6866 jan 2017. ISSN 1532-4435.
[33] L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, M. Hardt, B. Recht, and A. Talwalkar, A system for massively parallel hyperparameter tuning. arXiv preprint arXiv: arXh::1810.0592 1810.05934, 2018.
[34] L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-tzur, M. Hardt, B. Recht, and A. Talwalkar.
system for massively parallel hyperparameter tuning. In Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems, vol- ume 2, pages 230-246, 2020, URL https: //tp:::://rrreee //poceedd/s..cese.g...... mwsym. p22pp///2 Paper pdf
[35] X. Li and D. Roth. Learning question classifiers. In COLING 2002: The 19th International Con- ference on Computational Linguistics, 2002. URL https:// www.ww..wwbb........ wtalwwebbggg//e.... anthology/
C02- 1150
[36] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv: 2018.
[37] M. Minoux. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization techniques, pages 234-243, Springer, 1978.
[38] B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-efficient training of machine learning models, 2020.
[39] Y. Netzer, Wang. A. Coates, A. Bissacco, B. Wu, and Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsuper- vised Feature Learning 20/1, 2011. http:tt::
stanf ord. edu/housenumbers/
hrns201i_oouseeumbers pdf
[40] T.
Nickson, M. A. Oshorne, S. Reece, and J. Roberts. Automated machine learning on big data using stochastic algorithm tuning, 2014.
[41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy,, B. Steiner, L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style. high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, www.eerww....p.ppppppppeeeeeeeee 1ibrary.
pages 8024-8035. Curran Associates, Inc., 2019.
pdf
[42] Pennington, R. Socher, and C. Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empiricul Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar, Oct. 2014. Association for Computational Linguistics. doi: 10.3 I/D 14 4- 1162. URL httbpp:://aclaatholoyyy worg/D14-1162
12

---

[43] N. Pinto, D. Doukhan, J. DiCarlo, and D. D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS Computational Biology, 5, 2009.
[44] R. Schwartz, J. Dodge, N. Smith, and O. Etzioni. Green ai. Communications of the ACM, 63:54 63, 2020.
[45] L. N. Smith. A disciplined approach to neural network hyper-parameters: Part I-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv://03.098220 2018.
[46] Snoek, O. Rippel. Swersky, R. Kiros, N. Satish, N. Sundaram, M. M. A. Patwary, Prabhat, and R. P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, 15, page 2171-21 1171-2180. JMLR.org. 2015.
[47] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 1631- 1- 1642, Seattle, Washington, USA, Oct. 2013, Association for Computational Linguistics. URL
[48] E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 1906.0223, 2019.
[49] K. Swersky, J. Snoek, and R. P. Adams. Multi-task bayesian optimization. In C. J. C. Burges, L. Bottou, M. Welling, Ghahramani, and K. Q. Weinberger. ed- itors, Advances in Neural Information Processing Systems, volume 26. Curran As- sociates, Inc., 2013. URL hntps: httpp:://proeeeingn.neerr/ hww.eepps...cpppp////0111///11/. Paper. pdf
[50] K. Swersky, J. Snoek, and R. P. Adams. Freeze-thaw bayesian optimization. CoRR, abs/1406.3896, 2014. UtLttttp///aaxi org/ werg/abs//eee /a/aabbb//4..... 1101140063335 38966
[51] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackbexNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, Nov. 2018. Association for Computational Linguisties. doi: 10.18653/v1W18--1
URL htt.:/ttppe/cccaathooooooe wttp:/oooooooro/11---510100 org/
[52] K. Wei, Y. Liu, K. Kirchhoff, C. Bartels. and J. Bilmes. Submodular subset selection for large-scale speech training data. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3311-3315. IEEE, 2014.
[53] K. Wei, Y. Liu, K. Kirchhoff, and J. Bilmes. Unsupervised submodular subset selection for speech data. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4107-4111. IEEF, 2014.
[54] T. Yu and H. Zhu. Hyper- parameer optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689) 2020.
13


Reviews:
[{"review_id": "xI0QkwFdUT_", "reviewer": "Reviewer_Mepf", "summary": "This paper evaluates a gradient-based subset selection method in hyperparameter optimization and shows that the method can speed up the hyperparameter optimization process without significant performance deterioration compared to when using the entire dataset. The proposed method, AUTOMATA, consists of three components: hyperparameter search algorithm, data subset selection method, and hyperparameter scheduler. Each component of the proposed method can be selected from existing algorithms. The effectiveness of AUTOMATA is experimentally verified on datasets of text, computer vision, and tabular domain. The performance of  AUTOMATA is experimentally verified.", "questions": "* The effectiveness of the combination of data subset selection and hyperparameter scheduling is not so surprising because these techniques attempt to speed up model training from different aspects. It would be better to clarify the difficulty of incorporating the data subset selection into hyperparameter optimization.\n\n* I think that the value of the epoch interval for subset selection impacts the computational efficiency and accuracy in AUTOMATA. What is the impact of the epoch interval setting on the performance?\n\n* The hyperparameter search space treated in this paper seems to be not so large. It would be great to show the experimental results in large-scale search spaces. The NAS-Bench or HPO-Bench would be possible choices.\n\n* In Figure 3, the plot of \"FULL\" (red-colored) cannot be found.\n\n* The error bar should be plotted in Figure 3.", "limitations": "The limitations and potential negative societal impact are addressed.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths: The effectiveness of the data subset selection method in hyperparameter optimization is experimentally validated, and the proposed AUTOMATA succeeds in speedups of 3 to 30 times for several datasets.\n\n* Weaknesses: The approach is somewhat straightforward. The novelty of this paper is using the gradient-based data subset selection method in hyperparameter optimization. It is quite natural that subset selection methods are also useful in hyperparameter optimization because existing data subset selection methods succeed in the cost reduction of a single training process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "w1lIDVOzdrH", "reviewer": "Reviewer_9uDV", "summary": "This work proposed a framework to speed up HPO by gradient based dataset subset selection. The methodology contribution lies on the gradient based per batch subset selection using OMP. It seems to be a modification of  GARD-MATCH [23]. The authors provided extensive empirical results on different HPO search algorithms and HPO schedule algorithms, and in combination of different core set selection methods. The results suggested that one could speed up HPO significantly with the framework while keeping the final solution quality (mild decreasing).", "questions": "* It seems the results can be quite different depending on the HPO search algorithm and scheduler. Why is it the case and could the authors provide some insights? Does this mean I would better not to use the proposed method for certain HPO optimizer?\n\n* The author mentioned “Similar to GRAD-MATCH, we use a greedy algorithm called orthogonal matching pursuit (OMP)”, what is then the difference?\n\n* Figure 3 doesn't show any error bar but there are 3 or 5 repetitions. Is there some particular reason for it?\n\n* [Minor] Line 190, the summation over all the batches with subscript $i$ should be with something else as $i$ is already the hyperparameter configuration.\n", "limitations": "The authors have mentioned that choosing a subset size in advance can be difficult and it is hard to anticipate the loss of performance. I would also add how to make it more robust across different HPO optimizers.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper’s empirical work is very well done. It contains clear experiments setting, abundant real world examples, thoughtful baselines, detailed ablation studies and interesting analysis. The results are also encouraging. I like the presentation, the paper is well organized and easy to follow.\n\nOn the weakness side, the methodology contribution seems limited and the main merits of the paper are the empirical results. Also, the trade off between speed up and performance loss is hard to quantify in advance and it will be hard to choose a subset size in practice without trying a few options. In the end, the results seem sensitive to the HPO optimizers (the performance loss could differ significantly).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6R_mojAAK3h", "reviewer": "Reviewer_HuPX", "summary": "The paper presents AUTOMATA, a framework for hyper parameter tuning that relies on three main components for accelerating the search for the optimal hyper-parameter configuration. Automata was developed with the goal of accelerating the search for the optimal configuration in order to minimize the cost and environmental impact of the hyper-parameter tuning process. The key mechanism described by the authors as the main contributor to the savings and speed-ups achieved by Automata is DSS, a data subset selection approach that aims to find a subset of data with which to evaluate each configuration such that: i) it is faster to evaluate the quality of a configuration; ii) the performance order of the evaluated configurations when evaluated with the full training set is preserved.\n", "questions": "**a)** The expression \"retaining the ordering of hyper-parameters\" (line 73) is not explained well enough. From my understanding, what it means is that given that Automata uses less data to evaluate a specific configuration, you want to guarantee that if configuration A is worse than configuration B (when evaluated with a data subset selected by Automata), configuration A is actually worse than configuration B when evaluated with the full dataset. Is this correct? If yes, I could only understand this half-way through the paper and if you could clarify this earlier on, the paper would read much better.\n\n**b)** Figure 2 could be improved and further leveraged to aid with the explanation of the data subset selection procedure described in Section 2.3.1. For instance, why are there multiple DSS training loops? Do you evaluate multiple hyper parameter configurations at once? If yes, you could perhaps add this information to each of the input arrows to the DSS training loop component. It would also help if you added an example of the \"normal\" subset selection mechanism to visually compare it with the per-batch method.\n\n**c)** What is the meaning of k in equation 1) and in line 186? Is it the same k of line 209? If so, then how/where are coresets used by the data subset selection mechanism?\n\n**d)** Why do you use only MNIST to evaluate whether GSS retains the order of configurations if for the remaining evaluation you never use MNIST again? Since you have results for FULL for each pair of dataset and framework configuration (i.e., scheduler and hyper parameter tuning approach), why not evaluate the order retention for those cases as well? Moreover, since you claim Automata has better performance on larger datasets and models (line 351), it would be very interesting to see if the configuration order is retained in these cases.\n\n**e)** Can you clarify what you mean by relative test error? \n\n**f)** How do DSS and Grad-Match differ? If they differ, why is Grad-Match not a baseline? If they do not differ, this should be clarified in the submission.\n\n**Observations:** \n- checklist question - Did you include the total amount of compute and the type of resources used (e.g., type 570 of GPUs, internal cluster, or cloud provider)? --- I could not find this information in section F.4\n- Supplemental material, figure 10: it seems like the x-axis in the figure are not correctly labelled. \n", "limitations": "Limitations regarding any potential negative societal impact of the work were addressed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** the tasks and the methods are not new but the work provides a combination of well-known techniques which, as far as I know, is novel. It is not clear how DSS differs from Grad-Match [23].\n\n**Quality:** The evaluation could be improved by providing more experiments on the order of the hyper-parameters. If DSS differs from Grad-Match, why is Grad-Match not a baseline?\n\n**Clarity:** Overall, the paper is clear and well written. Some parts could be improved. First, if Automata is a framework, then more emphasis should be given to its components and to its use. For example, if I wanted to use Automata for my work: i) which components could I reuse? ii) which components would I need to replace? iii) what should I be careful about, for instance regarding integration of the 3 main components (search algorithm, DSS, and scheduler)? Second, the meaning of \"retaining the ordering of hyper-parameters\" is not clearly explained anywhere in the paper and when the authors mention that final accuracy is not their main concern (line 74) confuses the reader, since in the hyper parameter tuning context, improving accuracy is a primary concern. Third, it was not clear to me if variable 'k' was the same throughout section 3 nor what its meaning was. Figure 2 currently does not contribute much to understanding the paper. Suggestions on how to improve this are below. Finally, section 3.2 would read much better if there were more paragraphs, perhaps split per domain, like in section 3.4 (text datasets, image datasets, tabular datasets).\n\n**Significance:** the idea of having a framework that contains different search algorithms, data subset selection strategies, and schedulers that can be mixed and matched to speed-up the hyper-parameter tuning for different applications and domains is interesting and I can see it being used by practitioners/researchers.   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "24AeBbp_qo5", "reviewer": "Reviewer_Grjc", "summary": "This paper presents AUTOMATA, a gradient-based subset selection framework for hyper-parameter tuning. The authors evaluate the framework using various datasets and models. The experimental results show that the proposed framework can improve the speedup of parameter tuning significantly.  ", "questions": "- What are the hyper-parameter scope used in the experiments? \n- The experiment only compares with CRAIG. How about other approaches?  \n- It will be helpful for the authors to explain why the ASHA or Hyperband are used as scheduling algorithm? Are these two the most representative scheduling algorithms or the most commonly used ones? \n- Does this approach apply to more complex datasets, such as ImageNet? \n- How about showing the training wall clock time?\n- Have you evaluate the cases where the subsets are chosen at different number of epochs, e.g., 10, 40, etc?  \n", "limitations": "The authors have addressed the limitations of their works. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method proposed by the authors are very interesting.\n\nOriginality:\nThe novelty of this work limited. The design is more about a combination of some existing techniques.  \n\nQuality:\n- The writing can be improved. There is no explanation and justification on some design decisions. Please refer to Questions section for more details. \n\nClarity:\n- It will be helpful to provide more justification to the design. Why those three components are crucial? \n- The main text should be self-contained. \n- In Section 2.3.1, there is no theoretical proof or experiment to support idea of gradient selection. \n- Better to informative component names instead of Component-1, 2, and 3.\n\nSignificance:\n- This speedup-accuracy trade-off can be useful for other follow up works.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept