Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning
Authors: Martin Wistuba
Arlind Kadra
Josif Grabocka
Venue: ICLR 2022 Submitted (2022)
Keywords: hyperparameter optimization

Abstract: Gray-box hyperparameter optimization techniques have recently emerged as a promising direction for tuning Deep Learning methods. However, the multi-budget search mechanisms of existing prior works can suffer from the poor correlation among the performances of hyperparameter configurations at different budgets. As a remedy, we introduce DyHPO, a method that learns to dynamically decide which configuration to try next, and for what budget. Our technique is a modification to the classical Bayesian optimization for a gray-box setup. Concretely, we propose a new surrogate for Gaussian Processes that embeds the learning curve dynamics and a new acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization baselines through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse neural networks (MLP, CNN/NAS, RNN).

Under review as a conference paper at ICLR 2022
DYNAMIC AND EFFICIENT GRAY-BOX HYPERPARAM- ETER OPTIMIZATION FOR DEEP LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Gray-box hyperparameter optimization techniques have recently emerged as a promising direction for tuning Deep Learning methods. However, the multi- budget search mechanisms of existing prior works can suffer from the poor cor- relation among the performances of hyperparameter configurations at different budgets. As a remedy, we introduce DYHPO, a method that learns to dynami- cally decide which configuration to try next, and for what budget. Our technique is modification to the classical Bayesian optimization for a gray-box setup. Con- cretely, we propose a new surrogate for Gaussian Processes that embeds the learn- ing curve dynamics and a new acquisition function that incorporates multi-budger information. We demonstrate the significant superiority of DYHPO against state- of-the-art hyperparameter optimization baselines through large-scale experiments comprising 50 datasets (Tabular, Image. NLP) and diverse neural networks (MLP, CNN/NAS, RNN).
INTRODUCTION
Hyperparameter Optimization (HPO) is arguably acute open challenge for Deep Learning (DL), especially considering the crucial impact HPO has on achieving state-of-the-art empirical results. Unfortunately, HPO for DL is a relatively under-explored field and most DL researchers still op- timize their hyperparameters via obscure tril-ann-erros practices. On the other hand, traditional Bayesian Optimization HPO methods (Snock et al.) 2012 Bergstra et al.] (011]) are not directly applicable to deep networks, due to the infeasibility of evaluating large number of hyperparam- eter configurations. In order to scale HPO for DI., three main directions of research have been recently explored. (i) Online HPO methods search for hyperparameters during the optimization pro- cess via meta-level controllers (Chen et al.. 2017 Parker-Holder et al., 2020), however, this online adaptation can not accommodate all hyperparameters (e.g. related to architectural changes). (ii) Gradient- based HPO techniques, on the other hand, compute the derivative of the validation loss w.r.t. hyperparameters by reversing the training update steps (Maclaurin et al., 2013 Franceschi cet al. 2017, Lorraine et al.] 2020), however, the reversion is not directly applicable to all cases (e.g. dropout rate). The last direction, (iii) Gray-box HPO techniques discard sub-optimal configurations after evaluating them on lower budgets (e.g. after few epochs) (Li ct al. 2017 Falkner et al. 2018).
In contrast the online and gradient-baser alternatives, gray-box approaches can be deployed in an off-the-shelf manner to all types of hyperparameters and architectures. The gray-box concept is based on the intuition that a pporlyppefforiin hyperparameter configuration can be identified and terminated by inspecting the validation loss of the first few epochs, instead of waiting for the full convergence. The most prominent gray-box algorithm is Hyperband (Li et al.) 2017). which runs random configurations at different budgets (e.g. different number of epochs) and successively halves these configurations by keeping only top performers. Follow-up works, such as BOHB (Falkner al. 2018) DEHB (Aaad et al. 2021). replace the random sampling of Hyperband with more intelligent sampling based on Bayesian optimization or differentiable evolution.
Despite their great practical potential, existing gray-box methods suffer from major issue. The low-budgett (few epochs) performances are not always a good indicator for the full-budget (full convergence) performances. For example, properly regularized network converges slower in the first few epochs, however, typically performs better than a rog-ragulrrized variant after the full convergence. In other words, there poor rank correlation of the configurations' performances

---

Under review as a conference paper at ICLR 2022
at different budgets. As remedy, we introduce DyHPO, gray-box method that dynamically decides how many configurations to try and how much budget to spend on each configuration.
DYHPO is a Bayesian Optimization (BO) approach based on Gaussian Processes (GP), that extends and adapts BO to the multi-budget (a.k.a. multi-fidelitty) case. In this perspective, we propose a deep kernel GP that captures the learning dynamics (learning curve until the evaluated budget). As a result, we train a kernel capable of capturing the similarity of a pair of hyperparameter config- urations, even if the pair's configurations are evaluated at different budgets (technically, different learning curve lengths). Furthermore, we extend Expected Improvement (Jones et al. 1998) to the multi-budget case, by introducing new mechanism for the incumbent configuration
budget.
The joint effect of modeling GP kernel across budgets together with a dedicated acquisition func- tion leads to DYHPO achieving statistically significant empirical gain against state-of-the-as gray-box baselines (Falkner et al. 2018 Awad et al. 202T), including prior work on multi-budgen GPs (Kandasamy et al. 2017; Metz et al. (2020). We demonstrate the performance of DyHPO in three diverse popular types of deep learning architectures (MLP, CNN/NAS, RNN) and 50 datasets of three diverse modalities (tabular, image, natural language processing). We believe our method is a step forward towards making HPO for DL practical and feasible. Overall, our contributions can be
summarized as follows:
We introduce a novel Bayesian surrogate model that is based on Gaussian Process with a deep kernel. This surrogate model predicts the validation score of a machine learning model based on the hyperparameter configuration and the learning curve.
We derive a simple yet robust way to combine this surrogate model with Bayesian opti- mization, reusing most of the existing components currently used in traditional Bayesian optimization methods.
Finally, we demonstrate the efficiency of our method for hyperparameter optimization and neural architecture search tasks compared to the current state-of-the-ar methods. As an overarching goal, we believe our method is an important step towards scaling HPO for DL.
2 MOTIVATION
As mentioned earlier, major source of sub-optimality for the current gray-box techniques (Li al., 2017 Falkner et al., a 2018, Awad et al. 2021) is the poor rank correlation of performances at low and high budgets, which is endemic to the successive halving mechanism. In essence, prior methods mis- takenly discard good formance after few epochs. In contrast, our method fixes the endemic poor correlation of successive halving, by learning to decide which configuration to continue optimizing for one more time-budget step (e.g. one more epoch). In that perspective, our strategy resembles Freeze- Thaw (Swersky eet al. 2014), however, differs in fitting deep kernel GP surrogates with learning curves and a special multi-budget acquisition. DYHPO never discards a configuration as all options remain latent until the acquisition selects the most promising one and allocates one more budget unit of optimization.
configurations by myopically relying only on the early per-
We illustrate the differences between our strategy and successive halving with the experiment of Figure where we showcase the HPO progress of three different methods on the "Helena" dataset from the LCBench benchmark (Zimmer et al. 2021]. Random search is an example of a black-box approach that trains each candidate until completion without considering the intermediate scores. Hyperband (Li et al. 2017) is a gray-box approach that statically pre-allocates the budget for a set of candidates (Hyperband bracket) according to a predefined policy. Finally, DYHPO dynamically adapts the allocation of budgets for configurations after every HPO step.
The plots in the top row of Figure show the learning curves of multiple neural networks trained with different hyperparameter configurations. The darker the color of a learning curve, the later the model corresponding to the learning curve was trained during the optimization process. In an optimal scenario, there should be no learning curve of darker color trained for very long time if there is already a learning curve of lighter color with higher accuracy. Since black-box functions do not consider intermediate responses, this trend is not observed for random search. This is also not the case for gray-box methods such as Hyperband, because it is possible that no configuration selected for a Hyperband bracket is outperforming the current best configuration. Since at least one
2

---

Under review as a conference paper at ICLR 2022
Random Search
Hyperband Boxay-Box)
DyHPO (Gray-Box)
(Black-Box)
0.2
Accuracy
0.1
0.0
Temporal Order
15
75
75
10
50
60
25
0
20
Number of Epochs
Number of Epochs
Number of Epochs
Figure 1: Top row: Learning curves observed during the search. The darker the learning curve, the later it evaluated during the search. Bottom row: y-axis shows sequence of learning curve evaluations (bottom to top). The color indicates accuracy. The darker red the higher the accuracy.
of the bracket candidates must be fully trained according to the predefined policy, observe this suboptimal behavior in Figure] [ This motivates our idea of dynamically deciding when we want to continue training configuration based on the intermediate response. Clearly, DYHPO invests only a small budget into configurations that show little promise as indicated by the intermediate scores.
RELATED WORK ON GRAY-BOX HPO
Multi-Fidelity Bayesian Optimization. Bayesian optimization is a black-box function optimiza- tion framework that has been successfully applied in optimizing hyperparameter and neural archi- tectures alike (Snoek et al. 2012] Kandasamy et al. 2018 [Begsttra et al. 201T). To further improve Bayesian optimization, several works propose low cow-idelity data approximations of hyperparameter configurations. Low-fidelity approximations of hyperparameter configurations can be obtained by training on subset of the data (Swersky et al., 2013 Klein al. 2017) or terminating training early (Swersky et al.) 2014). Several methods extend Bayesian optimization to multi-fidelity data by engincering new kernels suited for this problem (Swersky et al. 2013 2014, Poloczek et al.) 2017). Kandasamy et al. (2016) extends GP-UCB (Srinivas et al. 2010) to the multi-fidelity setting by learning one Gaussian Process (GP) with a standard kernel for each fidelity. Their later work improves upon this method by learning one GP for all fidelities that enables the of continuous fidelities (Kandasamy et al. 2017). The work by Takeno et al. (2020) follows a similar idea but pro- poses use acquisition function based information gain instead of UCB. While most of the works rely on GPs to model the surrogate function, Li et al.) (202)) use Bayesian neural network that models the complex relationship between fidelities with stacked neural networks, one for each fidelity.
Multi-Fidelity Bandits. Hyperband (Li al.) 2017) is a bandits-based multi-fidelity method for hyperparameter optimization, that due to its simplicity and strong performance, enjoys great pop- ularity. The algorithm selects hyperparameter configurations at random and uses successive halv- ing (Jamieson & Talwalkar 2016) with different settings to early-stop less promising training runs. Several improvements have been proposed to Hyperband with the aim to replace the random sam- pling of hyperparameter configurations with more guided approach (Bertrand et al. 22017: Wang et al. 2018 Wistuba 2017). BOHB (Falkner et al.) 2018) uses TPE (Bergstra et |0.1 and builds surrogate model for every fidelity adhering
fixed-fie exed-fidelity selection scheme. DEHB (Awad) ct al. 2)21) samples candidates using differential evolution which handles discrete and large hyper- parameter search spaces better than BOHB.
3

---

Under review as a conference paper at ICLR 2022
Deep Kernel Learning with Bayesian Optimization. We among the first to deep kernll learning with Bayesian optimization and to the best of our knowledge the first to use it for multi- fidelity Bayesian optimization. [Rai et al. (2016) consider the of deep kernel instead of manually designed kernel in the context of standard Bayesian optimization, but, limit their exper- imentation synthetic data and do not consider its use for hyperparameter optimization. Perrone et al. (2018); Wistuba & Grabocka (2021) use a pre-trained deep kernel to warm start Bayesian op- timization with meta-data from previous optimizations. These approaches are multi-task or transfer learning methods that require the availability of meta-data from related tasks.
In contrast to prior work, we propose the first deep kernel GP for multi-fidelit, HPO that is able to capture the learning dynamics across fidelities/budgets, combined with an acquisition function that is tailored for the gray-box setup. Furthermore, our work represents an important step towards scaling HPO for Deep Learning (DI.), by demonstrating a statistically significant reduction in terms of HPO time on series of DI. network architectures and large set of diverse datasets.
PRELIMINARIES
Black-Box Optimization. As mentioned earlier, the problem of optimizing hyperparameters can be modeled as a black-box optimization problem. The objective is to maximize the response func- tion X E that returns the validation score for training a machine learning model with hyperparameter configuration x E X. In practice, this observation is noisy such that we observe in fact y/ - f(x,))++)) + E where g ~ (N o.)
Gray-Box Optimization. Since many machine learning algorithms allow to measure at various fi- delities, a relaxation of the black-box to the gray-box optimization problem is in many cases logical and allows for significantly faster optimization. The gray-box setting allows us to query configu- rations with budget smaller than the maximum budget B. Thus, we can query from the response function f : X' x N + I where fig J) is the response after spending budget of j on configuration x;- As before, these observations are noisy and we observe Yi.j = [xx,,) + Ej where Ej (0, oj,n). Please note, we assume that the budget required query fi.++ after querying Ji, is only b. While this is not necessarily the case for all problems, the models we consider are learned incrementally. Furthermore, we are able to make use of the learning curve when predicting fia.
Yij- (V.,,....,,
Bayesian Optimization (BO). BO is a general framework for solving black-box optimization problems and is very popular for optimizing hyperparameters. It has two ingredients, i.e. a surrogate model and an acquisition function. The surrogate is a probabilistic model which approximates the black-box function using the available information of function evaluations. The acquisition function returns the expected utility for a configuration given the surrogate model's prediction. The BO framework executes the following steps sequentially. First, the surrogate model is trained the available information about the black-box function, denoted as the history of evaluations D. Then, the configuration with the highest expected utility x; is evaluated and y/ is observed. The tuple (x;, yi) is added to D and the process is repeated until the HPO budget is exhausted. A common choice for the surrogate model are Gaussian Processes (Rasmussen & Williams, 2006). a popular choice for the acquisition function is expected improvement (Jones et al. 1998).
Gaussian Processes (GP). GPs are probabilistic machine learning models. Given training data set D = ((x, Ji = 1,...,m) n) with data points, the Gaussian Process assumption is that is random variable and the joint distribution of all y; assumed be multivariate Gaussian distributed as:
y N (m (X), kX X)) Furthermore, f, for test instances x, are jointly Gaussian with y as:
( K K.
y f.
~ m xX.x.)
(2)
K
K..
The mean function m is often set to O and its covariance function l: depends on parameters 0. For notational convenience, we use:
K, l (X, x + I K. k (X,xx100) K.. =k (X., ()
(3)
4

---

Under review as a conference paper at ICLR 2022
to define the kernel matrix. We can derive the posterior predictive distribution with mean and co- variance as follows:
y K, cov X.] K. K.
(4)
Often, the kernel function is manually engineered, popular example is the squared exponential kernel. However, in this work, we make use the idea of deep kernel learning (Wilson et al. 2016). The idea model the kernel neural network yp and learn the best kernel transformation
kdeep x, x' 0, w) k(p(x, w), (x w)l0).
As we will see later, this allows us to use convolutional operations as part of our kernel.
5 DYNAMIC MULTI-FIDELITY HYPERPARAMETIE OPTIMIZATION
In this section, we will describe DYHPO, our proposed method for hyperparameter optimization in the gray-box setting. At first, we will describe the surrogate model which Gaussian Process with deep convolutional kernel. Then, we describe a variation of the popular expected improvement acquisition function (Jones et al. 1998), modified consider multiple fidelities, and conclude with the final algorithm.
5.1 MULTI-FIDELITY SURROGATE WITH DEEP CONVOLUTIONAL KERNEL
We propose to use a Gaussian Process surrogate model that infers the value figj based on the hyperparameter configuration x;, the budget j well the past learning curve Y ijj-1- For this purpose, we use deep kernel which defined as
kdoop (XC Yi..-1,5), (x4, (e)-).5))) k(p(x,, Yx.-1.5), (x(=, Yvg-1.)))
(6)
We squared exponential kernel for and the neural network yp is composed of linear and convolutional layers as shown in Figure 2 We normalize the budget j to a range between and by dividing by the maximum budget B. Afterward, concatenated with the hyperparameter configuration x, and fed to linear layer.
The learning curve Yij-1 trans- formed by one-dimensional convo- lution followed by global max pool- ing layer. Finally, both representa- tions are fed another linear layer. Its output will be the input the ker- nel function k. Both, the kernel and the neural network consist of trainable parameters 0 and w, respec- tively. We find their optimal values by computing the maximum likeli- hood estimates
Yig-1
Convolution
Figure 2: The feature extractor &o used in deep kernel.
0, W = arg meggaapiin max 0. = arg max / ply|X, X, Y, O. w) w)p Y, O, w ddf 0.w
0,w
oc arg miny " y + log K,
8.w
In order to solve this optimization problem, we use gradient descent and Adam Kingma Ba 2015) with learning rate of 0.1. Given the maximum likelihood estimates, approximate the predictive posterior as formalized in Equation g and ultimately compute the mean and covariance of this Gaussian using Equation
p XJ Yig-1,j, = (Jis (ii Y Y,
(8)
5

---

Under review as a conference paper at ICLR 2022
5.2 MULTI-FIDELITY EXPECTED IMPROVEMENT
Expected improvement (Jones et al. 1998) is a commonly used acquisition function for the black- box setting and is defined as:
EI(x E Imax [f(x) - y 00]]
(9)
where y is the largest observed value of f. We propose a multi multi-fidelityy version of as:
EIMF jx.j] = E [mxx (f(x, - " 00
(10)
where:
max {y/ ((x, : 9). ) E D)
Ex ((x, -.).)) E
y mas
(11)
max {y/ (-, v) E D}
otherwise
Simply put, w/ is the largest observed value of for budget of j if exists already, otherwise, is the largest observed value for any budget. If there is only one possible budget, the multi-fidelity expected improvement identical to expected improvement.
5.3 THE DYHPO ALGORITIM
The DYHPO algorithm looks very similar to many black-box Bayesian optimization algorithms shown in Algorithm[] T The big difference is that each step dynamically decide which candidate configuration to train for a small additional budget. Possi- ble candidates are previously uncon- sidered configurations as well as con- figurations that did not reach the max- imum budget. In Line 2, the most promising candidate is chosen using the acquisition function introduced in Section [5.2 and the sur- rogate model's predictions. It is important to highlight that we do not maximize the acquisition function along the budget dimensionality. Instead, set j such that it is by exactly one higher than the budget used to evaluate x; before. x; has not been evaluated for any budget yet, j is set to 1. This ensures that explore configurations by slowly increasing the budget. After having selected the candidate and the corresponding budget, the function f is evaluated and we observe Vi.j (Line 3). This additional data point is added to D in Line 4. Then in Line the surrogate model updated according to the training scheme described in Section 5.1
Algorithm 1: DYHPO Algorithm
1: while not converged
2: f- arg x EIMF (x, j) (Section 5.2
3: Observe yk.j.
4: D f DU ((x,, Yi)- }i.))}
5: Update the surrogate model on D. (Section 5.1
6: return X; with largest Vi.j-
6 EXPERIMENTS
We evaluate DYHPO in three different settings on hyperparameter optimization for tabular, text, and image classification against several competitor methods. These include Hyperband (Li al. 2017), BOHB (Falkner et al. 2018), DEHB (Awad et al., 202T), and Dragonfly (Metz et al.) 2020). We use Dragonfly's multi-fidelityy optimizer (Kandasamy al. (017). For sanity check, also compare against random search (Bergstra & Bengio 2012). We use the publicly available implementations whenever available and implemented Hyperband and random search ourselves. We report the mean of ten repetitions and report two common metrics such as the regret and the average rank. The regret refers the absolute difference between the score of the solution found by an optimizer compared to the best possible score. If we report the regret as an aggregate result over multiple datasets, we report mean over all regrets. The average rank a metric we use to aggregate results over different datasets. For each dataset, the best performing method obtains a rank of 1. Ties are broken by using the average rank, e.g., if the methods have scores 0.9, 0.8, 0.8, 0.7, the ranks are 1, 2.5, 2.5, and 4. For both metrics, smaller is better.
6.1 FREDFORWARD NEURAL NETWORKS
In our first experiment, we evaluate the various methods on how well they optimize neural networks for tabular datasets. For this purpose, use the I.Cench learning curve benchmark (Zimmer al.

---

Under review as a conference paper at ICLR 2022
4.5-
Mean Regret
Average Rank
Random
4.0
Hyperband
BOHB
3.5
DEHB
Dragonfly
DyHPO
2.5
0.2
0.4
0.6
0.8
1,0
0.2
0.4
0.6
0.8
1.0
Normalized Wallclock Time
Normalized Wallclock Time
Figure 3: LCBench: Aggregated results over 35 different datasets. The normalized wallelock time represents the actual runtime divided by the total wallclock time of DYHPO including the overhead of fitting the deep GP. DYHPO achieves the best performance among all methods for both metries.
Random
DEHB
Random Dragantly
DyHPO
Random Dragontly Hyperioand
DyHPO
Dragorify
BOHB DyHPO
BOHB
BOHB DEHB
Hyperizend
Hyperionnd
DEHB
Figure 4: Critical difference diagram for LCBench for results corresponding to the time DyHPO took to complete 200, 600 and 1,000 epochs. DyHPO's improvement is statistically significant.
2021]. This benchmark contains learning curves for 35 different datasets where 2,000 neural net- works per dataset are trained for 50 epochs with Auto-PyTorch. The objective to optimize seven different hyperparameters of funnel-shaped neural networks, i.e., batch size, learning rate, momen- tum, weight decay, dropout, number of layers, and maximum number units per layer. We show the aggregated results in Figure 3 Here aggregate the normalized wallclock time by dividing the actual wallclock time of basclines by the total wallclock time our method DyHPO includ- ing the overhead incurred by fitting the deep GP. In that manner,
aggregate wallclock times across datasets. However, all the results for each individual dataset are reported in the Appendix (Figure 12 and 13 Furthermore,
use critical difference diagram report the pairwise statis- tical difference between all methods (Figure 4. Horizontal lines indicate groups of methods that are not significantly different from each other. As suggested in the work of Demsar (2006), we use the Friedman test to reject the null hypothesis followed by a pairwise posthoc analysis based on the Wilcoxon signed-rank test 0.05). While all gray-box methods have very similar performance in the very beginning, DYHPO quickly outperforms its competitors with respect to both evaluation metries with statistical significance. On this task, both BOHB and DEHB are significantly better than Hyperband but there is clear winner among these two methods.
6.2 RECURRENT NEURAL NETWORKS
We continue with evaluating all methods NLP tasks using search spaces provided in TaskSet (Metz et al. 2020). The objective is to optimize eight hyperparameters for a set of different recurrent neural networks (RNN) that differ in embedding size, RNN cell, and other architectural features. The set of hyperparameters consists of optiiieer-ppecfffc hyperparameters, such as the learning rate, the exponential decay rate of the first and second momentum of Adam, B1 and 82, and Adam's constant for numerical stability e. Furthermore, there are two hyperparameters controlling linear and exponential learning rate decays, as well as LI and L2 regularization terms. As before, provide the aggregated results in the main paper (Figure 5) and provide detailed results in the Appendix (Figure II). We show the critical difference diagram in Figure 6 As before, all gray-box methods have very similar performance in the very beginning. but, Dy HPO quickly outperforms its competitors with respect to both evaluation metrics. The difference is significant in the first inter- val measured, in fact, DYHPO is providing the best results across all different tasks for the majority

---

Under review as a conference paper at ICLR 2022
5
Mean Regret
Random
Hyperband
10"1
Average
BOHB
DEHB
Dragonfly
2-
DyHPO
0
200
400
600
800
1000
200
400
600
800
1000
Number of Steps
Number
Steps
Figure 5: TaskSet: Aggregated results over different NLP tasks. Again, DYHPO shows the best performance among all methods for both evaluation metrics.
3
Random
Drapontly
BOHB
Draponfly Hyperbend
BOHB DEHB
Draganfly DEHB
BOHB Hyporbard
Hyperbaind
DEHB
Figure 6: Critical difference diagram for TaskSet for results after 200, 600 and 1,000 epochs, re- spectively. DYHPO's improvement is statistically significant.
of the elapsed time. Given enough time, BOHB is able catch up but DyHPO's improvement is still statistically significant.
6.3 CONVOLUTIONAL NEURAL NETWORKS
Neural Architecture Search (Zoph & Le 2017) (NAS) raised a lot of interest in the deep learn- ing community in the last few years. Since it can be reformulated to hyperparameter op- timization problem, we can apply our method as well as many other standard hyperparame- ter optimization methods to it. We refrain from comparing against specialized NAS methods, since, this out-of-scope for our work but re- fer the interested reader to the comparison of Dong & Yang (2020). To evaluate methods in this scenario, we use NAS-Bench-201 (Dong YYnn 2020). This benchmark has precom- puted about 15,600 architectures trained for 200 epochs for the image classification datasets CIFAR-10, 10. CIFAR-100, and ImageNet. The objective to select for each of the oper- ations within the cell of the macro architecture one five different operations. All other hy- perparameters such as learning rate and batch size are kept fix. We report the results in Figure g the remaining results can found in the Appendix (Figure 10). Initially, DYHPO provides better results, but, given enough time, most methods perform no longer significantly different. We also no difference between Hyperband and BOHB or DEHB. A reason for this observation will be discussed in the next section.
ImageNat16-120
Regret
10"
10"
10"
Wallelock Time Seconds
Random
BOHB
Dragonfly
Hyperbond
DEHB
DyHPO
Figure 8: DYHPO quickly finds well vell-perfonning configurations. Given enough time. most methods find equally good architectures.
8

---

Under review as a conference paper at ICLR 2022
Mean Regret over all Datasets
ImageNet16-11 6-120
DyHPO
DyHPO wib LC
Regret
Ragret
10
DyHPO
DyHPO Wo'LC
10
10"
10"
10"
600
1000
Number Epochs
Training Time in Seconds
Figure 7: Left: Aggregated results for LCBench. Right: Results on ImageNet from NAS-Bench 201. Using the learning curve gives only little advantage on average for the LCBench problems. However, not using it significantly worsens for the NAS-Bench 201 tasks.
6.4 ABLATION STUDY
One of the main differences between DYHPO and similar methods such as the work by Kandasamy iet al.] (2017), is that the learning curve is input to the kernel function. For this reason, we investi- gate the impact of this design choice. We consider a variation of DYHPO, DYHPO w/o LC, which identical to its counterpart, and the only difference is that the learning curve is part of the input. We report the results in Figure 7 and refer to the Appendix for additional results (Figure 9 One of the striking observations is that the learning curve has only small impact for LCBench in contrast to NAS-Bench-200 where is significant. Our hypothesis is that for LCBench the hyperpa- rameter representation is valuable feature such that the additional use of the learning curve (which in fact is already implicitly considered by the Gaussian Process) is not required. For NAS- NAS-Bench 201, however, the architecture representation is poor and does not allow for good predictions on its own. This hypothesis aligns well with our previous experiments, where BOHB did better on LCBench compared to Hyperband whereas we have not seen any difference NAS-Bench 201. As reminder, BOHB is variation of Hyperband that considers the hyperparameter representation to sample candidates.
We conclude, that the additional use of an explicit learning curve representation might not lead to strong improvement in every scenario, however, also does not seem deteriorate it. Furthermore, there are cases where the explicit consideration leads to significantly better results.
LIMITATIONS OF OUR PAPER
Although DYHPO shows a convincing and statistically significant reduction of the HPO time on diverse Deep I Learning (DL.) experiments, we cautiously characterized our method only as a "step towards" scaling HPO for DI. The reason for our restrain is the lack of tabular benchmarks for HPO on very large deep learning models, such as Transformers-bast We hope our promising results will motivate the DL community, that own compute power, to invest in creating pre-computed tabular HPO benchmarks for very deep models, where HPO researchers can demonstrate the empirical performance of their proposed methods.
architectures (Devlin et al. 2019).
8 CONCLUSIONS
In this work, we present DYHPO, new Bayesian optimization (BO) algorithm for the gray-box setting. We introduced a new surrogate model for BO that uses a learnable deep kernel and takes the learning curve as explicit input. Furthermore, we motivated variation of expected improvement for the multi-fidelity setting. Finally, we compared our approach on diverse benchmarks on a total of 50 different tasks against the current state-of-the methods on gray-box hyperparameter opti- mization (HPO). Our method shows significant gains and has the potential to become the de facto standard for HPO in Deep Learning.

---

Under review as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Double-blind.
ETHICS STATEMENT
In our work, we use only publicly available data with no privacy concerns. Furthermore, our algo- rithm reduces the overall time for fitting deep networks, therefore, saving computational resources and yielding a positive impact on the environment. Moreover, our method can help smaller research organizations with limited access to resources be competitive in the deep learning domain, which reduces the investment costs on hardware. Although our method significantly reduces the time taken for optimizing machine learning algorithm that achieves peak performance, We warn against ning our method for an extended time only to achieve marginal gains in performance, unless is mission-criticee Last but least, in order save energy, we invite the community to create sparse benchmarks with surrogates, instead of dense tabular ones.
REPRODUCIBILITY STATEMENT
We attempt to facilitate reproduction of our results with the following measures:
We use only publicly available datasets and provide detailed description preprocessing (Section A.2) and the datasets themselves (Section A.I).
All our baselines are publicly available or trivial to implement. We provide all required details in Section A.5
We clearly describe our method in Section and provide additional details in Section A.4
Finally, we plan to make the source code of our method publicly available.
REFERENCES
Noor H. Awad, Necratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 1021, pp. 2147-2155 2021.
doi: 10. 000224963ijccai2221/999 URL httpp::://do..oor org/
10. 10.24963.1 cai 2ttp:63331jcca.22222222222 2021/296
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012. URL http::/cll act.://lllcmmmcccoocm www.eeo.rggggitttttoeoeoo cfm?id
2188395
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Sys- tems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pp. 2546
2554, 2011. URL https: ttttps:://rocceedingg neur www.rip..rr wtw.pppp....ppeeee wttpp/papee/221/aaassts Abstract atttaaala.b...0000aaaaa.........
.html
Hadrien Bertrand, Roberto Ardon, Matthieu Perrot, and Isabelle Bloch.
Hyperparameter opti- mization of deep neural networks: Combining hyperband with bayesian model selection. In Conférence sur I' l'Apprentissage Automatique, 2017.
Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy Lillicrap, Matthew Botvinick, and Nando de Freitas. Learning learn without gradient descent by gradient descent. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, pp. 748-756, 2017. URL MKKLittpp://rroceddin
m1 lr. wwwps/vws/////pptttppppppppwwwtt chen 7e. html
Janez Demsar. Statistical comparisons of classifiers over multiple data sets. Mach. Learn. Res., 7:1-30, 2006. URL ottp:////a//////loooooooo/ooo/// html
10

---

Under review as a conference paper at ICLR 2022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HLI 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171 2019. doi: 10. 00.18853/111111914225 httlpttt::: //op::://d.. .org/
1.0.8653/v1/nm 1423
Xuanyi Dong and Yi Yang. Nas-hench-201: Extending the scope of reproducible neural architec- ture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL HJxyžkBKDI
net fortm/orrm?d
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter opti- mization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1436 1445, 2018. URL TtlE:::/pprocedinggs.oo
html
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Leurning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1165-1173, ngs. mll.prres. / /v70 francesch i 77a html
2017.
Jacob Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew Gordon Wil- son. Gpytorch: Blackbox matrix-matrix gaussian process inference with GPU acceleration. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7587-7597, 2018. URL httt::::/procceeeeeee hew.eediiinnnneeuiusseeee CC/ hcc/pappr//018//eee -ww.all-bbstrrca html
Kevin G. Jamicson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, pp. 240-248, 2016. URL http://
proceedings mml. www.mm.r...s.1111111 mrs..ppress/r5//aaleeeooo htmp://1//ammmnnnnm.mmm 6, html
Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of ex- pensive black-box functions. J. Global Optimization, 13(4):455-49,, 1998. doi: 10.1023/A: 1008306431147. URL /ttpp:://do...ott /10 1023/:: 1083306411777
Kirthevasan Kandasamy, Gautam Dasarathy, Junier B. Oliva, Jeff G. Schneider, and Barnabás Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. In Ad- vances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016. December 5-10. 2016. Barcelona, Spain, pp. 992-1000, 2016. URL httpss://ppcccccigg.....e
http:aawwpp..cppp/////..... cc/paperr/216//aas/ 1B53F76646671aa22222220277776223 - html
Kirthevasan Kandasamy, Gautam Dasarathy, Jeff G. Schneider. and Barnabás Póczos. Multi-fidelity bayesian optimisation with continuous approximations. In Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 20/7, Sydney, NSW, Australia, 6-11 August 2017, mlr mwr.mress. / v70 100///kndaaamy// 17
pp. 1199-18998 2017. URL
html
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnaáás Póczos, and Eric P. Xing. Neural architecture search with bayesian optimisation and optimal transport. In Ad- vances in Neural Information Processing Systems 31: Annual Conference
Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 2020 2029, 2018. URL https: https::/prooeediags..o new.eidiidiggs.nnnisssss......s ce htte/aaprr/200 haaprr22188hash.. F33bal fa5c 0e8 ( -bt.taaa-Abbttaeee html
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
6980

---

Under review as a conference paper at ICLR 2022
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian opti-
mization of machine learning hyperparameters on large datasets. In Proceedings of the 20th In- ternational Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, mam.rrr... presss/
Fort Laudendale, FI, USA, pp. 528-536, 2017. v54/klein 7a. 7t..mmm
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18: httll.
185:1-185:52, 2017. URL http:// jmlr http.//o///ppppee//////---------
Shibo Li, Wei Xing, Robert M. Kirby, and Shandian Zhe. Multi-fidelitty bayesian optimization via deep neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 60eldeb043af3 wwwpd//bceeeeeeeeeeec Abstracttbbtttttttt. html
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In The 23rd International Conference on Artificial Intelligence and Statis- fics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. pp. 1540-1552, 2020.
URL http:://proceedinni/
mir wtt..ppres//11//orrrraaaaao
html
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- hyprrparameter opti- mization through reversible learning. In Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 2113-2122, 2015. URL
http::/prroceedingg mlr.. press/v37
http//maaaaaannnn.mmmm ppess//33/maclaarinnn// html
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using a thousand optimization tasks learn hyperparameter search strategies. CoRR, abs/2002. 11887, 2020. URL htt.::::///artt.. worg.iiwiiio..oooooooooooooooooe abs 2 2002. 11000021118777
Jack Parker-Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient online hyperparameter optimization with population-banl bandits. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, De- cember 6- 12, 2020, virtual, 2020. URL Absttaa www..pp..c../pppp............e
Valerio Perrone, Rodolphe Jenatton, Mathhis W. Seeger, and Cédric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewriPS 2018, December 3-8, 2018, het.rgss.erriii nwwrinurrp.... cc/ .ttl
Montréal, Canada, pp. 6846 6856, 2018. URL https http:::://oocceeeeeeeeeeoe /ttp://ttts///proeeeeeeeett
paper 2213://hae2227711aaaaaaa00110000
Matthias Poloczek, Jialei Wang, and Peter I. Frazier. Multi-information source optimization. In Advances Neural Information Processing Systems 30: Annual Conference on Neu-
ral Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA, pp. 4288 4298, 2017. URL https: df1 f1 12000 6 004411155111 79584 6 Awwttaaaabbbtttttaaaaaaaa htm]
/ttLlttttps::eeeeeeeee http:///pcceedeeeeeeeeee neur ips. hcc/paper/201////
Akshara Rai, Ruta Desai, and Siddharth Goyal. Bayesian opayeiiaanootimmizatiion with a neural network kernel, 2016. URL TRL/htp::/ //www. cs cmu. www.eduu/ueeeeee wwtp://utad/Eeeee//eeee. BwA_B/OO__N_______
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning- Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X.
Jasper Snock, Hugo Larochelle, and Ryan Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems
25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceed- ings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 2968,
12

---

Under review as a conference paper at ICLR 2022
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference Machine Learning (ICML-IO), June 21-24, 2010, Haifa, Israel, pp.
1015-1022, 2010. bttpttss:: cml wtte///ooeeee/////eeeee 201 0/ paperss 422. pdf
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Multi-task bayesian opti- mization. In Advances in Neural Information Processing Systems 26: 27th An-
nual Conference on Neural Information Processing Systems 2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2004 2012, 2013.
ps. cc httpppc...pppppeeeepepp...pppppp pttppap///2213/aahh.. E5S5aaJeccaacc007777188822222662 Abst.bbaat.bb-tt.ttttttt...aaaa. html
URL https: http:s////ppooceengg..sssse
Kevin Swersky. Jasper Snoek, and Ryan Presoott Adams, Freeze-thaw bayesian optimization. CoRR, abs/1406.3886, 2014. URL http://arxiv..rgg/aaaap.
3896
Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, and Masayuki Karasuyama. Multi- Multi-fidelity bayesian optimization with max-value entropy search Proceedings of the 37th International Conference on Machine Learn- ing, ICML, 2020, 13-18 July 2020, Virtual Event, pp. 9334-9345, 2020. URL. http:// proceedings mll..rr....... wwwss//l1l/taeeeeeeeeeeee
and its parallclization.
Jiazhuo Wang, Jason Xu, and Xuejun Wang. Combination of hyperband and bayesian optimization for hyperparamettr optimization in deep learning. CoRR, abs/1801.01566. 2018. (thhtp:
//aaxxv..rg/aabbb...e 1001 801 01596
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, pp. 370-378, 2016. URL http: html
//proceedings .mw...g...r..................... htt.://ewiaaaaai.pppee/e///////e
Martin Wistuba. Bayesian optimization combined with incremental evaluation for neural network architecture optimization. In @ PWEDHE/GDDDDDECCEEEEDC 2017.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. URL https: http://p/.pppppeeeeeeeeeeeeeeee/
Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch: Multi-fidelity metalearning for efficient and robust autodl. IEEE Trans. Pattern Anal. Mach. Intell,, 43(9):3079 3090, 2021. doi: 10. 10.110//AAM..221130077777
URL sttps:///oi.orq.// 111.00.11111PPMM1111111 2021.
3067763
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https:/ opw.review.e.cee httpe/viii.eetef.oreeeeeeeee net/forum? 1d=rIUe8Hcxq
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL SETUP
A.1 BENCHMARKS
LCBench. LCBenet feedforward neural network benchmark on tabular data which consists of 2000 configuration settings for each of the 35 datasets. The configurations were evaluated during HPO runs with AutoPyTorch. LCBench features search space of 7 numerical hyperparameters, where every hyperparameter configuration is trained for 50 epochs.
TaskSet. TaskSe is benchmark that features over 1162 diverse tasks from different domains and includes search spaces. In this work, we focus on NLP tasks and we use the Adam8p search space with continuous hyperparameters. We refer to Figure the exact task names considered in our experiments. The learning curves provided in TaskSet report scores after every 200 iterations. We refer to those as "steps" in Figure
NAS-Bench-201. NAS-Bench-20111 benchmark consisting of 15625 hyperparameter config- urations representing different architectures on the CIFAR-10, CIFAR- 100 and ImageNet datasets. NAS-Bench-201 features search space of 6 categorical hyperparameters and each architecture is trained for 200 epochs.
A.2 PREPROCESSING
In the following, we describe the preprocessing applied to the hyperparameter representation. For LCBench, we apply log-transform to batch size, learning rate, and weight decay, For TaskSet, apply it on the learning rate, L1 and L2 regularization terms, linear and exponential decay of the learning rate. All continuous hyperparameters are sealed to the range between 0 and using sklearn's MinMaxScaler. If mentioned otherwise, we one-hot encoding for the categorical hyperparameters. As detailed in the following, some baselines have a specific way of dealing with them. In that case, we use the method recommended by the authors.
A.3 FRAMEWORK
The framework contains the evaluated hyperparameters and their corresponding validation curves. The list of candidate hyperparameters is passed to the baseline-specifie interface, which in turn, optimizes and queries the framework for the hyperparameter configuration that maximizes utility. Our framework in turn responds with the validation curve and the cost of the evaluation. In case hyperparameter configuration has been evaluated previously up to a budget b and a baseline requires the response for budget 6 the cost is calculated accordingly only for the extra budget requested.
A.4 IMPLEMENTATION DETAILS
We implement the Deep Kernel Gaussian Process using GPyTorch 1.5 (Gardner et al. 2018). We use an RBF kernel and the dense layers of the transformation function & (Figure 2 have 128 and 256 units. We used convolutional layer with kernel size three and four filters. All parameters of the Deep Kernel are estimated by maximizing the marginal likelihood. We achieve this by using gradient ascent and Adam (Kingma & Ba 2015) with a learning rate of 0.1 and batch size of 64. We stop training as soon as the training likelihood not improving for 10 epochs in row or we completed 1,000 epochs. For every new data point, we start training the GP with its old parameters reduce the required effort for training.
set
14

---

Under review as a conference paper at ICLR 2022
A.5 BASELINES
Random Search & Hyperband. Random search and Hyperband sample hyperparameter config- urations at random and therefore the preprocessing is irrelevant. We have implemented both from scratch and use the recommended hyperparameters for Hyperband, i.e. y 3.
BOHB. For our experiments with BOHB, version 0.7.4 of the ffficially-reeease code
For experiments with DEHB, use the official public implementation We devel- oped an interface that communicates between our framework and DEHB. In addition to the initial preprocessing common for all methods, encode categorical hyperparameters with numerical value in the interval [0, II. For a categorical hyperparameter X,, we take N cqual-sized intervals, where N; represents the number of unique categorical values for hyperparameter x, and assign the value for categorical value n E N; to the middle of the interval [i, + 1] suggested by the authors. For configuring the DEHB algorithm we used the default values from the library.
DEHB.
Dragonfly. We use the publicly available code of Dragonfly No special treatment of categorical hyperparameters is required since Dragonfly has its own way to deal with them. We use version 0.1.6 with default settings.
B ADDITIONAL PLOTS
In this section, we provide additional plots for the performance comparison between all methods for the individual datasets in our benchmarks. In Figure 12 and 13 we show the performance comparison for all the datasets from LCBench regarding regret over the number of epochs. As can be seen, DYHPO manages to outperform the other competitors in the majority of the datasets, and in the datasets that it does not, it is always close to the nop-performing method and the difference between methods is marginal.
cifar10
cifar100
Regret
10"
Regret
10
10
DyHPO
DyHPO wid LC
DyHPO
DyHPO w/o LC
10
102
10
10"
10
10
10"
Training Time in Seconds
Training Time in Seconds
Figure 9: The learning curve as an explicit input is very important for each task of NAS-Bench 201.
Additionally, in Figure 10 we show the performance comparison over time of every method for the CIFAR-10 and CIFAR-TO0 datasets in the NAS-Bench-201 benchmark. As be seen, DYHPO converges faster and has a better performance compared to the other methods over the majority of the time, however, towards the end although is the optimal method or close the optimal method, the difference in regret is not significant anymore.
Furthermore, Figure 11 shows the performance comparison for the datasets chosen from TaskSet over the number of steps. Looking at the results, DYHPO is outperforming all methods convinc- ingly on the majority of datasets by converging faster and with significant differences in the regret evaluation metric.
15

---

Under review as a conference paper at ICLR 2022
offar10
dfar100
10"
Regret
Regret
80"
10
10
10"
10"
10
10"
10"
10"
Wallsbok Time Seconds
Wallolock Time Seconds
Random
BOHB DEHB
Dragorilly DyliPO
Rendom
BOHB
Deagonfly DyfIPO
Hyperband
Hyperband
Figure 10: NAS- MAS--ench-221 Regret Results.
Food NNNNNNG
indo pasch128 LSTM128 wvg bx64
inob patch112 LSTM126 5984
indb paich128 LSTMM28 embed128 ba64
Rugeat
10"
200
405
650
800
1003
200
<in
edo
f00
1003
0
200
<0c
clc
803
1003
Nambbe Sleps
Number
Staps
Number of Sleps
Randym
BOMB
Drigorly DysPOo
Random
BCHB SEHB
Dragorly DyNiFO
Random Hyperonnd
BOHB
Drigorty ByliFo
Hypeetard
DEHB
Hypertont
DENB
FaedToodRAANCaass Fxcaton mith petrndo GRU 28 hel28
NCCCCas Foadon
Food NCCaso Fiction Inch paictc2 8th8656 pwi,, avg. bet
Indb pwizh30 CaRUU meg, te
Regiut
1000
1003
250
Number
Number
Number Stops
Random hyperbard
BOMB
Dregarly
DEHB
DyHPO
DyHPO
DyHPO
FisecTlecl floviion
NoNNeee fradon
NNCeees fowioon Imeb patehd2 LSTM b5126
Incb patchae IRN roti 12126
Incb patch32 LSTM128 E bs125
10"
Rogeet
200
402
stc
803
1000
200
420
60c
600
1002
220
420
600
800
1002
Number
Stupa
Numbar
Shaps
Number of Stupa
Raadom Hisperband
Drigorty Dyniro
Itandom hyspertand
Drigority
Hardom Hypertand
Drepirty pyrro
DE10
cerd
NieeteedrWNGGss fhar
Floed Nolles focttoo mith paten32 VRNNS5 miu avg. be
Flaed ai ication
Indb, palena2, VRNN118 Jann bel 20
indo paocndiz VRNN664 Sanh weg. ban128
Raggent
Reguut
10"
800
10bo
ado
1053
200
Sleps
Steps
Steps
Random
BOHE
Dragorty Dyhо
Random Hyperbord
BOHB
Drigorty DyHPO
Drogorty DyHP
Hyperbood
Hypetard
Figure 11: Detailed results on per dataset level for TaskSet.
Lastly, in Figure 9 we ablate the learning curve input in our kernel, to see the effect it has on perfor- mance for the CIFAR-10 10 and CIFAR-100 100 datasets. The results indicate that the learning curve plays an important role in achieving better results by allowing faster convergence and a better anytime performance.
16

---

Under review as a conference paper at ICLR 2022
Arezzn enpoys
Audradio
13"
%%
250
600
1883
1000
Wadldock
Wallood Socords
Viall Time
Randor
pO i
Oogariy
Tandan
2010
Deggetty
Rardow
0010
Chigarty
KOCCup22 apelery
Reggit
1000
20th
4000
8900
199
2003
20:0
900
1951
1899
200
3000
1000
1900
Hoportand
Do16
Do10
0G-0
CYtPo
Addll
affisss
abort
Rogee
Rogget
3301
1250
Whuldbcc Time in Setunbh
Pldicach Irn in Secards
Nuuldcc Tind Sccon
Rindor
Rasday
MCHI
Ronday
300
1000
701
ein
100
Wolldvi Inde
Timss
Timw
Kondir
Ricder Dipetten
WOM
Kinder Iypertend
KD 0010
0010
oyeend
chrltee
maw
comadd
1908
70o
1940
1750
50
rdo
100
z/x
150
zbo
Watldock
Soconds
Well Back
Timee
Hig
Dy
Dytno
Sytro
conetype
medig
ebeb
Walldock Tiis Sacords
Weddbbos Time Secorde
Walldock Time in Seconde
Render
BOH6
201B
Mardor
RONB
Figure 12: Detailed results on a per dataset level for LCBench.
17

---

Under review as a conference paper at ICLR 2022
Nobd
Aayul
300
1000
2.90
300
XX
350
Wattoook Seoonds
Wilkook Time
Walcoos Timois boceres
Randor
D0:0
Oogariy
Bardin
9010
Drsgarty
Tanddan
2010
Degarll
Jento chow 2016 ondgane
900
1000
Ldle
2009
2510
3903
205
60
1846
1993
10l
50
180
2000
Hipertand
Do10
Sy-00
0010
0010
Jon
mfaai fydior
Reggi
120
Wallduc Time Seoondh
Phulcook Tru Second
Phuldock Time Sccon
Rinday
0016
Flandor
5016
12g
2000
133
200
2803
2001
41
120
Woldivk Tims Seovle
Kindir
ROM
Ricder
RDM
Kinder Ioputan
0010
0016
0010
Dytro
Apcroei
syire
190
sic
1004
41
800
1954
1204
Syeea
120-6
Syeen
12"
Walddoo Time Seconds
Wailibed Time Seconds
Nander
Bove
Figure 13: Detailed results on a per dataset level for LCBench (cont.)
18

---

Under review as a conference paper at ICLR 2022
Avergg Number fram Potthee
LCBendh
Averge Mam Number of Trum Potthee
Texicst
Aueenge Nu Number of True Pootfure
VAS-Bench-201
0.15
0.4
0.3
0.3
02
01
0.08-
20
30
40
20
30
40
100
150
200
Epochs
Random
60HB DC18
Diapority Dytaro
Randam Pypertand
BOHB DCPE
Drugontyy DytPo
Rarstmm Pypertand
aor DEMB
Emegorty bylino
Pypertend
Figure 14: DYHPO efficiently selects top-performing candidates and keeps training them, avoiding training poor confi cofiiuraations for long time.
LBBench
Theoset
NAS NAS-Bench-200
0:30
1:12-
0:15
Anstaga Rugget
025
Avssaga Rageat
0.10
8.08-
00.10
0.05
16
2h
sb
db
d2
1b
8b
Ab
do
160
160
zoo
Number of Eprocts
Number of Eppacts
Number of Eprace
Random Hyperbend
BOHE DEHB
Drogarty DyHPO
Rondom Hypertend
BOHB DEHB
Drogarty DyHPO
Random Hypertand
80HB DEHB
Drogerty DyHPO
Figure 15: DYHPO spends most its budget on hop-performing candidates.
C ON THE EFFECTIVENESS OF DYHPO
DyHPO effectively explores the search space and identifies promising candidates. This is visualized in Figure [14jn in which we plot the precision of each method for different considered budgets. The precision at an epoch i is defined as the number of top 1% candidates trained for at least i epochs divided by the number of all candidates trained for at least i epochs. The higher the precision, the less irrelevant candidates were considered and the less computational resources were wasted. For small budgets, the precision low since DYHPO spends budget to consider some candidates but then promising candidates are successfully identified and the precision quickly increases. For LCBench and Taskset, all other methods dedicate significantly more resources to irrelevant candidates which explains why DyHPO finds good candidates faster. For NAS-Bench Bench-201, DEHB can match the precision but only at a later stage. Simply put, the baselines select much more "poor" configurations (i.e. outside the top 1 1% performers) compared to our method DyHPO.
This argument is further supported by Figure 15 where we visualize the average regret of all the candidates trained for least the specified number of epochs the x-axis. In contrast to the regret plots in Section 6 here we do not show the regret of the best configuration, but the mean regret of all the selected configurations. The analysis deduces similar finding as in Figure [4] above. Our method DYHPO selects highly more qualitative hyperparameter configurations than all the baselines.
D PROMOTION OF POOR PERFORMING CANDIDATES
An interesting property of multi-fidelity HPO is the phenomenon of poor rank correlations among the validation performance of candidates at different budgets. In other words, a configuration that achieves poor performance at a small budget, might perform strongly instance, a well-regularized neural network will converge slower than an un-regularized network in the early optimization epochs, but eventually performs better when converged. We analyze this phenomenon and report the respective results in Figure [6] In this experiment we measure the percentage of "good" configurations at a particular budget, that were "bad" performers in at least one of the smaller budgets. We define "good" performance at budget B, when configuration achieves a validation accuracy ranked among the top I/3 compared to the validation accuracies of all the other configurations that were until that budget B. Similarly "bad" performance
larger budget. For
19

---

Under review as a conference paper at ICLR 2022
budget B represents configuration whose validation aceuracy belongs to the bottom 2/3 of all configurations run at that budget B.
LCBench
Toskeet
VAS Bench 201
Tutomer 2yoteoooe
Fraaton of Post
Fruston of Paat
0.c-
Puttomal
Fruction
0.
20
02
20
30
40
100
200
Number
of Epodas
Number of Siepp
Namber of Epoda:
Random Hypetaand
DENS
DyRPo Baselins
Randon Hyoebandd BOH3
DENO
DyFPP Conciiie
Random Hypebannd
DEND
DynPP
Drugonily
Eragortly
Drsponty
Cusclino
BOHS
20H3
Figure 16: Percentage of configuration belonging the top 1/3 configurations given budget, and ii) that were in the bottom 2/3 of configurations at one of the previous budgets. Here the budget represented by the number of steps or epochs.
In Figure 16 analyze the percentage of "good" configurations at each budget denoted by the x- axis, that were "bad" performers in at least one of the lower budgets. Such metric is proxy for the degree of the promotion of "bad" configurations towards higher budgets. We present the analysis for all the competing methods of our experimental protocol from Section [6] We have additionally included the ground-truth line annotated "Baseline", which represents the fraction of past poor performers among all the feasible configurations in the search space. In contrast, the respective methods compute the fraction of promotions only among the configurations that those methods have considered (i.e. selected within their HPO trials) until the budget indicated by the x-axis. We see that in all the search spaces I.Bench, TaskSet and NASBench-201 there high degree of "good" configurations that were "bad" at previous budget, with fractions of the ground-truth "Baseline" varying from ca. 40% LCBench, up to ca. 70% in the NASBench-20 datasets.
On the other hand, the analysis demonstrates that method DYHPO has promoted more "good" configurations that were "bad" a lower budget, compared to all the rival methods. In particular, ca. 80% of selected configurations at the datasets from the LCBench benchmark were "bad" performers at a lower budget, while the case of NASBench 201 this fraction approaches the level of 95%.
20


Reviews:
[{"review_id": "quoyrfXUwxd", "reviewer": "Reviewer_oS3h", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Multi-fidelity is very important to obtain practical HPO algorithms for deep learning. \n- The proposed deep kernel accounts for the correlation between learning curves to avoid naively stopping trials to early like HB does.\n- The modification proposed to account for the learning curve is fairly simple\n- The experiments are quite convincing, with DyHPO outperforming other HPO algorithms almost systematically. The baselines are good, with Hyperband, BOHB and DEHB being serious multi-fidelity contenders.\n\nWeaknesses:\n- It is very unclear to me how we can guarantee that the algorithm will often resample x to make it continue. If most dimensions of the search space are real, it seems to me x will most likely be different than previous ones, leading the algorithm to never continue trials. The empirical results clearly show that this is not the case, however the explanations do not make clear why it would not be the case.\n- Only figure 8 reports training time in seconds (I assume including HPO time to suggest new trials as well). With the frequent query on the algorithm (every 1 epoch) I assume there must be a significant amount of overhead, starting and stopping trials very often. Also the algorithm itself must be fairly slow compared to hyperband, with the deep kernel that requires training. It would be best to report more results on the running time of DyHPO.\n- There are no clear experiments with learning curves that are best later on to synthetically show that DyHPO performs well in this case. This would be extremely valuable to support strongly that the reason why DyHPO works so well is that it indeed let the best trials train even though they do not perform well at the beginning.", "summary_of_the_paper": "This paper present a new Bayesian Optimization algorithm that integrates a Deep Kernel over both the hyperparameters x and the fidelity budget j (typically number of epochs). It also present a slightly modified version of the expected improvement acquisition function to for the fidelity budget. The paper shows on several benchmarks that the proposed algorithm, called DyHPO is highly competitive, better performing than BOHB and DEHB on most of the benchmarks, otherwise performing similarly.  ", "summary_of_the_review": "The algorithm presented in this paper is an appreciable improvement to multi-fidelity variants of Bayesian Optimization, especially because it accounts for the correlation between the learning curves and avoid relying to strongly on low fidelity to make hard decisions on trials to stop or continue training. The experiments are convincing, they are broad and compare good baselines. The paper lacks important details in my opinion with respect to the optimization of the expected improvement and how it ensures a good fraction of the trials continue training. It also lacks analysis of the execution time of the algorithm and more explicit experiments showing it can avoid stopping good trials that progress slowly in the first epochs. I consider the work good enough for publication, but it could benefit from some clarifications and additional analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "n9PJ7Cu9hKG", "reviewer": "Reviewer_j16y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Post rebuttal\n\nGiven the detailed rebuttals by the authors, the updated baselines, I'm confident about increasing my rating for this paper. It might aid their arguments, if the authors were to move some sections of the appendix to the main paper. \n\n---------------------------------------------------------------------------------------------------------------------------------------\n## Pros: \n\nThe paper is quite clear (see the subsequent comments), and easy to follow. The method proposed is simple, and is an intuitive extension to the methods in the literature. The paper is also well placed in the context of previous methods. The experiments section is quite strong in the experiments and baselines covered. \n\n\n## Cons:\n\n\n1. The authors start with the motivation that the rank correlation of performance at various budgets is poor. However, this is seemingly contradicted in Fig 7 left where the exclusion of the learning curve in HPO doesn’t worsen the performance much over all the datasets. The authors show experiments where the inclusion of LC leads to better results for some datasets. The authors should either provide references for the poor correlation, or show statistics of how intermediate performance is a poor predictor of final rank. \n\n2. On training the deep convolutional kernel: The neural net used is a MLP with 128 and 256 hidden units. This is quite a large network. How do the authors reliably train this network with only a few (xi, j, Yi, j-1) tuples? In the initial phases, how reliable are the networks predictions to terminate a run? How are the hyperparameters of this training chosen? The authors should give details, and report ablations on network size, architecture, training parameters (lr, batch size etc). In the absence of these, it is hard to judge the merits of the proposed \\phi, as I do not understand how these specifics were arrived at. Also, the authors should report how the additional time of training the deep kernel changes the wall clock time measurements.\n\n3. Evaluations: The use of Epochs and Steps to describe the x-axis in various plots is a little confusing. Are these the same? Also, it is ideal that the authors include the true performance (say test accuracy on CIFAR/ImageNet exps), and the true wall-clock times somewhere in the paper, in addition to the regret plots presented. Also the proposed method’s rank fluctuates quite a bit in the initial steps in Fig 3 and 5; can the authors comment on this?\n4. Minor: \n“Reversing the training update steps” in the Intro para 1 makes it sound like undoing the update steps. \nThe authors might consider rewording Para 3 of Motivation to aid readability, as it took me a few reads to grasp the point. \nThe authors say “gradient descent and Adam” on Page 5 last para, and “gradient ascent and Adam” in A.4. \n\n5. Additional comments: These are general comments the authors might consider discussing. \nDo the authors find that the trained deep conv kernel is transferable across tasks? This might have interesting implications, if it can be. \nThe authors write at the end of Section 6 “the additional use of an explicit learning curve representation might not lead to a strong improvement in every scenario”. While experimental evidence has been provided, can the authors describe what factors determine if incorporating LC dynamics leads to better HPO?\n", "summary_of_the_paper": "The paper presents an extension to the commonly used Gaussian Process based HPO by incorporating the learning curve dynamics to decide the next HP configuration to be tried out. For this, the authors propose the use of a kernel that encodes the previous HP iterates using a neural network. The method is shown to reach lower regret values for the same computational budget compared to the baselines considered. \n", "summary_of_the_review": "The presented work is interesting, barring a few points commented on above. The motivation for this work needs further clarification from the authors. The experimental evidence of the efficacy of the method is strong. However, the paper in its current form misses some important details, and ablations. If the authors can address these points adequately in their rebuttal, I’d be quite happy to raise my score. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ry7WjHog53S", "reviewer": "Reviewer_RUAE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A potential strength of this paper is the proposal of a novel surrogate model for learning-curve data which, despite involving a neural network, seems to be operational on just the data observed during a single HPO experiment. There is a lot of prior work proposing learning-curve surrogates (see below), some cited here, but most are either quite simple (multi-task GP) or require \"warmstarting\" on data from previous HPO experiments. Having said that, I could not find any mentioning of this point, and I am really curious about the authors explaining how their \"deep GP\" model can be trained just on the very limited data observed during a single HPO experiment. For an expensive tuning problem, you probably have 20-40 configurations, most of which do not run for many epochs. And even if some configurations run for many epochs, learning curve data is exceedingly noisy.\n\nThe details really matter here. With a standard BO surrogate, I just need to refit the GP hyperparameters now and then, which can easily be done even for little data. With DyHPO, you need to presumably update a deep kernel, i.e. re-train a neural network. The paper does not say how this is done in a fully automated fashion. Is training started from scratch, or from the last recent weights? The first is expensive, while the latter is prone to get stuck at the previous solution and ignores the new data. How long do you re-train? Do you re-train after getting each new observation? While using complex (deep neural) surrogate models in BO is an obvious idea, many previous trials have failed, because complex NN models are just not easy/fast to update as part of sequential decision making, and in any case cannot be fit robustly to very small datasets. This fact has been clearly spelled out, for example in [7] for the closely related problem of bandit optimization. I'd be personally really surprised if this work was different and solved these difficult issues, but would be willing to give benefit of doubt, if a lot more information was provided here how the authors pulled it off. As it stands, the authors do not even mention there could be issues here.\n\nThe most obvious weakness of this paper is that very relevant prior work is ignored, namely on asynchronous multi-fidelity. Most prominently, ASHA [1] is well known and implemented in Ray Tune [3] or AutoGluon [4]. The baselines compared to against in this paper (Hyperband, BOHB) are all synchronous (and quite dated by now), meaning that many trials need to run to a certain level until another decision is taken. If you force methods to be synchronous, this puts them at a disadvantage. They need to delay decisions until some rung is completely filled, which delays decisions and slows them down. This is explained in the ASHA paper [1]. It is well known that for large scale multi-fidelity HPO, asynchronous scheduling works much better than synchronous, see for example the comparisons in [2]. Algorithms like ASHA are behind commercial automated tuning services [5]. It is quite astonishing that part of the research community is still considering synchronous methods like Hyperband or BOHB the state of the art.\n\nFor example, the paper claims that it is a new idea that DyHPO \"never discards a configuration\". That is precisely what ASHA [1] does as well (known as pause-and-resume scheduling), and what Freeze-Thaw BO suggested long ago. I'd be surprised if a well configured ASHA method (available in Ray Tune [3]) would not be competitive or beat the approach suggested here, despite not requiring a complex surrogate. When doing such comparisons, it is important to also take decision time into account, because updating a surrogate model can be very expensive. In DyHPO, this likely means re-running MLP fitting, which is probably really expensive.\n\nI also find the motivation as to why DyHPO works better than previous methods unconvincing. The authors claim that rank correlations between early evaluations (at few epochs) and late ones are poor. In my experience, this is just not the case, these correlations are in the majority pretty good, which is exactly why multi-fidelity methods work very well for DNN tuning. Sure, there are examples such as regularization, but the question is whether that matters. The authors should provide numerical evidence for such a claim. Now, even if these correlations are poor, it is not clear to me why DyHPO could do anything about that. Learning curve prediction is just hard, because by far most of the data is from early evaluations, but you are interested in late performance, so you need to extrapolate. Why would some vanilla deep kernel be good at that? The only way to really know about certain anti-correlations that can be exploited, is to either fit models to data from past HPO experiments (which DyHPO does not do), or to built the knowledge into the model (which they don't do either). Just because an NN is involved, does not mean it will do magic for you. The reason why DyHPO works better than competitors here is that it is asynchronous, but the others are synchronous, so at a disadvantage. Also, the reason why model-based HPO is better than random search based methods (like Hyperband) is mostly because the latter cannot exploit: they need to draw new configs always at random.\n\nThe experiments are pretty underwhelming. Apart from most relevant baselines missing (DyHPO is asynchronous, all competitors are synchronous), the curves are also not very meaningful, because the x axis is number of epochs instead of wall-clock time. DyHPO needs to update a complex surrogate model, including retraining a neural network, and the costs for doing that have to be taken into account. All experiments are also sequential, no parallel evaluations are used (this could easily be done by using Ray Tune [3]), again this falls far short of the current state of the art in automatic tuning of large neural models (e.g., methods like ASHA or PBT).\n\nFinally, there are quite some works on using complex surrogates to model learning curves in the context of HPO, for example [5]. It is not clear why this was not compared against, as code is available. The work of Wistuba and Grabocka is cited, which proposed deep kernel surrogates before (so this paper, against their claim, is not the first to do this in the context of multi-fidelity), and in fact Perrone etal (2018), cited here, did this even earlier, just not in the context of learning curve data. The paper of Wistuba is quite careful in explaining why a complex surrogate cannot be trained robustly on the data from a single experiment, and proposes an algorithm to warmstart from past data. It is dismissed here (as competitor) for doing so, but as I said above, I am not sure how DyHPO solves the apparent issue that complex NNs cannot be trained on the small amount of data observed in HPO.\n\n[1] ASHA: https://arxiv.org/abs/1810.05934\n[2] MOBSTER: https://arxiv.org/abs/2003.10865\n[3] Ray Tune: https://docs.ray.io/en/latest/tune/index.html\n[4] AutoGluon: https://auto.gluon.ai/stable/index.html\n[5] https://www.determined.ai/\n[6] https://openreview.net/forum?id=S11KBYclx\n[7] https://arxiv.org/abs/1802.09127\n", "summary_of_the_paper": "This paper is concerned with multi-fidelity HPO (the authors call this \"grey-box\", which is a non-standard term). They propose a surrogate model for learning curve data (e.g., metric values at each epoch) based on a deep GP kernel. Different to most previous work on synchronous multi-fidelity HPO, they decide for each running trial when it should be continued. Experiments are presented, where the method is compared to a range of synchronous HPO baselines. The experiment are fairly small-scale and do not use parallel evaluations.\n", "summary_of_the_review": "The paper may have some merits in suggesting a \"deep kernel\" surrogate model which, although quite related to previous work, is stated to work even if just fitted on the small amount of data from a single experiment, in an online fashion. However, this has been tried several times before with little success, and details explaining why the current approach should work are missing.\n\nThe proposed method uses asynchronous scheduling (much like Freeze-Thaw), but is compared against synchronous scheduling baselines, which have a major disadvantage. Comparisons to SotA methods like ASHA (or PBT) are missing, these are not cited. There is also quite a range of prior work on learning curve modeling for HPO, which is not compared against. Open source code for doing a better comparison is publicly available (for example, Ray Tune).\n\nExperiments are smallish scale, mostly on tabulated benchmarks, and again are not close to what is possible today with parallel computation. Compared to missing alternatives like ASHA, the proposed method is fairly complex and quite likely rather non-robust to handle. For example, it requires retraining a neural network model each time a bit of new data is obtained, which is very difficult to do.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H82D2oIhVMN", "reviewer": "Reviewer_cMi3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "major concerns\n\n1. There are several parts of this paper that are not clearly related to similar studies. \n    e.g. \n\t- multi-fidelity BO with deep models \n\t\t- Li+ \"Multi-Fidelity Bayesian Optimization via Deep Neural Networks\" NeurISP2020\n\t- EI for multi-fidelity BO \n\t\t- Picheny+ \"Quantile-Based Optimization of Noisy Computer Experiments with Tunable Precision\" Technometrics, 55(1):2-13\n\t\t- Lam+ \"Multifidelity Optimization using Statistical Surrogate Modeling for Non-Hierarchical Information Sources\" 56th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference. 2015. \n\n2. There is a lack of explanation about the architecture of the deep kernel (why this architecture, what makes this architecture effective for multi-fidelity optimization, etc.). It looks to be just a presentation of a kernel architecture that happens to work well.\n\n3. It is not appropriate to plot only the average riglet in the experimental results, so the variance should be plotted as well (if it is difficult to plot, it can be reported separately).\n\n4. Although the effectiveness of the multitask kernel is evaluated in ablation study, modeling using multi-task kernels in multi-fidelity optimization has already become popular and has been evaluated in various studies (e.g., https://arxiv.org/abs/1406.3896, https://arxiv.org/abs/1605.07079, https://arxiv.org/abs/1903.04703). Rather, what we should consider is what parts of the deep kernel structure are effective and why.", "summary_of_the_paper": "This paper proposes a gray-box optimization method for hyperparameter optimization of deep neural network models. In order to deal with the different budgets available for training NNs in the framework of multi-fidelity optimization, the proposed method uses a multi-task Gaussian process modeling that simultaneously measures the similarity between not only the inputs x but also the outputs y trained with different budgets. In particular, the multi-task Gaussian process model is constructed using a deep kernel with a feature extractor instead of the existing kernel function. The performance of the proposed method is evaluated by experiments on three types of neural nets: MLP, RNN, and CNN. MLP and RNN are treated as usual hyperparameter optimization in 7 and 8 dimensions, respectively, while CNN is treated as a rewrite of NAS as hyperparameter optimization.", "summary_of_the_review": "I cannot support the acceptance of this paper due to insufficient evaluation of the novelty and the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GrWNzrzpklz", "reviewer": "Reviewer_ty1k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Incorporating learning curve dynamics into the surrogate model is well motivated and supported by the ablation study on the NAS-Bench 201 dataset. \n- Extensive experimental results have been provided in terms of tabular datasets, NLP tasks, and NAS.\n\nCons:\n- Predicting learning curves is not new for HPO as it has been well explored by previous works like [1]. While the proposed method tries to involve the budget information for modeling curve dynamics, the technical novelty of this work is still somewhat limited since it seems like a direct combination between [Wilson et al., 2016] and [Kandasamy et al., 2017]. \n- The multi-fidelity acquisition function is not well supported by the ablation study. What is the comparison result between DYHPO and DYHPO w/o MF?\n- Some necessary baselines are missing in the current experiment, such as [1] and [Wilson et al., 2016]. \n\n[1] Learning Curve Prediction with Bayesian Neural Networks, ICLR’17.\n", "summary_of_the_paper": "A gray-box hyperparameter optimization framework has been developed based on a multi-fidelity acquisition function and a surrogate model incorporated with learning curve dynamics. The proposed method was built on top of deep kernel learning [Wilson et al., 2016] and multi-fidelity Bayesian optimization [Kandasamy et al., 2017]. Experimental results on three different settings were provided in optimizing hyperparameters for MLP, RNN, and CNN, respectively.", "summary_of_the_review": "Overall, the paper is easy to follow and well-motivated. While some ablated models and baselines are missing, the experimental results are comprehensive and seem to be solid. The main concern of this work is the lack of technical novelty compared with existing works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for performing Bayesian optimization for hyperparameter tuning that uses learning curve trajectories to reason about how long to train a model for (thus "grey box" optimization) and whether to continue training a model.  The reviewers seem to find the paper clear, well-motivated and the presented methodology sensible.  However, the reviews were quite mixed and leaning towards reject with 3, 6, 5, 3, 6.  A challenge for the authors is that there is already significant related literature on the subject of multi-fidelity optimization and even specific formulations for hyperparameter optimization that reason about learning curves.  A common criticism raised by the reviewers is that while there are extensive experiments, they don't seem to be the right choice of experiments to help understand the advantages of this method (e.g. epochs instead of wall-clock on the x-axis, choice of baselines, demonstration that early results are used to forecast later success, etc.).  Unfortunately, because there is significant related literature, the bar is raised somewhat in terms of empirical evidence (although theoretical evidence of the performance of this method would also help).  It seems clear that some of the reviewers are not convinced by the experiments that were presented.  Thus the recommendation is to reject the paper but encourage the authors to submit to a future venue.  It looks like the authors have gone a long way to address these concerns in their author responses.  Incorporating these new results and the reviewer feedback would go a long way to improving the paper for a future submission.
Decision: Reject