Supervising the Multi-Fidelity Race of Hyperparameter Configurations
Authors: Martin Wistuba
Arlind Kadra
Josif Grabocka
Venue: NeurIPS 2022 Accept (2022)
Keywords: hyperparameter optimization

Abstract: Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).

Supervising the Multi-Fidelity Race of Hyperparameter Configurations
Martin Wistuba*t
Arlind Kadra"
Amazon Web Services, Berlin, Germany marvistu@amazon.com
University of Freiburg, Freiburg, Germany kadraa@cs. uni -if-rieburg.de
Josif Grabocka
University of Freiburg, Freiburg, Germany grabockadcs uni uni-freiburg.de
Abstract
Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO bud- get to the hyperparameter configurations. In this work, we introduce DyHPO, Bayesian Optimization method that learns to decide which hyperparameter config- uration train further in dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budger information. We demonstrate the significant superiority of DyHPO against state-of-the-art hy- perparameter optimization methods through large-scale experiments comprising 50 datasets (Tabuulrr, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).
Introduction
Hyperparameter Optimization (HPO) is arguably acute open challenge for Deep Learning (DL), especially considering the crucial impact HPO has on achieving state-of-the-an empirical results. Unfortunately, HPO for DL is relatively under-explored field and most DL researchers still optimize their hyperparameters via obscure trial-and-error practices. On the other hand, traditional Bayesian Optimization HPO methods (Snoek al. 2012 Bergstra al. 2011 deep networks, due to the infeasibility of evaluating a large number hyperparameter configurations. In order scale HPO for DL, three main directions of research have been recently explored. (i) Online IPOO methods search for hyperparameters during the optimization process via meta-level controllers [Chen al. 2017 Parker-Holder et al.) 2220], however, this online adaptation can accommodate all hyperparameters (e.g. related to architectural changes). (ii) Gradient -basd HPO techniques. on the other hand, compute the derivative of the validation loss w.r.t. hyperparameters by reversing the training update steps [Maclaurin et al.) 2015 Franceschi et al. 2017 Lorraine et al.) 2020). however, the reversion is not directly applicable to all cases (e.g. dropout rate). The last direction, (ūi) Gray-box HPO techniques discard sub-optimal configurations after evaluating them on lower budgets [Li et al. 2017 Falkner al. 2018).
not directly applicable to
In contrast to the online and gradient-basse alternatives, gray-box approaches can be deployed in an off-the-shelf manner to all types of hyperparameters and architectures. The gray-box concept is based on the intuition that a poorly-performing hyperparameter configuration can be identified and
"quull contribution
work does not relate to position at Amazon
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

terminated by inspecting the validation loss of the first few epochs, instead of waiting for the full convergence. The most prominent gray-box algorithm is Hyperband [Li et al. 2017]. which is based on successive halving. It runs random configurations at different budgets (e.g. number of epochs) and successively halves these configurations by keeping only the top performers. Follow-up works, such as BOHB [Falkner et al., 2018] or DEHB (Awad et al. 2021]. replace the random sampling of Hyperband with a sampling based on Bayesian optimization or differentiable evolution.
Despite their great practical potential, gray-box methods suffer from a major issue. The low-budget (few epochs) performances are not always a good indicator for the full-budget (full convergence) performances. For example, a properly regularized network converges slower in the first few epochs, however, typically performs better than a non- rog-reeguari variant after the full convergence. In other words, there can be a poor rank correlation of the configurations' performances at different budgets.
We introduce DYHPO, a Bayesian Optimization (BO) approach based on Gaussian Processes (GP), that proposes a novel treatment the multi-budget (a.k.a. multi-fidelity) setup. In this perspective, we propose deep kernel GP that captures the learning dynamics. As a result, we train a kernel capable of capturing the similarity of pair of hyperparameter configurations, even if the pair's 's configurations are evaluated at differ- ent budgets. Furthermore, we extend Expected Improvement [Jones al., [998] the multi- budget case, by introducing new mechanism for the incumbent configuration a budget.
Hyperband
DyHPO
Temporal Order Accuracy
0.1
75
75
50
50
25
25
20
40 Number Epochs
Number of Epochs
Figure 1: Top: The learning curve for different hyperparameter configurations. The darker the learning curve, the later it was evaluated during the search. Bottom: The hyperparameter indices in temporal order evaluated during the opti- mization and their corresponding curves.
We illustrate the differences between our rac- ing strategy and successive halving with the experiment of Figure [] where, we showcase the HPO progress of two different methods on the "Helena" dataset from the LCBench bench-
mark (Zimmer et al., 2021]. Hyperband [Li et al.) [017] is grryy-box approach that stati- cally pre-allocates the budget for set of candidates (Hyperband bracket) according to a predefined policy. However, DYHPO dynamically adapts the allocation of budgets for configurations after every HPO step (a.k.a, a dynamic race). As a result, DYHPO invests only a small budget on configurations that show little promise as indicated by the intermediate scores.
The joint effect of modeling a GP kernel across budgets together with a dedicated acquisition function leads to DYHPO achieving a statistically significant empirical gain against state-of-the-ar gray-box baselines [Falkner et al. 2018 Awad al., [02T], including prior work on multi-budger GPs [Kandasamy et al. 2017 2020 or neural networks [Li et al., (020b]. We demonstrate the performance of DYHPO in three diverse deep learning architectures (MLP, CNN/NAS, RNN) and 50 datasets of three diverse modalities (tabular, image, natural language processing). We believe our method is a step forward toward making HPO for DL practical and feasible. Overall, our contributions can be summarized as follows:
We introduce a novel Bayesian surrogate for gray-box HPO optimization. Our novel surrogate model predicts the validation score of machine learning model based on both the hyperparameter configuration, the budget information, and the learning curve.
We derive simple yet robust way to combine this surrogate model with Bayesian opti- mization, reusing most of the existing components currently used in traditional Bayesian optimization methods.
Finally, we demonstrate the efficiency of our method for HPO and neural architecture search tasks compared to the current sfate-of-the-art methods in HPO, by outperforming seven strong HPO baselines with a statistically significant margin. As an overarching goal, we believe our method is an important step toward scaling HPO for DL.

---

2 Related Work on Gray-box HPO
Multi-Fidelitt Bayesian Optimization and Bandits. Bayesian optimization is a black-box func- tion optimization framework that has been successfully applied in optimizing hyperparameter and
neural architectures alike [Snock et al. 2012, Kandasamy et al.) 2018 Bergstra et al. 2011]. To
further improve Bayesian optimization, several works propose low-fidelity data approximations of
hyperparameter configurations by training on subset of the data (Swersky et al.] 2013 Klein et al.
2017a], or by terminating training early [Swersky et al. 2014). Additionally, several methods extend
Bayesian optimization to multi-fidelifg data by engineering new kernels suited for this problem (Swer- sky et al., 2013 2014 Poloczek et al.] [017]. Kandasamy et al. [2016] extends GP-UCX (Srinivas
ct al. [010]] to the mu multi-fielli fi-fidelityy setting by learning one Gaussian Process (GP) with a standard kernel
for each fidelity. Their later work improves upon this method by learning one GP for all fidelities that enables the use of continuous fidelities [Kandasamy et al. 2017]. The work by Takeno et al. [2020) follows similar idea but proposes to use an acquisition function based on information gain instead of UCB. While most of the works rely on GPs to model the surrogate function, Li et al. (2020b) use a Bayesian neural network that models the complex relationship between fidelities with stacked neural networks, one for each fidelity.
Hyperband [Li et al. 2017 is bandits-based multi- fidelity method for hyperparameter optimization that selects hyperparameter configurations at random and uses successive halving (Jamieson and Tal walkar 2016) with different settings to early-stop less promising training runs. Several improvements have been proposed to Hyperband with the aim to replace the random sampling of hyperparameter configurations with more guided approach [Bertrand et al. 2017 Wang et al. [018]] Wistuba 2017). BOHB [Falkner et al., 2018) uses TPE [Bergstra et al. [011 and builds a surrogate model for every fidelity adhering to fixedffidelit selection scheme. DEHB [Awad et al. [021] samples candidates
using differential evolution which handles large and discrete search spaces better than BOHB. Mendes
et al. [202I] propose variant of Hyperband which allows to skip stages.
Learning Curve Prediction A variety of methods attempt to extrapolate a partially observed learning curve in order to estimate the probability that configuration will improve over the current best solution. Domhan et al. (2015) propose to ensemble set of parametric functions to extrapolate a partial learning curve. While this method is able to extrapolate with a single example, it requires relatively long learning curve to do so. The work by Klein et al. [2017b] build upon the idea of using a set of parametric functions. The main difference is that they use a heteroscedastic Bayesian model to learn the ensemble weights. Baker al.) (2018) propose to use support vector machines (SVM) as an auto-regressive model. The SVM predicts the next value of a learning curve, the original learning curve is augmented by this value and we keep predicting further values. The work by Gargiani et al. (019) use a similar idea but makes prediction based on the last K observations only and uses probabilistic models. Wistuba and Pedapati [2020] propose to learn a prediction model across learning curves from different tasks to avoid the costly learning curve collection. In contrast to DyHPO, none of these methods selects configuration but limited to deciding when stop a running configuration.
Multi-Fidelity Acquisition Functions Klein et al. [2017a] propose an acquisition function which allows for selecting hyperparameter configurations and the dataset subset size. The idea is to reduce training time by considering only smaller part of the training data. In contrast to Elgr. this acquisition function designed to select arbitrary subset sizes whereas ElyF intended to slowly increase the invested budget over time. Mendes et al. (2020) extend the work of Klein et al. 2201aa to take business constraints into account.
Deep Kernel Learning with Bayesian Optimization. We are among the first to use deep kernel learning with Bayesian optimization and to the best of our knowledge the first to use it for multi- fidelity Bayesian optimization. |aii et al. (2016) consider the use of deep kernel instead of manually designed kernel in the context of standard Bayesian optimization. but, limit their experimentation to synthetic data and do not consider its use for hyperparameter optimization. Perrone et al. (2018), Wistuba and Grabocka 202T use a pre-trained deep kernel to warm start Bayesian optimization with meta-data from previous optimizations. The aforementioned approaches are multi-task or transfer learning methods that require the availability of meta-data from related tasks.
3

---

In contrast to prior work, we propose a method that introduces deep learning to multi multi-fidelity HPO with Bayesian Optimization, and captures the learning dynamics across fidelities/budgets, combined with an acquisition function that is tailored for the gray-box setup.
3 Dynamic Multi-Fidelity HPO
3.1 Preliminaries
Gray-Box Optimization.
The gray-box HPO setting allows querying configurations with a smaller budget compared the total maximal budget B. Thus, can query from the response function f: x + R where Júg [(x;, is the response after spending a budget of on configuration x,. As before, these observations are noisy and we observe Yi. = f(x,, j) E, where E ~ (0, o. Please note, assume that the budget required query fij+b after having queried fi,j is only b. Furthermore, we use the learning curve Yij = (Vi.1, --.j-) when predicting fi.j-
Gaussian Processes (GP). Given training data set D {(x, ) the Gaussian Process assumption that y/ is a random variable and the joint distribution of all yi is assumed to be multivariate Gaussian distributed as y ~ N (m (X), (x, Furthermore, f, for test instances X, are jointly Gaussian with as:
] N
(
K) K
K
K.
F.
~
m (X, (X,
K..
(1)
The mean function m is often set to and its covariance function depends on parameters For notational convenience, we use K,, k (X, X(0) + L K, k (X, X. 0) and K.. (X., X. 0) to define the kernel matrices. can derive the posterior predictive distribution with mean and covariance follows:
E X,, = K, cov [f, = K.. K,
Often, the kernel function is manually engineered. one popular example the squared exponential kernel. However, in this work, we make use of the idea of deep kernel learning [Wilson al. 2016). The idea is to model the kernel as a neural network & and learn the best kernel transformation K (0, w) k(p(x,, w), (xx'; 0)) which allows us convolutional operations in our kernel.
3.2 Deep Multi-Fidelity Surrogate
We propose use Gaussian Process surrogate model that infers the value of fig based on the hyperparameter configuration x;, the budget as well as the past learning curve Yij-1- For this purpose, we use deep kernel as:
K (0, w) == k(p((,, Yi w), pxxy, 3.':
(3)
We use squared exponential kernel for & and the neural network jo is composed of linear and convolutional layers as shown in Figure 2 We normalize the budget j to range between and 1 by dividing it by the maximum budget B. Afterward, is concatenated with the hyper- parameter configuration layer. The learning curve Y 1 is transformed by one-dimensional convolution followed by a global max pooling layer. Finally, both repre- sentations are fed to another linear layer.
and fed to a linear
Yij-1
Convolution
Figure 2: The feature extractor of our kemnl.
Its output will be the input to the kernel function k. Both, the kernel l and the neural network & consist of trainable parameters 0 and w, respectively. We find their optimal values by computing the maximum likelihood estimates as:
0, 1
arg maxp(yyXXppp X, Y, 0. w) oc arg miny T ToinnyKKK (0, w1 y + log KK (0, w)
(4)
8, a0
8.30

---

In order to solve this optimization problem, we use gradient descent and Adam [Kingma and Ba 2015] with a learning rate of 0.1. Given the maximum likelihood estimates, we can approximate the predictive posterior through p (fes) Xx, i,,, D, wt) and ultimately compute the mean and covariance of this Gaussian using Equation 2
3.3
Multi-Fidelity Expected Improvement
Expected improvement (Jones et al. 1998 is a commonly used acquisition function and defined as:
[D [max (f(x) ymT ,-"",(0)]
where yax the largest observed value of f. We propose multi-fidelity version as:
EIMF (x. j[D) = mmax {[(x, max ;
(6)
where:
max { ((x, .),,) E D} (((xx,,)), max {y (:: D}
(7)
otherwise
Simply put, Mrtu is the largest observed value of for a budget of j if it exists already, otherwise, the largest observed value for any budget. If there is only one possible budget, the multi-fidelity expected improvement identical to expected improvement.
3.4 The DyHPO Algorithm
The DYHPO algorithm looks very similar to many black-box Bayesian optimization algorithms as shown in Algorithm The big difference is that each step dynamically decide which candidate configuration to train for a small additional budget.
Possible candidates are previously un- considered configurations as well as configurations that did not reach the maximum budget. Line 2, the most promising candidate is chosen using the acquisition function introduced in Section 3.3 and the surrogate model's predictions. It is important to high- light that do not maximize the ac- quisition function along the budget di- mensionality. Instead, get such that it is by exactly one higher than the budget used to eval- uate x; before. This ensures that we explore configurations by slowly increasing the budget. After the candidate and the corresponding budget are selected, the function f is evaluated and we observe Yt. (Line 3). This additional data point is added to D Line 4. Then in Line the surrogate model is updated according to the training scheme described in Section 3.2
Algorithm DYHPO Algorithm
1:
b(x) = V E X
2:
while not converged do
3:
X; f- arg maxxex ElMF (x, b(x) - 1) (Sec. 3.3
4: Observe bx)
b(x,) - b(x;) +1
D DU ((x,, Y Y )., (i,(
set the bud-
7: Update the surrogate on D. (Sec. 3.2
return x, with largest yi...
4 Experimental Protocol
4.1 Experimental Setup
We evaluate DYHPO in three different settings on hyperparameter optimization for tabular, text, and image classification against several competitor methods, the details of which are provided in the following subsections. We ran all of our experiments an Amazon EC2 M5 Instance (m5.xlarge). In our experiments, we report the mean ten repetitions and we report two common metrics, the regret and the average rank. The regret refers to the absolute difference between the score of the solution found by an optimizer compared the best possible score. If we report the regret as an aggregate result over multiple datasets, report the mean over all regrets. The average rank is the
5

---

metric we use to aggregate rank results over different datasets. We provide further implementation and training details in Appendix A.4 Our implementation DYHPO is publicly available
4.2 Benchmarks
In our experiments, we use the following benchmarks. We provide more details in Appendix A.1
learning curve benchmark [Zimmer al. 2021 that evaluates neural network architec- tures for tabular datasets. LCBench contains learning curves for 35 different datasets, where 2,000 neural networks per dataset are trained for 50 epochs with Auto-PyTorch.
LCBench:
TaskSet: A benchmark that features diverse tasks Metz et al. [020]] from different domains and includes 5 search spaces with different degrees of freedom, where, every search space includes 1000 hyperparameter configurations. In this work, we focus on a subset of NLP tasks (12 tasks) and we use the Adam8p search space with continuous hyperparameters.
benchmark consisting of 15625 hyperparameter configurations representing different architectures on the 10. CIFAR- 100 and ImageNet datasets Dong and Yang [2020] NAS-Bench-22 201 features a search space of categorical hyperparameters and each architecture is trained for 200 epochs.
NAS-Bench-205
4.3 Baselines
Random Search: A random/stochasti black-box search method for HPO.
HyperBand: A multi-arm bandit method that extends successive halving by multiple brackets with different combinations of the initial number of configurations, and their initial budget [Li et al. 2017).
BOHB: An extension of Hyperband that replaces the random sampling of the initial configurations for each bracket with recommended configurations from a model-based approach [Falkner al.
2018 BOHB builds model for every fidelity that is considered.
DEHB: A method that builds upon Hyperband by exploiting differential evolution to sample the initial candidates Hyperband bracket (Awad et al. 202I]
An asynchronous version of successive halving (or asynchronous version of Hyperband if multiple brackets are run). ASHA [. et al. [2020a] does not wait for all configurations to finish inside successive halving bracket, but, instead promotes configurations to the next successive halving bracket in real-time.
ASHA:
MF-DNN: A multi-fidelity Bayesian optimization method that uses deep neural networks to capture the relationships between different fidelities Li al. [2020b].
Dragonfly: We compare against BOCA Kandasamy et al. 2017] by using the Dragonfly library Kan dasamy al. [2020]. This method suggests the next hyperparameter configuration as well as the budget it should be evaluated for.
4.4 Research Hypotheses and Associated Experiments
Hypothesis 1: DYHPO achieves state-of-the-art results in multi-fidelity HPO.
Experiment We compare DYHPO against the baselines of Section 4.3 on the benchmarks of Section 4.2 with the experimental setup of Section 4.1 For TaskSet we follow the authors' recommendation and report the number steps (every 200 iterations).
Hypothesis 2: DYHPO's ss runtime overhead has negligible impact on the quality of results.
We compare DYHPO against the baselines of Section 4.3 the wallclock time. The wallclock time includes both (i) the optimizer's runtime overhead for recommending the next hyperparameter configuration, plus (ii) the time needed evaluate the recommended configuration. In this experiment, we consider all datasets where the average training time per epoch is at least 10 seconds, because, for tasks where the training time is short, there no practical justification for
Experiment 2:

---

LCBench
TaskSet
ImageNet16-12
Random
Hyperband
Mean Regret
10
BOHB
10
DEHB
10
Dragonfly
ASHA
10
MF-ONN
DyHPO
0
500
1000
500
1000
2000
4000
Number of Epochs
Number of Steps
Number of Epochs
Figure 3: The mean regret for the different benchmarks over the number of epochs or steps (every 200 iterations). The results are aggregated over 35 different datasets for LCBench and aggregated over 12 different NLP tasks for TaskSet.
complex solutions and their overhead. In these cases, we recommend using random search. We don't 't report results for TaskSet because the benchmark lacks training times.
Hypothesis 3: DYHPO uses the computational budget more efficiently than baselines.
Experiment 3: To further verify that DyHPO is efficient compared to the baselines, we investigate whether competing methods spend their budgets on qualitative candidates. Concretely we: i) calculate precision of the top (w.r.t. ground truth) performing configurations that were selected by each method across different budgets, ii) compute the average regret of the selected configurations across budget, and iii) we compare the fraction of top-performing configurations at a given budget that were not top performers at lower budgets, i.c. measure the ability handle the poor correlation of performances across budgets.
Results
Experiment 1: DYHPO achieves state-of-the-art results. In our first experiment, we evaluate the various methods on the benchmarks listed in Section 4.2 We show the aggregated results in Figure the results show that DYHPO manages to outperform competitor methods over the set of considered benchmarks by achieving better mean regret across datasets. Not only does DYHPO achieve a better final performance, it also achieves strong anytime results by converging faster than the competitor methods. For the extended results, related to the performance of all methods on dataset level, we refer the reader to Appendix B
LCBench@50%%
LCBenchg 100%
TaskSeug250%
TaskSettg 100%
Randen
Dylipo
DyliFo
Randon NF
DyliPo O1iii
Dregunly OF DKN
DOPS
JP-
DEHA
Dasgertyy
Hydeooon
Drayyeil
Figure 4: Critical difference diagram for LCBench and TaskSet in terms of the number of HPO steps. The results correspond to results after 500 and 1,000 epochs. Connected ranks via bold bar indicate that performances are not significantly different (p > 0.05).
In Figure 4 we provide further evidence that DyHPO's improvement over the baselines is statistically significant. The critical difference diagram presents the ranks of all methods and provides information on the pairwise statistical difference between all methods for two fractions the number of HPO steps (50% and 100%). We included the ICCench and TaskSet benchmarks in our significance plots. NAS-Bench-200 was omitted because it has only 3 datasets and the statistical test cannot be applied. Horizontal lines indicate groupings of methods that are not significantly different. As suggested by the best published practices Demsar (2006). we use the Friedman test to reject the null hypothesis followed by pairwise post-hoc analysis based on the Wilcoxon signed-rank test (a 0.05).
For LCBench, DYHPO already outperforms the baselines significantly after 50% of the search budget, with statistically significant margin. As the optimization procedure continues, DYHPO manages to extend its gain in performance and is the only method that has a statistically significant improvement against all the other competitor methods. Similarly, for TaskSet, DYHPO manages to outperform all methods with a statistically significant margin only halfway through the optimization procedure and achieves the best rank over all methods. However, the optimization procedure continues, BOHB
7

---

manages to decrease the performance gap with DyHPO, although, it still achieves a worse rank across all datasets. Considering the empirical results, we conclude that Hypothesis 1 is validated and that DyHPO achieves state-of-the-art results on multi-fidelity HPO.
LCBench@50%
LOBandh
Image:Net66112
Random
Hyperband
Dregrnity
DysPO
Regier
BOHB
Hypeetond
ABHA
BOHS
DNN
DEHB
Dragonfly
ACBench@1000
ASHA
MF-DNN
DyHPO
Dragunily
DEHB
0- Normekosd Walicloc
10"
10"
Hypelbandd
ASHA
BOMB
Wallclock Time in Seconds
Figure 5: Left: The regret over time for all methods during the optimization procedure for the LCBench benchmark and the ImageNet dataset from the NAS-Bench-200 benchmark. The normalized wall clock time represents the actual run time divided by the total wall clock time of DyHPO including the overhead of fitting the deep GP. Right: The critical difference diagram for LCBench halfway through the HPO wall-clock time, and in the end. Connected ranks via a bold bar indicate that differences are not significant (p 0.05).
On the impact of DYHP''s overhead on the results. We present the results of our second experiment in Figure 5 (left), where, as it can be seen, DYHPO still outperforms the other methods when its overhead is considered. For LCBench, DyHPO manages to get an advantage fairly quickly and it only increases the gap in performance with the other methods as the optimization process progresses. Similarly, in the case of ImageNet from WAS--ench-200 DYHPO manages to gain an advantage earlier than other methods during the optimization procedure. Although in the end DyHPO still performs better than all the other methods, we believe most of the methods converge to good solution and the differences in the final performance are negligible. For the extended results, related to the performance of all methods on dataset level over time, we refer the reader to the plots in Appendix B Additionally, in Figure 5 (right), we provide the critical difference diagrams for LCBench that present the ranks and the statistical difference of all methods halfway through the optimization procedure, and in the end. As it can be seen, HyHPO has a better rank with significant margin with only half of the budget used and it retains the advantage until the end.
Experiment 2:
On the efficiency of DYHPO. In Figure 6 (left), we plot the precision of every method for different budgets during the optimization procedure, which demonstrates that DYHPO effectively explores the search space and identifies promising candidates. The precision at an epoch i defined as the number of top 1% candidates that are trained, divided by the number of all candidates trained, both trained for at least epochs. The higher the precision, the more relevant candidates were considered and the less computational resources were wasted. For small budgets, the precision is low since DYHPO spends budget to consider various candidates, but then, promising candidates are successfully identified and the precision quickly increases. This argument is further supported in Figure 6 (middle), where we visualize the average regret of all the candidates trained for at least the specified number of epochs on the x-axis. In contrast to the regret plots, here we do not show the regret of the best configuration, but the mean regret of all the selected configurations. The analysis deduces a similar finding, our method DYHPO selects more qualitative hyperparameter configurations than all the baselines.
Experiment 3:
An interesting property of multi-fidelity HPO is the phenomenon of poor rank correlations among the validation performance of candidates at different budgets. In other words, configuration that achieves a poor performance at a small budget can perform better at a larger budget. To analyze this phenomenon, we measure the percentage of "good" configurations at particular budget, that were "bad" performers in at least one of the smaller budgets. We define a "good"" performance at a budget B when a configuration achieves a validation accuracy ranked among the top 1/3 of the validation accuracies belonging to all the other configurations that were run until that budget B.
In Figure 6 (right), we analyze the percentage of "good" configurations at each budget denoted by the x-axis, that were "bad" performers in at least one of the lower budgets. Such a metric is a proxy for the degree of the promotion of "bad" configurations towards higher budgets. We present the analysis for all the competing methods of our experimental protocol from Section 4 We have additionally included the ground-truth line annotated as "Baseline", which represents the fraction of past poor
8

---

Predkion Top Cendlettes
LGflench
LGBench
LOBendh
Promattoon
0.2
Reinago Pagget
0.16
02-
210
Fraction
6.1
0.05
10
30
30
40
50
26
Number Eppchss
Numbor of Epochs
Numteral Ppochs
Rondom
DEHE
NF-D DNNN
Randon Hypolard
DEHB
MF DNN
Raston
an Dragonty ASHA
#yporbard
Drogerity
0y4P
Dragorty
DyHPO
Hypattua))
Dyiro
90HB
ASHA
BOKB
AS-A
90+B
Boodkre
Precsin of Top Candioans
0.16-
NAS Bench 201
0.12
NAS Bench 201
NAS 201
0.-
0.10
0.05-
100
150
100
100
200
100
Namber
Epocta
Namber of Epodda
Namberal Epochs
Randon
DE-13
NF-ONN
Fandon
DE-13
MF-ONN DyHiO
Random
DE
Hyperband
Dragordy
0y+io
Hyperfand
Disgorty
1hpasband
Oragonity
BOHB
ASHA
BOMB
ASHA
50MB
AEHA
Bocchee
The fraction of top- performing candidates from all candidates that were selected to be trained. Middle: The average regret for the configurations that were selected to trained a given budget. Right: The percentage of configurations that belong to the top 1/3 configurations at given budget and that were in the top bottom 2/3 of the configurations a previous budget. All of the results are from the LCBench and NAS-Bench-201 benchmark.
Figure The efficiency of DyHPO as the optimization progresses. Left:
performers among all the feasible configurations the search space. contrast, the respective methods compute the fraction of promotions only among the configurations that those methods have considered (i.c. selected within their HPO trials) until the budget indicated by the x-axis. We see that there is high degree of "good" configurations that were "bad" at previous budget, with fractions of the ground-truth -rtutttthh "Baseline" going up to 40% for the LCBench benchmark and up to 80% for the NAS-Bench-201 benchmark.
On the other hand, the analysis demonstrates that our method DYHPO has promoted more "good" configurations that were "bad" in lower budget, compared to all the rival methods. In particular, more than 80% of selected configurations from the datasets belonging to either benchmark were "bad" performers at lower budget. The empirical evidence validates Hypothesis 3 and demonstrates that DyHPO efficiently explores qualitative candidates. We provide the results of our analysis for DYHPO's s efficiency on the additional benchmarks (Taskset) in Appendix B
Ablating the impact of the learning curve
One of the main differences between DYHPO and similar methods Kandasamy al. 2017] is that the learning curve an input to the kernel function. For this reason, investigate the impact of this design choice. We consider variation of HYHPO w/o CNN, which is simply DYHPO without the learning curve.
ImageNet16-120
Regret
10-1
DyHPO
DyHPO w/o CNN
Random
It is worth emphasizing that both variants (with and without the learning curve) are multi-fidelity surrogates and both receive the budget informa- tion through the inputted index j in Equation 3 The only difference is that DYHPO additionally incorporates the pattern of the learning curve.
10
10
10"
10"
Training Time in Seconds
Figure 7: Ablating the impact of the learning curve on DYHPO.
We run the ablation on the largest dataset our collection. The ablation results are shown in Figure 7 while the remaining results the other datasets shown in Figure 8 of the appendix. Based on the results from our
benchmark and report the results for ImageNet, the

---

learning curve ablation, we conclude that the use of an explicit learning curve representation leads to significantly better results.
Limitations of Our Method
Although DYHPO shows convincing and statistically significant reduction of the HPO time diverse Deep Learning (DL. experiments, we cautiously characterized our method only as a "step towards" scaling HPO for DI.. The reason for restrain is the lack of tabular benchmarks for HPO on very large deep learning models, such as Transformers-baxse Additionally, the pause and resume part of training procedure the hyperparameters of parametric models, otherwise, the training of a hyperparameter configuration would have to be restarted. Lastly, for small datasets that be trained fast, the overhead of model-based techniques would make an approach like random search more appealing.
architectures [Devlin et al. 2019).
only be applied when tuning
7 Conclusions
In this work, we present DYHPO, a new Bayesian optimization (BO) algorithm for the gray-box setting. We introduced a new surrogate model for BO that uses a learnable deep kernel and takes the learning curve as an explicit imput. Furthermore, we motivated a variation of expected improvement for the multi-fidelit setting. Finally, compared approach diverse benchmarks on total 50 different tasks against the current seate-of-thh-art methods on gray-box hyperparameter optimization (HPO). Our method shows significant gains and has the potential to become the de facto standard for HPO in Deep Learning.
Acknowledgments
Josif Grabocka and Arlind Kadra would like to acknowledge the grant awarded by the Eva-Mayr-Stihh Stiftung. In addition, this research was funded by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under grant number 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, Josif Grabocka acknowledges the support of the BrainLinks. BrainTools center of excellence.
References
Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pages 2147-2153, 2021.
doi: (0.24963/jjaai.2022222
httpp:///::: org
10.249990 202222493/jjeee htt.iaaiii222229991222 2021/296
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture Search using performance prediction. In 6th International Conference Learning Representations, ICIR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings, 2018.
URL https:://ppnnrerie: net//oorumidd=H]jk33
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms hyper-parameter optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 201 Proceedings of a meeting held 12-14 December
2011, Granada, Spain, pages 2546-2554, 2011. URL https: /ttttpee///rrooeeiiggg.
neurips cc/ Abstract html
Hadrien Bertrand, Roberto Ardon, Matthieu Perrot, and Isabelle Bloch. Hyperparameter optimization of deep neural networks: Combining hyperband with bayesian model selection. In Conférence I' l'Apprentissage Automatique, 2017.
Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P. Lillicrap. Matthew Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In Proceedings of the 34th International Conference Machine Learning, ICML 2017,
10

---

Sydney, NSW, Australia, 6-11 August 2017, pages 748-756, 2017. URL http://proceeedngs
mlr. ptwss//77//cenn7e..ooo
Janez Demsar. Statistical comparison of classifiers over multiple data sets. J. Mach. Learn. Res., 7: 1-30, 2006. URL httt::ttp:/miiii http:///w...pppppp///ppapppessee /w//ar/maa..... html
Jacob Devlin, Ming-Wor -Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume / (Long and Short Papers), pages 4171 4186, 2019. doi: 10. 10.18833/11/19-111 1423. URL https:tts:///oo .rrg/
10. 118653/v1/n19-1 1423
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Qiang Yang and Michael J. Wooldridge, editors. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI Buenos Aires, Argentina, July 25-31, 2015, pages 3460-3468.
AAAI Press, 2015, URL http://i/i org/ /tt///bttttttt//aaae/e
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL htttttt//ppeenreeeeee
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter optimiza- tion at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockhollsmmsssan Stockholm, Sweden, July 10-15, 2018, pages 1436-1445, 2018. URL http://proceeddings www.mlrrpreee www.:presssvv/eeee watpss//voofaaaaerrrrr faw.nerlllerrrrrr 18a, html
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 1 August 2017, pages 1165-1173, mlr www.ppressssve http:///ooffaacccccciieeea
2017. URL http::::pppoceeediiiaaeaa
Jacob R. Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew Gordon Wil- son. Gpytorch: Blackbox matrix-matris gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural In- formation Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 7587-7597, 2018 URL httpss//prooeeddnng htt.:000777772000aa--bbsttrttttt Absraattttbttttt.hhhh html
Matilde Gargiani, Aaron Klein, Stefan Falkner, and Frank Hutter. Probabilistic rollouts for learning curve extrapolation across hyperparameter settings. CoRR, abs/1910.04522, 2019. URLhttp:
httpg///bb//11...0000 04522
Kevin G. Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, pages 240-248 2016. URL
http://proceedingg mlr wttppp/eese/sseessseseooo
html
Donald Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of ex- pensive black-box functions. J. Global Optimization. 13(4):455-99, 1998. doi: 10.1023/A: 1008306431147, URL/attps://do1 org/10. 10. 1023/ 1008364331447
Kirthevasan Kandasamy, Gautam Dasarathy, Junier B. Oliva, Jeff G. Schneider, and Barn- abás Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. In Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 992-1000, 2016. URL dtttps:/proceedie nwwrsseeerrps..om wtt.cc/pppp Abssrfrattt-bttttttt...a.aaaaa.. html
Absttact.
Kirthevasan Kandasamy, Gautam Dasarathy, Jeff G. Schneider, and Barnabás P6czos. Multi-fidelity bayesian optimisation with continuous approximations. In Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1799-1808, 2017. URL htttp://proceedinggs mlr /tress//70//Knndaaayyy.....

---

Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, and Eric P. Xing. Neural architecture search with bayesian optimisation and optimal transport. In Ad- vances in Newral Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2020-2029, 2018. URL https: httep::/pprcceednng wwwpeeediinnnn.uuurneneeee nwwrips cc/ paappr//0///aa//aa/ppppppppp (B3baaloeffaccleeerrrrraaadbdd waw.ee442aaaeeeaaaaaaataaa...... html
Kirthevasan Kandasamy, Karun Raju Vysyaraju. Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabás Póczos, and Eric P. Xing. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly. J. Mach. Learn. Res., 21: 81:1-81:27, 2020. URL htthtt//mmmml httpaa/.rrr/pppe///////////////e
223. html
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: /tttt:://riii..../////// owg aba/ 1412. 6980
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian
optimization of machine learning hyperparameters on large datasets. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, pages 528-536, 2017a. URL http:t//pcccceedinngg mlr. press/
v54/klein17a. hhmml
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017b. URL
Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. In Inderjit S. Dhillon, Dimitris S. Papailiopoolos, and Vivienne Sze, editors, Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020a. URL httLtttss///poceediing wwaeeeddiinnmmss......m.mmmmmmee wrggbook//03..ppppoooog
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18: 185:1-185:52, 2017. URL httt:t/tt//mmmi org httpg//aaprss////////// 551665558.. html
Shibo Li, Wei Xing. Robert M. Kirby, and Shandian Zhe. Multi-fidelit bayesian optimization via deep neural networks. In Advances Neurul Information Processing Systems 33: An- nual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020b. URL htttps://proceeddnggs www.inng...uuiis.. httppig..ee.rps..pepeeee/e////ee wwtppp/pppp///020000// / Saeelab44arttaasseeeeeeedddeeedd hww.aaaddeeeeeedeeaea---aara.e.e Abstract htttiaaattttmll......
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiaaion. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], pages 1540-1552, 2020. URL.attp://proceedingg mlr. prwss/resssseee v108 http://orrraaanaaantetmmmae
Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient-based hyperparameter opti- mization through reversible learning. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 2113-2122, 2015. URL htttp://proceedini mlr. wwwpppreer/vvv arees//33//mcclaurint. /tcm/maaalannnaa/m html
Pedro Mendes, Maria Casimiro, Paolo Romano, and David Garlan. Trimtuner: Efficient optimization of machine learning jobs in the cloud via sub-sampling. In 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, MASCOTS 2020, Nice, France, November 17-19, 2020. pages 1-8. IEEE, 2020. doi: 10. 10..MA/COOOTS50086 2020.9285971. https:::: ptttps::/ddor //oii//oiiooo org/ 10. 1009 1190999AAA /MASCOSSSSOTT....... 2020. 9285971
Pedro Mendes, Maria Casimiro, and Paolo Romano. Hyperjump: Accelerating hyperband via risk modelling, CoRR, abs/2108.02477 024799 2021. /ttpss//////ax www.aawwwww....raaeeeeeeee
www./ogg//////oooooooooooooeesr/ ab////bs////00 024799
12

---

Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using a thousand optimization tasks to lear hyperparameter search strategies. CoRR, abs/2002. 11887, 2020. URL httph::tts/aaaae
org/ ab22/002 11887
Jack Parker-Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient online hyperparameter optimization with population-based bandits. In Advances in Neural Information Processing
Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. December 6-12, 2020, virtual. 2020. URL stttts::/pproot httpss://proceeddnggg www.gng........... wwtpppppp//aaeeeeeeceeee /
1obpbee1330---a0 Abstract.. html
Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6846-6856, 2018. https:ttts///reeeeeeee wwwroceedingg.nnuriis....
1 hatp://aaaaa11113333000000000000 -btpr///tttcattaatttttt.c....... html
Matthias Poloczek, Jialei Wang, and Peter I. Frazier. Multi-informatii source optimiza- tion. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4288 4298, 2017. URL https: htttps://proceedinie Abstaaat.bbbtattttac.. html
neurips cc/
Akshara Rai, Ruta Desai, and Siddharth Goyal. Bayesian optimization with a neural network kernel, 2016. URLhttpp:// cB cmu. www.cdu/ http///utaadfilee//O.OO1 pdf
Jasper Snoek, Hugo Larochelle, and Ryan Adams. Practical bayesian optimization of machine learn- ing algorithms. In Advances in Neural Information Processing Systems 25: 26th Anmual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012. URL http:::t/tsssceeeeeeeee
2012, Lake Tahoe, Nevada, United States, pages neurips. http:///ww//////1111aaa////66660
html
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 1015-1022, 2010. URL.httpo ht/m/mmmml.aa httppp//iicmm..ccc/aeeeeeeeeeeee Cct.a///eeeoeeeeeee/e/0000 pttp:/eeeeeeee///ppppprraa/////e
pd
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Multi-task bayesian optimization. In Advances in Neural Information Processing Systems 26: 27th Annual Conference Neural
Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2004-2012, 2013.
ieeuuu neurips Absstattt html
cc/
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. CoRR, abs/1406.3896, 2014. /ttp:///// argg/bs//406/6 3896
Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, and Masayuki Karasuyama. Multi-fidelity bayesian optimization with max max-value entropy search and its parallelization. In Proceedings of the 37th International Conference on Machine Learning, 1020, 13- 18 July 2020, Virtual Event, pages mlr. www.ppesssviii vwtssss/vii1/ v119 takeno20a. html
2020. /ttp/htttt//prooeedings
Jiazuuo Wang, Jason Xu, and Xuejun Wang. Combination of hyperband and bayesian optimization for hyperparameter optimization in deep learning. CORR, abs/1801 .01596, 2018. URL. http:
01596
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Proceedings of the 19th International Conference Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, pages 370-378, 2016. URL httt:://poccee/eiie/ press/ http//////wwl//// html
mlr
Martin Wistuba. Bayesian optimization combined with incremental evaluation for neural network architecture optimization. In AwW.MMLL 2017.
13

---

Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. CtIl.htts:/// httpp:://openreviee http://www....ooorooooooooooo/e/
Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10303- 10312. PMLR, 2020.
URL ptttp::p/proceediggss mlr press/ /ttsss/vv1/// http1///iisuuaaaaaa html
Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch: Multi-fidelity metalearning for efficient and robust autodl. IEEE Trans. Pattern Anal. Mach. Intell., 43(9):0079-3090 2021. doi: 10.11009/PAMM.2201. 3067763.. URL (ttp:tt::: //tp::///ooiio org/ 1109 /PMMII 202..0022222..
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section "Societal Implications" in the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] We provide our main algorithm in Section 3 and we additionally provide the detailed implementation details in Appendix A for all methods and benchmarks. We will release the code for the camera-ready version of our work.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report the statistical significance of the performance difference between our method and the baselines in Section
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 1.
4. If you using existing assets (e.g., code, data, models) or curating/releasigg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.2 and
Section 4.3
(b) Did you mention the license of the assets? [Yes] See Appendix A. and A.5 where we provide references the assets where the license included.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] The benchmarks and baselines are open-sourced.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information offensive content? [N/A] The data does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "yS_bRES0C2U", "reviewer": "Reviewer_XM9p", "summary": "This paper proposed a novel multi-fidelity grey-box Bayesian optimization method based on deep kernel GP.", "questions": "Could the authors discuss more the efficiency in terms of time?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n\nThe motivation is clear and well-defined. Good summary of related work.  The experiments are comprehensive. Limitations were also discussed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XmwiMb4khk", "reviewer": "Reviewer_MHNh", "summary": "In the paper \"Supervising the Multi-Fidelity Race of Hyperparameter Configurations\" an HPO method for neural networks is proposed based on the framework of Bayesian optimization. To this end, the authors propose a kernel function, which is mainly implemented via a neural network, that accounts for different fidelities and is thus able to make use of all the data in contrast to other methods that employ a distinct surrogate model for each fidelity level.", "questions": "* ll.16ff: What about SMAC? Which is a method particularly tweaked towards the configuration of algorithms and therewith also for HPO?\n* What are the drawbacks and limitations of DyHPO?", "limitations": "Conceptual limitations are not discussed. However, the scope of the proposed HPO method is more or less clearly limited to the hyperparameter optimization for neural networks.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe authors address a severe issue of current multi-fidelity methods which are prone to reject/cancel out hyperparameter configurations which perform bad on early budget stages but would outperform configurations promoted to the next multi-fidelity levels on higher budgets. As an example the authors explicitly mention neural networks which are trained with regularization and known to perform better in the long run but are dominated by configurations that are not regularizing for lower budgets. I agree with the authors that this problem is one of the main drawbacks of multi-fidelity methods. Another example would be to compare more complex networks, which require more data for training, to simpler networks, maybe even linear ones, which perform sufficiently well on lower budgets already. In this case, the configurations representing the simpler candidates would seem beneficial in the first place and be promoted to subsequent stages.\n\nThe proposed kernel function seems to me like a reasonable and original contribution. At least I am not aware of any such method which would pursue a very similar strategy which is not discussed in the paper.\n\nIn the results the proposed approach is found to yield state-of-the-art performance. More specifically, DyHPO shows outstanding performance.\n\nOverall, the paper is clearly written and well structured. Especially the experimental part, I really enjoyed as it was well structured and the addressed research questions have been made very explicit.\n\n## Weaknesses\nThe authors promise to release their code upon acceptance. However, for a thorough review process it would be more appropriate to also hand in the source code.\n\nFigures and algrotihms are positioned inconveniently throughout the document. First, text floats around some of the figures, I assume to save some space. Honestly speaking, I do not know whether this is against the style guidelines. In a hurry, I could not find anything about this in the guidelines, so I assume it is okay, still I would consider it a bad practice and \"cheating on the space limitations\" to some extent. Figures should be aligned to the top to avoid outcomes as on page 7 where two lines of text are given above the figure. As a reader I did not expect that and first missed the two lines. Also the fonts in the figures are sometimes too small and thus hard to read as a printed version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PEe5eJjs_KB", "reviewer": "Reviewer_NJxE", "summary": "The authors consider multi-fidelity hyperparameter optimization (HPO). They introduce a new method, DyHPO, which uses Bayesian optimization (BO) with a newly proposed deep kernel that takes as input the full learning curve. Their approach also uses a new multi-fidelity version of expected improvement. At every iteration, DyHPO decides which hyper parameter set to train for an additional fidelity (epoch). The authors run extensive experiments on 50 datasets spanning tabular, image, and NLP data.", "questions": "The authors say the will release the code in the camera ready version. Can the authors release their code during the author response period? For example, by using https://anonymous.4open.science/?  This would improve the contribution.\n\nCan the authors run more ablation studies, such as some of the ones mentioned in the weaknesses?\n\nCan the authors comment on Freeze-thaw Bayesian optimization and ideally run empirical comparisons?", "limitations": "The authors do include limitations and societal impact, but this paper is technically over the page limit, since the societal impact section is supposed to fit within the first 9 pages. The authors should get the societal impact section into the first 9 pages.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThis is a clean method that performs well. Intuitively, methods such as Hyperband or BOHB could make use of learning curve extrapolation to improve performance, so it is great to see some progress in this direction.\n\nExtensive experiments: the diversity and the quantity of the experiments is well-done. They chose seven other benchmarks, from baselines to state-of-the-art techniques.\n\nThey have evaluated three research questions by giving hypotheses and then designing experiments to test the hypotheses.\n\n### Weaknesses\nIt is not particularly novel. All of the pieces are similar to existing work (e.g., multi-fidelity Bayesian optimization, deep kernel learning, freeze-thaw Bayesian optimization, and learning curve extrapolation work). However, it is still non-trivial to design this method.\n\nThe approach seems pretty similar to Swersky et al. 2014 (freeze-thaw Bayesian optimization). That approach also uses Bayesian optimization to decide whether to pause or continue training dynamically over all hyperparameter settings. (The difference is deep kernels, using the full learning curve, and the specific acquisition function.) Ideally, the authors should compare to this method in the experiments, given that it is so similar.\n\nI would have liked to see more ablation studies. For example, the authors do not really mention some related works on extrapolating learning curves, such as the following:\n- Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves\n- Learning curve prediction with Bayesian neural networks\n- Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings\n- Accelerating neural architecture search using performance prediction\n\nThe authors could discuss and compare to these other techniques that implicitly or explicitly use the full learning curve dynamics. \n\nThe authors could also run an ablation study on the acquisition function, and the deep kernel vs another surrogate.\n\nLines 241-242 seems strange because you can simply decrease the amount of fidelities. For example, instead of treating every epoch separately in DyHPO, you could only allow DyHPO to query epochs 5, 10, 15, 20, etc. \n\nNo code. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8LrpI3iQK6B", "reviewer": "Reviewer_1bWR", "summary": "The paper presents DyHPO, a new multi-fidelity method for hyper parameter optimisation. They key novelty at the basis of DyHPO is its new deep kernel GP. By relying on a deep kernel (based on a CNN) DyHPO can leverage the learning curve to recommend the next configuration to explore. DyHPO relies on an expected improvement based acquisition function to decide which configuration to recommend and evaluates each configuration at different fidelities, incrementally. The evaluation based on well-known datasets and state-of-the-art baselines shows that DyHPO achieves state-of-the-art results on the considered benchmarks, demonstrating its superiority and the advantage of its deep kernel.\n", "questions": "**a)** I would have liked to see a plot with the ablation line (DyHPO without CNN) and with the other baselines, because it is harder to evaluate whether DyHPO's gains are all due to the deep kernel (i.e., it is hard to compare DyHPO without the deep kernel with the baselines).\n\n**b)** How does multi-fidelity EI contribute to the gains achieved by DyHPO? Did you try other acquisition functions? For example, [2] proposes an acquisition function based on entropy search (ES)\n\n**c)** I assume \"regret\" is the same as \"loss\". If this is true perhaps you could clarify this in the text, as \"loss\" is a commonly used metric.\n\n**d)** Why do you sometimes use \"number of epochs\" and others \"number of steps\" in the x-axis?\n\n**e)** In some of the LCBench datasets (e.g., ppendix, figure 12, \"Amazon_employee_access\", \"bank-marketing\", \"blood-transfusion-service-center\", \"credit-g\") DyHPO is no longer as competitive with state-of-the-art methods. Do you have an intuition as to which special characteristics of these datasets may lead to a worse performance by DyHPO?\n\n**f)** When you evaluate the precision (experiment 3, line 289), you explain your definition of precision (which is consistent with the \"usual\" definition of precision = TP / (TP + FP)) however in the plot the y-axis says true positives. Is this a typo or are you actually showing the true positives instead of the precision?\n\n**g)** I'm assuming the budget is the number of epochs during which you evaluate a configuration (I don't think this is explicit in the text or I may have missed it) and also that you \"maintain\" a j for each configuration, and when that configuration is selected you increment it. Are these assumptions correct? This is not very clear in the text.\n\n**h)** Given the previous assumptions, I wonder how many configurations are explored (and on how many budgets) when DyHPO has already found promising configurations. That is, once DyHPO finds the configuration it ends up recommending, how much more does it explore instead of exploit?\n\n**i)** What is the total cost spent by DyHPO compared to the baselines?\n\n**j)** The title of the submission is not very clear to me. Why \"Supervising\"?\n\n**Comments:**\n- maybe you could add some markers to each baseline to make it easier to distinguish each line. I had to zoom in quite a bit (e.g. on figure 3) to see the results easily.\n- Figures 12 and 13: legend mentions steps, but x-axis has epochs", "limitations": "Given that the kernel is based on a CNN, which is more computationally expensive that \"traditional\" (non-deep) kernels, I would have liked to see a discussion on how relying on such a kernel may prevent DyHPO from being widely adopted (due to the compute power required to deploy it). Second, the problem of setting the parameters of the CNN is also not discussed. For example, if I were to use a shallower NN to reduce complexity, how would that impact performance? Similarly, what would the performance gains be if I made the network even deeper?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** The paper presents a new method for for hyper parameter optimisation, which is a novel combination of known techniques. The main difference from existing work is the fact that it relies on a deep kernel. Related work is adequately cite, [1] and [2] may also be of interest to the authors. [1] presents a new HPO approach based on HyperBand, while the one presented in [2] is a multi-fidelity approach that creates one model that exploits all fidelities, evaluating a configuration on a specific fidelity as a single \"entity\".\n\n**Quality:** The claims are well supported, the evaluation methods and baselines are appropriate. The limitations of the proposed work are also discussed.\n\n**Clarity:** The paper is clearly written and well structured. It's easy to follow and mostly easy to understand. It could do a better job at explaining how j is maintained and updated for each configuration.\n\n**Significance:** The results are important as hyper parameter optimisation is an active area of research and given the demonstrated results of the proposed method, it is likely to have a high impact on the community. This is especially true if the source code is made available (as the authors say they will do).\n\n[1] Mendes, Pedro, Maria Casimiro, and Paolo Romano. \"HyperJump: Accelerating HyperBand via Risk Modelling.\" arXiv preprint arXiv:2108.02479 (2021).\n\n[2] Mendes, Pedro, et al. \"Trimtuner: Efficient optimization of machine learning jobs in the cloud via sub-sampling.\" 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept