PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Authors: Neeratyoy Mallik
Eddie Bergman
Carl Hvarfner
Danny Stoll
Maciej Janowski
Marius Lindauer
Luigi Nardi
Frank Hutter
Venue: NeurIPS 2023 poster (2023)
Keywords: Hyperparameter Optimization
Deep Learning
TL;DR: We propose a hyperparameter optimization algorithm tailored to the needs of deep learning researchers & practitioners that draws on expert beliefs and cheap preliminary explorations, and show its robustness and efficiency.

Abstract: Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. 
While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL.
Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations.
To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. 
Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs.

Practical Hyperparameter Optimization in the Age of Deep Learning
Neeratyoy Mallik
Edward Bergman University of Freiburg
University of Freiburg
mallikēcs feas-freiburg.de
bergmane@cs. uni meiifreibuurg de
Carl Hvarfner
Danny Stoll
Lund University
University of Freiburg stolldšcs. uni freiburguggrr de
carl. carl.hvarre www.waerrrrss...oommm....e se
Maciej Janowski
Marius Lindauer
University of Freiburg
Leibniz University Hannover
janowski@cs, uni frii-freeburg.. de
m. 1 lindauer@ai. wnw.lannovvut haanovver.... de
Luigi Nardi
Frank Hutter
Lund University
University of Freiburg
Stanford University DBtune
fh@cs, uni uni-freiburg. de
luigi www.iirrds...... 1th. se
Abstract
Hyperparameters of Deep Learning (DL) pipelines crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers. we propose Pri orBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate Pri PriorBand efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs.
Introduction
The performance of Deep Learning (DL) models erucially depends on dataset-specific settings of their hyperparameters (HPs) 11.2 Therefore, Hyperparameter Optimization (HPO) is an integral step the development DL models [1 HPO methods have typically been applied traditional machine learning models (including shallow neural networks) that focus on fairly small datasets 87] Current DL practitioners, however, often utilize much larger models and datasets (e.g., a single training of GPT-3
[9] was estimated to require 3 1023 FLOPS, i.e., months on thousand V100 GPUs 10). As recent HPO practices continue to apply simple techniques like grid, random, or manual search [[T+4]] existing HPO methods seem misaligned with DL practice.
To make the misalignment between existing HPO methods and DL practice explicit, we identify the following desiderata for an efficient, scalable HPO method suitable for current DL:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table Comparison with respect to identified desiderata for DL pipelines. Shown our algorithm, PriorBand, along with Grid Search, multi-fidelity random search (MF-RS), e.g. HyperBand, ASHA, Bayesian Optimization (BO), BO with Expert Priors (rBO), multi-fidelity BO (MF-BO), e-g. BOHB, Mobster. A checkmark and red cross indicates whether they satisfy the corresponding desider- atum. Parenthesized checkmark indicates the desideratum is partly fulfilled, or requires additional modifications to be fulfilled. Model-based methods require customized kernels for discrete hyperpa- rameters and asynchronous parallel approaches [5] to achieve speed-up under (asynchronous) parallelism. Multi-fidelity algorithms' final performance is contingent on the budget spent HPO.
Criterion
Grid Search
MF-RS BO
BO with Priors
MF-BO
PriorBand
Good anytime performance
*
Good final performance
Proxy task integration
x
Integrate expert belief's
x
Mixed search spaces Model free Specdup under parallelism
5X5
S
Language Model (Transformer 1B)
Ranks on all benchmarks
Error Rate (%)
Relative rank
3
66
2
64
1
3
5
10
1
5
7
10
Full trainings
Full trainings
Prior
HB
PriorBand
Figure 1: Both plots compare Random Search (RS), sampling from prior (RS+Prior), HyperBand (HB), and our method PriorBand which utilizes good prior as defined by an expert. [Left] Tuning a large transformer on the word benchmark. PriorBand leverages the prior, achieving strong anytime performance. [Right] Ranks on all our 12 benchmarks;
consistently ranks best.
1. Strong performance under low compute budgets: As DL pipelines become larger and more complex, only a few model training can be afforded to find performant configurations.
2. Integrate cheap proxy tasks: To save resources, cheap evaluations must be used to find promising configurations early while maintaining robustness to uninformative proxies.
3. Integrate expert beliefs: DL models often come with competitive default setting or an expert may have prior beliefs on what settings to consider for HPO. Incorporating this information is essential while maintaining robustness to such beliefs being inaccurate.
4. Handle mixed type search spaces: Modern DL pipelines tend to be composed of many kinds of hyperparameterss e.g. categorical, numerical, and log hyperparameters.
5. Simple to understand and implement: The HPO method should be easy to apply to different DL pipelines and the behavior should be conceptually easy to understand.
6. Parallelism: Scaling to modern compute resources must be simple and effective.
Existing HPO algorithms satisfy subsets of these desiderata (see Table D. For example, methods that utilize cheaper proxy tasks exploit training data subsets [6]. fewer epochs/updates [7 [8], and proxy performance predictors [99 20 Previous works also consider how to combine expert knowledge into the optimization procedure in the form of the optimal value (2], meta or transfer learned knowledge (22 configurations and regions that known to the expert have worked well previously [26 28 Nonetheless, no HPO method exists that meets all of the desiderata above, especially, simple model-free approach that can cheaper proxy tasks, letting the DL expert integrate their domain knowledge for HPO.
We propose PriorBand, the first method to fulfill all desiderata. Our contributions follows:

---

1. We are the first to develop an approach to leverage cheap proxy tasks with an expert prior imput (Section ( and we show the need beyond naive solution to do (Section 4
2. We contribute general HPO algorithm with PriorBand. fulfilling all desiderata (Table required for application to DL (Section []] also see Figure
3. We demonstrate the efficiency and robustmess of PriorBand on a wide suite of DL tasks under practically feasible compute resources (Section
4. We highlight the flexibility of the method behind PriorBand by improving different multi- fidelity algorithms and their model-based extensions (Section 7.2 & (7.3, which highlights the flexible, modular design and possible extensions of PriorBand.
We conclude, highlight limitations, and discuss further research directions in Section 10 Our code for reproducing the experiments is open-sourced at https:: /tt:://ggtttbbtt com/ //auummmmmmii /f exp
Problem statement
We consider the problem of minimizing expensive-to-evalue objective function f R, i.e., A E arg minlee f()). where a configuration À comes from a search space which may consist of any combination of continuous, discrete and categorical variables. In HPO particularly, f refers to training and validating some model based on hyperparameters defined by A.
Multi-fidelity optimization of (()) requires a proxy function, namely f(X, =). that provides cheap approximation of f at given fidelity 2, e.g., the validation loss of a model only trained for epochs. Our methodology considers a fidelity scale z E [zmin =max] such that evaluating at mmax coincides
with our true objective: f(A) = f(A, =max).
To take all desiderata into account, additionally include user-specified belief A + R. where (A)= = P (A = X*) represents a subjective probability that given configuration is optimal. Thus, the overall objective is
X E arg min f(x, Amax), guided by π m(A).
(I)
Our problem is to efficiently solve Equation T Tunder a constrained budget, e.g., 10 full trainings of a DL model. Given DL training can diverge, the incumbent return is the configuration-iide pair (A, 2) that achieved the lowest error while optimizing for Equation] T
Background
While the first target Equation in its full form, below, introduce the necessary background on methods that use either multi-fidelinn optimization or expert beliefs.
Successive Halving (SH) optimizes Equation I (sans π) as a best-arm identification problem in multi-armed bandit setting (29.. Given a lower and upper fidelity bound (2miin Zmax] and a reduction factor (17), SH discretizes the fidelity range geometrically with log-factor of 1). For example, if the inputs to SH are z [3, 81] and 17 = the fidelity levels in SH are z = {3. 9, 27, 81) with accompanying enumeration called rungs, {0, 1, 2, 3). Any < Amax is called a lower fidelity, cheaper proxy task DL training. Every iteration of each round of SH involves uniform random sampling of 17 froe configurations at the lowest fidelity or r = 0, where Smax [log, (zmax/ //in)/mmn)) After evaluating all samples, only the top-performing (1/n) configurations are retained, discarding the rest. Subsequently, these T7" /n configurations are evaluated at r = 1, or the next higher discretized fidelity. This is repeated till there is only one surviving configuration Sampling and evaluating many configurations for cheap, performing a tournament selection to retain strong samples, and repeating this, ensures that the high fidelity evaluations happen for relatively stronger configurations only. SH proves that such an early-stopping strategy guarantees a likely improvement over random search under the same total evaluation budget. Due to its random sampling component, SH is an example of MF-RS in Table [
the highest
See Appendix D.2 for an illustrative example.
HyperBand (HB) attempts to address SH's susceptibility to poor fidelity performance correla- tions (30. has the same HPs as SH, and thus given SH's fidelity discretization, HB iteratively executes multiple rounds of SH where miin is set to the next higher discrete fidelity. Using the
3

---

H3d (high cor- good prior)
H3d (high corr-bad prior)
H3d car-good prior)
H3d (low corr-bar prior)
Objective
-2
-2
6 Full Evaluations
12
12
12
Full Evaluations
Full Evaluations
Full Evaluations
Mode
Optimum
HB
HB + Prior
(B+Pri)r(500%
Figure 2: comparison of naive solutions to Equation on the 3-dimensional multi-fidelity Hart- mann benchmarks. We compare different versions of HB, utilizing different strengths of the prior distribution T(-): HB with 0% influence the prior (HB), HB with 50% sampling from the prior (Il++Prr0550%) and HB with 100% sampling from prior (HB+Prior).
example from SH. given 2 E [3, 81] and 3, one full iteration of HB corresponds to running
SH(2mmn 3, mmax 81) followed by SH(Zmin = 9, 2max 81), SH (2min 27, 2mmx = 81) and SH(Zmin 81, mmax = 81). Notably, such different instantiations of SH in HB do not share any information with each other implicitly. Each execution of SH( zmax) called an SH bracket. The sequence of all unique SH brackets run by HB in one iteration is called an HB bracket.
BBO utilizes the unnormalized probability distribution (A) in BO context accelerate the optimization using the knowledge provided by the user [28 In BBO, the initial design is sampled entirely from π until Bayesian Optimization (BO) begins, where the acquisition function a is
augmented with a prior term that decays over time t: (A) = ax)) (A) where 3 is an HP. In this work, we borrow x(.) the expert prior interface the HPO problem. TBO relies the decaying prior on the acquisition ox, (A) to gradually add more weight to the model and thus allow recovery from poor initial design under bad prior. We choose the model-free setting and adapt the strength of the prior based on the evidence of its usefulness.
4 The need to move beyond a naive solution
To solve Equation 1 an intuitive approach is to augment an existing multi-fidelity algorithm, e.g., HyperBand (HB), with sampling from the prior (--). In this section, however, we show that this naive approach is not sufficient, motivating the introduction of our algorithm, PriorBand, in Section 5
To study the naive combination of HyperBand with prior-based sampling, use the parameterized 3-dimensional multi-fidelity Hartmann benchmarks [31] (Appendix D.I.T] and compare HB with uniform sampling. HB with 100% prior-based sampling, and HB with 50% uniform and prior-based sampling (Figure 2). Unsurprisingly, 100% sampling from priors works best when the prior is helpful and not sampling from the prior works best for misleading priors. 50% random sampling works quite robustly, but is never competitive with the best approach. To achieve robustness to the quality of the prior and rival the best approach for each case, introduce PriorBand below.
PriorBand: Moving beyond a naive solution
The key idea behind PriorBand is to complement the sampling strategies used in the naive solution, uniform sampling, and prserd sampling, with a third strategy: incumbent- based sampling. Thereby, we overcome the robustness issues of the naive solutions presented in Section while still fulfilling the desideratum for simplicity. We first introduce and motivate sampling around the incumbent (Section 5.1). and then describe the ensemble sampling policy E, that combines all three sampling strategies and incorporates E,, into HB (Section 5.2) for PriorBand.
5.1 Incumbent-band based sampling strategy, X(-)
Prior Band leverages the current incumbent to counter uninformative priors while supporting good priors, as the current incumbent can be seen as indicating likely good region to sample from. Note that this view on the region around the incumbent is close to the definition of (T)) in =BO (Section

---

B). where the prior distribution encodes the expert's belief about the location of the global optima and thus good region to sample from.
To construct the incumbent-based sampler X()), we perform local perturbation of the current best configuration. Each hyperparameter (HP) is chosen with probability for perturbation. If chosen, continuous HPss are perturbed by c (0, o²). For discrete HPs, we resample with a uniform probability for each categorical value except the incumbent configuration's 's value which has a higher probability of selection, discussed further in Appendix E.2.5] For PriorBand, we fix these values at p 0.5 and σ = 0.25. This perturbation operation is simple, easy to implement, and has constant time complexity. We show ablations with two other possible local search designs in Appendix F.2.3
5.2 PriorBand: The E Ex-augmented HyperBand
PriorBand exploits HB for scheduling and replaces its random sampling component with a com- bination of random sampling, prior-based sampling, and incumbent-basee sampling. We denote the proportions these individual sampling components as Pts. Px. and py. respectively, and their combination the ensemble sampling policy (ESP) En,
Figure 3 (left) illustrates PriorBand as an extension of HB that, next to the HB hyperparameters (Emin: maax 1/, budget) accepts the expert prior as additional input and uses the ESP & in lieu of random sampling. This sampling from & illustrated in Algorithm [ Note that & has access to the optimization state (st) and can thus reactively adapt its individual sampling probabilities prs- Px. and PA based on the optimization history. We now discuss how we decay the random sampling probability Pu (Section 5.2.1 and the proportion of incumbent and prior sampling (Section 5.2.2
PriorBand
Rmin
Current HyperBand state, 8
Px 1/" Pu
Pu
Probability of choosing a strategy to draw a sample
Zanxx
P
tnp/1hg) 60 60
Probability ratios
seample
PX
budgel
HyperBand
X
Figure 3: PriorBand schema; [Left] in the base algorithm vanilla-HB, we replace the random sampling module by E,, which can interface an expert prior и and access the state of HB; [Right] & reads the state every iteration and determines the probabilities for selecting sampling strategy.
Algorithm 1 Sampling from &
Algorithm Dynamic weighting of w'
1: Input: 8, Smax: 17, observations H. prior æ Smax, 8 input from HB 3: pu = 1/(1 + 7"), Pa 11 1-pts. =
I: Input: observations H. rung r, pold
2: Smax s
2: A; = =Aiiion + top_(1/n)HH,
4: if activate conate_incumben((s.. then
3: Sx+ + Exz wi \(X,)
D m
(n 1)
5: Pr. Ps +- Algorithm 2 H. r. Pa)
4: S, + Ex, wi T(;;)
6: end if
5: PY pold SS + +S²)
7: d(-) é sample strategy by (pte. Pa. P
6: Px pold Sn/(S, + Sx)
8: X f sample from d(-)
7: return Pr. Pi
9: return A
5.2.1 Decaying proportion of random sampling
Given the premise that we should initially trust the expert prior, yet with the benefít of incorpo- rating random sampling (from Section 4. we make two additional assumptions, namely (i) we trust the expert's belief most at the maximum fidelity Zmax: and (ii) we would like to use cheaper fidelities to explore more. Given HB's discretization of the fidelity range [Zmin Zmar] into rungs r (0, (O...... .Fmaz) ) we geometrically increase our sampling probability from x (-) over U(()) by
Px 17 Pu.
(2)
5

---

Figure 4: A visual example of how configurations contribute to the scores S and Sx The prior distribution here is Gaussian (M,, o²), with matching distribution X(-) ~V o²) placed on the incumbent. The labels 1-6 indicate configurations and their rank in A. Here, e.g., X2 has densities such that x (A2) X(x2). contributing more to S than to Sx- The configurations X3, A4. Xs are between both distributions and Ås has low density under either, whereas A (the incumbent) will be the primary influence such that Sx Sx. This implies that Pa > pr and demonstrates that all else being relatively equal, we will be more likely to sample from 2(.).
This naturally captures our assumptions, cqually favouring (-)) and (1(-) initially but increasing trust in TT-) according to the rung r we sample at. This geometric decay was inspired by similar scaling used in HB's scheduling and backed by ablations over a linear decay and constant proportion (Appendix E.2.I
with the constraint that Pt + = (see L.3, Algorithm D.
5.2.2 Incumbent-based sampling proportion
under bad, unin- formative, or adversarial priors. In PriorBand, initially P1, = until both (i) budget equivalent to the first SH bracket has been exhausted (RS 1) 2mmx): ); and (ii) least configuration has been evaluated at 2max- This is captured in Line 4 of Algorithm by activate_ activate_incumbent incumbent ()
Incumbent-basse sampling intends to maintain strong anytime performance
Once incumbent-basee sampling is active we need to decide how much to trust X(-) vs. trusting (-)) Essentially, we achieve this by rating how likely the best seen configurations would be under X(-) and (t), respectively. A shows the detailed steps calculate pa and Ps. and Figure provides an example. We first define an ordered set N {\i,... }. ordered by performance, of the top-performing (1/1) configuration for the highest rung with least 7) evaluated configurations (Line 2 in Algorithm 2) Given A', we compute two weighted scores Sr capturing how likely these top configurations are under the densities of x--) and X(-), respectively. We also weigh top-performing configurations more highly. which is accomplished with simple linear mapping uy = (n + 1) (Lines 3-4 in Algorithm 2p. Finally, we obtain p= and Pş by normalizing these scores ratio of their sum (Lines 5-6 in Algorithm 2). We note the rates are adaptive based on how the set Nr is spread out relative to the prior and the incumbent. We observe that this adaptive behavior is crucial in quickly recovering from bad prior inputs and thus maintaining strong final performance.
Experimental setup
We now describe experiment design, benchmarks, and baselines and demonstrate the robustness of PriorBand in handling varying qualities of an expert prior input. Additionally, we also showcase the generality and practicality of the ESP, E
6.1 Benchmarks
We curated a set of 12 benchmarks that cover a diverse set of search spaces, including mixed- type spaces and log-scaled hyperparameters, and wide range of downstream tasks, e.g., language modeling, image classification, tabular data, a medical application, and translation. We select 4 of the PD1 benchmarks (4 HPs) 33 that train large models such transformers with batch sizes commonly found on modern hardware, and fit surrogates on them. Further, we select 5 benchmarks from LCBench (7 HPs) B3 34 and consider all 3 JAHSBench [35 surrogate benchmarks that offer 14 dimensional mixed-type search space for tuning both the architecture and training hyperparameters. All benchmarks and their selection are described in further detail in Appendix D.
We want at least configurations; hence max(n) of configurations the rung / /)) selected.

---

6.2 Baselines
We choose a representative from each of the families of optimizers listed in Table leaving out grid search. We use the official implementation of BOHB, while all other algorithms were implemented by us (and are available as part of our repository). The prior-bascd baselines (BBO and RS+Prior) sample the mode of the prior distribution as the first evaluation in our experiments to ensure fair comparison where the prior configuration is certainly evaluated, irrespective of it being good or bad. For all the HB based algorithms use 7) 3 and the fidelity bounds (Zmin= zmax) as per the benchmark. Further implementation and hyperparameter details for the baselines can be found in Appendix D.2 In principle, PriorBand only needs an additional input of iT-) as the user belief, other than the standard HB hyperparameters. However, for a discussion on the hyperparameters of the incumbent-based local search (Section 5.1 please refer Appendix E.2.5
6.3 Design and setup
We show experiments both for single and multi-worker cases (4 workers in our experiments). For the single workers, we report the mean validation error with standard error bars for 50 seeds; for multi- worker experiments, we use 10 seeds. The plots show average relative ranks achieved aggregated across all 12 benchmarks when comparing the anytime incumbent configuration at 2max- We group the runs on benchmark under different qualities of an expert prior input. We also compare the average normalized regret per benchmark under good priors. The prior construction procedure follows the design from the work by Hvarfner al. [28), and we describe our procedure in detail in Appendix [33] In the main paper, we only evaluate the good and bad prior settings as we believe that this reflects practical setting of prior qualities; in Appendix Dee also evaluate the near-optimal prior settings and also evaluate all 3 prior settings over high and low-perfone now-performance correlation problems. In Appendix F.5) we also report additional experiments for different budget horizons. Further experiment design details can be found in Appendices D.55 As an example of potential post-hoc analysis possible with PriorBand, we show how the dynamic probabilities calculated for each of the sampling strategies in PriorBand can be visualized (Appendix E.3) revealing the quality of the prior used.
7
Results
We now report and discuss the results of our experiments.
7.1 Robustness of PriorBand
Here, we demonstrate the robustness of PriorBand over wide range of prior qualities by comparing them to the nearest non- con-prior-based algorithms: random search (RS) and HyperBand (HB). Fig- ure3 showcases PriorBand be anytime equal better than HB on each of our benchmarks under our good prior design. Note that the quality of a good prior varies per benchmark. The aggre- which illustrates that PriorBand (HB+E,) can utilize good priors and gain strong anytime performance over HB. Moreover, Figure biitttom-righh) clearly shows PriorBand's 's ability to recover from bad prior information and match vanilla-HB's performance in this adversarial case (the bad prior was intentionally chosen to be an extremely poor configuration the worst of 50k random samples). In most practical scenarios, the expert has better intuition than this setup and one can expect substantial speedups over HB, like in Figure 5 middle). Figure [5 bottomm-llff demonstrates that when using an unknown quality of prior, even including the adversarial prior, Prior is still the best choice on average. We show budget of 12 function evaluations here, which is approximately the maximal budget required for completing at least one HB iteration for the benchmarks chosen. In Appendix F. we show more comparisons to prior-based baselines and highlight PriorBand's robustness.
gation of Figure 5 top) is
7.2 Generality of Ensemble Sampling Policy &
The ESP, &,, only needs access to an SH-like optimization state as input. In this section, we show that other popular multi-fidelity algorithms, such as ASHA or asynchronous-HIB (36) that build on SH, can also support expert priors with the help of & In Figure [ we compare the vanilla algorithms with their Ex -aggmeentdd versions. Given that these algorithms were designed for parallel setups,
7

---

[+]PD] WMT (good)
LLCBench-18990 (good)
[+]CIFARI00 (good)
[+]mmagenet (good)
Nomalized Regret
10"
10a
10a
10
10
10
10-)
3
J.CBench-168330 (good)
[+ 1-IDDD-Lmib (good)
-LLLBBncc-1ee99 1e99nch-168910111 (good)
ILC3ench-167190 (good)
Normalzee Sodnalized Regret
10
109
10
10
ww.com
10-1
12
3
[CIAAR 10 (gond)
Colorctaal Histalogy (gead)
CCench 126026 (goad)
Faasioo MNIST (good)
Normalized Regret
100
10
10
10
10-)
10-1
10-1
10
12
Full trai ni ngs
trainings
Full traininos
Full trainings
Mode
RS
HB
PriorBand
All
Good prior
Bad prior
Relative rank
2
3
9
12 0
6
12
3
12
Full trainings
Full trainings
Full trainings
RS
HB
PriorBand
Figure 5: [TTpp] Comparing normalized regret in the good prior setting; The [Bottom] figure Good (middle) is ranking aggregate view of the [Top] figure. [Bottom] Comparing average relative ranks of FrriorBand to Random Search (RS) and HB, over single worker runs across benchmarks. Each benchmark-alge aalgoran- pair was run for 50 seeds where priors per benchmark are the same across seed. We show mean rank and standard error. The All (left) plot averages the benchmark across Good and Bad priors
compare them evaluations. Similar to the previous section, the FSP-augmented (+ESP) algorithms can leverage good priors and recover under bad priors. Under bad priors, asynchronous-HBB (+ESP) starts worst owing to bad prior higher fidelity evaluations at the start but shows strong recovery. In Appendix F2 we compare these variants over different correlation settings of the benchmarks.
distributed among 4 workers, running for total budget of 20 function
All
Good prior
Bad prior
Relative rank
-3
3
2
2
3
5
0
2
3
5
Full trainings
trainings
Full trainings Async-HB(+ESP)
ASHA
AsyncHB
ASHA(+ESP)
Figure Comparing the average relative ranks of Asynchronous-SH (ASHA) and Asynchronous-HB (Async-HB) and their variants with the Ensemble Sampling Policy (+ESP) when distributed over 4 workers for a cumulative budget of 20 function evaluations.

---

7.3 Extensibility with models
Although our focus is the model-free low compute regime, we also show how PriorBand can be optionally extended with model-based surrogates to perform Bayesian Optimization, especially when longer training budgets are available. We compare BO, TBO, BOHB, and PriorBand with its model-extension, Prior Band+BB (Gaussian Processes as surrogates and Expected Improve- ment (37 for acquisition). All BO methods use an initial design of 10 function evaluations, except BOHB which sets this implicitly Ndim + 2. Compared to other prior-based algorithms (RS+Prior, BOO) PriorBand+BO)
consistently robust on average (All) under all priors. The anytime per- formance gain of Pri PriorBand+BO under good priors is evidence of ESP & 's utility. We note that PriorBand+BO recovers quickly in the bad prior setting and even outperforms all other algorithms af- ter as little budget as 6 full trainings. The fact that, until 10x function evaluations, PriorBand+BD actually just PriorBand and model search has not begun, highlights the effectiveness of ESP in adaptively trading off different sampling strategies and the initial strength of incumbent sampling. Appendices
F3 contain details on our modeling choices and more experimental results.
Good prior
Bad prior
Relative rank
-
2
2
1
5
10
15
20
0
10
15
20
5
10
15
20
Full trainings
Full trainings
Full trainings
PSior
nBO
BOHB
PriorBand+ +DO
Figure 7: Comparing the average relative ranks of model-based algorithms and sampling from the prior (RS+Prior) under single-worker runs, aggregated over all benchmarks-per-prio. quality.
7.4 Ablation studies
In Figure 8 compare choice of using density scores (Section 5.2.2
trade-off prior- based
and incumbent- based sampling against other similar or simpler heuristics, validating our decision. PriorBand(constant employs fixed heuristic trade off prior and incumbent-bisg based sampling as = 1)' Px- In Pri (dier Pa is decayed as a function of iterations completed. More precisely, p3 = 2b. subject to - Ps + the index of the current SH being run HB. Appendix E.2.2 contains more ablations.
(from Section 5.2) where b E N and indicates
Good prior
Bad prior
3
Relative rank
2-
2
1
5
15
0
10 Full trainings
20
5
15
Full trainings
Full trainings
(riorBand(constant))
PriorBand(decay)
PriorBand
Figure 8: A comparison of Priorrann with different strategies for trading off incumbent vs. prior distribution sampling. PriorBand's default of using density scores shows dominant performance in almost all scenarios. However, a bad prior induces a marked difference in performance. In general, the plot highlights that each variant reacts differently to bad priors. The adaptive nature of Pri orBand clearly is robust to different scenarios comparatively.
8
Related work
While using expert priors and local search (38 for hyperparameter optimization has been explored previously, few works considered priors over the optimum [27] 28 39], and they all target the

---

single-fideliy setting. The expert priors we consider should not be confused with the priors natively supported by BO, priors over the function structure determined by kernels 40 41
In deep learning, training epochs and dataset subsets 17 42
are frequently used as fidelity variables to create cheap proxy tasks, with input resolution, network width, and network depth also occasionally used [35]. SH (2 and HB [30] are effective randomized policies for multi-fidelity HPO that use early stopping of configurations on a geometric spacing of the fidelity space and can
also be extended to the model-based setting 441
Limitations
We acknowledge that PriorBand inherits pathologies of HB such as poor fidelity correlations and can be sensitive to the choice of fidelity bounds and early stopping rate. Despite that, our results across different correlation settings suggest a relatively strong performance of PriorBand (Appendix (F). Though PriorBand supports any kind of prior distribution as input, in our experiments we only considered the Gaussian distribution (with fixed standard deviation of 0.25), as it is a natural choice and was used previously in the literature (28). However, the expert free to represent the prior with other distributions. We expect PriorBand to show similar behaviors we report when comparing to other prior-based algorithms under a similar distribution. We note that PriorBand is not entirely free of hyperparameters. In our experiments, remove confounding factors while comparing different prior strengths and correlations. Moreover, hyperparameter choices for the experiments, such 7 3 and Gaussian priors are largely borrowed from existing literature [28 36 (44. Depending on the context of the specific deep learning expert, the compute budgets (~ 10 12 full trainings) used in our experimental setting may not always be feasible. However, the key insight from the experiments is that PriorBand can interface informative priors, and provide strong anytime performance under short HPO budgets (also in less than 5 full trainings). Longer HPO budgets for PriorBand ensure recovery from potential bad, uninformative priors. In practice, deep learning experts often have good prior knowledge and thus PriorBand is expected to retain strong anytime performance under low compute.
keep all PriorBand hyperparameters fixed to
10 Conclusion
We identify that existing HPO algorithms are misaligned with deep learning (DL) practice and make this explicit with six desiderata (Section[]) [T]. To overcome this misalignment, our solution, PriorBand, allows a DL expert to incorporate their intuition of well-performing configurations into multi-fidelity HPO and thereby satisfies all desiderata to be practically useful for DL. The key component in PriorBand, the ESP, &,, is modular, flexible, and can be applied to other multi-fidelity algorithms.
11 Acknowledgments and disclosure of funding
N. Mallik, D. Stoll and F. Hutter acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. E. Bergman was partially supported by TAILOR, project funded by EU Horizon 2020 research and innovation programme under GA No 952215. C. Hvarfner and L.. Nardi were partially supported by the Wallenberg AI, Autonomous Sytems and Software Program (WASPP funded by the Knut and Alice Wallenberg Foundation. L. Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348. M. Lindauer was funded by the European Union (ERC, "ixAutoML", grant no. 101041029). The research of Maciej Janowski was supported by the Deutsche Forschungsgemeinnccaaft (DFG, German Research Foundation) under grant number 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, Maciej Janowski acknowledges the BrainTook Center of Excellence.
support of the BrainLinks-
Funded by the European Union
10

---

References
[1] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.
[2] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference Learning Representations (ICLR' 19), 2019, Published online: iclr.cc
[3] Y. Chen, Huang, Z. Wang. I. Antonoglou. J. Schrittwieser, D. Silver, and N. de Freitas. Bayesian optimization in alphago. 1rXiv:181.00885: [cs.LG], 2018.
[4] G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In Proceedings of the International Conference Learning Representations (ICLR' 18), 2018. Published online: iclr.cc
[5] B. Zhang, R. Rajan, Pineda, N. Lambert, A. Biedenkapp. K. Chua, Hutter, and R. Calandra. On the importance of Hyperparameter Optimization for model-based reinforcement learning- In A. Banerjee and K. Fukumizu, editors, Proceedings of the 24th International Conference on Artificial Intelligence and Statisties (AISTATS 21), pages 4015-4023. Proceedings of Machine Learning Research, 2021.
[6] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. In Ranzato et al. 45
[7] Snoek, Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett. F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Pro- ceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 12), pages 2960-2968. Curran Associates, 2012.
[8] M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-Sklean 2.0: Hands- free automl via meta-learning. JMLR, 23(261):1-61, 2022.
[9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Larochelle et 446] pages 1877-1901.
[10] Lambda. Demystifying gpt-3. httper//aaakalaasssccc/o//leeegs June 2020. Accessed on May 4, 2023.
[11] X. Bouthillier and G. Varoquaux. Survey of machine-leaming experimental methods NeurIPS2019 and ICLR2020. Research report [hal-02447823), Inria Saclay Ile de France, 2020.
[12] N. Hasebrook, F. Morsbach, N. Kannengielser, M. Zöller, J. Franke, M. Lindauer, F. Hutter, and A. Sunyaev. Practitioner motives to select hyperparameter optimization methods. arXiv preprint 2rXiv:2203.01717, 2022.
[13] F. Schneider, Z. Nado, N. Agarwal, G. E. Dahl, and P. Hennig. Hity workshop poll, neurips 2022.
2022. https////gttaaaaccco////cccceeee
[14] K. Van der Blom, A. Serban, H. Hoos, and J. Visser. Automl adoption in ml software. In K. Eggensperger, F. Hutter, E. LeDell, M. Lindauer, G. Shala, J. Vanschoren, and C. White, editors, ICML workshop Automated Machine Learning (AutoML workshop 2021), 2021.
[15] Javier Gonzalez, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages -48-657, Cadiz, Spain, 09-11 May 2016. PMLR. URL https://proceediin mlr prwss/ v51 http///onzaeeze.... presss/vlllgmmzalee1.... html

---

[16] A. Klein, S. Falkner. S. Bartels, P. Hennig, and F. Hutter. Fast Bayesian optimization of machine learning hyperparameterss on large datasets. In A. Singh and Zhu, editors, Proceedings of the Seventeenth International Conference Artificial Intelligence and Statistics (AISTATS volume 54. Proceedings of Machine Learning Research, 2017.
[17] K. Swersky, J. Snoek, and R. Adams. Freeze-thaw Bayesian optimization. arXiv:1406.3896 MLL, 2014.
[18] Domhan, Springenherg. and F. Hutter. Speeding up automatic Hyperparameter Optimiza- tion of deep neural networks by extrapolation of learning curves. In Q. Yang and M. Wooldridge, editors, Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJ-
CAI'15), pages 3460-3468, 2015.
[19] C. White, A. Zela, B. Ru, Y. Liu, and F. Hutter. How powerful are performance predictors in neural architecture search? In Ranzato et al. [45).
[20] M. Abdelfattah, A. Mehrotra, L. Dudziak, and N. Lane. Zero-cost proxies for lightweight NAS.
In Proceedings of the International Conference on Learning Representations (ICLR'21) 12), 2021.
[21] V. Nguyen and M. A. Osborne. Knowing the what but not the where in bayesian optimization. In H. Daume III and A. Singh, editors, Proceedings of the 37th Internations Conference on Machine Learning (ICMI. ((C),, volume 98, pages 7317-7326. Proceedings of Machine Learning Research, 2020.
[22] M. Feurer, J. Springenberg, and F. Hutter. Initializing Bayesian Hyperparameter Optimization via meta-learning. In B. Bonet and S. Koenig, editors, Proceedings of the Twenty-ninth AAAI Conference on Artificial Intelligence (AAAI' 15), pages 1138-1135. AAAI Press, 2015.
[23] V. Perrone and H. Shen. Learning search spaces for bayesian optimization:
Another view of hyperparameter transfer learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alche Buc, E. Fox, and R. Garnett, editors, Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurIPS' 19), pages 12751-12761. Curran Associates, 2019.
[24] M. Feurer, B. Letham, F. Hutter, and Bakshy. Practical transfer learning for bayesian optimization. 1802.022 19v3 [stat.ML]. 2018.
[25] M. Wistuba and J. Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In Proceedings of the International Conference on Learning Representations (ICLR" (CCLR221), 2021.
[26] D. Stoll, J. Franke, D. Wagner, S. Selg, and F. Hutter. Hyperparameter transfer across developer adjustments. In Larochelle et al. [46].
[27] A. Souza, I.. Nardi, L. Oliveira, K. Olukotun, M. Lindauer, and F. Hutter. Bayesian optimization with a prior for the optimum. In N. Oliver, F. Pérez-Cruz, S. Kramer, J. Read, and J. Lozano, editors, Machine Learning and Knowledge Discovery in Databases (ECMIJPKDID) 21), volume 12975 of Lecture Notes in Computer Science, pages 265-296. Springer-Veriee Verlag, 2021.
[28] C. Hvarfner, D. Stoll, A. Souza, L. Nardi, M. Lindauer, and F. Hutter. =BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In Proceedings of the International Conference on Learning Representations (ICLR 22), 2022. Published online:
iclr.cc
[29] K. Jamieson and A. Talwalkar. Non-stochastic best arm identification and Hyperparameter Optimization. In Gretton and C. Robert, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS' 16)) volume 51. Proceedings of Machine Learning Research, 2016.
[30] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to Hyperparameter Optimization. JMLR, 18(185):1-52, 2018.
[31] K. Kandasamy, G. Dasarathy, J. Schneider, and B. Póczos. Multi-fidelity Bayesian Optimisation with Continuous Approximations. In D. Precup and Y. Teh, editors, Proceedings of the 34th International Conference on Machine Learning (ICMI. 17), volume 70, pages 1799-1808 Proceedings of Machine Learning Research, 2017.
12

---

[32] Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee. Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization, 2021. URL https://arxivv .rg. httg/abb/21109.00222
[33] Lucas Zimmer. Marius Lindauer, and Frank Hutter. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079 3090, 2021.
[34] Pfisterer, L. Schneider, J. Moosbauer, M. Binder, and B. Bischl. YAHPO Gym - design criteria and new multifidelity benchmark for hyperparameter optimization. arXiv:2109.03670 [cs.LG], 2021.
[35] A. Bansal, D. Stoll, M. Janowski, A. Zela, and F. Hutter. JAHS-bench-201: A foundation for research on joint architecture and hyperparameter search. In Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22 2022.
[36] L. Li, K. G. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-tzur, M. Hardt, Recht, and A. Tal- walkar. A system for massively parallel hyperparameter tuning. In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems 2. volume 2, 2020.
[37] D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black box functions. JGO, 13:455-492, 1998.
[38] Q. Wu, C. Wang, and S. Huang. Frugal optimization for cost-related hyperparameters. In Q. Yang, K. Leyton-Brown, and Mausam, editors, Proceedings of the Thirty-Fifth Conference on Artificial Intelligence (AAAI'21). Association for the Advancement of Artificial Intelligence, AAAI Press, 2021.
[39] Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Proceedings of the 24th International Conference on Advances Neural Information Processing Systems Curran Associates, 2011.
(NeurIPS' 11), pages
[40] K. Swersky, J. Snoek, and R. Adams. Multi-task Bayesian optimization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Proceedings of the 26th International Conference Advances in Neural Information Processing Systems (NeurlPS 13), pages 2004- 2012. Curran Associates, 2013.
[41] C. Oh, E. Gavves, and M. Welling. BOCK International Conference on Machine Learning, pages 3865-3874, 2018.
Bayesian optimization with cylindrical kernels. In
[42] A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian Hyperparameter Optimization on large datasets. Electronic Journal of Statistics, 11:4945-4968 2017.
[43] V. Nguyen, S. Schulze, and M. Osborne. Bayesian optimization for iterative learning. In Larochelle et al. [46].
[44] S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML'I8) volume 80, pages 1437-1446. Proceedings of Machine Learning Research, 2018
[45] M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors. Proceedings the
the 34th International Conference on Advances Neural Information Processing Systems (NeurIPS'2I), 2021. Curran Associates.
[46] H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balean, and H. Lin, editors. Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS' 20), 2020. Curran Associates.
[47] I.. Zimmer, Lindauer, and F. Hutter. Auto-Pytorch: Multi-fidelity metalearning for efficient and robust Au. TPAMI, pages 3079-3090, 2021.
13

---

[48] A. Falkner, G. Friedrich, K. Schekotihin, R. Taupe, and E. Teppan. Industrial applications of answer set programming. KI- Künstliche Intelligenz, pages 1-12, 2018.
[49] X. Dong, M. Tan, A. Yu, D. Peng. B. Gabrys, and Q. Le. Autohas: Differentiable hyper- parameter and architecture search. FrXx::2006.03636 (cs.CV) 2020.
[50] Justin M. Gilmer, George E. Dahl, and Zachary Nado. init2winit: a jax codebase for initialization, optimization, and tuning research, 2021. URL httt:://githuub.com/gooo//
t2winit.
[51] Y. Nesterov. A method of solving a convex programming problem with convergence rate O(1/sqr(k)) Soviet Mathematics Doklady, 27:372-376, 1983.
[52] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content- based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 98, 2021.
[53] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. One billion word benchmark for measuring progress in statistical language modeling, CoRR, abs/1312 131.3035 2013. /att://////rx org abs/ 11///3112..3 3005
[54] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren. Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https://g/t gighub...... com/
facebookreesearch/zooooooo ormers, 2022.
[55] Ondřcj Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop on statistical machine translation. Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1 1-46, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL http::/aclee org/ http///ttoooooo11--11eo
[56] Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.0714. 2016.
Wide residual networks. arXiv preprint
[57] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[58] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR'16,, pages 770-778. CVF and IEEE CS, IEEE, 2016.
[59] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 115(3):111-252, 2015.
[60] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, and R. Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD' pages 785-794. ACM Press, 2016.
[61] N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient Hyperparameter Optimization. In Zhou, editor, Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI'21, pages 2147-2153, 2021.
[62] Siems, L. Zimmer, A. Zela, J. Lukasik, M. Keuper, and Hutter. NAS-bench-301 and the case for surrogate benchmarks for Neural Architecture Search. arXiv:2008.09777> [cs.LG], 2020.
[63] B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. N. van Rijn, and J. Vanschoren. Openml benchmarking suites and the openml100. 7rX1vv/708.0373lv/ [stat.M.... 2019.
14

---

[64] J. Kiili, E. Laaksonen, R. Turner, D. Eriksson, S. Park, M. Mccourt, Z. Xu, and I. Guyon. Black box optimization challenge. 2020.
[65] A. Klein, L. Tiao, Lienart, C. Archambeau, and M. Seeger. Model-based Asynchronous Hyperparameter and Neural Architecture Search. arXiv:2003. 10865 [cs.LG], 2020.
[66] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carle Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL
http://arxiv.oorgaaaa 1tt.s./991...6601. 06403
[67] B. Matern. Spatial Variation. Lecture Notes in Statistics Springer New York, 2013. ISBN 978 1461578925.
[68] Frank Hutter, Lin Xu, Holger Hoos, and Kevin Leyton-Brown. Algorithm runtime prediction: Methods & evaluation, 2013.
15


Reviews:
[{"review_id": "pksgN7xEu4", "reviewer": "Reviewer_ejik", "summary": "This paper presents PriorBand, a hyperparameter optimization (HPO) algorithm designed specifically for deep learning models. PriorBand fulfills six key requirements and tackles the shortcomings of existing HPO methods that are unsuitable for DL. It leverages cheap proxy tasks while considering expert input. The algorithm eliminates the need for a naive solution for integrating expert domain knowledge into HPO. Experiments across a wide array of DL tasks are conducted to demonstrate the efficiency and robustness of PriorBand.", "questions": "See Weakness.", "limitations": "See Weakness.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The proposed algorithm is simple yet effective. The augmented prior based on the existing HyperBand algorithm is technically sound and exhibits clear improvement over baselines.\n\n+ The paper is well-written and well-structured. Motivation is clearly stated before the introduction of detailed algorithms (e.g., Sect. 4). ", "weaknesses": "- Though the experiments conducted in the paper are quite comprehensive, I found most of them are moved to the appendix where the figures in the main paper are of limited information (most are about robustness to bad priors). I would suggest authors rearrange the paper by moving figures from Appendix F to the main text.\n\n- In Table 1, given the fact that all the baseline methods satisfy \"Mixed search spaces\" + \"Speedup under parallelism\", I would suggest the authors remove these two rows as they are also not the major technical contributions of the proposed method. Maybe instead, replace with one sentence in the caption highlighting both the baselines and the proposed method can satisfy these two criteria.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank the reviewer for their kind remarks of appreciation for the method itself and we are delighted to read that the effort put into the presentation, structure, and motivation made the paper clear.\n\n---\n> Though the experiments conducted in the paper are quite comprehensive, I found most of them are moved to the appendix where the figures in the main paper are of limited information (most are about robustness to bad priors). I would suggest authors rearrange the paper by moving figures from Appendix F to the main text.\n---\n\n1.\n  **a)** We appreciate that the reviewer took the time to read our Appendix and we agree with the reviewer's remarks at large. We decided on a set of plots aggregated across tasks in the main paper, that are focused on the primary message of how the Ensemble Sampling Policy (ESP) improves performance. For the camera-ready, we plan to move _Fig. 21_ and _Fig. 22_ to the main paper.\n\n  **b)** As an alternative, we now also created average normalized regret plots which we show on the _additional page_ we can upload during the rebuttal. They could substitute average relative ranks as an alternative way of communicating aggregated results over benchmarks.\n\n  We kindly request the reviewer to check these plots to see if they are a more suitable set of visualizations for the final main draft.\n\n---\n> In Table 1, given the fact that all the baseline methods satisfy \"Mixed search spaces\" + \"Speedup under parallelism\", I would suggest the authors remove these two rows as they are also not the major technical contributions of the proposed method. Maybe instead, replace with one sentence in the caption highlighting both the baselines and the proposed method can satisfy these two criteria.\n---\n\n2. Thank you for pointing this out, it is indeed correct. However, we would like to clarify that we do not intend the desiderata to be technical contributions, but an overview of the HPO approach landscape. In fact, in terms of technical contributions, the Ensemble Sampling Policy, which is our major contribution, improves both anytime performance and final performance and can therefore not be isolated into any one of the 7 criteria in Table 1. In the end, we believe users of an HPO system are agnostic to *how* a desideratum such as efficiency is fulfilled, as long as it is indeed fulfilled.\n\n\n### ___\n\nWe value the reviewer's input and recommendations and invite the reviewer to explore the additional PDF we've provided. We are happy to address additional questions to increase your confidence in the review.\n"}, {"review_id": "CniFWxw6Ds", "reviewer": "Reviewer_QGiN", "summary": "In the paper \"PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning\" the authors propose an extension to the well-known HPO methods Hyperband by adapting the way how candidate hyperparameter settings are sampled. To this end, the authors propose to use a weighting mechanism to balance between three sampling distributions: random, locally random close to the incumbent, and a prespecified prior. The latter allows experts to inject beliefs about the optimum. In the empirical study PriorBand is found to perform best on average among the considered methods.", "questions": "- What is the individual effect of incumbent-based and prior-based sampling?\n- What is the impact on the theoretical guarantees for Hyperband? Are all the assumptions still fulfilled?\n- What is the definition of \"relative rank\"? Is it only a rank or does it also include performance differences?\n- Where do these desiderata stem from? Although they all seem intuitive it is not clear whether they are exhaustive and what the coverage is like.", "limitations": "Limitations of PriorBand are well discussed. However, it is not entirely clear to what extent the desiderata for HPO in the age of deep learning is limited or maybe even exhaustive. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "+ PriorBand compares favorably to its competitors and allows for hyperparameter optimization with comparably low budgets.\n+ The paper is very well written and the presentation in general is excellent.\n+ The authors put very much effort into making their work reproducible, openly accessible and easy to understand.\n+ For the self-imposed desiderata, PriorBand offers the most desired properties.", "weaknesses": "- The work is pretty incremental. The only original contribution of this work is to weight the different probability distributions from which candidates are sampled.\n- It is not clear what effect the incumbent-sampling has on the overall performance. At least I could not find any ablation studies in this regard. Is it really necessary to include incumbent-sampling? What would happen if only random sampling is balanced against prior-sampling? Is it then harder to retain decent performance in the light of bad priors?\n- Nothing is stated about the impact on theoretical guarantees that are known to hold for Hyperband. Does the theoretical framework of Hyperband still apply for the changed distributions?\n- When comparing different hyperparameter optimizers only relative ranks are ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and for appreciating the strong performance of PriorBand, our paper presentation and our efforts to make research open-source and reproducible.\n\n---\n> What is the individual effect of incumbent-based and prior-based sampling?\n---\n1.\n  **a)** We apologize that this was not sufficiently clear from the main paper. We actually do report multiple ablation studies on the individual components of PriorBand in the appendix. The incumbent-based ablation can be found in _Appendix E.2, Fig. 17_, showing that, for any prior quality and fidelity correlation, the incumbent-based sampling improves performance. _Appendix E.3_, _Fig. 19_ also highlights the role of how the weights of each sampler adapt to lend robustness to PriorBand. \n\n  **b)**  The effect of prior-based sampling can be found in _Fig. 1_ and _Fig. 2_, (as a substitute to RS) and in _Fig. 5_ (the effect of good/bad priors on the outcome of optimization) with complementary results in _Fig. 21_ and _Fig. 22_. While a bad prior naturally degrades performance initially, its long-run impact is negligible compared to RS.\nIf the reviewer has additional ablations in mind, we would be happy to add them to the camera-ready.\n\n---\n> The work is pretty incremental. The only original contribution of this work is to weight the different probability distributions from which candidates are sampled.\n---\n2. While your observation holds merit, we believe our contribution facilitates new directions in hyperparameter optimization (HPO), adding novelty to the existing literature\n\n    Our work is the first, to our knowledge, to empower experts in enhancing multi-fidelity optimization through their knowledge and intuition. It's principled, well-motivated, with thorough ablations, enabling effective model-based extensions. Based on the gap it identifies and fills, we believe our work is novel.\n\n    We would also like to mention that the established HPO algorithms we compare to may be considered to be incremental: ASHA is a simple if-condition different from SuccessiveHalving, BOHB is Bayesian Optimization (BO) as a sampler in HyperBand (HB), etc. However, they all provide unique benefits and are extensively used algorithms.\n\n    PriorBand adds to this list by uniquely supporting expert priors, additionally empowering _all_ such algorithms to interface with expert priors through our contribution of the Ensemble Sample Policy (ESP), making our approach novel in scope, design, and application. Moreover, its robustness and simplicity make it extremely practical - a largely overlooked criterion in the BO/HPO community.\n\n---\n> Nothing is stated about the impact on theoretical guarantees that are known to hold for HB. Does the theoretical framework of HB still apply to the changed distributions?\n\n> What is the impact on the theoretical guarantees for HB? Are all the assumptions still fulfilled?\n---\n3.\n  **a)** Thank you, we would like to clarify that PriorBand ensures a constant proportion of random sampling at the highest fidelity (_Eq. 2_) and thus can trivially be proven to be no more than a constant factor worse than HB in the worst case. We did not add this theorem due to its simplicity and since the important practical reductions of anytime regret over HB come from the local search component and the user prior, NOT through random sampling, any theoretical results for these speedups remain elusive.\nWe are hopeful that we can apply a multi-armed bandit formulation to the selection of sampling ratios in the future to obtain a regret bound compared to the best of the candidate sampling strategies.\n\n  **b)** While it's feasible to apply HB's multi-armed bandit formulation to PriorBand by considering the local search as a form of exploitation, we believe this topic deserves its own dedicated paper. As such, we consider it as potential future work.\n\n---\n> When comparing different hyperparameter optimizers only relative ranks are _(used?)_\n\n> What is the definition of \"relative rank\"? Is it only a rank or does it also include performance differences?\n---\n4.\n  **a)** Our ranking procedure is aggregated to show robustness across benchmarks and is described in detail in _Appendix D.5_.\n  \n  We agree that ranking plots only show part of the picture and therefore have now also uploaded plots showing average normalized regret in our _rebuttal PDF_, whose conclusions remain the same.\n\n  **b)** It ranks algorithms based on their best solutions at a given time. We compare these rankings across benchmarks for one seed, average across benchmarks for robustness, and repeat across seeds to calculate mean and standard errors for overall ranking.\n---\n> Where do these desiderata stem from? Although they all seem intuitive it is not clear whether they are exhaustive and what the coverage is like.\n\n> ...it is not entirely clear to what extent the desiderata for HPO in the age of deep learning is limited or maybe even exhaustive.\n---\n5. Our desiderata are extending the desiderata stated in the influential paper on BOHB, Falkner et al. [1] which combines BO and HB, in combination with recent insights [2, 3] that manual search is preferred to (more sophisticated) HPO. When HPO is adopted, simple, yet effective, algorithms like HB are preferred. As such, “Expert Beliefs” and “Simplicity” were well-grounded additions to that list. We believe our list is thorough, but we invite the reviewer to suggest anything that might be missing to make it truly exhaustive.\n\n### ___\nWe'd appreciate knowing if our responses have satisfied you and if our clarified perspective might improve your evaluation. Any further questions, comments or suggestions for enhancing the paper are most welcome.\n\n### References:\n\n[1] Falkner et al., BOHB: Robust and Efficient Hyperparameter Optimization at Scale, 2018\n\n[2] Bouthillier and Varoquaux, Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020, 2020\n\n[3] Schneider et al., HITY workshop poll, NeurIPS, 2022\n"}, {"review_id": "WzSneyWKwI", "reviewer": "Reviewer_zHth", "summary": "This paper proposes PriorBand, an extension of HyperBand that adds expert priors and a novel sampling technique to replace random sampling in HB, called the Ensemble Sampling Policy (ESP). The ESP allows the algorithm to lean on the expert prior, but also use the current incumbent in case the prior is non-optimal. Under good and bad priors, the authors demonstrate the efficacy of this approach, showing additionally that the ESP also improves the performance of other Successive Halving (SH)-based methods.\n\n**Rebuttal:** The authors have clearly addressed my comments in their rebuttal. I have updated my score from 5 to 7.", "questions": "- In Alg. 2, L5, can you explain why $p_\\pi$ is used instead of $p_{\\hat{\\lambda}}$?\n- In Fig. 14, it seems the lines have confidence intervals. How are these computed, and how many repeats of the experiment were used?\n- In Appendix D.3, the authors describe the way a \"good\" prior is generated, using the best of 25 configurations. In practice, this would add computational overhead to the overall process of training a model with good hyper-parameters. In that sense, perhaps it should be emphasized that the Bad prior results are more useful to a practitioner, who, following their intuition/expertise/random choice, might possibly use a non-optimal prior (i.e., without running anything). Can you comment on this?", "limitations": "The authors have addressed the limitations of their work appropriately in the main text and the supplementary materials.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- My favorite part of this paper is Section 7.2, which shows that the ESP proposed in this paper extends to other SH-like HPO methods as well. This contribution should perhaps be highlighted more.\n- The paper's verbiage is very clear, and examples presented in the introductory sections are useful aids to readers who may not be familiar with HyperBand's workings.", "weaknesses": "- My major concern is in the use of average relative rank to demonstrate efficacy. While a lower rank indicates better performance, it does not indicate whether that performance delta is statistically significant. As opposed to confidence intervals over *ranks*, I would much rater see the actual improvement in per-dataset metrics. For example, the original HyperBand paper [1] showed the test error over wall time for multiple HPO algorithms.\n- minor: In L130, you're missing a period.\n- L182: typo: modelling --> modeling\n\n[1] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research, 18(1), 6765-6816.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad to hear that the reviewer appreciated our presentation and found a favorite part too! Thank you for your review and comments.\n\n---\n> My major concern is in the use of average relative rank to demonstrate efficacy. While a lower rank indicates better performance, it does not indicate whether that performance delta is statistically significant. As opposed to confidence intervals over ranks, I would much rather see the actual improvement in per-dataset metrics. For example, the original HyperBand paper showed the test error over wall time for multiple HPO algorithms.\n---\n\n1. Thank you for raising this important point. We agree that relative ranks only show part of the picture, and we used it throughout for ease of aggregation across far more benchmarks than, e.g., the Hyperband paper, for additional robustness of results. Aggregating raw performance across benchmarks has to be done carefully to avoid individual benchmarks being over-represented due to different scales of performances; we now did this using average normalized regret. Fig. 1 and Fig. 2 in the _rebuttal PDF_ include per-dataset regret plots for good priors and average normalized regret for different prior strengths, respectively. Due to page restrictions, in the main paper, we can only include either the aggregated average rank or the average regret. Your suggestion on how to effectively communicate and persuade the reader with Appendix support is welcome, but we will make sure to add regret plots in a camera-ready version, including a pair (good-bad priors) of per-dataset normalized regret plots.\n\n---\n> In Alg. 2, L5, can you explain why $p_\\pi$ is used instead of $p_{\\hat{\\lambda}}$?\n---\n\n2. We acknowledge any potential confusion that may have arisen from our use of $p_\\pi$ in Alg. 2. We shall rename $p_\\pi$ in L1 and RHS of L5-6 in Alg. 2 to $p_\\pi^{\\mathrm{old}}$ in the camera-ready. \n\n     L3 in Alg. 1 computes the probability of sampling uniformly random and the remaining probability is assigned to $p_\\pi$. The role of Alg. 2 now (taking the current $p_\\pi$ as input) is to split this probability between the prior sampler and the incumbent-based sampler. L5 in Alg. 2 thus assigns the determined proportion of this probability to $p_{\\hat{\\lambda}}$ and in L6 $p_\\pi$ is updated as well.\n\nPlease let us know if this clarifies the confusion or whether you have any further questions.\n\n\n---\n> In Appendix D.3, the authors describe the way a \"good\" prior is generated, using the best of 25 configurations. In practice, this would add computational overhead to the overall process of training a model with good hyper-parameters. In that sense, perhaps it should be emphasized that the Bad prior results are more useful to a practitioner, who, following their intuition/expertise/random choice, might possibly use a non-optimal prior (i.e., without running anything). Can you comment on this?\n---\n\n3. This is a crucial misunderstanding; please let us explain.\n\n    **a)** The best-of-25-samples methodology is strictly an experimental setup protocol to obtain a sense of what a good configuration is. That is, we are not suggesting running 25 random configurations before running PriorBand (with that budget we want to long be done with the optimization), but rather we assume that a practitioner may have existing knowledge based on earlier tuning experience or the literature. (If you’re, e.g., optimizing GPT-4, you use your knowledge gathered from GPT-2 and GPT-3.) For our experimental evaluation, we require a simple, general and reproducible protocol and therefore followed prior work on Bayesian optimization with expert priors [1]. However, we acknowledge that this type of prior may not perfectly match a practitioner’s prior. In _Appendix D.3, Fig 13_, we show the performance of random samples produced from these priors to help judge their relative quality.\n\n    **b**) We agree that the issue of prior quality is a relevant one. _Tab. 10_ displays PriorBand’s performance across prior strengths, compared to other algorithms. PriorBand’s performance degrades gracefully even in the face of a highly adversarial prior (the worst of 50000 random samples), as it maintains the performance of HyperBand on a 12-evaluation budget and only lags behind slightly on a 5-evaluation budget. This adversarial prior is highly unrealistic; if a user doesn’t know which hyperparameter values would work well they can always specify a uniform prior, in which case prior samples are additional random samples.\n\n  Given the prevalence of manual tuning, we ultimately believe that practitioners are substantially more well-informed than a vanilla (random) approach. As such, we believe that they should generally have a positive influence on PriorBand’s performance.\n\n---\n> In Fig. 14, it seems the lines have confidence intervals. How are these computed, and how many repeats of the experiment were used?\n---\n4. We would like to refer you to _Section 6.3_, _Appendices D.4_ and _D.5_ where we detail our experimental setup. We hope this addresses your question about Fig. 14. If not, we are happy to clarify further.\n\n### ___\n\nWe appreciate your feedback and believe to have addressed all the points you raised. Given our additional results for your main concern of only showing rank-based results and our clarification of the crucial misunderstanding of the source of priors (25 random points being far beyond our total budget), we would greatly appreciate it if you were to increase your score correspondingly. If you have additional questions we would be more than glad to discuss them.\n\nWe shall certainly fix the typos you pointed out, in the final draft.\n\n### References:\n\n[1] Hvarfner et al. (2022). $\\pi$BO: Augmenting acquisition functions with user beliefs for Bayesian Optimization. arXiv preprint arXiv:2204.11051.\n"}, {"review_id": "EyNfisydYi", "reviewer": "Reviewer_meuK", "summary": "This paper presents a method to enhance the random sampling component of hyperband. The authors propose replacing it with a combination of random sampling, prior-based sampling, and incumbent-based sampling. They also suggest adjusting the proportion of these samplers based on the current state in the hyperparameter optimization process. The authors perform experiments on a series of benchmarks and compare the proposed method with multiple classical HBO baselines.", "questions": "Why modifying the random sampling component of HyperBand can help decrease the expenses associated with tuning deep learning models?", "limitations": "As a hyperparameter optimization algorithm, the proposed method encompasses several hyperparameters that may significantly influence its final outcomes. These include the hyperparameters associated with the perturbation operation and those control the proportion of the three sampling methods. However, there is currently a dearth of corresponding ablation studies that specifically investigate the individual contributions of these hyperparameters.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This work might be useful in some particular situations.", "weaknesses": "- The motivation for this work may appear artificial, aiming to reduce the cost of hyperparameter optimization in the age of deep learning. However, the paper lacks a detailed explanation of how the proposed adjustments to the sampling component can make hyperparameter optimization more practical and cost-effective in the era of deep learning. Can the authors provide a more comprehensive reasoning process to support this claim?\n\n- Technically, this method is essentially a combination of prior work, including multi-fidelity optimization [1], expert priors [2], and local search [3]. The authors' contributions primarily build upon and benefit from these existing approaches, rather than introducing original ideas. The authors declare that their approach fulfills all the desired requirements for application to deep learning, but this claim is essentially derived from the benefits provided by multi-fidelity optimization.\n\n[1] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1), 6765-6816.\n\n[2]Hvarfner, C., Stoll, D., Souza, A., Lindauer, M., Hutter, F., & Nardi, L. (2022). $\\pi $ BO: Augmenting acquisition functions with user beliefs for bayesian optimization. arXiv preprint arXiv:2204.11051.\n\n[3]Wu, Q., Wang, C., & Huang, S. (2021, May). Frugal optimization for cost-related hyperparameters. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 10347-10354).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments.\n\nWe would request you elaborate on your thoughts on the situations in which our work could be useful. Understanding this could give us perspective on how to address your concerns.\n\n---\n> The motivation for this work may appear artificial...the paper lacks a detailed explanation of how the proposed adjustments to the sampling component can make HPO more practical and cost-effective in the era of deep learning. Can the authors provide a more comprehensive reasoning process to support this claim?\n\n> Why modifying the random sampling component of HyperBand (HB) can help decrease the expenses associated with tuning deep learning models?\n---\n\n1.\n  **a)** We respectfully disagree with the statement of artificial motivation. In many cases, human experts have strong knowledge about hyperparameter settings that are likely to perform well. E.g., if you’re optimizing GPT-4, you do not run HyperBand (HB), but you rather reuse previously known good hyperparameters, maybe with small local adjustments. PriorBand exploits the same prior sampling approach and marries it with multi-fidelity optimization.\n\n  **b)** In the original Hyperband paper, the following is stated in Future Work:\n    \"_Finally, **Hyperband can benefit from different sampling schemes aside from simple random search** ... meta-learning can be used to **define intelligent priors informed by previous experimentation**.  Finally, as mentioned in Section 2, exploring ways to combine Hyperband with **adaptive configuration selection strategies** is a very promising future direction._\" \n\n  As such, the original authors share the opinion that this is a good direction for increased performance; the results confirm this hypothesis. \n\n  **c)** _Fig. 2_ shows that augmenting HB’s random sampling with prior sampling improves anytime performance. To account for failure modes, a local search around the incumbent is introduced as the third sampler, and a dynamic weighting algorithm is designed to trade off the 3 samplers. _Fig. 1_ illustrates the improved anytime performance of this method. _Section 7_ and _Appendix F_ show strong anytime performance for PriorBand under good priors. _Tables 10-14_ show per-dataset performance gains even under budgets equivalent to 4-5 model training.\n\n  **d)** Improving anytime performance directly translates to lower compute requirements for HPO, allowing practitioners to find suitable model configurations at lower costs than previously existing methods. Our work thus contributes to making HPO practical for DL.\n\n  **e)** Our experiment design illustrates results for a maximum budget of 5 model training when 4 parallel model training are possible. Whether such a budget is feasible for HPO is heavily subjective. However, we believe it captures tractable hardware and compute resources in many more cases than previously possible.\n\n\n---\n> Technically, this method is essentially a combination of prior work, including multi-fidelity optimization [1], expert priors [2], and local search [3].\n> The authors declare that their approach fulfills all the desired requirements for application to deep learning, but this claim is essentially derived from the benefits provided by multi-fidelity optimization.\n---\n\n2.\n  **a)** In the realm of HPO, Multi-fidelity Optimization (MFO) and expert priors represent two key paradigms, with HB [1] and $\\pi$BO [2] serving as their respective representatives in our work. Neither one of these in isolation is sufficient to fulfill all desired requirements. To our knowledge, there is no existing work which efficiently merges these two paradigms. Our contribution, the Ensemble Sampling Policy (ESP), is the first to propose a general yet systematic approach to doing so robustly, allowing for multi-fidelity algorithms to interface with expert priors and fulfill the required desiderata. Moreover, the ESP enables an explicit expert prior interface to an entire family of multi-fidelity algorithms.\n\n  **b)** The referenced paper [3] can be integrated into our related literature, thank you. Our proposed method’s contribution over [3] is that we are agnostic to the exact form of local search used. Sampling from the unit sphere [3] aligns with one of our local search ablations (\"hypersphere\" in _Fig. 16_), but we found it to exhibit higher variance and to be less robust than the local mutation in PriorBand. Sampling from a neighborhood sphere offered more hyperparameters and issues mentioned in _L811-816 (Appendix E.2.3)_.\n\n\n---\n> As a HPO algorithm, the proposed method encompasses several hyperparameters … a dearth of corresponding ablation studies that specifically investigate the individual contributions of these hyperparameters.\n---\n\n3. In Fig. 3 of the rebuttal PDF, we have added ablation studies on the local search hyperparameters (Appendix E.2.5) as pointed out. We ablate over the standard deviation of the Gaussian around the incumbent configuration and the mutation rate, that is, the probability of selection of a hyperparameter for perturbation. Our chosen default setting for these hyperparameters, which we keep fixed across all our experiments, is amongst the better choices, but performance is very robust to these hyperparameters. \n\n\n### ___\n\nWe hope that our explanations were satisfactory and brought about more clarity. If so, we would appreciate it if you consider increasing our score. If you have any further questions and comments we would be very glad to discuss them.  \n\n### References\n\n[1] Li et al. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1), 6765-6816.\n\n[2] Hvarfner et al. (2022). $\\pi$BO: Augmenting acquisition functions with user beliefs for Bayesian Optimization. arXiv preprint arXiv:2204.11051.\n\n[3] Wu et al. (2021, May). Frugal optimization for cost-related hyperparameters. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 10347-10354).\n"}]

Author rebuttal:
To all reviewers and chairs,

We upload the permitted extra PDF with plots addressing a few points raised across the reviews.

**Figure 1)** shows the influence of a "good" prior on PriorBand, per-dataset under the reproducible protocol for prior generation in our experiments (_Appendix D.3_). This is an alternative to the mean relative ranks in the draft. 

In this plot, we show the mean normalized regret over iterations for PriorBand per benchmark. The regret is calculated by normalizing all values between the minimum and maximum values seen across all included algorithms and seeds for each benchmark. 

**Figure 2)** shows that the absolute performance of PriorBand is significant over the baselines, something not visible in relative ranking plots. It shows the mean normalized regret averaged across all benchmarks under different strengths of priors. This demonstrates both strong anytime and final performance of PriorBand across different prior qualities, under tractable budgets.

The second from right plot in this figure is the aggregated view of Figure 1.

**Figure 3)** highlights that our chosen hyperparameters for PriorBand offer a good balance between exploration and exploitation. This figure shows two ablation studies over the PriorBand local search hyperparameters: (i) the standard deviation of the local perturbation, and (ii) the mutation rate, which determines the chance of selection of each hyperparameter for perturbation. 

More exploitative local search hyperparameters naturally give gains under strong priors. However, given the goal of being robust across all possible scenarios, a conservative local search (our default for PriorBand) offers steady anytime gains under different prior qualities. This allows PriorBand to be truly practical for HPO.

We have provided further captions to explain the figures but we are happy to clarify further if required.

We are looking forward to an engaged discussion period!


Meta-review: The paper provides an approach for hyper paramater optimization (HPO), suited for deep learning. The method provided by the authors doesn’t contain an entirely new technique, rather carefully combines several existing methods. A major concern raised by some of the reviewers points this out as a lack of novelty. This being said, the reviewers agree that the provided combination of techniques is sensible and well described. The paper provides a good survey of existing works and a thorough set of experiments analyzing the technique and proving its efficiency compared to SoTA techniques. Given that multi-fidelty HPO already has many published paper, being able to innovate there is not an easy task. With this in mind, along with the paper’s mentioned strengths, I think the paper would be of intererst to the NeurIPS community.
Decision: Accept (poster)