Title: PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Hyperparameter Optimization

Deep Learning
Abstract: Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. 
While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL.
Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations.
To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. 
Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs.

PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

Abstract

Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand , an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate PriorBand 's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs.

1 Introduction

The performance of Deep Learning (DL) models crucially depends on dataset-specific settings of their hyperparameters (HPs) [1, 2]. Therefore, Hyperparameter Optimization (HPO) is an integral step in the development of DL models [3-6]. HPO methods have typically been applied to traditional machine learning models (including shallow neural networks) that focus on fairly small datasets [7, 8]. Current DL practitioners, however, often utilize much larger models and datasets (e.g., a single training of GPT-3 [9] was estimated to require 3 · 10 23 FLOPS, i.e., months on a thousand V100 GPUs [10]). As recent HPO practices continue to apply simple techniques like grid, random, or manual search [11-14], existing HPO methods seem misaligned with DL practice.

To make the misalignment between existing HPO methods and DL practice explicit, we identify the following desiderata for an efficient, scalable HPO method suitable for current DL:

1

3

5

7

10

Full trainings

64

66

68

Error Rate (%)

1

3

5

7

10

Full trainings

1

2

3

4

Relative rank

Ranks on all benchmarks

RS

RS+Prior

HB

PriorBand

Strong performance under low compute budgets : As DL pipelines become larger and more complex, only a few model training can be afforded to find performant configurations.

Integrate cheap proxy tasks : To save resources, cheap evaluations must be used to find promising configurations early while maintaining robustness to uninformative proxies.

Integrate expert beliefs : DL models often come with a competitive default setting or an expert may have prior beliefs on what settings to consider for HPO. Incorporating this information is essential while maintaining robustness to such beliefs being inaccurate.

Handle mixed type search spaces : Modern DL pipelines tend to be composed of many kinds of hyperparameters, e.g. categorical, numerical, and log hyperparameters.

Simple to understand and implement : The HPO method should be easy to apply to different DL pipelines and the behavior should be conceptually easy to understand.

Parallelism : Scaling to modern compute resources must be simple and effective.

Existing HPO algorithms satisfy subsets of these desiderata (see Table 1). For example, methods that utilize cheaper proxy tasks exploit training data subsets [16], fewer epochs/updates [17, 18], and proxy performance predictors [19, 20]. Previous works also consider how to combine expert knowledge into the optimization procedure in the form of the optimal value [21], meta or transfer learned knowledge [22-25] or configurations and regions that are known to the expert to have worked well previously [26-28]. Nonetheless, no HPO method exists that meets all of the desiderata above, especially, a simple model-free approach that can run cheaper proxy tasks, letting the DL expert integrate their domain knowledge for HPO.

We propose PriorBand , the first method to fulfill all desiderata. Our contributions are as follows:

We are the first to develop an approach to leverage cheap proxy tasks with an expert prior input (Section 2), and we show the need beyond a naive solution to do so (Section 4).

We contribute a general HPO algorithm with PriorBand , fulfilling all desiderata (Table 1) required for application to DL (Section 5; also see Figure 1)

We demonstrate the efficiency and robustness of PriorBand on a wide suite of DL tasks under practically feasible compute resources (Section 7).

We highlight the flexibility of the method behind PriorBand by improving different multifidelity algorithms and their model-based extensions (Section 7.2 & 7.3), which highlights the flexible, modular design and possible extensions of PriorBand .

We conclude, highlight limitations, and discuss further research directions in Section 10. Our code for reproducing the experiments is open-sourced at https://github.com/automl/mf-prior-exp .

2 Problem statement

We consider the problem of minimizing an expensive-to-evaluate objective function f : Λ → R , i.e., λ ∗ ∈ arg min λ ∈ Λ f ( λ ) , where a configuration λ comes from a search space Λ which may consist of any combination of continuous, discrete and categorical variables. In HPO particularly, f refers to training and validating some model based on hyperparameters defined by λ .

Multi-fidelity optimization of f ( λ ) requires a proxy function, namely ˆ f ( λ , z ) , that provides a cheap approximation of f at a given fidelity z , e.g., the validation loss of a model only trained for z epochs. Our methodology considers a fidelity scale z ∈ [ z min , z max ] such that evaluating at z max coincides with our true objective: f ( λ ) = ˆ f ( λ , z max ) .

To take all desiderata into account, we additionally include a user-specified belief π : Λ → R , where π ( λ ) = P ( λ = λ ∗ ) represents a subjective probability that a given configuration λ is optimal. Thus, the overall objective is

Our problem is to efficiently solve Equation 1 under a constrained budget, e.g., 10 full trainings of a DL model. Given DL training can diverge, the incumbent we return is the configuration-fidelity pair ( λ , z ) that achieved the lowest error while optimizing for Equation 1.

3 Background

While we are the first to target Equation 1 in its full form, below, we introduce the necessary background on methods that use either multi-fidelity optimization or expert beliefs.

Successive Halving (SH) optimizes Equation 1 (sans π ) as a best-arm identification problem in a multi-armed bandit setting [29]. Given a lower and upper fidelity bound ( [ z min , z max ] ) and a reduction factor ( η ), SH discretizes the fidelity range geometrically with a log-factor of η . For example, if the inputs to SH are z ∈ [3 , 81] and η = 3 , the fidelity levels in SH are z = { 3 , 9 , 27 , 81 } with accompanying enumeration called rungs, r = { 0 , 1 , 2 , 3 } . Any z < z max is called a lower fidelity, a cheaper proxy task in DL training. Every iteration of each round of SH involves uniform random sampling of η s max -1 configurations at the lowest fidelity or r = 0 , where s max = ⌊ log η ( z max /z min ) ⌋ . After evaluating all samples, only the top-performing (1 /η ) configurations are retained, discarding the rest. Subsequently, these η s max -1 /η configurations are evaluated at r = 1 , or the next higher discretized fidelity. This is repeated till there is only one surviving configuration at the highest r. Sampling and evaluating many configurations for cheap, performing a tournament selection to retain strong samples, and repeating this, ensures that the high fidelity evaluations happen for relatively stronger configurations only. SH proves that such an early-stopping strategy guarantees a likely improvement over random search under the same total evaluation budget. Due to its random sampling component, SH is an example of MF-RS in Table 1. See Appendix D.2 for an illustrative example.

HyperBand (HB) attempts to address SH's susceptibility to poor fidelity performance correlations [30]. It has the same HPs as SH, and thus given SH's fidelity discretization, HB iteratively executes multiple rounds of SH where z min is set to the next higher discrete fidelity. Using the

0

3

6

9

12

Full Evaluations

4

2

0

Objective

H3d (high corr-good prior)

0

3

6

9

12

Full Evaluations

4

2

0

H3d (high corr-bad prior)

0

3

6

9

12

Full Evaluations

4

2

0

H3d (low corr-good prior)

0

3

6

9

12

Full Evaluations

4

2

0

H3d (low corr-bad prior)

Mode

Optimum

HB

HB+Prior

HB+Prior(50%)

example from SH, given z ∈ [3 , 81] and η = 3 , one full iteration of HB corresponds to running SH( z min = 3 , z max = 81 ) followed by SH( z min = 9 , z max = 81 ), SH( z min = 27 , z max = 81 ) and SH( z min = 81 , z max = 81 ). Notably, such different instantiations of SH in HB do not share any information with each other implicitly. Each execution of SH( z min , z max) is called an SH bracket . The sequence of all unique SH brackets run by HB in one iteration is called an HB bracket .

π BO utilizes the unnormalized probability distribution π ( λ ) in a BO context to accelerate the optimization using the knowledge provided by the user [28]. In π BO, the initial design is sampled entirely from π until Bayesian Optimization (BO) begins, where the acquisition function α is augmented with a prior term that decays over time t : α t π ( λ ) = α ( λ ) · π ( λ ) β t , where β is an HP. In this work, we borrow π ( · ) as the expert prior interface to the HPO problem. π BO relies on the decaying prior on the acquisition α t π ( λ ) to gradually add more weight to the model and thus allow recovery from a poor initial design under a bad prior. We choose the model-free setting and adapt the strength of the prior based on the evidence of its usefulness.

4 The need to move beyond a naive solution

To solve Equation 1, an intuitive approach is to augment an existing multi-fidelity algorithm, e.g., HyperBand (HB), with sampling from the prior π ( · ) . In this section, however, we show that this naive approach is not sufficient, motivating the introduction of our algorithm, PriorBand, in Section 5.

To study the naive combination of HyperBand with prior-based sampling, we use the parameterized 3 -dimensional multi-fidelity Hartmann benchmarks [31] (Appendix D.1.1) and compare HB with uniform sampling, HB with 100% prior-based sampling, and HB with 50% uniform and prior-based sampling (Figure 2). Unsurprisingly, 100% sampling from priors works best when the prior is helpful and not sampling from the prior works best for misleading priors. 50% random sampling works quite robustly, but is never competitive with the best approach. To achieve robustness to the quality of the prior and rival the best approach for each case, we introduce PriorBand below.

5 PriorBand: Moving beyond a naive solution

The key idea behind PriorBand is to complement the sampling strategies used in the naive solution, uniform sampling, and prior-based sampling, with a third strategy: incumbent-based sampling. Thereby, we overcome the robustness issues of the naive solutions presented in Section 4, while still fulfilling the desideratum for simplicity. We first introduce and motivate sampling around the incumbent (Section 5.1), and then describe the ensemble sampling policy E π that combines all three sampling strategies and incorporates E π into HB (Section 5.2) for PriorBand .

5.1 Incumbent-based sampling strategy, ˆ λ ( · )

PriorBand leverages the current incumbent to counter uninformative priors while supporting good priors, as the current incumbent can be seen as indicating a likely good region to sample from. Note that this view on the region around the incumbent is close to the definition of π ( · ) in π BO (Section

3), where the prior distribution encodes the expert's belief about the location of the global optima and thus a good region to sample from.

To construct the incumbent-based sampler ˆ λ ( · ) , we perform a local perturbation of the current best configuration. Each hyperparameter (HP) is chosen with probability p for perturbation. If chosen, continuous HPs are perturbed by ϵ ∼ N (0 , σ 2 ) . For discrete HPs, we resample with a uniform probability for each categorical value except the incumbent configuration's value which has a higher probability of selection, discussed further in Appendix E.2.5. For PriorBand , we fix these values at p = 0 . 5 and σ = 0 . 25 . This perturbation operation is simple, easy to implement, and has constant time complexity. We show ablations with two other possible local search designs in Appendix E.2.3.

5.2 PriorBand : The E π -augmented HyperBand

PriorBand exploits HB for scheduling and replaces its random sampling component with a combination of random sampling, prior-based sampling, and incumbent-based sampling. We denote the proportions of these individual sampling components as p U , p π , and p ˆ λ , respectively, and their combination as the ensemble sampling policy (ESP) E π .

Figure 3 (left) illustrates PriorBand as an extension of HB that, next to the HB hyperparameters ( z min , z max , η, budget) accepts the expert prior π as an additional input and uses the ESP E π in lieu of random sampling. This sampling from E π is illustrated in Algorithm 1. Note that E π has access to the optimization state ( s t ) and can thus reactively adapt its individual sampling probabilities p U , p π , and p ˆ λ based on the optimization history. We now discuss how we decay the random sampling probability p U (Section 5.2.1); and the proportion of incumbent and prior sampling (Section 5.2.2).

PriorBand

Zmin

π

Current

pπ=n"·pu

pu

Probability

Zmax

TT

HyperBand

pπ

pπ

ofchoosing

astrategy

state,St

top(1/m)

Probabilityratios

to drawa

n

St

sample

configurations

sample

budget.

HyperBand

1*

Algorithm 1 Sampling from E π

1: Input: s, s max , η, observations H , prior π

2: r = s max -s ▷ s max , s input from HB

4: if activate_incumbent( s t ) then

3: p U = 1 / (1 + η r ) , p π = 1 -p U , p ˆ λ = 0

5: p π , p ˆ λ ← Algorithm 2( H , r , p π )

6: end if

7: d ( · ) ← sample strategy by { p U , p π , p ˆ λ }

8: λ ← sample from d ( · )

9: return λ

Algorithm 2 Dynamic weighting of w ˆ λ

1: Input: observations H , rung r , p old π

2: Λ ′ z = { λ i } 1: n ← top_ (1 /η )( H , r )

3: S ˆ λ ← ∑ Λ ′ z w i · ˆ λ ( λ i ) ▷ w i = ( n +1) - i

4: S π ← ∑ Λ ′ z w i · π ( λ i )

5: p ˆ λ ← p old π · S ˆ λ / ( S π + S ˆ λ )

6: p π ← p old π · S π / ( S π + S ˆ λ )

7: return p π , p ˆ λ

5.2.1 Decaying proportion of random sampling

Given the premise that we should initially trust the expert prior, yet with the benefit of incorporating random sampling (from Section 4), we make two additional assumptions, namely (i) we trust the expert's belief most at the maximum fidelity z max; and (ii) we would like to use cheaper fidelities to explore more. Given HB's discretization of the fidelity range [ z min , z max ] into rungs r ∈ { 0 , . . . , r max } , we geometrically increase our sampling probability from π ( · ) over U ( · ) by

4

6

3

2

5

1

Figure 4: A visual example of how configurations contribute to the scores S π and S ˆ λ . The prior distribution here is Gaussian N ( µ π , σ 2 ) , with a matching distribution ˆ λ ( · ) = N ( ˆ λ , σ 2 ) placed on the incumbent. The labels 1 -6 indicate configurations and their rank in Λ ′ z . Here, e.g., λ 2 has densities such that π ( λ 2 ) > ¯ λ ( λ 2 ) , contributing more to S π than to S ˆ λ . The configurations λ 3 , λ 4 , λ 5 are between both distributions and λ 6 has low density under either, whereas λ 1 (the incumbent) will be the primary influence such that S ˆ λ > S π . This implies that p ˆ λ > p π and demonstrates that all else being relatively equal, we will be more likely to sample from ˆ λ ( · ) .

with the constraint that p U + p π = 1 (see L3, Algorithm 1). This naturally captures our assumptions, equally favouring π ( · ) and U ( · ) initially but increasing trust in π ( · ) according to the rung r we sample at. This geometric decay was inspired by similar scaling used in HB's scheduling and backed by ablations over a linear decay and constant proportion (Appendix E.2.1).

5.2.2 Incumbent-based sampling proportion

Incumbent-based sampling intends to maintain strong anytime performance even under bad, uninformative, or adversarial priors. In PriorBand , initially p ˆ λ = 0 until both (i) a budget equivalent to the first SH bracket has been exhausted ( ≈ η · z max); and (ii) at least one configuration has been evaluated at z max. This is captured in Line 4 of Algorithm 1 by activate_incumbent() .

Once incumbent-based sampling is active we need to decide how much to trust ˆ λ ( · ) vs. trusting π ( · ) . Essentially, we achieve this by rating how likely the best seen configurations would be under ˆ λ ( · ) and π ( · ) , respectively. Algorithm 2 shows the detailed steps to calculate p π and p ˆ λ , and Figure 4 provides an example. We first define an ordered set Λ ′ z = { λ 1 , . . . , λ n } , ordered by performance, of the top-performing (1 /η ) configurations 1 for the highest rung r with at least η evaluated configurations (Line 2 in Algorithm 2). Given Λ ′ z , we compute two weighted scores S π , S ˆ λ , capturing how likely these top configurations are under the densities of π ( · ) and ˆ λ ( · ) , respectively. We also weigh top-performing configurations more highly, which is accomplished with a simple linear mapping w i = ( n +1) -i (Lines 3-4 in Algorithm 2). Finally, we obtain p π and p ˆ λ by normalizing these scores as a ratio of their sum (Lines 5-6 in Algorithm 2). We note the rates are adaptive based on how the set Λ ′ z is spread out relative to the prior and the incumbent. We observe that this adaptive behavior is crucial in quickly recovering from bad prior inputs and thus maintaining strong final performance.

6 Experimental setup

We now describe our experiment design, benchmarks, and baselines and demonstrate the robustness of PriorBand in handling varying qualities of an expert prior input. Additionally, we also showcase the generality and practicality of the ESP, E π .

6.1 Benchmarks

We curated a set of 12 benchmarks that cover a diverse set of search spaces, including mixedtype spaces and log-scaled hyperparameters, and a wide range of downstream tasks, e.g., language modeling, image classification, tabular data, a medical application, and translation. We select 4 of the PD1 benchmarks (4 HPs) [32] that train large models such as transformers with batch sizes commonly found on modern hardware, and fit surrogates on them. Further, we select 5 benchmarks from LCBench (7 HPs) [33, 34] and consider all 3 JAHSBench [35] surrogate benchmarks that offer a 14 dimensional mixed-type search space for tuning both the architecture and training hyperparameters. All benchmarks and their selection are described in further detail in Appendix D.1.

6.2 Baselines

We choose a representative from each of the families of optimizers listed in Table 1, leaving out grid search. We use the official implementation of BOHB, while all other algorithms were implemented by us (and are available as part of our repository). The prior-based baselines ( π BO and RS+Prior) sample the mode of the prior distribution as the first evaluation in our experiments to ensure a fair comparison where the prior configuration is certainly evaluated, irrespective of it being good or bad. For all the HB based algorithms we use η = 3 and the fidelity bounds ( z min , z max) as per the benchmark. Further implementation and hyperparameter details for the baselines can be found in Appendix D.2. In principle, PriorBand only needs an additional input of π ( · ) as the user belief, other than the standard HB hyperparameters. However, for a discussion on the hyperparameters of the incumbent-based local search (Section 5.1), please refer to Appendix E.2.5.

6.3 Design and setup

We show experiments both for single and multi-worker cases ( 4 workers in our experiments). For the single workers, we report the mean validation error with standard error bars for 50 seeds; for multiworker experiments, we use 10 seeds. The plots show average relative ranks achieved aggregated across all 12 benchmarks when comparing the anytime incumbent configuration at z max. We group the runs on benchmark under different qualities of an expert prior input. We also compare the average normalized regret per benchmark under good priors. The prior construction procedure follows the design from the work by Hvarfner et al. [28], and we describe our procedure in detail in Appendix D.3. In the main paper, we only evaluate the good and bad prior settings as we believe that this reflects a practical setting of prior qualities; in Appendix F we also evaluate the near-optimal prior settings and also evaluate all 3 prior settings over high and low-performance correlation problems. In Appendix F.5, we also report additional experiments for different budget horizons. Further experiment design details can be found in Appendices D.4 andD.5. As an example of potential post-hoc analysis possible with PriorBand , we show how the dynamic probabilities calculated for each of the sampling strategies in PriorBand can be visualized (Appendix E.3), revealing the quality of the prior used.

7 Results

We now report and discuss the results of our experiments.

7.1 Robustness of PriorBand

Here, we demonstrate the robustness of PriorBand over a wide range of prior qualities by comparing them to the nearest non-prior-based algorithms: random search (RS) and HyperBand (HB). Figure 5(top) showcases PriorBand to be anytime equal or better than HB on each of our benchmarks under our good prior design. Note that the quality of a good prior varies per benchmark. The aggregation of Figure 5(top) is Figure 5(bottom-middle), which illustrates that PriorBand (HB+ E π ) can utilize good priors and gain strong anytime performance over HB. Moreover, Figure 5(bottom-right) clearly shows PriorBand 's ability to recover from bad prior information and match vanilla-HB's performance in this adversarial case (the bad prior was intentionally chosen to be an extremely poor configuration - the worst of 50 k random samples). In most practical scenarios, the expert has better intuition than this setup and one can expect substantial speedups over HB, like in Figure 5(bottommiddle). Figure 5(bottom-left) demonstrates that when using an unknown quality of prior, even including the adversarial prior, PriorBand is still the best choice on average. We show a budget of 12 function evaluations here, which is approximately the maximal budget required for completing at least one HB iteration for the benchmarks chosen. In Appendix F.1 we show more comparisons to prior-based baselines and highlight PriorBand 's robustness.

7.2 Generality of Ensemble Sampling Policy E π

The ESP, E π , only needs access to an SH-like optimization state as input. In this section, we show that other popular multi-fidelity algorithms, such as ASHA or asynchronous-HB [36] that build on SH, can also support expert priors with the help of E π . In Figure 6, we compare the vanilla algorithms with their E π -augmented versions. Given that these algorithms were designed for parallel setups,

1

3

6

9

12

10

1

10 0

Normalized Regret

[+]PD1-WMT (good)

1

3

6

9

12

10

1

10 0

[+]LCBench-189906 (good)

1

3

6

9

12

10

1

10 0

[+]CIFAR100 (good)

1

3

6

9

12

10

1

10 0

[+]Imagenet (good)

1

3

6

9

12

10 0

2×10 1

3×10 1

4×10 1

6×10 1

Normalized Regret

[+]LCBench-168330 (good)

1

3

6

9

12

10

1

10 0

[+]PD1-Lm1b (good)

1

3

6

9

12

10

1

10 0

[+]LCBench-168910 (good)

1

3

6

9

12

10

1

10 0

[+]LCBench-167190 (good)

1

3

6

9

12

Full trainings

10

2

10

1

10 0

Normalized Regret

[-]CIFAR-10 (good)

1

3

6

9

12

Full trainings

10

1

10 0

[-]Colorectal-Histology (good)

1

3

6

9

12

Full trainings

10

1

10 0

[-]LCBench-126026 (good)

1

3

6

9

12

Full trainings

10

1

10 0

[-]Fashion-MNIST (good)

Mode

RS

HB

PriorBand

0

3

6

9

12

Full trainings

1

2

3

Relative rank

All

0

3

6

9

12

Full trainings

1

2

3

Good prior

0

3

6

9

12

Full trainings

1

2

3

Bad prior

RS

HB

PriorBand

we compare them on runs distributed among 4 workers, running for a total budget of 20 function evaluations. Similar to the previous section, the ESP-augmented (+ESP) algorithms can leverage good priors and recover under bad priors. Under bad priors, asynchronous-HB (+ESP) starts worst owing to bad prior higher fidelity evaluations at the start but shows strong recovery. In Appendix F.2 we compare these variants over different correlation settings of the benchmarks.

0

1

2

3

4

5

Full trainings

1

2

3

4

Relative rank

All

0

1

2

3

4

5

Full trainings

1

2

3

4

Good prior

0

1

2

3

4

5

Full trainings

1

2

3

4

Bad prior

ASHA

AsyncHB

ASHA(+ESP)

Async-HB(+ESP)

7.3 Extensibility with models

Although our focus is the model-free low compute regime, we also show how PriorBand can be optionally extended with model-based surrogates to perform Bayesian Optimization, especially when longer training budgets are available. We compare BO, π BO, BOHB, and PriorBand with its model-extension, PriorBand +BO (Gaussian Processes as surrogates and Expected Improvement [37] for acquisition). All BO methods use an initial design of 10 function evaluations, except BOHB which sets this implicitly as N dim +2 . Compared to other prior-based algorithms (RS+Prior, π BO), PriorBand +BO is consistently robust on average ( All ) under all priors. The anytime performance gain of PriorBand +BO under good priors is evidence of ESP E π 's utility. We note that PriorBand +BOrecovers quickly in the bad prior setting and even outperforms all other algorithms after as little a budget as 6 full trainings. The fact that, until 10 × function evaluations, PriorBand +BO is actually just PriorBand and model search has not begun, highlights the effectiveness of ESP in adaptively trading off different sampling strategies and the initial strength of incumbent sampling. Appendices E.4, F.3 contain details on our modeling choices and more experimental results.

0

5

10

15

20

Full trainings

1

2

3

4

5

Relative rank

All

0

5

10

15

20

Full trainings

1

2

3

4

5

Good prior

0

5

10

15

20

Full trainings

1

2

3

4

5

Bad prior

RS+Prior

BO

BO

BOHB

PriorBand+BO

7.4 Ablation studies

In Figure 8, we compare our choice of using density scores (Section 5.2.2) to trade-off prior-based and incumbent-based sampling against other similar or simpler heuristics, validating our decision. PriorBand (constant) employs a fixed heuristic to trade off prior and incumbent-based sampling as p π = η · p ˆ λ . In PriorBand (decay), p π is decayed as a function of iterations completed. More precisely, p ˆ λ = 2 b · p π , subject to p π + p ˆ λ + p U = 1 (from Section 5.2), where b ∈ N and indicates the index of the current SH being run in HB. Appendix E.2.2 contains more ablations.

0

5

10

15

20

Full trainings

1

2

3

Relative rank

All

0

5

10

15

20

Full trainings

1

2

3

Good prior

0

5

10

15

20

Full trainings

1

2

3

Bad prior

PriorBand(constant)

PriorBand(decay)

PriorBand

8 Related work

While using expert priors and local search [38] for hyperparameter optimization has been explored previously, few works considered priors over the optimum [27, 28, 39], and they all target the

single-fidelity setting. The expert priors we consider should not be confused with the priors natively supported by BO, i.e., priors over the function structure determined by kernels [7, 40, 41].

In deep learning, training epochs and dataset subsets [17, 42, 43] are frequently used as fidelity variables to create cheap proxy tasks, with input resolution, network width, and network depth also occasionally used [35]. SH [29] and HB [30] are effective randomized policies for multi-fidelity HPO that use early stopping of configurations on a geometric spacing of the fidelity space and can also be extended to the model-based setting [44].

9 Limitations

We acknowledge that PriorBand inherits pathologies of HB such as poor fidelity correlations and can be sensitive to the choice of fidelity bounds and early stopping rate. Despite that, our results across different correlation settings suggest a relatively strong performance of PriorBand (Appendix F). Though PriorBand supports any kind of prior distribution as input, in our experiments we only considered the Gaussian distribution (with a fixed standard deviation of 0 . 25 ), as it is a natural choice and was used previously in the literature [28]. However, the expert is free to represent the prior with other distributions. We expect PriorBand to show similar behaviors as we report when comparing to other prior-based algorithms under a similar distribution. We note that PriorBand is not entirely free of hyperparameters. In our experiments, we keep all PriorBand hyperparameters fixed to remove confounding factors while comparing different prior strengths and correlations. Moreover, our hyperparameter choices for the experiments, such as η = 3 and Gaussian priors are largely borrowed from existing literature [28, 36, 44]. Depending on the context of the specific deep learning expert, the compute budgets ( ∼ 10 -12 full trainings) used in our experimental setting may not always be feasible. However, the key insight from the experiments is that PriorBand can interface informative priors, and provide strong anytime performance under short HPO budgets (also in less than 5 full trainings). Longer HPO budgets for PriorBand ensure recovery from potential bad, uninformative priors. In practice, deep learning experts often have good prior knowledge and thus PriorBand is expected to retain strong anytime performance under low compute.

10 Conclusion

We identify that existing HPO algorithms are misaligned with deep learning (DL) practice and make this explicit with six desiderata (Section 1). To overcome this misalignment, our solution, PriorBand , allows a DL expert to incorporate their intuition of well-performing configurations into multi-fidelity HPO and thereby satisfies all desiderata to be practically useful for DL. The key component in PriorBand , the ESP, E π , is modular, flexible, and can be applied to other multi-fidelity algorithms.

11 Acknowledgments and disclosure of funding

N. Mallik, D. Stoll and F. Hutter acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. E. Bergman was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. C. Hvarfner and L. Nardi were partially supported by the Wallenberg AI, Autonomous Sytems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. L. Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project - Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348. M. Lindauer was funded by the European Union (ERC, 'ixAutoML', grant no.101041029). The research of Maciej Janowski was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, Maciej Janowski acknowledges the support of the BrainLinks- BrainTools Center of Excellence.

References

I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning . MIT Press, 2016.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR'19) , 2019. Published online: iclr.cc .

Y. Chen, A. Huang, Z. Wang, I. Antonoglou, J. Schrittwieser, D. Silver, and N. de Freitas. Bayesian optimization in alphago. arXiv:1812.06855 [cs.LG] , 2018.

G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In Proceedings of the International Conference on Learning Representations (ICLR'18) , 2018. Published online: iclr.cc .

B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp, K. Chua, F. Hutter, and R. Calandra. On the importance of Hyperparameter Optimization for model-based reinforcement learning. In A. Banerjee and K. Fukumizu, editors, Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS'21) , pages 4015-4023. Proceedings of Machine Learning Research, 2021.

R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. In Ranzato et al. [45].

J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS'12) , pages 2960-2968. Curran Associates, 2012.

M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-Sklearn 2.0: Handsfree automl via meta-learning. JMLR , 23(261):1-61, 2022.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Larochelle et al. [46], pages 1877-1901.

Lambda. Demystifying gpt-3. https://lambdalabs.com/blog/demystifying-gpt-3 , June 2020. Accessed on May 4, 2023.

X. Bouthillier and G. Varoquaux. Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020. Research report [hal-02447823], Inria Saclay Ile de France, 2020.

N. Hasebrook, F. Morsbach, N. Kannengießer, M. Zöller, J. Franke, M. Lindauer, F. Hutter, and A. Sunyaev. Practitioner motives to select hyperparameter optimization methods. arXiv preprint arXiv:2203.01717 , 2022.

F. Schneider, Z. Nado, N. Agarwal, G. E. Dahl, and P. Hennig. Hity workshop poll, neurips 2022. https://github.com/fsschneider/HITYWorkshopPoll , 2022.

K. Van der Blom, A. Serban, H. Hoos, and J. Visser. Automl adoption in ml software. In K. Eggensperger, F. Hutter, E. LeDell, M. Lindauer, G. Shala, J. Vanschoren, and C. White, editors, ICML workshop on Automated Machine Learning (AutoML workshop 2021) , 2021.

Javier Gonzalez, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51 of Proceedings of Machine Learning Research , pages 648-657, Cadiz, Spain, 09-11 May 2016. PMLR. URL https://proceedings.mlr.press/v51/gonzalez16a.html .

A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast Bayesian optimization of machine learning hyperparameters on large datasets. In A. Singh and J. Zhu, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS'17) , volume 54. Proceedings of Machine Learning Research, 2017.

K. Swersky, J. Snoek, and R. Adams. Freeze-thaw Bayesian optimization. arXiv:1406.3896 [stats.ML] , 2014.

T. Domhan, J. Springenberg, and F. Hutter. Speeding up automatic Hyperparameter Optimization of deep neural networks by extrapolation of learning curves. In Q. Yang and M. Wooldridge, editors, Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI'15) , pages 3460-3468, 2015.

C. White, A. Zela, B. Ru, Y. Liu, and F. Hutter. How powerful are performance predictors in neural architecture search? In Ranzato et al. [45].

M. Abdelfattah, A. Mehrotra, L. Dudziak, and N. Lane. Zero-cost proxies for lightweight NAS. In Proceedings of the International Conference on Learning Representations (ICLR'21) , 2021.

V. Nguyen and M. A. Osborne. Knowing the what but not the where in bayesian optimization. In H. Daume III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning (ICML'20) , volume 98, pages 7317-7326. Proceedings of Machine Learning Research, 2020.

M. Feurer, J. Springenberg, and F. Hutter. Initializing Bayesian Hyperparameter Optimization via meta-learning. In B. Bonet and S. Koenig, editors, Proceedings of the Twenty-ninth AAAI Conference on Artificial Intelligence (AAAI'15) , pages 1128-1135. AAAI Press, 2015.

V. Perrone and H. Shen. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche Buc, E. Fox, and R. Garnett, editors, Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurIPS'19) , pages 12751-12761. Curran Associates, 2019.

M. Feurer, B. Letham, F. Hutter, and E. Bakshy. Practical transfer learning for bayesian optimization. arXiv:1802.02219v3 [stat.ML] , 2018.

M. Wistuba and J. Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In Proceedings of the International Conference on Learning Representations (ICLR'21) , 2021.

D. Stoll, J. Franke, D. Wagner, S. Selg, and F. Hutter. Hyperparameter transfer across developer adjustments. In Larochelle et al. [46].

A. Souza, L. Nardi, L. Oliveira, K. Olukotun, M. Lindauer, and F. Hutter. Bayesian optimization with a prior for the optimum. In N. Oliver, F. Pérez-Cruz, S. Kramer, J. Read, and J. Lozano, editors, Machine Learning and Knowledge Discovery in Databases (ECML/PKDD'21) , volume 12975 of Lecture Notes in Computer Science , pages 265-296. Springer-Verlag, 2021.

C. Hvarfner, D. Stoll, A. Souza, L. Nardi, M. Lindauer, and F. Hutter. π BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In Proceedings of the International Conference on Learning Representations (ICLR'22) , 2022. Published online: iclr.cc .

K. Jamieson and A. Talwalkar. Non-stochastic best arm identification and Hyperparameter Optimization. In A. Gretton and C. Robert, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS'16) , volume 51. Proceedings of Machine Learning Research, 2016.

L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to Hyperparameter Optimization. JMLR , 18(185):1-52, 2018.

K. Kandasamy, G. Dasarathy, J. Schneider, and B. Póczos. Multi-fidelity Bayesian Optimisation with Continuous Approximations. In D. Precup and Y. Teh, editors, Proceedings of the 34th International Conference on Machine Learning (ICML'17) , volume 70, pages 1799-1808. Proceedings of Machine Learning Research, 2017.

Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization, 2021. URL https://arxiv.org/abs/2109.08215 .

Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(9):3079 - 3090, 2021.

F. Pfisterer, L. Schneider, J. Moosbauer, M. Binder, and B. Bischl. YAHPO Gym - design criteria and a new multifidelity benchmark for hyperparameter optimization. arXiv:2109.03670 [cs.LG] , 2021.

A. Bansal, D. Stoll, M. Janowski, A. Zela, and F. Hutter. JAHS-bench-201: A foundation for research on joint architecture and hyperparameter search. In Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22) , 2022.

L. Li, K. G. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-tzur, M. Hardt, B. Recht, and A. Talwalkar. A system for massively parallel hyperparameter tuning. In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems 2 , volume 2, 2020.

D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black box functions. JGO , 13:455-492, 1998.

Q. Wu, C. Wang, and S. Huang. Frugal optimization for cost-related hyperparameters. In Q. Yang, K. Leyton-Brown, and Mausam, editors, Proceedings of the Thirty-Fifth Conference on Artificial Intelligence (AAAI'21) . Association for the Advancement of Artificial Intelligence, AAAI Press, 2021.

J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems (NeurIPS'11) , pages 2546-2554. Curran Associates, 2011.

K. Swersky, J. Snoek, and R. Adams. Multi-task Bayesian optimization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS'13) , pages 20042012. Curran Associates, 2013.

C. Oh, E. Gavves, and M. Welling. BOCK : Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning , pages 3865-3874, 2018.

A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian Hyperparameter Optimization on large datasets. Electronic Journal of Statistics , 11:4945-4968, 2017.

V. Nguyen, S. Schulze, and M. A. Osborne. Bayesian optimization for iterative learning. In Larochelle et al. [46].

S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML'18) , volume 80, pages 1437-1446. Proceedings of Machine Learning Research, 2018.

M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors. Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS'21) , 2021. Curran Associates.

H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors. Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS'20) , 2020. Curran Associates.

L. Zimmer, M. Lindauer, and F. Hutter. Auto-Pytorch: Multi-fidelity metalearning for efficient and robust AutoDL. TPAMI , pages 3079-3090, 2021.

A. Falkner, G. Friedrich, K. Schekotihin, R. Taupe, and E. Teppan. Industrial applications of answer set programming. KI-Künstliche Intelligenz , pages 1-12, 2018.

X. Dong, M. Tan, A. Yu, D. Peng, B. Gabrys, and Q. Le. Autohas: Differentiable hyperparameter and architecture search. arXiv:2006.03656 [cs.CV] , 2020.

Justin M. Gilmer, George E. Dahl, and Zachary Nado. init2winit: a jax codebase for initialization, optimization, and tuning research, 2021. URL http://github.com/google/init2winit .

Y. Nesterov. A method of solving a convex programming problem with convergence rate O(1/sqr(k)). Soviet Mathematics Doklady , 27:372-376, 1983.

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics , 9:53-68, 2021.

Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. One billion word benchmark for measuring progress in statistical language modeling. CoRR , abs/1312.3005, 2013. URL http://arxiv.org/abs/1312.3005 .

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https://github.com/ facebookresearch/xformers , 2022.

Ondˇ rej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 1-46, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL http://aclweb.org/anthology/W15-3001 .

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR'16) , pages 770-778. CVF and IEEE CS, IEEE, 2016.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. IJCV , 115(3):211-252, 2015.

T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, and R. Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'16) , pages 785-794. ACM Press, 2016.

N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient Hyperparameter Optimization. In Z. Zhou, editor, Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI'21) , pages 2147-2153, 2021.

J. Siems, L. Zimmer, A. Zela, J. Lukasik, M. Keuper, and F. Hutter. NAS-bench-301 and the case for surrogate benchmarks for Neural Architecture Search. arXiv:2008.09777v4 [cs.LG] , 2020.

B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. N. van Rijn, and J. Vanschoren. Openml benchmarking suites and the openml100. arXiv:1708.03731v1 [stat.ML] , 2019.

J. Kiili, E. Laaksonen, R. Turner, D. Eriksson, S. Park, M. Mccourt, Z. Xu, and I. Guyon. Black box optimization challenge, 2020.

A. Klein, L. Tiao, T. Lienart, C. Archambeau, and M. Seeger. Model-based Asynchronous Hyperparameter and Neural Architecture Search. arXiv:2003.10865 [cs.LG] , 2020.

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33 , 2020. URL http://arxiv.org/abs/1910.06403 .

B. Matern. Spatial Variation . Lecture Notes in Statistics. Springer New York, 2013. ISBN 9781461578925. URL https://books.google.se/books?id=HrbSBwAAQBAJ .

Frank Hutter, Lin Xu, Holger H. Hoos, and Kevin Leyton-Brown. Algorithm runtime prediction: Methods & evaluation, 2013.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to enhance the random sampling component of hyperband. The authors propose replacing it with a combination of random sampling, prior-based sampling, and incumbent-based sampling. They also suggest adjusting the proportion of these samplers based on the current state in the hyperparameter optimization process. The authors perform experiments on a series of benchmarks and compare the proposed method with multiple classical HBO baselines.\n\nSTRENGTHS:\nThis work might be useful in some particular situations.\n\nWEAKNESSES:\n- The motivation for this work may appear artificial, aiming to reduce the cost of hyperparameter optimization in the age of deep learning. However, the paper lacks a detailed explanation of how the proposed adjustments to the sampling component can make hyperparameter optimization more practical and cost-effective in the era of deep learning. Can the authors provide a more comprehensive reasoning process to support this claim?\n\n- Technically, this method is essentially a combination of prior work, including multi-fidelity optimization [1], expert priors [2], and local search [3]. The authors' contributions primarily build upon and benefit from these existing approaches, rather than introducing original ideas. The authors declare that their approach fulfills all the desired requirements for application to deep learning, but this claim is essentially derived from the benefits provided by multi-fidelity optimization.\n\n[1] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1), 6765-6816.\n\n[2]Hvarfner, C., Stoll, D., Souza, A., Lindauer, M., Hutter, F., & Nardi, L. (2022). $\\pi $ BO: Augmenting acquisition functions with user beliefs for bayesian optimization. arXiv preprint arXiv:2204.11051.\n\n[3]Wu, Q., Wang, C., & Huang, S. (2021, May). Frugal optimization for cost-related hyperparameters. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 10347-10354).\n\nQUESTIONS:\nWhy modifying the random sampling component of HyperBand can help decrease the expenses associated with tuning deep learning models?\n\nLIMITATIONS:\nAs a hyperparameter optimization algorithm, the proposed method encompasses several hyperparameters that may significantly influence its final outcomes. These include the hyperparameters associated with the perturbation operation and those control the proportion of the three sampling methods. However, there is currently a dearth of corresponding ablation studies that specifically investigate the individual contributions of these hyperparameters.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PriorBand, an extension of HyperBand that adds expert priors and a novel sampling technique to replace random sampling in HB, called the Ensemble Sampling Policy (ESP). The ESP allows the algorithm to lean on the expert prior, but also use the current incumbent in case the prior is non-optimal. Under good and bad priors, the authors demonstrate the efficacy of this approach, showing additionally that the ESP also improves the performance of other Successive Halving (SH)-based methods.\n\n**Rebuttal:** The authors have clearly addressed my comments in their rebuttal. I have updated my score from 5 to 7.\n\nSTRENGTHS:\n- My favorite part of this paper is Section 7.2, which shows that the ESP proposed in this paper extends to other SH-like HPO methods as well. This contribution should perhaps be highlighted more.\n- The paper's verbiage is very clear, and examples presented in the introductory sections are useful aids to readers who may not be familiar with HyperBand's workings.\n\nWEAKNESSES:\n- My major concern is in the use of average relative rank to demonstrate efficacy. While a lower rank indicates better performance, it does not indicate whether that performance delta is statistically significant. As opposed to confidence intervals over *ranks*, I would much rater see the actual improvement in per-dataset metrics. For example, the original HyperBand paper [1] showed the test error over wall time for multiple HPO algorithms.\n- minor: In L130, you're missing a period.\n- L182: typo: modelling --> modeling\n\n[1] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research, 18(1), 6765-6816.\n\nQUESTIONS:\n- In Alg. 2, L5, can you explain why $p_\\pi$ is used instead of $p_{\\hat{\\lambda}}$?\n- In Fig. 14, it seems the lines have confidence intervals. How are these computed, and how many repeats of the experiment were used?\n- In Appendix D.3, the authors describe the way a \"good\" prior is generated, using the best of 25 configurations. In practice, this would add computational overhead to the overall process of training a model with good hyper-parameters. In that sense, perhaps it should be emphasized that the Bad prior results are more useful to a practitioner, who, following their intuition/expertise/random choice, might possibly use a non-optimal prior (i.e., without running anything). Can you comment on this?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work appropriately in the main text and the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper \"PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning\" the authors propose an extension to the well-known HPO methods Hyperband by adapting the way how candidate hyperparameter settings are sampled. To this end, the authors propose to use a weighting mechanism to balance between three sampling distributions: random, locally random close to the incumbent, and a prespecified prior. The latter allows experts to inject beliefs about the optimum. In the empirical study PriorBand is found to perform best on average among the considered methods.\n\nSTRENGTHS:\n+ PriorBand compares favorably to its competitors and allows for hyperparameter optimization with comparably low budgets.\n+ The paper is very well written and the presentation in general is excellent.\n+ The authors put very much effort into making their work reproducible, openly accessible and easy to understand.\n+ For the self-imposed desiderata, PriorBand offers the most desired properties.\n\nWEAKNESSES:\n- The work is pretty incremental. The only original contribution of this work is to weight the different probability distributions from which candidates are sampled.\n- It is not clear what effect the incumbent-sampling has on the overall performance. At least I could not find any ablation studies in this regard. Is it really necessary to include incumbent-sampling? What would happen if only random sampling is balanced against prior-sampling? Is it then harder to retain decent performance in the light of bad priors?\n- Nothing is stated about the impact on theoretical guarantees that are known to hold for Hyperband. Does the theoretical framework of Hyperband still apply for the changed distributions?\n- When comparing different hyperparameter optimizers only relative ranks are\n\nQUESTIONS:\n- What is the individual effect of incumbent-based and prior-based sampling?\n- What is the impact on the theoretical guarantees for Hyperband? Are all the assumptions still fulfilled?\n- What is the definition of \"relative rank\"? Is it only a rank or does it also include performance differences?\n- Where do these desiderata stem from? Although they all seem intuitive it is not clear whether they are exhaustive and what the coverage is like.\n\nLIMITATIONS:\nLimitations of PriorBand are well discussed. However, it is not entirely clear to what extent the desiderata for HPO in the age of deep learning is limited or maybe even exhaustive.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents PriorBand, a hyperparameter optimization (HPO) algorithm designed specifically for deep learning models. PriorBand fulfills six key requirements and tackles the shortcomings of existing HPO methods that are unsuitable for DL. It leverages cheap proxy tasks while considering expert input. The algorithm eliminates the need for a naive solution for integrating expert domain knowledge into HPO. Experiments across a wide array of DL tasks are conducted to demonstrate the efficiency and robustness of PriorBand.

STRENGTHS:
+ The proposed algorithm is simple yet effective. The augmented prior based on the existing HyperBand algorithm is technically sound and exhibits clear improvement over baselines.

+ The paper is well-written and well-structured. Motivation is clearly stated before the introduction of detailed algorithms (e.g., Sect. 4).

WEAKNESSES:
- Though the experiments conducted in the paper are quite comprehensive, I found most of them are moved to the appendix where the figures in the main paper are of limited information (most are about robustness to bad priors). I would suggest authors rearrange the paper by moving figures from Appendix F to the main text.

- In Table 1, given the fact that all the baseline methods satisfy "Mixed search spaces" + "Speedup under parallelism", I would suggest the authors remove these two rows as they are also not the major technical contributions of the proposed method. Maybe instead, replace with one sentence in the caption highlighting both the baselines and the proposed method can satisfy these two criteria.

QUESTIONS:
See Weakness.

LIMITATIONS:
See Weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06