Deep Power Laws for Hyperparameter Optimization
Authors: Arlind Kadra
Maciej Janowski
Martin Wistuba
Josif Grabocka
Venue: Submitted to ICLR 2023 (2023)
Keywords: hyperparameter optimization
multi-fidelity optimization
power laws
deep neural networks
deep power laws.

Abstract: Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the issue of hyperparameter optimization, however, most of the methods do not exploit the scaling law property of learning curves. In this work, we propose Deep Power Law (DPL), a neural network model conditioned to yield predictions that follow a power-law scaling pattern. Our model dynamically decides which configurations to pause and train incrementally by making use of multi-fidelity estimation. We compare our method against 7 state-of-the-art competitors on 3 benchmarks related to tabular, image, and NLP datasets covering 59 diverse search spaces. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors.

Under review as a conference paper at ICLR 2023
DEEP POWER LAWS FOR HYPERPARAMETER OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the is- sue of hyperparameter optimization, however, most of the methods not ex- ploit the scaling law property of learning curves. In this work, we propose Deep Power Law (DPL), neural network model conditioned tions that follow power-law scaling pattern. Our model dynamically decides which configurations to pause and train incrementally by making use multi- fidelity estimation. We compare our method against 7 state-of-the-art com- petitors on benchmarks related to tabular, image, and NLP datasets cover- ing 57 diverse search spaces. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors. We open-source our implementation and make our code publicly available at: attps::/ /anonmoosnn 4oper. sciencee//Deeeeeeeeeeee
yield predic-
INTRODUCTION
Hyperparameter Optimization (HPO) is a major challenge for the Machine Learning community. Unfortunately, HPO is not yet feasible for Deep Learning (DL) methods due to the high cost of eval- uating multiple configurations. Recently, Gray-box HPO (a.k.a. multi-fidelity HPO) has emerged as promising paradigm for HPO in DL, by discarding poorly-perfomming hyperparameter configura- tions after observing the validation error on the low-level fidelities of the optimization procedure (Li al. 2017: Falkner al. 2018, Awad al. 2021
Li al.) 2020). The advantage of gray-box Parker-Holder et al. 2020), or meta-gradient
HPO compared to online HPO (Chen et al.] 2017
HPO (Maclaurin et al.) 2015 Franceschi et al. 2017 Lorraine et al. 2020 the ability tune all
types of hyperparameters.
In recent years, a stream of papers highlights the fact that the performance of DL methods is pre- dictable (Hestness et al. 2017), concretely, that the validation error rate is a power law function of the model size, or dataset size (Rosenfeld et al. 2020 202I). Such a power law relationship has been subsequently validated in the domain of NLP, too Ghoorbani et al. 2022). In this paper, demonstrate that the power-law principle has the potential to be gameechanger in HPO, because can evaluate hyperparameter configurations in low-budget regimes (e.g. after few epochs), then estimate the performance on the full dataset using dataset-specinf power law models.
We introduce Deep Power Law (DPL) ensembles, a probabilistic surrogate for Bayesian optimiza- tion (BO) that estimates the performance of hyperparameter configuration at future budgets using ensembles of deep power law functions. Subsequently, novel proposed flavor of BO dynamically decides which configurations to pause and train incrementally by relying on the performance esti- mations of the surrogate. We demonstrate that our method achieves the new state-of-the-ar in HPO for DL by comparing against strong HPO baselines, and 57 datasets of three diverse modalities (tabular, image, and natural language processing). As result, we believe the proposed method has the potential to finally make HPO for DL a feasible reality. Overall, our contributions can be summarized as follows:
We introduce novel probabilistic surrogate for gray-box HPO based on ensembles of deep power law functions.

---

Under review as a conference paper at ICLR 2023
We derive simple mechanism to combine surrogate with Bayesian optimization. Finally, we demonstrate the superiority of our method against the current state-of-the-ae in HPO for Deep Learning, with a very large-scale HPO experimental protocol.
2 RELATED WORK
Multi-fidelity HPO relaxes the black box assumption by assuming it has access to the learning curve of hyperparameter configuration. Such learning curve is the function that maps either training time or dataset size, to the validation performance. The early performance of configurations (i.e. first segment of the learning curve) can be used to discard unpromising configurations, before waiting for full convergence. Successive halving (Jamieson & Talwalkar (016) is a widely used multi-fidelity method that randomly samples hyperparameter configurations, starts evaluating them, and ends fraction of them upon reaching a predefined budget. Afterward, the budget is multiplied by the fraction of discarded hyperparameter configurations and the process continues until the maximum budget is reached. Although the method relies only on the last observed value of the learning curve, very efficient. In recent years, various flavors of successive halving have been elaborated, including Hyperband (Li et al. 2017), which effectively runs successive halving in parallel with different settings. A major improvement to Hyperband is replacing random search with a more efficient sampling strategy (Awad et al. 2021 Falkner et al. (2018). However, the only assumption these methods make about the learning curve is that it will improve over time. contrast, we fit surrogates that exploit power law assumption on the curves.
Learning curve prediction is a related topic, where the performance of a configuration predicted based partially observed learning curve. Typically, the assumptions about the learning curve are much stronger than those described above. The prediction is often based on the assumption that the performance increases at the beginning and then flattened towards the end. One way to model this behavior is to define a weighted set of parametric functions (Domhan et al., 2015 Klein al. 2017]. Then, the parameters of all functions are determined so that the resulting prediction best matches the observed learning curve. Another approach is to use learning curves from already evaluated configurations and to find an affine transformation that leads to well-matched learning curve (Chandrashekaran & Lane 2017). A more data-driven approach is to learn the typical learning curve behavior directly from learning curves across different datasets (Wistuba & Pedapati] 2020).
Learning curve prediction algorithms can be combined with successive halving (Baker et al. 2018)
In contrast to this line research, we actually fit ensembles power law surrogates for conducting multi-fidelity HPO with Bayesian optimization.
Scaling laws describe the relationship between the performance of deep learning models as a func- tion of dataset size model size. Concretely, Hestness et al. (2017) show empirically for differ- ent data modalities and neural architectures that power law re relationship holds when growing the dataset. Further work confirms this observation and extends it by demonstrating the power law lationship also with regard to the model size (Rosenfeld et al.] 2020 2021, Ghorbani et al.) 2022). From a practical angle. Yang et al.) (2022) propose tune hyperparameters on small-scale model and then transfer it to a large-scale version. In contrast to these papers, we directly use the power law assumption for training surrogates in Bayesian optimization for HPO.
3 PRELIMINARIES
Hyperparameter Optimization (HPO) demands finding the configurations À E A of a Machine Learning method that achieve the lowest validation loss ((Val) of a model (e.g. a neural network), which is parameterized with and learned to minimize the training loss (Trii)) as:
X == arg min [(Val) (2, 0r (A)). XEA
s.t. 8" (A) = arg min (Trrai)) (2,
(1)
DEe
For simplicity we denote the validation loss as our function of interest f(X) = [(Va)) (A, 8" (A)).
The optimal hyperparameter configurations! X* of Equation 1 are found via HPO policy A (also
2

---

Under review as a conference paper at ICLR 2023
called an HPO method) that given history of N evaluated configurations H fA, (A:)) suggests the (N + 1)-th configuration to evaluate as AN+1 := A(H) where A: [A x [R.]" A. The search for an optimal HPO policy is bi-objective problem in itself, aiming at (i) finding con- figuration out of evaluations that achieves the smallest validation loss f()), and (ii) ensuring that the costs of evaluating the N configurations do not exceed total budget Q2 as shown in Equation 2
arg min min / (Ai=
A ((1((-1))
(2)
iE Ny
[(A)))
1>0
where:
1160
0
i=
N
subject to: Ω
>
cost (f (A:))
Bayesian optimization (BO) is the most popular type of policy for HPO, due to its ability balance the exploration and exploitation aspects of minimizing the loss f. Technically speaking, BO fits surrogate f(A; 0) parametrized with 0 to approximate the observed loss f()) using the history H, as 8" = arg ming (())))))) ~px P(f(A) X 0). Afterwards, BO uses an acquisition/utility function
a: A + 2 to recommend the next configuration as AN+1 == A (H())) = arg maxXeλ a (A; 0*). typical acquisition choice is the Expected Improvement (Mockus et al. 1978). For more detailed introduction to BO and HPO we refer the interested reader to Hutter et al. (2019)
Gray-box fiulti- HPO refers the case where an approximation of the validation loss can be measured at a lower budget b E B, where B = (0, max budget]. For instance in Deep Learning we can measure the validation loss after few epochs (0 b < c), rather than wait for full convergence (b = max budget). Throughout this paper the term budget refers to the number of optimization epochs. The evaluation g(Val) (X, 8" (>, b)). where f (>, b): A + R2+ The concept of budgets alters the HPO problem definition slightly. The history of N configurations evaluated at different budgets becomes set of N triples (config, budget, eval) defined H == {(),, b,, f (A;, b,)) A gray-box HPO policy is still optimized for Equation however, the constraint is altered as Ω > E cost (f (Ax, b.)).
configuration À for a budget & defined as ʃ (A, b) :=
POWER LAW SURROGATES FOR BAYESIAN OPTIMIZATION
Prior work has demonstrated that the performance of Machine Learning methods as a function of budgets (i.e. dataset size, number of optimization epochs, model size, image resolution) follows a power law relationship (Rosenfeld et al. 2020 202T). In this work, we employ this power law dependence between the validation loss and the number of optimization epochs in Deep Learning. We propose novel gray-box Hyperparameter Optimization method which is based on power law surrogates. We assume that every learning curve I (A, can be described by power law function defined by (a,, 3, 7). Concretely, we define a power law function modelling the validation loss for a configuration budget (a.k.a. number of epochs) as shown in Equation 3
(A, + ax Bx, VA R
We propose to use surrogate models employing the power law Bayesian optimization (BO) to optimize hyperparameters. We suggest learning shared power law function across all configu- rations by conditioning the power law coefficients a, 3 y on À using parametric neural network g that maps configuration the power law coefficients of its learning curve as g A + R3. The network has three output nodes, corresponding to the power law coefficients, denoted as g(A)a, (())))))))))))) (N)s, {()),- The confiuraaion--oondtionee power law surrogate becomes:
f (A,b) = g())a + +(()a b-a(A), g g:A- A 3
(4)
3

---

Under review as a conference paper at ICLR 2023
Using history of learning curve evaluations H := {A,,, b f (A,,)))))= law surrogate to minimize the following loss function using stochastic gradient descent:
train the power
arg min (x,)).(( (Ac, b,) (o(A)aa + 9(Ne) 3 b, s(Ao)a))
(5)
BO surrogates need to be probabilistic regression models because the acquisition functions require the posterior variance of the predictions. As a result, train ensemble of K diverse surrogates (A,b),.... (2, (x,b) b) with the Deep Ensemble strategy (Lakshminarayanan et al., 2017), by initializing each surrogate with different weights and by training with a different sequence of mini- batches. The posterior mean μ and the posterior variance o² of the power law ensemble are trivially computed as:
(2(A,
b)
(A,
K
km
1 03((,,b) ス K
Σ
The acquisition function of our approach relies on selecting the configuration X with the lowest estimated loss at the full budget. other words, we select the hyperparameter configurations that are expected to achieve the best performance at the end of the optimization procedure.
commonly used acquisition function the domain Expected Improvement (EI) which incor- porates both the mean and uncertainty of predictions, applying a trade-off between exploration and exploitation. Consequently, in our work, we use the Expected Improvement (EI) acquisition with the estimated full budget's (b max_ budget) posterior mean and variance.
EI(A, bHH) = E max ( (1)(A, - f(beatt (b= max max.eudget) soodgett).o)] 00]
X pext := arg max EI ()))
The best observed loss min {f (A,8) (A of (A, H A < b}. We briefly define the acquisition Equation and refer the reader to Mockus et al. (1978) for the details of the EI.
until
budget b is denoted as (best) (b)
However, after selecting a configuration with our variant of the EI acquisition, we do not naively run it until convergence. Instead, we propose a novel multi-fidelity strategy that advances the selected next of Equation 7 by small budget of Jeste, e.g. epoch of training. Therefore, the selected uext will be evaluated at joest as defined in Equation g Notice our proposed strategy also covers new configurations with no learning curve evaluations in H.
batpp btep
next (ADoat ,
; E H
next
max b, otherwise (A D.)
A PROOF-OF-CONCEPS EXAMPLE
To visually demonstrate the power law surrogate, we created a 1-dimensional search space where we train a Preact ResNet (He et al. 2016) on the CIFAR-10 dataset (Krizhevsky et al. 2009). We train multiple versions of the model applying a different dropout hyperparameter À E 0.1, for 50 epochs, with a cosine annealed learning rate and an initial value of 10- We train our power law surrogate curves for the aforementioned hyperparameter subspace, except for the hyperparameter configura- tion corresponding to 0.65, where, only subset of the learning curve. Ideally, our power
subspace of the hyperparameter search space. We use the full validation
4

---

Under review as a conference paper at ICLR 2023
0.90
0.85
0.8
0.7
0.80
Accuracy
0.6
0.75
0.5
Real values
0.70
Predicted
Evaluated Points
Unobserved Values
0.3
Partial Evaluated Points
Surrogate Prediction
0.60
0.2
40
Drapoul rate
Number of epachs
The power law predictions and uncertainty estimations the CIFARIO 1-D search space. Left: The maximal budget across different hyperparameter configurations (dropout rates). Right: Across different budgets for the same hyperparameter configuration that observed partially.
Figure 1:
law surrogate should model a low uncertainty in the observed hyperparameter configurations and a higher uncertainty in the partial or unobserved regions.
As shown in Figure T our power law surrogate fits the training data well by generalizing correctly across different hyperparameter configurations. Furthermore, the power law surrogate models low uncertainty in the region of observed hyperparameter configurations, and higher uncertainty in the region of partially observed configurations that scales more the further observed points. Additionally, Figure [] (right) conveys that when we evaluate the hyperparameter configuration that is only observed partially, the uncertainty scales with the budget. More concretely, the surrogate is more uncertain the higher the budget (epoch) for which we predict compared to the last observed value.
from the region of
EXPERIMENTAL PROTOCOL
In our experiments, we standardize the data by performing min-max scaling for our method and every baseline included. If a bascline has specific preprocessing protocol, do not apply min- max scaling but we apply the protocol as suggested by the authors. In the following experiments, report the regret of the best configuration found > best which is defined as:
R (Xtmart max (0, florade) (b max budget) -f (thest) (b max mrax_budget)
(9)
where (oraalee) (b = max budget) == min (A) (A. E HAA b). In short, the regret is the difference the metric performance from the best possible hyperparameter configura- tion (oracle) the dataset to the best-found hyperparameter configuration by method. The metric benchmark-speciffic, since the benchmarks do not support a common metric. On a dataset level, report the average regret across 10 repetitions with different seeds. When reporting results over all datasets, we report the averaged normalized regret. The normalized regret for each dataset is calculated by dividing the regret by the difference between the best value and the worst value spe- cific to the dataset. We normalize the regret because over all datasets, there are regret distances of different scales, which in results for all datasets. For more information regarding the detailed implementation of our method, refer the reader Appendix
lead to low regret values that dominate the overall averaged
6.1 BENCHMARKS
LCBench: A benchmark that features 2,000 hyperparameter configurations that parametrize the
architecture of simple feedforward neural networks, as well as, the training pipeline (Zimmer et al. 2021). The benchmark features numerical hyperparameters. The hyperparameter configurations
were evaluated for epochs on 35 different datasets from the AutoML benchmark (Gijsbers et al,
2019). When reporting results for LCBench, the balanced accuracy metric.
5

---

Under review as a conference paper at ICLR 2023
PD1: A deep learning benchmark (Wang al. 2022) that consists of recent DL architectures on large vision datasets such as CIFAR-I0, 10, CIFAR-IOO, ImageNet, as well as statistical modeling corpora and protein sequences datasets from the bioinformatics domain. Every search space fea- tures varying learning curve lengths, ranging from 5 to 1414, and a different number of evaluated hyperparameter configurations ranging from 807 2807. The search space includes hyperparame- ter configurations that parametrize the learning rate, the learning rate scheduler and the momentum. When reporting results for the PDI benchmark, we use the accuracy metric.
TaskSet: A benchmark that features different optimization tasks evaluated in 5 different search spaces (Metz al. 2020). For our work, we focus on the Adam8p search space, which among the largest search spaces in the benchmark with 1000 hyperparameter configurations, every hyper- parameter configuration featuring continuous hyperparameters. The hyperparameters control the learning rate, the learning rate schedulers and the optimizer. Every run consists of 50 steps, where, every step corresponds to 200 training iterations. For variety among our benchmarks, we focus on 12 NLP tasks. When reporting results for TaskSet use the loss metric since the benchmark does not offer information regarding accuracy.
6.2 BASELINES
Random Search:
Randomly samples hyperparameter configurations for the largest possible budget.
Uses multiple brackets with different trade-off's of the initial budget and number of epochs to initially train (Li al. 2017). then further applies Successive Halving (SH) (Jamieson & Talwalkar 2016) every individual bracket to decide which configurations to further train and which configurations to stop training.
Hyperband:
ASHA: An asynchronous version of SH (Li et al. 2018) that does not wait for all configurations
finish in a bracket before running for the next fidelity.
BOHB: An extension of Hyperband that replaces the random sampling of hyperparameter
figurations in the initial brackets with model-based sampling (Falkner et al.] 2018). BOHB uses
TPE (Bergstra et al., 2011) as an inner model and has individual model for every fidelity.
DEHB: An additional extension of Hyperband, which differs from BOHB by using evolutionary strategies sample the initial hyperparameter configurations (Awad et al. 2021).
SMAC: A method that extends Hyperband but uses random forests to sample the initial hyperpa- rameter configurations for bracket (Lindauer et al. 2022).
Dragonfly: We use the Dragonfly Library (Kandasamy et al. 2020 to compare against BOCA (Kandasamy et al. 2017). multi-fidelity method that uses Gaussian processes to predict the next hyperparameter to evaluate and the fidelity for which it should be evaluated.
For all the baselines, we use their official public implementations. We provide additional details in Appendix C
7 RESEARCH HYPOTHESES AND EXPERIMENTAL RESULTS
Hypothesis 1:
The power law assumption improves the quality of the learning curve predictions.
Initially, we compare power laws against GPs and simple feedforward neural networks by fitting in- dividual instances of every algorithm on partially observed learning curve for every corresponding hyperparameter configuration. We repeat the procedure with different fractions of partially observed learning curves for every dataset in the I.Ceech benchmark. We check the accuracy of each model in predicting the final performance by calculating the correlation. Lastly, investigate the perfor- mance of a simple feedforward neural network and Deep Power Laws (DPLs) conditioned on the hyperparameter configurations. In particular, every hyperparameter configuration, but, we fit a model for all partial learning curves of different hyperparameter configurations, learning shared model for all configurations.
do not fit model on the partial learning curve of
As can be seen in Figure 2 DPLs manages to have the highest accuracy compared to all the con- sidered model algorithms. Observing the results, fitting an individual power law learning curve

---

Under review as a conference paper at ICLR 2023
Average Noralizzd Regret
LCBench
TaskSet
10
DPL
BOHB
10
10"
DEHB
Hyperband
Random
10
SMAC
10"
Epochs
500
1000
Steps
500
1000
0.2
Fraction tutal Banaiorna
Figure 3: A comparison of all methods considered in the experiments over the number of epochs/steps for all the considered benchmarks. We report the average normalized regret for all methods. A step corresponds to 200 iterations in the case of TaskSet.
achieves predictions of higher quality compared to fitting a simple feedforward neural network or Gaussian processes.
Furthermore, conditioning neural network to output based on a power law formulation yields DPI. that can share information across hyper- parameter configurations while improving ac- curacy compared to the individual power laws. As expected, the effect is more noticeable in the small data regime, where, small subset of observed points from the learn- ing curve. Lastly, DPL is less expensive to fit on benchmark compared to fitting an in- dividual power law for every hyperparameter configuration which does not scale and it over- comes the issue of fitting an individual power law, which requires data points for every hyper- parameter configuration. Based on the results, consider Hypothesis 1 to be validated and that DPLs yield qualitative predictions that surpass different models fitted on individual hyperparameter configurations.
Correiation: Est. vs. True
0.8
0.6
observe only
0.4
0.5 Fraction of Leaming Curve Length
0.1
0.2
0.3
0.4
DAL
Cond NN
PL NN
GP
The correlations of individual mod- els, and the DPL. for different fractions of ob- served learning curves across datasets. DPL.: Deep Power Law, Cond NN: Conditioned neural network, PL: Power Law, NN: Neural Network, GP: Gaussian processes.
Figure 2:
Hypothesis 2:
Our method DPL achieves state-of-the-art results in HPO.
Initially, we compare the average normalized regret of our method against all competitors over the number epochs for the benchmarks. Figure we show the performance all the methods considered in the experiments over the number of epochs, observed, DPLs manage to outperform in all the considered benchmarks. In the case of LCBench, DPLs converge faster to better solution compared to the competitor methods and continue optimization process ends. Furthermore, we observe the same trend with TaskSet, only in 25% of the optimization process the DPLs converge to a better solution compared all the considered baselines and increase the lead until the optimization process ends. Lastly, since PD1 features datasets with different learning curves, consider budget of 20 full function evaluations for every dataset (roughly the same number of full function evaluations as in the other two benchmarks). We normalize the budget by dividing the number of epochs by the total number of epochs per dataset. As seen in Figure our method converges slower compared to DragonFly, the closest baseline in the benchmark, however, when 75% of the optimization process reached, our method matches the baseline in performance and continues to improve until the end of the optimization process, once again, performing best compared to all basclines.
increase the gap in performance until the
In addition to the average normalized regret, Figure provides the critical difference diagrams that show the average dataset ranks of all methods for the different benchmarks, where, DPL consistently achieves the best rank with statistically significant difference in the majority of cases. In particular, we provide the critical difference diagrams during half the optimization procedure and at the end of the optimization procedure. As observed, for I. LCBench and TaskSet, DPL. achieves the best per-
7

---

Under review as a conference paper at ICLR 2023
LCBench@50%
TaskSet@50%
PD1
Random
Random ASHA Dragonfly
DPL BOHB
ASHA Hypamann
BOHB Dmgontly Random
Dragonily Hape SMAC
Rypebaad
SMAC
LCBench@1000 100%
TaskSet@ 100%
PD1 100%
6
7
Random
Random ASHA
ASHA
DPL ROMB Random
ASHA SMAC
BOHB Ryperband
Hyperbans
Dragentl
DEMB
Draganily
Figure 4: The critical difference diagrams for DPL and the other baselines in all the considered benchmarks. Top: The The numbers indicate the average ranks across all datasets (the lower the better). Connected lines correspond to methods whose result difference not statistically significant.
LCBench
PD1
Average lormalized Regret
ASHA
BOHB
DEHB
Dragonfly
10"
Hyperband
Random
SMAC
Normllized Total Walltime
Nomalized Total Walkime
Figure 5: The performance of rival methods over the normalized time for all the benchmarks con- sidered. We report the average normalized regret for all baselines.
formance with a statistically significant difference in results only halfway through the optimization procedure, retaining the statistical significance until the end of the HPO procedure. In the case of PDI, our method is among the best three methods during half of the optimization procedure, with non-significant difference compared to the other two top-3 methods. Although at the end of the optimization procedure our method does not have a statistical difference in the results compared to the other methods, it still achieves the best rank over all datasets.
Lastly, we analyse the performance of DPL and all the other methods considered in the experiments in 5 where, it can be observed, DPL manages to outperform the competitors even when method's overhead time is included, showing that the time overhead of DPL is negligible in the results. In more detail, the total time includes the time to evaluate hyperparameter configuration and the time taken by each respective method in its inner process to suggest the next hyperparameter configurations evaluate. Additionally, before the results are averaged for all datasets, the per- dataset time is normalized by the time it took random search to finish the optimization procedure. TaskSet included in Figure since does not offer information regarding the time. Given the results, we conclude that Hypothesis 2 is validated and that DPL manages to achieve state-of-the. art performance.
Hypothesis 3: DPL explores the search space more efficiently compared to the baselines.
We conduct further analyses to understand the source of the efficiency of our method DPL versus the baselines. As a result, we analyze two important aspects, the quality of the evaluated configurations, well as the exploration capability of our gray-box HPO. Initially, Figure 6 (left) shows that the top 1% configurations (ranked by accuracy) our method considers. method discover significantly more top configurations in the datasets of the LCBench benchmark, compared to baselines.
measure what fraction of
until convergence
The middle plot of Figure shows the average regret for each configuration promoted to the respec- tive budget. According to this plot, DPL is more efficient by assigning budget only to configurations with lower regret compared to the other methods. The precision and regret plots demonstrate that
8

---

Under review as a conference paper at ICLR 2023
Precssor of Top Creddtiens
LGilendh
LCBench
LCBench
0.88
0.4
Brunuge Raggne
0.16
Frecttion of Piaor
0.6
0.4-
02-
0.0
0.0
50
Numberot
Number Epochs
DRL
Dragonty
Hyseebond
Diagonty
hypettond
Drogority dy
Booeline
60HB
BOHB
ASI
Figure 6: Study of DPL's efficiency over the course of HPO. Left: Share of the best candidates selected during training. Middle: Average regret of configurations chosen to be trained at each budget. Right: Share of top third configurations given budget which were bottom two third configurations at previous budget.
the quality of the evaluated configurations is largely better than all baselines, therefore, giving our method significant lift in the performance rank. Last but not least, the right plot shows the percent- age of configurations that were performing poor in an earlier epoch (i.e. accuracy-wise in the bottom 2/3 of configurations up to the epoch indicated at the x-axis) but performed better at later epochs (i.e. at the top 1/3 of configurations). Furthermore, we added a line labeled with "Baseline", which represents the fraction of previously poor-performing configurations of all configurations. Such be- havior is observed often with learning curves, for instance, strongly regularized networks converge slowly. The results indicate that our method can better explore unpromising early configurations, by giving them chance through the uncertainty estimation of our ensemble, and the respective Bayesian optimization mechanism.
LIMITATIONS
Our proposed ensemble of power law functions achieves an important gain in performance concern- ing the state-of-the-ar in gray-box HPO and highlights the efficiency of modeling learning curves with the power law assumption. However, we believe that further research needed calibrate the power law model for the beginning and the end parts of the learning curves. Recent work highlighted that the error rate at very small fidelities (c.g. after few mini-batches) is not a power law (Rosenfeld et al. 202I). Contrary to the common expectations, we experienced that the uncertainty estimation arising from the Deep Ensemble (Lakshminarayanann et al. 2017) is not very qualitative compared to standard BO surrogates such Gaussian Processes. In addition, having train ensemble has additional computational costs, due to the necessity of training multiple power law models. In the future, plan re-conceptualize our surrogate, by combining power laws with Gaussian Processes.
CONCLUSIONS
In this work, we introduce Deep Power Law (DPL), a probabilistic surrogate based on an ensemble of power law functions. The proposed surrogate is used within a novel gray-box HPO method based on Bayesian optimization. In contrast to the prior work we exploit the recently-discovered scaling laws for estimating the performance of Deep Learning models. Through extensive experiments comprising 7 baselines, 57 datasets, and search spaces of diverse deep learning architectures, we showed that DPL outperforms strong HPO baselines for DL by large margin. an overarching contribution, we advanced the state-of-the-art in the important field of HPO for Deep Learning.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pp. 2147-2155 2021.
doi: 10.24963/ijcai.2222.
URL [RLlhttps///cc..oo
10. 24.249633/ jcai ptt.:aaa.......................a 2021/296
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representa- fions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings, 2018. URL ftttp:://ppereviiin http://peneeeeeeeeeeeeeeeereeeee www.foorm//eee. HJck HJqk3N1vG
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
Akshay Chandrashekaran and Ian R. Lane. Speeding up hyper-parameter optimization by extrapo- lation of learning curves using previous builds. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I, volume 10534 of Lecture Notes in Computer Science, pp. 477-492. Springer, 2017.
Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy Lillicrap, Matthew Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, pp. 748 2017. URL httttt:://prrcecings....
m r. press/v70/chert 7e html
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 3460-3468. AAAI Press, 2015.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter opti- mization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsssnn Stockholm, Sweden, July 10-15, 2018, pp. 1436-1445, 2018. URL http::://pproccenngg..... http://ppoceeecnnn....eeeeeeeees
html
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1165-1173, /vtres//v0///ee vv0 franceschi 17a. html
2017. URL http:://proceeeennggg mlr prwss/
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtaal Event, April 25-29, 2022. OpenReview.net, 2022. URL /tttps://openreviest htt.://ee.....oooo.............. SMM
Pieter Gijsbers, Erin LeDell, Janek Thomas, Séhastien Poirier, Bernd Bischl, and Joaquin Van- schoren. An open source automl benchmark. arXiv preprint arXiv: 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianine- jad, Md. Mostofa Ali Patwary, Yang Yang, and Yangi Zhou. Deep learning scaling is predictable, empirically, CoRR, abs/17 101.00092, 2017. URL httttttt//aaaae
www//bgg//aaoooooooo..oo... awaaaas/lii 00409
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning Meth- ods, Systems, Challenges. Springer, 2019.
10

---

Under review as a conference paper at ICLR 2023
Kevin G. Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016. pp. 240-248, 2016. URL http://
proceedi ngs ml www.prresssee... sitassvvl... jmmiaammm eson 6. ht.ml
Kirthevasan Kandasamy, Gautam Dasarathy, Jeff G. Schneider, and Barnabás Póczos. Multi-fidelity bayesian optimisation with continuous approximations. In Proceedings of the 34th Interna- lional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1809,1888, 2017. URL BRRLttp::// http:::/ppocccednnneeeeeooo/e/ee www.mmrr.e.... peress/v700 2770/aanaaaayy html
Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabás Póczos, and Eric Xing. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly. J. Mach. Learn. Res.,
21:81:1-81:27, 2020. Utt.ttt::// htt.htttpp//mil...rrr/rrrrr/r//s
223. 3.3ml
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg. and Frank Hutter. Learning curve predic- tion with bayesian neural networks. In 5th International Conference on Learning Representa- lions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https::/epenreviiee net fttp://rmmmid-111kYYY
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL (ef22ed47ffd2c5 1022c11064700 waw.:///7aaaaaateeee--------
wwweeriissccepppper.eeeep wttpappeee//////111 pdf
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint 1XXv:18100.0934, 5, 2018.
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A novel bandit-based approach to hyperparameter optimization. J. Mach, Learn. Res., 18:
185:1-185:52, 2017. URL http: / / jmmr. wwt.oog/////// wttp:pppppp///.///eeeeee// 558. .html
Shibo Li, Wei Xing, Robert M. Kirby, and Shandian Zhe. Multi-fidelity bayesian optimization via deep neural networks, In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. (ogoleoo4...7tssseeeeeeeeeee---.
newrpps hctpp//ccacapper/////sssst
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp. Difan Deng, Car- olin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian op- timization package for hyperparameter optimization. Journal of Machine Learning Research MLOSS, 23(5)):1-9, 2022.
(JMLR) -
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. The 23d International Conference on Artificial Intelligence and Statis- fics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. pp. 1540-1552, 2020. URL http:://proceecinggg/e mlr. wtt.speersss//1ooaaaaaaem
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter opti- mization through reversible learning. In Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 2113-2122, 2015. URL http::/ppoceedee ngs ml press vv / /a caaaclurring 15.html
Luke Metz, Niru Maheswaranathan. Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using a thousand optimization tasks to leam hyperparameter search strategies. CoRR, Abs/2002.118877 2020. URL hhttps://aaxt::
hbsp///a///a 2002. 11887

---

Under review as a conference paper at ICLR 2023
Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization. 2(117-129):2, 1978.
Jack Parker-Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient online hyperparameter optimization with population-based bandits. In Advances in Neural Information Processing Sys- tems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, De- cember 6- 12, 2020, viriual, 2020. URL
Abstracchh
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive pre- diction of the generalization enror across scales. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httptt:://ppeeeeiiieeeeee
Jonathan S. Rosenfeid, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th ternational Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 9075-9083, PMLR, 2021. URL http::/procceeiingg
www.ppres//vlll hwtp://ooe.......eeemeeee ldl d html
David Salinas, Matthiss Seeger, Aaron Klein, Valerio Perrone, Martin Wistuba, and Cedric Archam- beau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In First Conference on Automated Machine Learning (Main Track), 2022.
Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-training helps bayesian optimization too. arXiv preprint 2000:22077.00084, 2022.
Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 10303-10312, 2020. URL http://////t http:://proceedingg ngs. ml wet.ress//vl//wiee..
html
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, 2awp/2203.03466. 2022. doi: 2203. 03466
10.48550 03466. URL/attps://aoi. org.o0..o....o.ooooo 48550/ /ttpattpp.//aolooo/10.....8
I Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch tabular: Multi-fidelig fidelity metalearn- ing for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 43(9):3079 3090, 2021.
A IMPLEMENTATION DETAILS
For our method, use a 2-layer feedforward neural network with 128 units per layer. Additionally, we use batch normalization after every linear layer. We use Leaky ReLU our non-linearity. Our network has 3 output units, which, are then combined to yield the power law output. We apply the Sigmoid non-linearity activation only on the 3 and output units. For the experiments in the pd1 benchmark, we apply a Leaky ReLU non-linearity to improve convergence, since, the benchmark features low eиrоr rate values. We use PyTorch version 1.12 as the main library on top of which we build our method.
We use the MSE loss train our network, coupled with Adam featuring initial learning rate of 10 We train every network of our ensemble for 250 epochs only in the beginning of the hpo optimization process and we continuously refine the model for 20 epochs every hpo iteration. Lastly, we use 5 models to build our ensemble of DPLs. For the experiments, we use an initial history H of 1 randomly sampled hyperparameter configuration evaluated for 1 epoch.
12

---

Under review as a conference paper at ICLR 2023
B BENCHMARKS
LCBench We use the official implementation as the interface for the LCBench benchmark | As suggested by the authors, we use the benchmark information starting from the second step and we skip the last step of the curve since it is repeat of the preceding step.
The TaskSet benchmark features 1000 diverse tasks. We decide focus on only 12 NLP tasks from the TaskSet benchmark to add variety to our entire collection of datasets. Our limitation on the number of tasks included is related to the limited compute power, as we are unable to run for the entire suite of tasks offered in TaskSet. TaskSet features set of 8 hyperparameters, that consists of i) optimizer-speciffe hyperparameters, such as the learning rate, the exponential decay rate, 31 and 32, and Adam's constant for numerical stability ii) hyperparameters that control the linear and exponential decay schedulers for the learning rate decay, and lastly iii) hyperparameters that control the L1 and 1.2 regularization terms. Every hyperparameter in TaskSet except 31 and (32 is sampled logarithmically.
TaskSet:
PD1: We use the synetune library (Salinas al., 2022 for our interface the PD1 benchmark. From the benchmark, we only include datasets that have a learning curve of length greater than 10. We furthermore only include datasets that have a learning curve lower or equal to 50 to have fair comparison between all benchmarks by having approximately 20 full function evaluations. PDI features numerical hyperparameters, Ir_initial_val _vluue Ir-power, Ir tr.decay_steps_ factor and one. minus momentum, where Ir initial value and one minus momentum are log sam- pled. The learning rate decay is applied based on a polynomial schedule.
C BASELINES
Random Search: We implemented random search by randomly sampling hyperparameter configu- rations from the benchmarks with the maximal budget.
Hyperband, BOHB, LCNet: We use version 0.7.4 of the HpBandSter library as common code- E
base for all baselines the method, we could not get stable results across all the benchmarks and hence dropped the method from our comparison.
For the last approach mentioned, despite heavy hyperparameter tuning of
ASHA: For the implementation of ASHA we use the public implementation from the optuna library version 2010...
DEHB: We use the public implementation offered by the authors 5
MF-DNN: In our experiments we used the official implementation from the authors However, the method crashes which does not allow for full results on all benchmarks.
SMAC: For our experiment with SMAC used the official code base from the authors Fl
Dragonfly: We use version 0.1.6 of the publicly available Dragonfly library.
For all the multi-fidelity methods considered in the experiments, we the same minimal and maximal fidelities. In more detail, for the LCBench and TaskSet benchmarks we use a minimal fidelity lower bound of I and a maximal fidelity lower bound equal to the max budget. In the case of PDI, where, the learning curves have different lengths, we use a minimal bound that allows for maximal amount of 4 brackets, if more than 4 brackets can be achieved we use minimal budget of 1.
D PLOTS
htteer//lttaab.com/attt/////a/lt
ottess//ithubb.cc//aatmm////
htteass/iiibbbb.cccc/aa111///1/1 MFRO
http:://githbbbccmm/attt/////
13

---

Under review as a conference paper at ICLR 2023
0.4
Absolute Relative Error
0.3
0.2
0.1
0.0
0.1
0.2
0.3 0.4 0.5
Fraction of Learning Curve Length
Figure 7: The dataset absolute relative error distribution of DPL over the different learning curve fractions. The distribution is calculated from the ground truth and prediction values, averaged over all configurations of dataset.
14


Reviews:
[{"review_id": "z9zF2Qz0pE", "reviewer": "Reviewer_vKZP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the very important topic of hyperparameter selection, by marrying scaling laws and the Bayesian optimization hyperparameter search --- under the assumption of learning curves taking a powelaw form. \n\nThe central assumption in the paper is that training curves, and in particular validation error as function of number of epochs, are well approximated by powerlaws, with the coefficients dependent on the hyperparameter selection.\n\nBased on this assumption the authors propose a bayesian optimization method for hyperparameter search which uses a neural network (NN) estimator of the expected coefficients value of unseen hyperparameters.\nThe NN is trained on a batch of hyperparameters pre-trained configurations (learning curves), multiple times, with seed and sample order permutations, to form an ensemble from which mean and variance are predicted.\n\nThe authors showcase a preliminary demonstration on a CIFAR10 task of predicting the performance while scanning dropout level and show consistent uncertainty behavior with respect to the distance (in dropout level, and in number of epochs) between measured and predicted points.\n\nThe authors then continue to assess the validity of the framework and its performance on 3 benchmarks, relative to several alternative hyperparameter optimization baselines.\n\nFirst, the agreement of the powerlaw-base learning curve is assessed relative to other learning curve prediction methods. Superior relative correlation is demonstrated.\n\nSecond, the authors examine the averaged regret of the proposed method relative to the baselines. And finally the authors examine the exploration / exploitation of the method.\n", "summary_of_the_review": "Hyperparameter efficient optimization remains one of the practical/economical thorns in the field. The paper offers an interesting direction of capitalizing on predictable phenomena in the form of scaling laws for the leveraging in the context of bayesian inference for hyperparameter search.\n\nWhile preliminary results are encouraging, there are numerous weaknesses pertaining to the robustness of the underlying assumptions, performance and practical (namely compute) expected utility.\n\nSignificant work is required for addressing these above mentioned issues, and I will be happy to upgrade my review and score if they are adequately resolved.\n\n%%%%%% updated score following rebuttal clarifications %%%%%%\nsee reply for details --- core issue is concern with respect to actual applicability due to mixed results. Strongly encourage to resubmit with demonstration of the utilization of method claimed superiority to improve performance in the wild", "strength_and_weaknesses": "$\\bf{Strengths: }$\n\nScaling laws and specifically their power law characteristics have emerged as a powerful tool for both practical and theoretical investigation. Attempting to apply this very strong prior for improving in a principled baysian framework for the gains in the efficiency of hyperparameter search is an important and potentially very useful direction.\n\nThe preliminary results in the paper are encouraging relative to current methods examined in some of the cases presented in terms of performance (though not convincingly in all — e.g. no statistically significant difference between the random baseline and DPL as shown in figures 4).\nSimilarly, the claimed efficiency (and demonstrated relative to waltime — figure 5) is of paramount importance, as at the large scale hyperparameter tuning becomes practically prohibitive.\n\n$\\bf{Weaknesses: }$\n\nHowever, there are several methodological improvements which merit attention:\n\n1. The core assumption of power-law learning curves, is poorly qualified.\n\nIt is still very often the case, in practice, that multiple schedules are involved in the training configuration. These, in particular, are subject of (hyperparameter) tuning and materially violate the powerlaw assumption. To name several such very common schedules one may consider warmup and learning rate scheduling (stepped or otherwise as in this paper where cosine scheduling with no repetition is applied at least in part). \n\nThe proof of concept provided on CIFAR10 suffers from several limitations, not contributing to the further the confidence in the power-law assumption:\nThe uncertainty of the estimation diverges very quickly outside the interpolation area (area within the range of measurements) — figure 1 left. Further, the learning curve itself *does not* agree with the prediction, even on the measured area, further indicating that it is probably not actually a powerlaw in this case (as, by construction, the prediction is a powerlaw).\n\n2. The analysis conducted in figure 2, does show the relative superiority of the correlation, however correlation may be too blunt an instrument for the purposes of interest. High correlation may be present with significant (perhaps even rank non-conservation) deviations at the accuracies of interest (where small absolute error differences are large relative differences, in the low error regime).\n\nThese limitations are not merely of aesthetics, qualifying when the results of this method may be trusted is very important in order to substantiate it as a good hyperparameter optimization method. \n\nFor example — from what point in the learning curve is the approximation reliable? (and consequently, what is the expected computational/budget gain?) . and what limitations of applicability does this method impose with respect to the optimization protocol? \n\nAt the least a relative deviation (e.g. error in log scale correlation) should be added, and with a measure not only of the average over all datasets, but rather a breakdown per dataset and with both average (over learning curve agreement with different hyperparameters) and deviations.\nAs well as a discussion and examples of where this prior actually does not agree with the learning curves and fails.\n\n3. Evaluation of performance:\n\nThe authors use existing benchmarks, but show average regret or average normalized regret over a multitude of datasets. It is not clear from such an analysis what is the individual dataset performance or whether the standard deviation is very large as to hide phenomena which may adversely affect the performance of the proposed method in different scenarios. For example, a behavior which makes one wonder as to its origin is figure 3, PD1 average normalized regret which is worse than the random baseline up until >0.5 of iterations. \n\nThis same concern is mirrored in the rank statistical parity of the random baseline even at full budget (figure 4, right).\n\nCan it be explained in these cases why the method does so poorly (even if others are as poor).\n\nAgain, this is of practical importance --- hyper parameter optimization is an exercise which one engages in the context of a specific, single task. A method should be shown superior separately on all, or called out for its failure modes when not universally applicable.\n\n4. Finally, it is unclear to this reviewer if this method has strong predictive power in the sense that it can be effectively used with a low budget in order to reliably precede large scale (prohibitively expensive) experiments. The benchmarks used may answer this question, but it remains opaque in the body of the work.\nSpecifically, since the DPL requires training on multiple degrees of freedom (hyperparameter dimensions) in order to form a reliable prediction along these degrees of freedom, to what extent is this done at small fidelity? What is the expected computational gain relative to current methods which involve e.g. grid search at the small scale and then hyperparameter scaling?\n\nIt is crucial (especially if this work remains in the purely empirical domain, as is currently governed by the NN at the heart of the hyperparameters-loss estimation), that the computational gain be fleshed out for this venue to offer a path towards superior hyperparameter optimization.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dUhEiYOHz7b", "reviewer": "Reviewer_KFvH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-fidelity Bayesian optimization approach to hyperparameter optimization called Deep Power Law (DPL) that exploits the power law assumption of learning curves to quickly identify strong hyperparameter settings.  Results on 3 benchmarks with tabular, image, and NLP datasets show DPL to outperform 7 state-of-the-art competitors.  ", "summary_of_the_review": "While DPL outperforms many leading HPO methods on the 3 studied benchmarks, the experiments do not sufficiently establish DPL as a viable general HPO method due to missing experiments on non-lookup based benchmarks and lack of search spaces with step LR schedules.  Additionally, the novelty of DPL is limited since it is similar to prior power law performance prediction methods and has a very simple fixed allocation multi-fidelity component.", "strength_and_weaknesses": "Pros:\n- The empirical results are strong on the 3 benchmarks studied.\n- Paper is well written and easy to understand.\n\nCons:\n- Empirical analysis limited to pretrained benchmarks with a finite number of trained configurations.  In truly continuous search spaces, it is unclear how often DPL will repeat the same hyperparameter setting to be trained for more batches without additional discretization.  \n- From what I could tell, experiments do not contain search spaces with step wise learning rate schedules where learning rates decrease drastically at fixed intervals.  These types of LR schedules often result in learning curves that do now follow the power law and are used frequently in practice.  \n- DPL is limited in novelty since power law based performance prediction methods have been studied before and the multi-fidelity aspect just allocates a small unit of work per decision step.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c2CzQy9mBf3", "reviewer": "Reviewer_3aYj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hyperparameter optimization algorithm that leverages the power law phenomenon of the learning curves. In particular, the proposed method DPL assumes the surrogate $\\hat f(\\lambda)$  of the true learning curve $f(\\lambda)$ obeys a power law function, which can be parameterized by a neural network. The ensemble of K diverse learned surrogates can be used to determine the next hyperparameter. Empirical studies show that 1) the power law assumption improves the quality of the learning curve prediction; 2) DPL achieves state-of-the-art results in HPO; 3) DPL explores the search space more efficiently than baselines.", "summary_of_the_review": "The proposed method is novel in leveraging the power law in learning curves, but the clarity of the paper is a main concern. Besides, the evaluation is not convincing enough to show the efficacy of the proposed method.", "strength_and_weaknesses": "Strength:\n\n1. Adopting the observation that the performance of machine learning methods as a function of budgets usually follows a power law to hyperparameter optimization is quite novel.\n2. The performance of DPL seems to be better than most of the competitors in terms of average normalized regret in several benchmarks, e.g. LCBench. \n\nWeaknesses:\n\n1. The paper is not very well written that some parts of it is hard to read.\n\n    1.1. Notation issues: 1) It should be $\\alpha_\\lambda, \\beta_\\lambda, \\gamma_\\lambda\\in \\mathbb{R}$ in eqn (3); 2) Why is $b$ bolded in eqn (4)(5)(6)(7)(8) but not bolded before and after?\n\n    1.2. The paper is not self-contained that several techniques are used without introduction. For example, what is deep ensemble strategy? What is expected improvement? They seem to play an important role in the algorithm, but readers do not know what they are without referring to literature.\n\n2. The empirical studies are puzzling.\n\n    2.1. What is average normalized regret? Why is it an indicator of the efficacy of the HPO algorithms?\n\n    2.2. I think a more convincing setting would be the following: 1) Choose a task with randomly initialized hyperparameter; 2) Apply DPL and other HPO methods; 3) Compare the final results (e.g. accuracy) of different methods; 4) Repeat these experiments to many tasks (NLP, CV etc.). After all, what we really care is the ultimate performance on the tasks. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NTg4TPuOa3m", "reviewer": "Reviewer_KEEA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work propose a Deep Power Law (DPL) which uses the scaling law property of learning curves for hyperparameter optimization, by improving fidelity of HOP for deep learning. DPL ensembles predicts the performance of hyperparameter configurations in low-budget regimes as a probabilistic surrogator for Bayesian optimization (BO). This work demonstrated the effectiveness of the proposed method by conducting extensive experiments on 62 datasets under various domain such as image, tabular, and NLP domains.", "summary_of_the_review": "They uses scaling law property for improving the fidelity of HPO and demonstrated the proposed method on multiple HPO benchmarks.", "strength_and_weaknesses": "- Strengths\n  - The writing is clear and easy to follow.\n  - Applying scaling law property of learning curves for HPO is novel.\n  - This work demonstrated their method on extensive experiments under various domain such as image, tabular, and NLP domains.\n\n- Weaknesses\n  - In the experimental section, I think that the title 'Hypothesis' is not appropriate. For example, even if this work demonstrated the performance of their work empirically not theoretically, they regarded Hypothesis 'Our method DPL achieves state-of-the-art results in HPO' is achieved.\n  - With the most experiments, this work focus on 'regret' metric. Could the authors show the experiments with the metric 'accuracy'? For example, by applying the proposed method and multiple HPO methods on ResNet and cifar10/imagenet, we can see the model performance. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject