Scaling Laws for Hyperparameter Optimization
Authors: Arlind Kadra
Maciej Janowski
Martin Wistuba
Josif Grabocka
Venue: NeurIPS 2023 poster (2023)
Keywords: hyperparameter optimization
multi-fidelity hyperparameter optimization
multi-fidelity hpo
power laws
deep neural networks
deep power laws
deep ensemble
deep learning
large language models
scaling laws
llm
TL;DR: Multi-fidelity hyperparameter optimization with deep power laws that achieves state-of-the-art results across diverse benchmarks.

Abstract: Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the issue of hyperparameter optimization, however, most of the methods do not exploit the dominant power law nature of learning curves for Bayesian optimization. In this work, we propose Deep Power Laws (DPL), an ensemble of neural network models conditioned to yield predictions that follow a power-law scaling pattern. Our method dynamically decides which configurations to pause and train incrementally by making use of gray-box evaluations. We compare our method against 7 state-of-the-art competitors on 3 benchmarks related to tabular, image, and NLP datasets covering 59 diverse tasks. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors.

Scaling Laws for Hyperparameter Optimization
Arlind Kadra Representation Learning Lab University of Freiburg
Maciej Janowski Representation Learning Lab University of Freiburg
kadraa@cs uni-freiburg
j janowski@cs uni uni-freiburg.dde
Martin Wistuba
Josif Grabocka
Amazon Web Services
Representation Learning Lab University of Freiburg
Amazon Berlin
marwistu@amazon.co
grabocka@cs. uni uni-freiburg.com
Abstract
Hyperparameter optimization is important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been stream of methods that tackle the issue of hyperparameter optimization, however, most of the methods do not exploit the dominant power law nature of learning curves for Bayesian optimization. In this work, we propose Deep Power Laws (DPL), an ensemble of neural network models conditioned to yield predictions that follow a power-law scaling pattern. Our method dynamically decides which configurations to pause and train incre- mentally by making use of gray-box evaluations. We compare our method against 7 state-of-the -f-tth--rrrrrr art competitors on 3 benchmarks related to tabular, image, and NLP datasets covering diverse tasks. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors.
Introduction
Hyperparameter Optimization (HPO) is major challenge for the Machine Learning community. Unfortunately, HPO is not yet feasible for Deep Learning (DL) methods due to the high cost of evaluating multiple configurations. Recently multi-fidelity HPO (a sub sub-problem of gray-box HPO) has emerged promising paradigm for HPO in DL. by discarding poorly-performing hyperparameter configurations after observing the validation error on the low-level fidelities of the optimization
procedure 288
29 The advantage of multi-fidelity HPO compared to online HPO is the ability to tune all types hyperparameters.
38 or
meta-gradient HPO
10 31
In recent years, stream of papers highlights the fact that the performance of DL methods is predictable [14], concretely, that the validation error rate a power law function of the model size, or dataset size 41 40) Such power law relationship has been subsequently validated in the domain of NLP, too M. In this paper, we demonstrate that the power-law principle has the potential to game-changer in HPO, because we can evaluate hyperparameter configurations in low-budget regimes (e.g. after few epochs), then estimate the performance on the full budget using -ataset-specific power law models. Concretely, we hypothesize and empirically demonstrate that optimization curves (training epochs versus accuracy, or loss) can be efficiently modeled as simple power law functions.
As result, we introduce Deep Power Law (DPL.) ensembles, a probabilistic surrogate for Bayesian optimization (BO) that estimates the performance of hyperparameter configuration future budgets using ensembles of deep power law functions. Subsequently, our novel formulation of BO dynami- cally decides which configurations pause and train incrementally by relying on the performance
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

estimations of the surrogate. We demonstrate that our method achieves the new state-of-the ahe- in HPO for DL by comparing against 7 strong HPO baselines, and 59 datasets of three diverse modalities (tabular, image, and natural language processing). Our experimental protocol contains state-of-the-ar deep learning architectures such as Transformers [4), XFormer [26), ResNeXt [49] and large language models like GPT-2 [39]. As result, we believe the proposed method has the potential to finally make HPO for DL a feasible reality.
Overall, our contributions can be summarized as follows:
We introduce novel probabilistic surrogate for multi-fidelity HPO based on ensembles of deep power law functions.
We derive a simple mechanism to combine our surrogate with Bayesian optimization.
Finally, demonstrate the empirical superiority of our method against the current state-of- the-art HPO for Deep Learning, with large-scale HPO experimental protocol.
2 Related Work
Multi-fidelity HPO assumes method has access to the learning curve of a hyperparameter con- figuration. Such a learning curve is the function that maps either training time or dataset size, to the validation performance. The early performance of configurations (i.e. first segment of the learning curve) is used to discard unpromising configurations, before waiting for full convergence. Successive halving [17 is widely used multi-fidelit method that randomly samples hyperparameter configurations, starts evaluating them, and ends a fraction of them upon reaching a predefined budget. Afterwards, the budget is divided by the fraction of discarded hyperparameter configurations and the process continues until the maximum budget is reached. Although the method relies only on the last observed value of the learning curve, it is very efficient. In recent years, various flavors of successive halving have been suggested, including Hyperband (28). which effectively runs successive halving in parallel with different settings. A major improvement Hyperband is replacing random search with a more efficient sampling strategy 9 A more efficient approach is to allocate the budget dynamically to the most promising configurations 47.. However, the only assumption these methods make about the learning curve is that it will improve over time, while recent work [4] exploits a power law assumption on the curves. Similarly, we surrogates that exploit power law assumption, however, our method is able to estimate the performance of unobserved configurations through probabilistic surrogates learned jointly for all hyperparameter configurations.
Learning curve prediction is a related topic, where the performance of a configuration is predicted based on a partially observed learning curve (36). Typically, the assumptions about the learning curve are much stronger than those described above. The prediction is often based on the assumption that the performance increases at the beginning and then flattened towards the end. One way to model this behavior is to define a weighted set of parametric functions [8] 23]. Then, the parameters of all functions are determined so that the resulting prediction best matches the observed learning curve. Another approach is to use learning curves from already evaluated configurations and to find an affine transformation that leads to a well-matched learning curve (6. more fata-driven approach is to learn the typical learning curve behaviour directly from learning curves across different datasets [48] Learning curve prediction algorithms can be combined with successive halving (2). contrast to this research line, we fit ensembles of power law surrogates for conducting multi-fidelity HPO with Bayesian optimization.
Scaling laws describe the relationship between the performance of deep learning models as function of dataset size model size power law [4 T6
40
20
15 33
Another work tunes
hyperparameters on a small-scale model and then transfers them to a large-scale contrast these papers, we directly use the power law assumption for training surrogates in Bayesian optimization for HPO.
scale version [50). In
3 Preliminaries
Hyperparameter Optimization (HPO) demands finding the configurations À E A of Machine Learning method that achieve the lowest validation loss ((vi)) of model (e.g. neural network), which is parameterized with 0 e θ and learned to minimize the training loss C (Taii)

---

X == arg min ((Val) (1, (2)), s.t. 0* (A) = arg min ((Train) (2,
(I)
λEA
BEB
For simplicity, we denote the validation loss our function of interest f(A) [(Val) (A. 0" (A)). The optimal hyperparameter configurations X of Equation [ are found via an HPO policy A E (also called an HPO method) that given history of N evaluated configurations H {fr. (Au))i= suggests the (N + 1)-th configuration to evaluate as AN+1 = A (H)()) where A: [A [2]x* + A. The search for optimal HPO policy is bi-objective problem in itself, aiming at (i) finding a configuration out of N evaluations that achieves the smallest validation loss f()), and (ii) ensuring that the costs of evaluating the N configurations do not exceed a total budget
? as shown in Equation 2
arg min min AEP i6 iE (Ai= (H
(2)
{() ((A))))
i>
where:
H
i=
subject to: Ω 22
cost (f (A:))
Bayesian optimization (BO) the most popular type of policy for HPO, due to its ability balance the exploration and exploitation aspects of minimizing the validation loss f in terms of hyperparame- ters À A. Technically speaking, BO fits surrogate f(A; Φ) parametrized with ø E + to approximate the observed loss f(A) using the history H(N), 6² == arg max ((A))(A)) <PN(N ((]()) Φ). where, represents the probability. Afterwards, BO uses acquisition/utiiity function A + R.
to recommend the next configuration as AN+1 := A (H)()) arg max (f(A:
Multi-fidelit HPO refers to the case where approximation of the validation loss be measured at a lower budget b E B, where B == (0, max] For instance, in Deep Learning we can measure the validation loss after few epochs
b < e), rather than wait for full convergence (b bmax). Throughout this paper, the term budget refers to a learning curve step. The evaluation of a configuration for budget b is defined as f (A, A R.
4 Power Law Surrogates for Bayesian Optimization
Prior work has demonstrated that the performance of Machine Learning methods as a function of budgets (i.e. dataset size, number of optimization epochs, model size, image resolution) follows power law relationship [41
15]. In this work, we employ this power law dependence between the validation loss and the number of optimization epochs in Deep Learning. We propose novel multi-fidelity Hyperparameter Optimization method which is based on power law surrogates. We assume that every learning curve f (A,-) -) can be described by a power law function defined by (a., B3 7). Concretely, we define a power law function for the validation loss of a configuration À at a budget ka. the number of epochs) as shown in Equation
ax ax Bx, 7A E
Instead of fitting one separate power law function to each learning curve, we fit single shared power law function across all configurations by conditioning the power law coefficients a; 3, 7 on using a parametric neural network g that maps a configuration to the power law coefficients of its learning curve as g - R The network has three output nodes, corresponding the power law coefficients, denoted as g(A)a, g(A)s, g()),, as defined in Equation
(A,b) = 9()), + o())s b b-o(A), 9:A A R
(4)
3

---

Using a history of learning curve evaluations H(N) : {(A,, b,, f (Az, b,)) we can train the power law surrogate minimize the following loss function using stochastic gradient descent:
arg min PAL (1,, (Ai,
(5)
BO surrogates need to be probabilistic regression models because the acquisition functions require the posterior variance of the predictions. As result, train ensemble of K diverse surrogates (x, ..... ((K)) (X, b) by initializing each surrogate with different weights and by training with different sequence of mini-batches as in prior work [25] [22]. The posterior mean μ and the posterior variance o² of the power law ensemble trivially computed as:
my(A,b) = K (A, 03 = K Σ (A.
(6)
A commonly used acquisition function in the domain is Expected Improvement (EI)
35 which
incorporates both the mean and uncertainty of predictions, applying a trade-off between exploration and exploitation. Consequently, in our work, we use the El acquisition with the estimated full budget's (bmax) posterior mean and variance. We briefly define the acquisition function in Equation
AN+1 = arg max EI (2, Drnux ( AEa
(7)
max f(A,, Umuxx) f (Abests D of]
(A, bmaa H) = ((A,baa) (N (2
where, f (Ahest, bmmx) corresponds to the best observed loss for any budget 6' < bmux from the history (NN. After selecting configuration with our variant of the El acquisition, we do not naively run until convergence. Instead, we propose a novel multi-fidelity strategy that advances the selected AN+1 of Equation by small budget of dstep e.g. epoch of training. Therefore, the selected AN+1 will be evaluated at + as defined in Equation S Notice, our proposed strategy also covers new configurations with no learning curve evaluations in (N),
Dssep: Deee +
ANN+1 (AN+1,);)) E H(N)
bx+1 ==
b,
otherwise
(8)
(Aw HH))
We provide the detailed pseudocode of our method at Algorithm
5
Experimental Protocol
5.1
Benchmarks
LCBench: A benchmark that features 2,000 hyperparameter configurations that parametrize the archi- tecture of simple feedforward neural networks, as well as, the training pipeline [5T]. The benchmark features numerical hyperparameters and 35 different datasets from the AutoML benchmark [12].
PD1: deep learning benchmark [45] that consists of recent DL (including Transformers) architec- tures run large vision datasets such CIFAR-10, CIFAR- 100, ImageNet, well as statistical modeling corpora and protein sequence datasets from bioinformatics. Every search space includes varying learning curve lengths, ranging from 5 to 1414, and different number of evaluated hy- perparameter configurations ranging from 807 to 2807. The search space includes hyperparameter configurations that parametrize the learning rate, the learning rate scheduler, and the momentum.
TaskSet: A benchmark that features different optimization tasks evaluated different search spaces 34] For our work, we focus the Adam8p search space, which is among the largest search spaces

---

Algorithm 1: Multi-Fidelity HPO with Deep Power Laws
Input :Search space A, initial design H (imit), budget increment Step
Output Best hyperparameter configuration X
Iteration i <- 0; Evaluate initial configurations and budgets H Hiiitt)
2
while still budget do
Fit DPL ensemble (A) b). ((K) b) from history H using Equation
Recommend the next configuration A4+1 and its budget by+1 using Equation 6 7 and 8
5
Train A4+1 until b,+1 and measure the validation loss I (A++1; b(41):
Append to history H <- H' U [(1+1. b4t1s (Au+1: b(41))};
i-i+l:
end
return Best configuration X with the smallest validation loss
min (A+,b, f M)E
f(*,b);
in the benchmark with 1000 hyperparameter configurations. Every hyperparameter configuration
features 8 continuous hyperparameters. The hyperparameters control the learning rate, the learning rate schedulers, and the optimizer. For variety among our benchmarks, we focus on 12 RNN text classification tasks that feature different RNN cells, embedding sizes, and batch sizes.
For a more detailed explanation the benchmarks, we refer the reader to Appendix
5.2 Baselines
Random Search stochastically samples hyperparameter configurations for the largest possible budget. Hyperband uses multiple brackets with different trade-offs of the initial budget and number of epochs to initially train [28]. then applies Successive Halving (SH) [17] on every bracket. ASHA is an asynchronous version of SH [27] that does not wait for all configurations
finish in SH bracket before starting the next one. Furthermore, BOHB is a follow-up of Hyperband that uses TPE 3 to sample the initial hyperparameter configurations of a bracket [9] DEHB, the other hand, modifies Hyperband by using evolutionary strategies to sample the initial hyperparameter configurations [ Similarly, multi-fidelity SMAC extends Hyperband but random forests sample the initial hyperparameter configurations for bracket [30]. We also use the Dragonfly Library [19] to compare against BOCA (8), a multi-fidelity method that uses Gaussian Processes to predict the next hyperparameter to evaluate, as well as the fidelity for which it should be evaluated. For all the baselines, we use their official public implementations. We provide additional details in Appendix G
5.3 Architecture & Training
For our method, we use an ensemble of 5 models, where every model consists of a 2- 2-layer feedforward neural network with 128 units per layer and Leaky ReLU for the non-linearity. The architecture of our method is motivated by prior work [25]. Our network has 3 output units, that are then combined with the budget & yield the power law output. We apply the GLU non-linearity activation only on the 3 and output units, allowing a to take any value.
We use the LI loss to train our network, coupled with Adam featuring an initial learning rate of 10-3, For the first 10 iterations of our multi-fidelity HPO method in Algorithm ] we train every network of our ensemble for 250 epochs with randomly sampled initial weights. This choice helps the convergence of the weights in the early stage of HPO. Next, we continuously refine the model for 20 epochs every HPO iteration. However, if the optimization stagnates (surrogate fitting loss does not improve) for more than the LC Length + a buffer of 0.2 x LC Length, the training procedure restarted with random weights, where every model is trained again for 250 epochs and then only refined. During the refining phase, the new data point at an HPO iteration (Line 9 at Algorithm [ is sampled with repeat on every batch, to learn new data points equally compared to older data points. Since we are working with discrete search spaces, evaluate the acquisition function exhaustively
5

---

on every hyperparameter configuration. When dealing with continuous search spaces, the acquisition function can either be evaluated exhaustively on a discretized version of the search space, or in a gradient-based way. Our implementation of DPL. is publicly available
5.4 Protocol
In our experiments, we standardize the hyperparameter values by performing min-max scaling for our method and every baseline. If baseline has specific preprocessing protocol, we do not apply min-max scaling but we apply the protocol as suggested by the authors.
The benchmarks do not support a common evaluation metric for configurations (i.e. the function f). As consequence, the evaluation metric for LCBench is the balanced accuracy, for TaskSet the log-likelihood loss, while for PD1 the accuracy. Moreover, the benchmarks do not offer learning curves with a common step size. For LCBench and PDI, one step size is equivalent to one epoch, while for TaskSet one step size is 200 batches. The HPO budget defined as the maximum number of steps needed to fully evaluate 20 hyperparameter configurations. In that context, one unit step of the HPO budget signifies training a particular configuration for one more optimization step (e.g. 200 batches in TaskSet or epoch in LCBench).
In the following experiments, for all methods, we report the regret of the best-found configuration as shown in Equation 9
R = f (Abest, bmax) - (AAoracte bmax)
(9)
where the oracle is given as f (Aoracte, Dmax) == min (a, | (2,b, E H(D) A S 8mix). and H(D) corresponds to the set of all the exhaustivel--evaluaee hyperparameter configurations' performances on a dataset D. If the oracle confi inniiuraaionn is not known in advance for the search space, (D))) can be replaced with the history H(N) at the end of the HPO procedure. The only difference between ff (Aaess, mmx)) ) and f (Aoructe, mmxx) is that the former only considers the history at the HPO step for which are reporting the results.
In short, the regret is the difference the evaluation metric performance from the best-found hyperparameter configuration during optimization to the best possible hyperparameter configuration (oracle) on the dataset (in a minimization setting). On a dataset level, we report the average regret across 10 repetitions with different seeds. To be able aggregate results over datasets, report the averaged normalized regret. As normalization, we divide the regret by the difference between the performances of the best and the worst hyperparameter configuration the mean of the normalized regrets across all the datasets of a benchmark.
dataset. Then we compute
Moreover, in the experiments that report the average normalized regret over time, we provide results over normalized wall clock time. The wall clock time includes both the method's overhead (i.e. training the surrogate and selecting the next hyperparameter configuration to evaluate) and the time taken to evaluate the selected hyperparameter configuration (i.e. evaluating f). Since the methods have different run times, we normalize the individual times by the time took Random Search (the fastest non non-model-based method) to complete the HPO optimization process. To provide fair any-time comparison, we report results until the time it took Random Search to evaluate 20 hyperparameter configurations.
Furthermore, when reporting the learning curve (LC) length fraction, we imply the fraction of the total learning curve length. LCBench and TaskSet have LCs of a fixed length for all datasets, corresponding to 51 epochs for LCBench and 50 epochs for TaskSet. In contrast, PD1 has varying LC lengths for different datasets.
In our experiments, all methods start with a history (inst) of 1 randomly sampled hyperparameter configuration evaluated for step/epoch in the case of multi-fidelity techniques (Hyperband, BOHB, DEHB, SMAC, ASHA, Dragonfly; descriptions Section 5.2). or for the full budget for the black- box technique (Random Search). We ran experiments on a CPU cluster, where every node contains two Intel Xeon 2--2640v4 CPUs with 20 CPU cores running at 2.2 GHz. The total memory of every node is 120GB, and every experiment is limited to 2 cores which offer 12GB.

---

Research Hypotheses and Experiments
The power law assumption im- proves the quality of learning curve forecasting..
Hypothesis 1:
Correlation; Est. vs. True
In this experiment, we evaluate the predictive performance of forecasting models that given a fraction the observed learning curve, esti- mate the remaining unobserved segment of the curve, on the LCBench benchmark. The results of Figure compare three different forecasting models, concretely, neural networks (NN), Gaus- sian Processes (GP), and Power Law functions (PL). For the three variants (PL., NN, GP) we fitted one model on every learning curve of each hyperparameter configuration (i.e. given b in the x-axis estimate one f(b) separately for every A). For the other two variants (DPL and Cond NN) we fit a single forecasting model (not an ensem- ble) for all configurations, by conditioning the surrogate on the configuration (i.e. given and X, estimate f (A, b)). The purpose of the experiment is to assess whether power law function regressor leads to superior predictive accuracy, compared to generic forecasting models, such as neural networks, or Gaussian processes. The evaluation metric of the experiment highlighted in Figure is the rank correlation between the estimated performances at the end of the learning curve and the true performances.
0.8
0.6
0.1
0.2
0.3
0.4
0.5
LC Length Fraction
DPL
Cond NN
PL NN GP
Figure 1: Rank correlations of learning curve fore- casting models, which are given a fraction of the learning curve and estimate the remaining curve segment. DPL: Deep Power Law, Cond NN: Con- ditioned neural network, PL: Power Law, NN: Neu- ral Network, GP: Gaussian processes.
We notice that although a Power Law regressor has significantly fewer parameters than neural network (3 to 288 parameters), PL still achieves higher predictive performance than NN. Furthermore, our conditioning of the power law function to the hyperparameter configuration further improves the predictive quality, as the difference between DIPP and PI. demonstrates.
Lastly, we refer the reader to Appendix H where we provide an analysis of the distributions for the absolute error rate between the DPL predictions and the ground truth values over the different LC length fractions, showing that DPL does not only retain the ranks but, it also accurately predicts the final performance. Based on the results, we consider Hypothesis I to be validated and that DPL is accurate in terms of learning curve forecasting.
What about learning curves that do not follow power law pattern? Although the provided empirical evidence in this section strongly suggests that the presented power law model can accurately forecast learning curves, it is also true that some learning curves have divergent behaviour that does not follow the power law assumption. As a consequence, we investigate two different ways to handle curves that do not follow the power law assumption: i) recently- -ronntly-- power law functions that include breaking points (5), or shifts in the curve [8), and ii) min-smoothing the learning curves transform them into monotonically decreasing time series. In Appendix A we provide ample empirical evidence showing that although the alternative formulations achieve comparable performances, still they do not outperform our simpler power law formulation. The findings indicate that even though not all learning curves are power laws, most of them are, therefore a power law surrogate is efficient in forecasting the final performance of a partially-observe learning curve. As result, the forthcoming experiments will provide further empirical evidence that our power law surrogates lead state-of-the-arr HPO results when deployed in the proposed multi-fidelity Bayesian optimization setup.
Hypothesis 2:
Our method DPL achieves state-of- oftte-of-the-ae the-art results in HPO.
In Figure show the performance of the considered methods over the HPO budget, where DPL manages to outperform all the rival baselines. In the case of ICCBnch, DPL. quickly finds well-performing hyperparameter configurations compared to the competitor methods and continues to discover even better configurations until the HPO process ends. Furthermore, we observe the same trend with TaskSet and PDI, where after ca. 25% of the HPO budget, our method DPL converges to better regret compared to the baselines and increases the lead until HPO ends. For a detailed overview of the performances of all methods on all individual datasets, we point the reader Appendix H
7

---

Average Normalzzed Regret
TaskSet
PD1
10"
10"
10"
10
10
10
10
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Budget
Budget
Budget
DPL
ASHA
BOHB
DEHB
Dragonfly
Hyperband
Random
SMAC
Figure 2: DPL. discovers better hyperparameter configurations than all rival baselines in terms of regret (distance to oracle). Solid curves and shaded regions represent the mean and standard error of the averaged normalized regret.
LCB
TaskSeng@00%%
PD1660%
Random
Random ASHA Dragonlyy
ASF2 Dragor Hyperband
Apprband DEHB
LCBench@1000%
TaskSaig
PD1 100%
haperband
ASTA
BOTB
Ixperband Suare
DEHB
BOHB
CEHB
Figure The critical difference diagrams 50% and 100% of the HPO budget. The ranks indicate the sorted position in terms of regret, aggregated across datasets (the lower the better). Thick horizontal lines highlight differences that are not statistically significant.
In addition, providess the critical differ- ence diagrams of the per-dataset regret ranks at 50% and 100% of the HPO budget. Our method DPL outperforms all baselines in 5 out of 6 cases (in 4 of which with a statistically significant mar- gin). while being second best only at the 50% of the HPO budget on the PD1 benchmark. We investigate the lack of statistical significance in PD1, by analyzing the individual dataset perfor- compared
LGBendh
Average Normalized Regret
10 10
DPL
ASHA
BOHB
DEHB
mances where DPL performs to other baselines. We notice that the datasets have skewed distribution of hyperparameter configuration performances, where, a majority of the configurations achieve top performance. Based on the results, we conclude that a lack of statistical significance is the outcome of a search space that includes relatively simple optimiza- tion tasks and not a specific failure state of our method. We provide the detailed results of our analysis in Appendix[D) [D]
PD1
Draponfly
Hyparband
Avarae Namallzzd Regret
Random
SMAC
10
10
0.00
0.60
Normalized Wallime
Figure 4: The average normalized regret of DPL and rival methods over the normalized time for all the considered benchmarks. Solid curves and shaded regions represent the mean and standard average normalized regret.
Lastly, we analyse the performance of DPL over time in Figure 4 As it can be observed, DPL manages to outperform the competitors even when the method's overhead time included, showing that the overhead of DPL (i.c. fitting
surrogate and running the acquisition) negligible in terms of the quality the HPO results. For more detailed information, regarding the DPL overhead time, we point to Appendix E. TaskSet is not included in Figure since the benchmark does not offer runtimes. Given the results, we conclude that Hypothesis 2 is validated and that DPL achieves state-of-the-an performance in HPO.

---

LCBench
LCBgrch
LGBench
4.15
0.0
Awerge
2.10
0.4
0.66
0.0
0.0
04
08
0.
044
0.6
LG Langii Freeiion TaskBBt
LG LGLenghh Fiecdon
Lonoli Fradoon
TaskBat
Taakk Bot
0.58
0.30
0.25
Precdon of
Rectage Regeed
2.25
Fog0 Of
0.4
0.06
0.15
0.00
0.0
0.2
04
08
0.8
1.0
0.0
0.2
C.A
0.5
0.5
1.0
0.0
0.2
B4
0.5
0.5
1.
L.C Langth Finetion
L.C. anghh Fraidioon
Lergt Friedion
Figure 5: Post-hoc analysis study DPL's efficiency, Left: Share of the best candidates selected during training. Middle: Average regret of configurations chosen to be trained at each budget. Right: Share of top third configurations given budget which were bottom two third configurations at previous budget.
Hypothesis 3: DPL explores the search space more efficiently compared to the baselines.
We conduct further analyses to understand the source of the efficiency of DPL versus the baselines. As a result, we analyze two important aspects, the quality of the evaluated configurations, as well as the exploration capability of our multi-fidelity HPO method. Initially, we measure what fraction of the top 1% configurations (ranked by accuracy) can our method discover. 51gute55(eett) shows that until convergence our method can discover significantly more top configurations compared to the baselines. The middle plots of Figure 5 show the average regret for each configuration promoted to the respective budget. According to the plot, DPL more efficient and assigns the budget only to configurations with lower regret compared to the other methods. The precision and regret plots demonstrate that the quality of the evaluated configurations is largely better than all baselines, therefore, giving our method significant lift in the performance rank. Last but not least, the right plot shows the percentage of configurations that were performing poorly in an earlier epoch (i.e. accuracy-we -wise in the bottom 2/3 of configurations up to the epoch indicated at the x-axis) but performed better at later epochs (i.e. at the top 1/3 of configurations). Furthermore, we added a line labeled with "Baseline", which represents the fraction of previously poor-performing configurations of all configurations. This behavior is observed often with learning curves, where for instance, strongly regularized networks converge slowly. For the same analysis regarding the PD1 benchmark, we point the reader to Appendix H
The results indicate that our method explores well the unpromising early configurations, by consider- ing them through the uncertainty estimation of our ensemble and the respective Bayesian optimization mechanism. The results validate Hypothesis 3 and confirm that DPL, explores the search space more efficiently.
Hypothesis 4: Our method DPL offers an effective tool for HPO in Large Language Models.
In this experiment, we consider the case of tuning the hyperparameters of transformers in Large Language Models. To this end, we computed a tabular benchmark by training a smaller GPT-2 [39] model on the OpenWeb tune three learning rate hyperparameters: the fraction of warmup steps, the maximum learning rate at the end of warmup, and the minimum learning rate at the end of the decay. We repeat the experiments for seven model sizes ranging from 0.3M to 30M total parameters, ablating the embedding size the multi-head attention layers (details in Appendix B).
dataset [13 for series of different hyperparameter configurations. We
We follow the common practice of conducting HPO with small transformers and then deploying the discovered optimal configuration on the full-scale transformers (50) As result, we search for the optimal hyperparameters of small transformers (embedding size of (6, 12..... 96, 192)) and then evaluate the discovered configurations at full-scale transformer with embedding size of 384.

---

Embedding size
Embedding size 48
Embedding size 192
7.00
6.80
Validation
6.60
6.00
6.40
5.00
6.20
HPO Budget [GPU-hours]
Error at Larger Scale
Normalized HPO budget = 25%
Normalized HPO budget = 50%
Normalizer HPO budget 100%
**
3.98
3.96
3.95
3.95
3.94
3.92
3.92
3.92
3.90
3.90
3.90
12
48
192
12
48
192
12
48
192
Small Transformer Scale [embedding size]
DPL BOHB
Random
oracle small
oracle at largest scale
Figure 6: HPO for Transformer architectures. Top: HPO small-scale transformers in terms of the embedding size. Bottom: Error on the full-scale transformer, using the hyperparameter configuration discovered by conducting HPO using the small transformers. We present three analyses, ablating the HPO time on the small-scale transformer up to the HPO budget of full function evaluations.
Figure 6 shows the HPO results of DPL against Random Search and BOHB rival multi-fidelif HPO baseline). In the top row of plots, we observe the performance of the discovered configurations the small transformers for the indicated embedding size. We observe that method finds better configurations than the baselines at any proxy space with small embedding sizes.
On the other hand, the bottom row of plots presents the performance of the discovered configurations in the small embedding space, by applying such hyperparameter configurations to the full-scale transformers. We observe that the configurations discovered by DPL the small search space achieve very competitive results on the full-scale transformers, finding the oracle configuration the full-scale transformers in the majority of cases. It takes D.L 3.6 hours to find the oracle configuration for the largest model via HPO for the smallest model. In turn, takes 21.52 hours to train the largest model only once. For more details, 4 and confirm that DPL is an efficient HPO technique for tuning the hyperparameters of large language models when the HPO is conducted using smaller transformer model sizes.
refer the reader Appendix B The results validate Hypothesis
7 Conclusions
Summary. In this work, we introduce Deep Power Law (DPL), probabilistic surrogate based on an ensemble of power law functions. The proposed surrogate used within novel multi-fides filtiifiieeity Hyperparameter Optimization (HPO) method based on Bayesian optimization. In contrast to the prior work, we exploit scaling laws for estimating the performance of Deep Learning (DL.) models. Through extensive experiments comprising 7 baselines, 59 datasets, and search spaces of diverse deep learning architectures, we show that DPL, outperforms strong HPO baselines for DL by large margin. As an overarching contribution, advance the state-of-the- the-rrr in the important field of HPO for DL.
Limitations and future work. Contrary to the common perception, we experienced that the uncertainty estimation arising from the Deep Ensemble approach (25) suboptimal compared to standard BO surrogates such as Gaussian Processes. In addition, having to train an ensemble has additional computational costs, due to the necessity of training multiple power law models. In the future, we plan to investigate the combination of power laws with Gaussian Processes, as well as investigate additional fidelity types.
10

---

Acknowledgements
JG, AK and MJ would like to acknowledge the grant awarded by the Eva-Mayr-Sttin Stiftung. In addition, this research was funded by the Deutsche Forschungsgermeinschann (DFG, German Research Foundation) under grant number 4 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, JG and AK acknowledge the support of the ChinnLink--BrinnToo center of excellence.
References
[1] Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 2147-2153, 2021.
[2] Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural archi- tecture search using performance prediction. In 6th International Conference on Learning Reprexentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings, 2018.
[3] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.
[4] Benedetto J. Buratti and Eli Upfal. Ordalia: Deep learning hyperparameter search via general- ization error bounds extrapolation. In 2019 IEEE International Conference on Big Data (Big Data), pages 180-187, 2019.
[5] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, 2022.
[6] Akshay Chandrashckaran and Ian R. Lane. Speeding up hyper-parameter optimization by extrapolation of learning curves using previous builds. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part I, volume 10534 of Lecture Notes in Computer Science, pages 477-492 Springer, 2017.
[7] Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P. Lillicrap. Matthew Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In Proceedings of the 34th International Conference Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 I August 2017, pages 748-756, 2017.
[8] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hy- perparameter optimization of deep neural networks by extrapolation learning curves. In Proceedings of the Twenty-Fourth International Joint Conference Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages
AAAI Press, 2015.
[9] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm. Sweden, July 10-15, 2018, pages 1436-1445, 2018.
[10] Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1165-1173, 2017.
2017.
[11] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[12] Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren. An open source automl benchmark. arXiv preprint arXiv:1907.00909 2019.

---

[13] Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skkylion007, htt.h//....//peeeeeeeetttoorrrpp 2019.
[14] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, abs/1772..00499 2017.
[15] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[16] Achin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, and Stefano Soatto. A meta-learning approach to predicting performance and data requirements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3623-3632, June 2023.
[17] Kevin G. Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyper- parameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9- 2016, pages 240-248, 2016.
[18] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff G. Schneider, and Barnabás Póczos. Multi- fidelity bayesian optimisation with continuous approximations. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1799-1808, 2017.
[19] Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christo- pher R. Collins, Jeff Schneider, Barnabás Póczos, and Eric P. Xing. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly. Mach. Learn. Res., 21:81:1-81:21 2020.
[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830 08366) 2020.
[21] Andrej Karpathy. nanoGPT.
2023.
[22] Abdus Salam Khazi, Sebastian Pineda Arango, and Josif Grabocka. Deep ranking ensembles for hyperparameter optimization. In The Eleventh International Conference on Learning Representations, 2023.
[23] Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In 5th International Conference Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[24] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[25] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Informa- tion Processing Systems, volume 30. Curran Associates, Inc., 2017.
[26] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: modular and hackable transformer modelling library. https: htt///tttuut thub com/ facebookresearchh crceeaarccchfooooooo orrersss. 2022.
12

---

[27] Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.0553 1810.0593, 5, 2018.
[28] Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18:185:1-185:55 2017.
[29] Shibo Li, Wei Xing. Robert M. Kirby, and Shandian Zhe. Multi-fidelity bayesian optimization via deep neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[30] Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp. Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research (JMLR) MLOSS, 23(54):1 9, 2022.
[31] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit diffcrentiation, In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. pages 1540-1552, 2020.
[32] Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient-based hyperparameter optimization through reversible learnin. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 2113-2122, 2015.
[33] Rafid Mahmood, James Lucas, Jose M. Alvarez, Sanja Fidler, and Mare Law. Optimizing data collection for machine learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 29915-29928. Curran Associates, Inc., 2022.
[34] Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies. CoRR, abs/2002. 1887, 2020.
[35] Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2(1 17-19)2), 1978.
[36] Felix Mohr and Jan N van Rijn. Learning curves for decision making in supervised machine learning-a survey, arXiv preprint arXiv:2201.12155 2022.
[37] OpenAI. Gpt-4 technical report, 2023.
[38] Jack Parker-Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient online hyperparam- eter optimization with population- based bandits. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[39] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[40] Jonathan S. Rosenfeld, Jonathan Frankle. Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9075-9082 PMLR, 2021.
[41] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[42] David Salinas, Matthias Seeger, Aaron Klein, Valerio Perrone, Martin Wistuba, and Cedric Archambeau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In First Conference Automated Machine Learning (Main Track), 2022.
13

---

[43]
Mingxing Tan and Quoe Le. EfficientNerV2:: Smaller models and faster training. github com/ 2021.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez,
E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[45] Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-training helps bayesian optimization too. arXiv preprini arXiv:2207 030844 2022.
[46]
Ross Wightman. PyTorch image models.
httpe://githno..co/rvvgggggau
senic-immage-modee 2019.
[47] Martin Wistuba, Arlind Kadra, and Josif Graboeka. Supervising the multi-fidelity race of hyperparameter configurations. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[48] Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 10303-10312, 2020.
[49] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[50] Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.034466, 2022.
[51] Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions Pattern Analysis and Machine Intellignce, 43(9):3079 - 3090, 2021.
14

---

A
Modeling
10
LCBench
10"
LCBench
Average Nornalized Regret
Average Normalized Regret
10"1
10-1
0.00
0.25
0.50
0.75
HPO Budget
0.00
0.25
0.50
0.75
1.00
HPO Budget
DPL
Dragonfly
Candidate 1
ASHA
Hyperband
Candidate 2
DPL
Random
BOHB
Random
Broken
ASHA
Dragonfly Hyperband
SMAC
DEHB
SMAC
BOHB
SPL
Figure 7: Left: The performance of different power law formulations, as well as, the baselines the
LCBench benchmark over the HPO budget. Right: The performance of the power law formulation when min-smoothing is applied to the learning curve ensure monotonically decreasing learning curve.
To inspect the modeling choices of the power law functions used as our surrogate, we con- sider different formulations for the ensemble members of our surrogate as shown in Table [ Initially, consider Candidate which can handle shifts in the learning curve by introduc- ing additional parameter d. Furthermore, consider a more complex version, Candidate 2, that allows to additionally scale the budget, Table 1: Alternative power law formulations for by introducing variable e. Lastly, we consider the DPL surrogate.
Label
Formula
DPL
a + 8.b
Candidate
a a B (b 1b-00 d]
Candidate
d
Broken Law
a 8
Broken Laws [5] (BPL), which can handle breaking points in the learning curve. We use version that can handle one breaking point since the authors the method suggest as a sufficient formulation to approximate the majority of cases. We the DPI. surrogate with every individual formulation the LCBench benchmark to investigate their empirical performance.
Figure 7 presents the results, where, chosen surrogate formulation (DPL) despite being the simplest, outperforms all other formulations. The Candidate formulation does not manage to outperform all competitor methods, the Candidate formulation manages outperform all rival methods, however, only with marginal difference. The Broken Law formulation manages to outperform all the rival baselines by considerable margin, however, it still performs worse than the simple power law formulation used for DPL.
We would like to point out that the alternative power law formulations are more difficult to opti- mize/run since they are prone to diverge and fall into a dead state given different combinations of parameter values. The most common is division by zero for e.g. d term in the Broken Law formulation, taking the root a negative number b + term in Candidate e 6 + d for Candidate and the term in Broken Law.
We additionally consider min- smoothing the learning curve y, by taking each step of the learning curve the minimal value of the learning curve yb ymmooth where, y ymmooth min (3oo V1..... 36). The min-smoothing allows diverging curves be formulated as power laws since the diverging phase will be substituted with stagnation. Figure 7 shows that incorporating learning curve min-smoothing for our surrogate (SPL.) performs comparably to DPL. without learning curve smoothing and manages to beat the other HPO baselines.
15

---

B nanoGPT-Bench
In recent years, deep learning research has increasingly focused large-scale models, particularly
Large L.anguage Models (LLMs) like the Generative Pre-trained Transformers (GPTs). To evaluate the effectiveness of search algorithms, propose benchmark based the nanoGPT model (2]]
reproducing the performance of the small GPT-2 39 model trained on the OpenWebText dataset 13
Our experimental setup is designed with the practicalities of real-world hardware constraints in mind. The common practice in the field is to perform Hyperparameter Optimization (HPO) on an informative proxy task that adheres to model size scaling laws [37]. and then to apply these optimized parameters to larger models. This pragmatic approach is necessitated by the fact that training larger models would require significantly more expensive computational resources and time. Within these constraints, we focus our experiments on NVIDIA RTX 2080 GPUs.
Max
Learning Rate Min
Warmp
Max
Steps
Figure 8: nanoGPT-Bench search space parametrization.
Baseline: We train a small nanoGPT model, a scaled-down variant of the small GPT-2, reducing the parameter count to approximately 30 M from the original 124 M parameters, The model architecture includes transformer layers and attention heads, and the embedding size set to 384. The Adam W optimizer is utilized for training, with the first and second moment estimates configured to 0.9 and 0.98, respectively. The weight decay is set to 10-1, and we apply gradient clipping at a value of 1.0 prevent large gradients from causing instability in the model training. For the training process, optimize the cross-entropy loss for next-token prediction. The process involves 350 steps, with each step encompassing 1000 random samples, with a batch size of 12. Each data point has context size of 512 tokens, encoded using OpenAl's ''s token embeddings (sized of 50304). This procedure ensures that even the most resource-intensive experiments stay within the limits of single GPU day.
Search Space: Our hyperparameter search space con- struction involves varying the number of warmup steps, along with the maximum and minimum learning rates for
HP
Values
the cosine scheduler. The specific parametrization of the
Max LR
[10 1005:100 10:4, 10 31
scheduler is illustrated in Figure & The discretized choices are presented in detail in Table
Min LR
1%; 10% of Max LR
Warmup Steps
[10%; 20%] of Budget
Table 2: Search space of nanoGPT- Bench.
Fidelity Space: To construct the fidelity space, we focus on two key dimensions: the number of training steps and the transformer's 's embedding size, serving as a proxy for model size. In this exploration, the natural fidelity of the number of training steps is visualized by the validation curves during model training as depicted in Figure On the other hand, the end performance correlation between the different fidelities is reflected by the Pearson correlation in Table
3 We establish proxy tasks by sampling embedding size from a log scale, {6, 12, ... 96, 192}, with the maximum being 384. Consequently, each configuration has proxy and one target task, leading to a total of 84 unique configurations in our full benchmark. Every configuration is trained for 350 steps.
16

---

Embedding size
Embedding size 12
Embedding size 24
10
10
8
6
200
350
50
200
350
50
200
350
Steps
Embedding size
Embedding size 96
Embedding size 192
Validation Error
10
10
200
350
50
200
350
200
Steps
Embedding size 384
10
6-
200
350
Steps
Figure 9; Validation loss curves during model training for all nanoGPT-Bench configurations across model fidelity values.
Runtime [GPU-hours]
1.00
20
Size [bytes]
0.75-
FLOPS
0.25
1.--
0.00
96
192
384
6
96
192
384
6
48
96
192
364
Small Transformer Scale [embedding size]
Figure 10: Scaling of model size in relation to bytes, FLOPS, and runtime based on average values across all nanoGPT-Bendd configurations.
Due the GPU under-utilization of small model sizes, the runtime scales linearly the model size scales exponentially. This relationship can be observed in Figure I0 and Figure We expect the runtime scale in linear proportion to the model size when larger models are considered.
Figure 12 illustrates the effectiveness of DPL, particularly when the number of training steps considered as a fidelity dimension. Vertical dotted lines denote the iteration which an algo- rithm identifies the oracle value with an absolute tolerance of 0.01.
Embedding Size
Correlation
0.951
12
0.880
24
0.971
48
0.955
Figure 13 depicts the results over the different values of the embedding fidelity. We utilized DPL, BOHB, and random search in proxy tasks, incrementing the budget allocation over each run, up to a horizon of 6 full-function evalua- tions. From these proxy tasks, extracted the incumbent hyperparameters and evaluated their performance on the target task, that correponds to the maximum embedding size of 384.
96
0.987
192
0.994
384
1.000
Table 3: Pearson correlation across fidelities.
Despite operating within short regimes, DPL consistently outperforms baselines in terms of the mean incumbent value and the explored regime, as evidenced by error bars indicating the range between best and worst incumbents across 10 seeds. It should be noted, however, that the correlation between
17

---

Embedding size
Embedding size
Embedding size
24
2
6.7
6.8
6.9
7.0
7.0
7.1
7.2
7.3
7.2
7.3
7.4
Runtime [GPU-hours]
Embedding size
48
Embedding size
Embedding size
192
Count
1
7.95
8.00
8.05
8.10
8.15
8.90
8.95
9.00 9.05 9.10
9.15
13.6
13.7
13.8
13.9
14.0
Runtime (GPU-hours]
Embedding size
384
21.4
21.6
21.8
Runtime [GPU-hours]
Figure 11: Distribution of GPU-hours required for training across different model fidelity values.
Embedding
Embedding
12
Embedding size
7.00
7.00
7.00
6.90
6.80
6.70
6.40
6.00
6.60
6.20
10
20
40
30
40
40
HPO Budget [GPU-hours]
7.00
Embedding size
48
7.00
Embedding size
96
7.00
Embedding size
192
Validation Errt
6.00
6.00
5.00
5.00
10
20
30
15
45
20
40
80
HPO Budget [GPU-hours]
Embedding size
384
6.00
4.00
30
60
90
120
HPO Budget [GPU-hours]
DPL
ВOHB
Random
oracle small scale
The incumbent performance of DPL. and other baselines during the HPO budget full function evaluations for different values of the embedding size fidelity. Dashed lines indicate the point at which the oracle been evaluated for every algorithm. Solid curves and shaded areas stand for mean value across runs and standard error.
Figure 12:
proxy and target tasks not always perfect. This can result in proxy task incumbent that does not translate to the oracle in the target task, which can be observed particularly at lower fidelity levels.
18

---

Error Largest Scale
HPO budget
full func evals
HPO budget full func evais
HPO budget 3 full func evals
4.40
4,00
4,00
4.20
3.95
3.95
4.00
3.90
3.90
12
48
96
192
12
24
96
192
24
96
192
Small Transformer Scale lembedding size]
Error Largest Scale
HPO budget full func evals
HPO budget full func evals
HPO budget 6 full func evals
4.00
4,00
4,00
3.95
3.95
3.95
3.90
3.90
3.90
12
48
192
12
96
24
96
192
Small Transformer Scale lembedding size
DPL
BOHB
Random
oracle largest scale
Figure 13: The target task performance distribution for DPL and other methods over different HPO budgets ranging from full function evaluations.
C Continuous Search Space
0.9
Incumbent Error
0.6
0.1
0.00
0.25
0.50
0.75
1.00
HPO Budget
DPL
BOHB
Random
SMAC
Figure 14: The incumbent error of DPL, as well as, the baselines on the CIFARIO task over the HPO budget.
The primary objective of this study is to investigate the efficacy of Deep Power Laws (DPL) trading off exploration vs exploitation in a continuous HPO search space. In this study, we do not make use of pre-computed tabular tables, but instead we optimize the hyperparameters model online, by iteratively pausing unpromising configurations and moving forward only promising hyperparameter configurations during the HPO optimization procedure.
an EfficientNetV2
To benchmark our findings, we contrast the results against established baseline algorithms such as random search, BOHB, and SMAC.
We employ EfficientNetV2 43 as benchmarking model and train on the CIFARIO dataset (24). Specifically, we train the lightweight variant of EfficienNNerV2-b0 from scratch for 50 epochs, using the RMSprop optimizer. The learning rate is initiated 10 6 and gradually increased over span of five warmup epochs to reach the learning rate value of 5 10-4, Following the warmup phase, we employ a cosine learning rate scheduler, with a decay factor 0.97 applied every 10 epochs. The weight decay is set at 10 with no momentum used. Furthermore, the dropout rate is configured to be 10-6 and the model's moving average exponential decay is set at 0.9996. During the training phase, the batch size is set to 64, while for the validation phase, it is reduced to 8. All experiments are performed using the timm library [46]
Baseline:
19

---

Search Space: In our experiment, we concen- on optimizing the two most critical hy- perparameters, learning rate, and weight decay, while keeping the remaining hyperparameters fixed as per the baseline model. We construct a search space for these two hyperparameters in accordance with common practices (Table To emulate continuous search space for the acquisition function, we generate 100 equally-sized steps on a logarithmic scale from the lower bound to the upper bound of each dimension. This process yields a search space comprising 10* potential configurations.
HP
Values
LR weight decay
[10-5 10
[0. 10
Table 4: Search space of CIFARIO task.
Our method demonstrates a substamtial speedup in terms of anytime performance when compared to baseline algorithms (Figure [4p. Acknowledging the practical constraints of evaluating large-scale models, we pragmatically allocated an HPO budget for maximum of 3 full-function evaluations, equivalent 150 epochs. The results of our exploration underline the compelling potential of the DPL. algorithm to effectively manage HPO tasks within continuous search space. Exhibiting significant speedup gains, DPL proves itself to be not only viable but an efficient method for identifying optimal hyperparameters. The findings further underscore the adaptability and efficacy of DPL in addressing complex HPO tasks, reinforcing its standing as a valuable tool in the machine learning toolbox.
D PD1 Investigation
ser at sota, Adl msted 18M
atvite 254
100
300
200
180
02
2.4
02
65
0.
Figure 15: Configuration test performance histograms for datasets where DPL does not outperform baselines.
We investigate the datasets: fashion_mmise _aarrtta______-- pal__poollngoom _asiiingggnnn_nnn can__cnntttrraaa battlbbbtc_____eeee size, _i2__266, ci for100_wiideeeeeee _w__wu_eeeeeee______eeeeeeee___e sasl__bacc__s___eeeeeeeeeeeeseee Bttcch_siee______ 256, suhn_noo cetto_no_eztttee ha___tooeeraaaaeeeeeeeeeeeeeaeee wet_eee_aiddeeeseeeeeeee wwdeeresnet, sertt__btch__eeee _isee_ssse_e__ 1024,
where DPL does not outperform other methods for the PD1 benchmark as shown in Figure 23 We analyze the test performance of the individual hyperparameter configurations that belong to the aforementioned datasets. Figure [5] shows that the search spaces of the datasets have a skewed distribution of performances, where, there exist large number of hyperparameter configurations that achieve top performance. In such datasets, even a non-model based technique will quickly find a well-performing configuration, since, there is high chance for randomly-sampler configuration to achieve the top performance. For this reason, there is little room for sophisticated HPO techniques in these datasets.
To further validate our hypothesis, we investigate two additional datasets dionis from the LCBench benchmark and uniref50 trans former sat__bbbtt_ssseee 128 from the PDI benchmark, where DPL achieves a strong performance compared to baseline HPO methods. The results in Figure show that DPL excels tasks that complex and that require optimizations find hyperparameter configurations that achieve top performance.
Based on the results, we conclude that the lack of statistical significance in PD1 is not specific failure mode of DPI., but a consequence of multiple PD1 datasets where the majority of configurations achieve the top performance.
20

---

dioris
uniref50 transformer batch_szee
1200
403
350
300
Frquerey
Fresuency
400
160
103
Nosuraay
Accuracy
Figure 16: Histograms of the distribution of performances for datasets where DPL, performs strongly.
Embedding size
6
Embedding size
12
Embedding size
24
20
20
Percentage of Iteration Time
0.0
1.0
0.0
0.5 HPO Budget
0.0
0.5
1.0
Embedding size 48
Embedding size
Embedding size
192
20
10
10
0.0
0.5 HPO Budget
0.0
0.5
1.0
Embedding
384
10
0.0
0.5
1.0
HPO Budget
Figure 17: The percentage taken by the DPL overhead in the total time per iteration for the different embedding sizes in nanoGPT-Bench.
E DPL Overhead
To investigate the efficiency of DPL in terms method runtime, we investigate the percentage of time that the DPL overhead contributes in the total time taken to perform one HPO iteration in the nanoGPT-Bench. The total time taken constitutes of the DPL overhead (training the DPL surrogate, calling the acquisition function to suggest the next hyperparameter configuration) and the time taken to run the target algorithm for one more step. Figure 17 shows that the impact the DPL overhead is negligible in the total time taken. For the smallest embedding of size 6, DPL takes only 10% of the total time taken to perform one HPO iteration after spending half of the optimization budget. At the end, after circa 40 hours of HPO optimization, DPL has impact of 20% in the total time taken to perform one HPO iteration. The impact is even smaller for the largest embedding of size 384, where DPL has impact of only 5% in the total time taken per iteration after spending half of the optimization budget and it has an impact of only 10% % in the total time per iteration after more than 120 hours of HPO optimization.
The findings validate our claim that DPI. has a minor time overhead performing hyperparameter optimization, which explains the strong any-time performance of our method.
21

---

F Details of Considered Benchmarks
LCBench: We use the official implementation as the interface for the LCBench benchmark P As suggested by the authors, we use the benchmark information starting from the second step and we skip the last step of the curve since it is repeat of the preceding step.
The TaskSet benchmark features 1000 diverse tasks. We decide focus only 12 NLP tasks from the TaskSet benchmark to add variety to our entire collection of datasets. Our limitation on the number of included tasks related to the limited compute power, the entire suite of tasks offered in TaskSet. TaskSet features a set of 8 hyperparameters, that consists i) optimizer-specific hyperparameters, such as the learning rate, the exponential decay rate, B1 and S2. and Adam's 's constant for numerical stability ii) hyperparameters that control the linear and exponential decay schedulers for the learning rate decay, and lastly iii) hyperparameters that control the LI and L2 regularization terms. Every hyperparameter in TaskSet except 81 and G2 is sampled logarithmically.
TaskSet:
are unable run for
We the synetune library we only include datasets that have a learning curve of length greater than 10. We furthermore only include datasets that have learning curve lower or equal to 50 to have fair comparison between all benchmarks by having approximately 20 full- full-funtiion evaluations. PD1 features 4 numerical hyper- parameters, Ir "r_initial_uabue Ir power, Ir rr_decay_ste seteaa_ssssp____e steps_factor factor and one_ mme_minss___ wtw.mmuuuummmnnuuuununma _omennumm where Ir_ L__jinttaal__eee _aalue and one_minee tate_minnssmmmee momentum are log sampled. The learning rate decay is applied based polynomial schedule, its hyperparameters taken from the search space.
PD1:
42 for our interface the PD1 benchmark. From the benchmark,
G Baselines
We implemented random search by randomly sampling hyperparameter configura- tions from the benchmarks with the maximal budget.
Random Search:
We use version 0.7.4 of the HpBandSter library as common codebase for all 3 baselines 5 For the last approach mentioned, despite heavy hyperparameter tuning of the method, we could not get stable results across all the benchmarks and hence dropped the method from our comparison.
Hyperband, BOHB, LCNet:
ASHA: For the implementation of ASHA we use the public implementation from the optuna library, version 2.1..0.
DEHB:
We use the public implementation offered by the authors 4
ME-DNN: In our experiments we used the official implementation from the authors However, the method crashes which does not allow for full results on all benchmarks.
SMAC: For our experiment with SMAC we used the official code base from the authors
Dragonfly: We version 0. of the publicly available Dragonfly library.
For all the multi multi-fidelity methods considered in the experiments, we use the same minimal and maximal fidelities. In more detail, for the LCBench, TaskSet and PD1 benchmarks we use minimal fidelity lower bound of and maximal fidelity lower bound equal to the max budget.
H Plots
In Hypothesis 1, we prove that DPL achieves better performance in comparison to other models in estimating the final performance for different hyperparameter configurations based on partial observations. Our analysis shows that DPL manages to retain the ranks of different hyperparameter
http:://giccabb.ccc/attt/t//////
http:://ggttabb..om////bbbbb//1p
https///ggthab..co//uttt///////
22

---

Absolute Relative Error
0.4
0.3
0.2
0.1
0.0
0.1
0.2
0.3
0.4 0.5
LC Length Fraction
Figure 18: The absolute relative error distribution of DPL over the different learning curve fractions in the LCBench benchmark. The distribution is calculated from the ground truth and prediction values, averaged over all configurations dataset. Every distribution is over the datasets.
configurations. We complement Hypothesis by additionally investigating the absolute relative We measure the difference between the DPL. estimation of the final learning curve value for different fractions of available partial observations the actual end performance. Figure T8 shows that DPL. does not only retain the ranks of the final performances of different hyperparameter configurations, but it also correctly estimates the final performance by attaining small relative error, where the error is reduced the more partial observations we have from the learning curve.
In Hypothesis 3, we investigate the efficiency of DPL in exploring more promising configurations compared to other HPO methods. In Figure 19 we provide the same comparison with regards the PD1 benchmark. Based on the results, we conclude that DPI. explores more promising configurations compared to other HPO methods.
PD1
PO1
PD1
0.8
0.20
0.00
Fractton of Roor
0.6
0.15
Thecaten
0.10
0.04
0.05
0.02
Length
Lenget
LCengghhh
ASHA
DEHB
Hyperband
Random
SMAC
Basetine
Figure 19: Post-hoc analysis study DPL's efficiency. Left: Share of the best candidates selected during training. Middle: Average regret of configurations chosen to be trained at each budget. Right: Share of top third configurations a given budget which were bottom two third configurations at previous budget.
Lastly, we provide the per-dataset performances all methods, where we present the mean regret of the incumbent trajectory and the standard error over 10 runs in LCBench (Figure 20 and 2T). TaskSet (Figure (2). and PDI (Figure 23). The results show that DPL consistently outperforms other methods in the majority of cases achieving strong any-time performance and not only strong final performance.
23

---

10"
10%
16
10
Budge
Buege.
0%
00:00
DPL
DCHB
Rasdon
Randor
Dragarty
Dragerfy
2010
DONG
Itpetend
90:0
Fastion
K000spp sapebonc
Midhoun
16
HRO Aacort
IPO Burger
0%%
DK-10
Rarcon DMAC
DTL
DC-15
Hander
DFL
DEMS
Randon
ASH-
Deggrrty
45
Drigonty
SMAG
ADHA
Dragrity
50AG
pepertend
BORS
Hypertedd
hyperburd
10
09
622
0A
0.4
E.A
RC44
BC445
Hypwited
B0--
Hyperorrd
Infuraabtta
10
HPO dadgel
HPG Dodgi
HCG Bugge
0%
DC-10
Rarcen
DTL
Handen
DnL
Hander
Dragunity
Dragurry
SMAND
ASIA
Dragorfy
SUAC
244
E4
E3
BP1
Ravion
(9)
DEHA
Readiv
Ade-A
Drsgerity
SMAC
A8r4h
Drugorty thpettend
RMAC
ARHA
Dragerty
AMAC
DOrt
BONS
03:0
hypeowrd
dadgel
Sudget
Dugge.
DPL
D0:0
Rancon
DPL
DCHB
Raxdon
CPL
DENG
Randor
Draporiy
Dragorty
Bragerfy
S010
BONG
0310
Figure 20: Performance comparison over the number of epochs on dataset level for LCBench. We plot the mean value over 10 runs and the standard error.
24

---

1/-
107
Budgd
Burge.
0%%
00:00
DPL
DCHB
Rasdon
Randor
Dragariy
Dragorty
2010
DONG
00:0
Rypetarr
pareen
J:ge drees 20st, endgres compaey
10"
10.7
HPO Aarort
HP. Rudge
IP0 Burget
D%%
DK-10
Harcon
DTL
DC-15
Hander DMAG
DPL
DEMS
Randon 50AG
ASH-
Dragerity
Drigonly
ADMA
Dragrity
pepertend
BORS
Hyyetted
83m0
Hyperburd
Ideat fedtre
10
62
0A
022
0.4
E.A
R044
BC445
Hypwited
82-B
Hypenura
16
HBO dadge
HPG Bedgi
HCG Bugge
0%
DC-10
Rerce
DTL
Handen
D%L
DEMS
Hander
Dragunty
Dragurry
SMAC
ASHA
Dragorfy
S0AC
ryblis
22
244
31
E4
E2
0P1
(6--0
[P1
091
DEHA
Rzadiv
Ade-A
Drsgerity
A8-4h
Drugorly
RMAG
ARHA
Dragorty
AMAC
DOrt
popeband
BONS
thpertend
hypeowar
dudge
bedge
DP.
DENG
Rarcer
DPL
DENG
Sandon
Daporty
S010
Figure 21: Performance comparison over the number of epochs on a dataset level for LCBench (cont.). We plot the mean value over 10 runs and the standard error.
25

---

inch prcc122 LS1 M11212 say bolf
Floed CoaddriCaassttooooccoo
TeedTocONNGG osalcation
mdb paich? 28 LS 511M188 unbwel22 bet
10"
10
Regnet
10
Reget
10
0.5
HPO Budget
HPd Budiel
HPO Budget
DPL
Random
DPL
DEHB
Randem
DPL
DEHB
Random
ASHA
Dragonity
SMAC
ASHA
Dragoniyy
BMAC
ASHA
Diagenfly
SMAC
BSHB
Hyjeroend
BOHES
Hypersond
BOHB
Hyperoend
Floed ToXFFNNC codiocooo
Flxed TONRMNNG ospilcadion
Inch paten57 GRU118 nx17
Insh patro33 GRBI1 ang hx12
mrb pairt12 INNN relo man_bet?
10"
Regget
Regret
10"
10"
0.0
0.
0.4
0.0
0.8
1.0
HPO Budget
HPO Budget
HPO Budgut
DEHB
Random
DP.
Random
DEHB
Drogontty
Dragonily
SMAC
Daggnnty
SMAC
BOMB
Hypertend
BOHB
Hypprrend
BOHB
Hypertand
Fioed TudR&NNC cariooin
loaion
FleedTauRRN asstrradon
10
to"
Regiet
Rigre
10
10
10
10
10
0.0
0.2
0.4
0.6
0.3
1.0
0.0
0.2
0.4
6.6
0.8
1.0
0.0
0.2
044
0.6
0.3
1.0
Budget
Budget
Budget
DPL
DEHB
Random SMAC
DRL
DEHB
Random
DPL
DEHB
Random
ASHA
Drogoniyy
SMAC
Diagantty
SMAC
Hypertond
Hyperoand
Hypertond
TEMIRNNG asaterdon
Info calnte1 vibuiaso wsy txr12
kndo patch12 ViNha54 berh asp, te:12
10
Ieth catrest Vitaae 26 bes 2e12
8'10
70
Regnet
10
3x10
10
2"10"
10
0.
64
0.8
1.0
0.0
02
0.8
1.0
0.4
0.6
1.0
HIO Butgel
DPL
DE/t3
Handom SMAC
DPs
DEHB
Raedon SMAC
DPL
DEMB
Raendo SMAC
ASHA
ASHA
Diagertly
ASHA
Diagenfly
*youtband
BOHB
Myper
Hypweband
Figure 22: Performance comparison over the number of steps on a dataset level for TaskSet, We plot the mean value over 10 runs and the standard error.
26

---

boton,
pooling onn
206
teshton max pooling ono, tonn 206
Regen
Reghet
Regnet
10
10
0.0
02
0.4
0.6
0.3
1.0
0.0
0.2
0.4
0.6
0.
1.0
00
02
0.4
0.8
0
1.0
HPO Burigel
HPO Budgel
Budiel
DPL
DEHB
Random
DP:
DEHB
Random
DEHB
Random SMAD
Dragorily
Draganlly
SMAC
Daggnnfy
BOHB
Hyportand
Hyperoand
Hypertond
meat smpple con, bwich sire, 259
Into Incrafome Jwten drw 2045
maad mas pootigg onn relu beich eim. 258.
10
10"
Regiet
Regnt
10
0.0
0.
0.4
0.0
0.8
1.0
HPO Budget
HPO Budget
HPO Budgut
Random
DPL
Random
DEHB
Drogenty
Dragonfly
Dagenty
BOMB
Hypertend
BOHB
Hyperzend
BOHB
Hypertand
m=te__ max wwt.ppoolgggooomm mmmmm, Iann, botch ston, 2016
mrist prrtt_smpphpgg ccm botch, wtt.hh_ste_ee 2.50
9hhn, /7, extra, wide, resnet batch sttee_ 1024
10"
Ragget
Pegget
10"
10
10
10"
0.0
0.2
0.0
0.6
0.8
1.0
HPO Budgel
HPO Budipl
MPO Budiel
ASHA
Dragorfly
SMAC
Bragarily
SMAC
Drugonfly
SMAC
BOHB
Hyperoend
8OHB
Hyperoand
BOHB
Hypecband
unnel50 ze-aforer batch diz 125
Reght
10
HPO Budget
DP.
ASHA
Di Disgorfly
SMAC
BCHB
Hypwrzand
Figure 23: Performance comparison over the fraction the total optimization iterations on a dataset level for PD1. We plot the mean value over 10 runs and the standard error.
27


Reviews:
[{"review_id": "Jjlvn1wDGN", "reviewer": "Reviewer_7buU", "summary": "The paper proposes deep power laws ensembles for hyperparameter optimization. More precisely, it is used as a surrogate for Bayesian Optimization (BO) to estimate the performance of a hyperparameter configuration leveraging ensembles of deep power law functions. Furthermore, it is combined with multi-fidelity optimization to estimate the performance for an upcoming budget, enabling incremental training that can be paused for individual configurations based on the performance estimate of the surrogate.", "questions": "For easier reference during the rebuttal, the questions are enumerated below:\n1. Do you agree or disagree with any of the remarks from section Weaknesses?\n2. Do you agree or disagree with the suggested corrections of the mathematical formulation listed in the Weaknesses section?\n3. l. 171-172: “HPO budget is defined as the maximum number of steps needed to fully evaluate 20 hyperparameter configurations.” How about the overall CPU hours needed to execute the experiments with different approaches? Do they differ? Are there any additional ecological insights, e.g. from a Green AutoML perspective [3], like one might save wall clock time compared to others? Does this relate to performance improvement? \n4. l. 205-206: The description of the used hardware is incomplete as memory is missing. \n5. I am wondering why there is no bold conclusion for Hypothesis 4?\n\nFinal remark regarding the overall rating: I am strongly convinced that the weaknesses can be easily addressed during the rebuttal phase. If the authors do so, I am more than happy to increase my score to an accept as I think that the paper is really good besides the weaknesses above.\n\n[1] Buratti, B., & Upfal, E. (2019). Ordalia: Deep Learning Hyperparameter Search via Generalization Error Bounds Extrapolation. In 2019 IEEE International Conference on Big Data (Big Data) (pp. 180-187). IEEE.\n\n[2] Mohr, F., & van Rijn, J. (2022). Learning Curves for Decision Making in Supervised Machine Learning--A Survey. arXiv preprint arXiv:2201.12150.\n\n[3] Tornede, T., Tornede, A., Hanselle, J., Mohr, F., Wever, M., & Hüllermeier, E. (2023). Towards Green Automated Machine Learning: Status Quo and Future Directions. Journal of Artificial Intelligence Research, 77, 427-457.\n", "limitations": "Limitations are not explicitly given in the paper if I have not overseen something. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "### Originality and Quality\nThe presented approach is interesting, as it combines promising directions of HPO: Multi-Fidelity and Bayesian Optimization based on power laws. It is original in the way that it is the first work (to the best of my knowledge) that leverages Deep Ensembles for power laws as surrogates for BO.Moreover, a simple but original multi-fidelity strategy is presented to dynamically adjust the budget a configuration is evaluated on, allowing incremental training of configurations by pausing and continuing the process dependent on the performance estimate of the surrogate. Overall, none of the parts of the presented approach is innovative on its own, but the combination thereof is.\n\n### Clarity\nThe paper is well structured and written in an understandable manner. The different components and their interaction could be clearly separated and emphasized more.\n\n### Significance\nThe presented method is evaluated on 3 state-of-the-art benchmarks, as well as compared to 7 state-of-the-art HPO tools. The evaluation of 4 well selected hypotheses lead to the conclusion that assumptions are accurate, the approach is working correctly, and the presented method outperforms competitors in most cases.\n", "weaknesses": "### Originality\nThe originality of this paper is partly hard to define, as related work or citations are missing at key aspects of this paper. Especially regarding the usage of power laws in the context of HPO, existing work is not investigated / cited [1, 2]. Furthermore, motivation, explanation, and limitation of multi-fidelity and power laws are mixed up (see e.g. l. 57-59). Moreover, the presented multi-fidelity strategy is not put into context of existing strategies. Furthermore, the authors mix up gray-box HPO and multi-fidelity HPO (e.g. l. 95: “Gray-box (multi-fidelity)”). Multi-fidelity can be classified as gray-box, but not the other way around.  In addition, the assumption that every learning curve can be described by a power law function should be verified by a reference or two. Overall, the work could be better embedded into existing work on learning curves, see [2]. \n\n### Clarity and Quality\nThe clarity and quality of the paper can be improved. On one hand, the paper should be self contained, which mainly refers to a missing formal introduction and short background of power law functions. Related work and approach description are partially mixed or not suitably placed, e.g. exploiting a power law assumption with the presented method would have been expected to be explained within the description of the method and not as part of related work of multi-fidelity HPO. On the other hand, the mathematical formulation can be improved. Details below:\n1. l. 52-53: “[...] the budget is multiplied by the fraction of discarded hyperparameter configurations and the process continues until the maximum budget is reached.” This would reduce the budget with every step, but it should be increased to reach the maximum budget. Suggestion: “[...] budget is divided by the fraction [...]”.\n2. l. 57-59: “However, the only assumption these methods make about the learning curve is that it will improve over time. In contrast, we fit surrogates that exploit a power law assumption on the curves.” There is already work leveraging power-laws for performance prediction of learning curves [1, 2].\n3. l. 79f / Eq. 1: Too much space between $\\theta^*$ and the next bracket\n4. l. 82: The basic definition of $H$ should contain $N$, as it is defined later in Eq. 2 (l. 87f), otherwise multiple usages of $H$ are ambiguous.\n5. l. 83: $\\mathcal{A}$ is defined, and directly afterwards the normal $A$ is used. Furthermore, the set definition of all possible $\\mathcal{A}$ is missing which should be used for $\\arg\\min$ (in l. 87f)\n6. l. 87f / Eq. 2: “$\\lambda_i = \\mathcal{A}$” is an unusual notation and more common in programming than math, “$\\lambda_i =$” could be removed here to avoid confusion.\n7. l. 87f / Eq. 2: $\\Omega$ should be $\\geq$ (greater equals) the sum of the cost, as the budget should not be exceeded, but can be matched.\n8. l. 91: $p$ is not defined, as well as the set definition of all possible $\\phi$.\n9. l. 138: What is “the loss”?\n10 l. 201: SMAC is not necessarily an extension of HB, but only the mf part of SMAC uses HB. This should be properly described. \nLastly, there is a grammar issue in l. 55: “elaborated” → “elaborated on”, and Figure 5 is misplaced as it belongs to the earlier section.\n\n[1] Buratti, B., & Upfal, E. (2019). Ordalia: Deep Learning Hyperparameter Search via Generalization Error Bounds Extrapolation. In 2019 IEEE International Conference on Big Data (Big Data) (pp. 180-187). IEEE.\n\n[2] Mohr, F., & van Rijn, J. (2022). Learning Curves for Decision Making in Supervised Machine Learning--A Survey. arXiv preprint arXiv:2201.12150.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful and detailed review of our work. We provide the following clarifications on the questions raised by the reviewer.\n\n- **Regarding the remarks from the section Weaknesses**:\n    - **In terms of related work:** \n        \n        We will cite both [1] and [2] in our related work.  Additionally, in the case of [1], we will highlight the difference in terms of proposing a \n        conditioned probabilistic power law surrogate that needs no observations per LC in order to estimate the performance. The ability to \n        estimate the performance of unobserved configurations through probabilistic surrogates is essential for Bayesian optimization, \n        therefore, making our method the first to propose power law surrogates for uncertainty-driven HPO.\n    - **Regarding “motivation, explanation, and limitation of multi-fidelity and power laws are mixed up (see e.g. l. 57-59)”:** \n\n        We understand the reviewer’s concern, however, we believe the related work is structured in a consistent manner. At the end of every \n        related work paragraph, we have a sentence that delineates the novelty of our paper from the prior work of that paragraph. In terms of \n        multi-fidelity HPO, our novelty is in proposing a novel surrogate with the power law assumptions. This is the reason why the sentence \n        refers to both multi-fidelity HPO (problem definition) and power laws (novelty).\n    - **Regarding “the presented multi-fidelity strategy is not put into context of existing strategies” and “Overall, the work could be better embedded into existing work on learning curves, see [2].”:**  \n         Our work is positioned as in the context of well-established multi-fidelity HPO (problem definition) and Bayesian optimization (BO, strategy for solving the problem). Given our novel power-law surrogate and acquisition, which are very standard BO components, we assess the method to be positioned within a well-established BO strategy. Additionally, the surrogate function with power laws is connected to existing work on **learning curves (Line 60)**. Furthermore, we cite multiple papers in the learning curves prediction paragraph, the majority of which are cited by [2]. However, we agree with the reviewer in further extending/strengthening the related work with the aforementioned works [1]-[2].\n    - **Regarding “mix up gray-box HPO and multi-fidelity HPO (e.g. l. 95: “Gray-box (multi-fidelity)”)”:** \n        \n        We agree with the reviewer. We will make it clear in our work that we address multi-fidelity HPO, and specify that it is a sub-problem of gray-box HPO as the reviewer suggests.\n\n- **Regarding the \"suggested corrections of the mathematical formulation\":** \n    \n    We agree with the suggested changes on the formalism from the reviewer and we will update the camera-ready accordingly.\n\n- **Regarding the \"time performance and ecological insights\":**\n    \n    The reviewer is correct in his/her understanding that the budget for the wallclock experiments is the runtime equivalent of fully evaluating 20 randomly selected configurations. This budget limit is set as a practical compromise considering our available computational resources. All methods, therefore, need to search for an incumbent (best per method) configuration within this budget. For instance, if 20 randomly selected hp configurations are evaluated for a total of say 100 hours, then we set the HPO budget for all methods to 100 hours. Within this budget, methods are free to partially or fully evaluate as many configurations as they decide within their search mechanisms. The best model's performance discovered by each method within the budget is used to compute the comparative metrics; namely regret and ranks. \n\n    Regarding the wallclock HPO time, we point the reviewer to **Hypothesis 2 L283-289 (Figure 4)** where we fairly compare all methods over time [surrogate fit (if it is a model-based method) + hyperparameter evaluation]. **The setup for the experiment is described in Section 5 L159-166**. Summarized, the time of every algorithm is normalized by the random search time and the total time shown in Figure 4 is the time it took random search to perform 20 HPO trials.\n\n    Regarding ecological insights, our method uses fewer computational resources to achieve the same/better performance compared to other methods.  In the case of LCBench, our method matches the final performance of random search in approximately 10% of the total time and the final performance of the closest competitor (BOHB) in 20% of the total time. While for PD1, our method matches the final performance of random search in 30% of the total time and matches the performance of DragonFly the closest competitor in 80% of the total time.\n\n- **Regarding \"the memory description of the used hardware\":**\n    \n    We would like to thank the reviewer for pointing out the missing information. The total memory of every node is 120GB, and every experiment is limited to 2 cores which offer 12GB. We are going to modify Section 5 accordingly for the camera-ready version of our paper since the rebuttal phase does not allow modifications to the submitted manuscript.\n\n- **Regarding \"the missing bold conclusion for Hypothesis 4\":** \n    \n    We agree with the reviewer, and we will add a bold conclusion for Hyp. 4 along the lines of \"The results validate Hypothesis 4 and confirm that DPL is an efficient HPO technique for tuning the hyperparameters of large language models when the HPO is conducted using smaller transformer model sizes.\"\n\n\nIf the reviewer is satisfied with the clarifications and proposed changes we would appreciate a reflection of the discussion to the score. In case there are more questions, we are happy to answer them.\n\n\n"}, {"review_id": "6tzhUjbhYu", "reviewer": "Reviewer_sM9u", "summary": "This paper proposes to combine power law patterns in learning curves, which have been recently popularized by scaling laws, to improve efficiency and performance of Bayesian optimization (BO) based hyperparameter optimization (HPO). Authors provide detailed discussions on how to model the surrogate function and accuracy of their power law based surrogate function in predicting final performance. Tested on three HPO benchmarks, authors empirically demonstrate that DPL achieves better performance given a limited budget compared to other BO-based HPO baseline algorithms. ", "questions": "1. I am curious about the applicability of DPL to different ML techniques such as random forest or XGBoost, which is arguably the most popular option for tabular data. I am not an expert in this domain, but I doubt the concept of learning curves also plays an important role in those types of models. If not, I believe stating it clear and focusing writing on the area where their method shines most would actually improve the clarify of the paper.\n2. I want to know author's opinion on the applicability of their method on training recent large models. The training cost of recent large models can be astronomical, so multiple training runs assumed by Bayesian optimization may not be realistic despite DPL's improved efficiency compared to the baseline. Even though authors included some transformer experiments, I still think the size of their transformers are quite smaller than recent models. This question is not specifically about DPL, but more about BO-based HPO. Any insights would be much appreciated.", "limitations": "Authors haven't clearly discussed the limitations of their work. Having one or two sentences on the limitations would be helpful for people who are interested in this direction.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I believe the strategy of incorporating power law patterns from scaling laws into the surrogate function of Bayesian optimization is neat and smart in that it enables an improved prediction of learning curves.\n2. In addition, authors have empirically justified the soundness of their method through diverse analyses and experiments. \n3. Given its simplicity, I expect this method would generalize across different tasks not explored in this paper.\n4. Authors described their experiment settings in detail, which was very helpful in evaluating the significance of their method.", "weaknesses": "1. I believe the clarify of the paper could be potentially improved. For example, abstract of this paper is too simple, and I wasn't able to grasp the general direction or the concept of the paper by reading them. Also, I believe moving the algorithm box from Appendix to the main text would help readers to better understand their method. I currently see quite a lot of empty spaces in the paper, so expect, with minor formatting efforts, it would be easy to move the algorithm box to the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful review. We provide the following clarifications on the questions raised by the reviewer.\n- **Regarding “the abstract of this paper is too simple, and I wasn't able to grasp the general direction or the concept of the paper by reading them.”:**\n\n    We will improve the abstract in two ways for the camera-ready version by better describing: i) the problem (describing the multi-fidelity HPO approach with learning curves) we are addressing and ii) the proposed method (introducing a probabilistic scaling law performance predictor with a dynamic acquisition function for Bayesian optimization). \n\n    If the reviewer has other suggestions we would be happy to incorporate them.\n- **Regarding “I believe moving the algorithm box from Appendix to the main text would help readers to better understand the method”:**\n\n    We agree with the reviewer’s suggestion and will move the algorithm to the main paper for the camera-ready version.\n\n- **Regarding the “applicability of DPL to different ML techniques such as random forest or XGBoost”:**\n\n    The reviewer is correct. It would be possible to apply DPL for ML techniques such as random forest or XGBoost, by considering the number of ensemble models as a budget/fidelity. This is a very interesting future work that we will add to a new section “Limitations and Future Work”.\n\n- **Regarding the \"the applicability of the DPL method in training recent large models\":**\n\n    Searching directly on a model with billions of parameters might be indeed challenging. However, in practice, we can tune the hyperparameters on a smaller version and transfer it to the larger version.\n    \n    We would like to refer the reviewer to [1] who motivated a possible transfer theoretically and demonstrated empirically that models with **a few million parameters transfer to LLMs with billions of parameters**. Other AutoML works employ the same idea. Most cell-based NAS methods (for example the following early work [2]) are training small networks and later scale them by increasing width and depth. Similarly, work on transformer search makes use of the same technique [3].\n\n[1] Yang, Ge, et al. \"Tuning large neural networks via zero-shot hyperparameter transfer.\" Advances in Neural Information Processing Systems 34 (2021): 17084-17097.\n\n[2] Pham, Hieu, et al. \"Efficient Neural Architecture Search via Parameter Sharing.\" ICML 2018: 4092-4101\n\n[3] So, David, et al. \"The Evolved Transformer.\" ICML 2019: 5877-5886\n\nIf the reviewer is pleased with the clarifications and proposed changes we would appreciate a reflection of the discussion to the score. In case there are more questions, we are happy to answer them.\n"}, {"review_id": "7Vy9OIrnoF", "reviewer": "Reviewer_rbxf", "summary": "The paper proposes a novel surrogate for multi-fidelity HPO that uses an ensemble of power-law models to estimate the future validation loss of hyper-parameter configurations at intermediate stages of training. The principal novelty of the work lies in exploiting the observation (made in other work) that learning curves tend to follow power laws. \nThorough experiments are presented across diverse datasets and tasks, comparing against publicly available strong HPO baselines; SOTA performance is demonstrated. Analysis demonstrates an improvement in learning-curve forecasting for the surrogate over non-power law techniques, as well as an increase in efficiency for DPL over HPO baselines. \n", "questions": "The results on PD1 seem weaker than those for LCBench and TaskSet in Figures 2,3,4, and PD1 is left out of Figure 5 and pushed to the appendix. Do you have any analysis or commentary on the relative performance of the method as a function of the architectures or tasks involved in each benchmark? From the description in Sec 5.1, it seems PD1 is a quite diverse benchmark. While the analysis presented quite reasonably focuses on the aggregated performance of each benchmark, was any analysis performed as to the relative performance on specific architectures, tasks, or hyper-parameter spaces? ", "limitations": "The practical limitations of the applicability of this work towards LLMs may be under-explored. While the extension of the method towards giant models is not necessary for the impact of this generally strong contribution, it remains unclear if the efficiency gains over existing HPO methods are sufficient to overcome the general concerns of computational inefficiency which have typically precluded Bayesian HPO methods from general use in larger models.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper presents a novel contribution to the HPO methodology literature. The claims of the paper are well supported by experiment, and generally well analyzed. The specification of the separate hypotheses of the work in the analysis gives a pleasantly clear structure to the paper. The results demonstrably advance the SOTA in HPO; this work seems very likely to be used and built upon in future HPO work.", "weaknesses": "The final analysis section, evaluating the use of DPL for LLMs, is less convincing than the previous sections. Significant work is done by line 319 “We follow the common practice of conducting HPO with small transformers and then deploying the discovered optimal configuration on the full-scale transformers”. Citations of prior work which discuss this convention or apply the ‘common practice’ should be included (perhaps the paper on Tensor Programs V (Yang et al.2022)). Following the convention of that paper, it would be valuable to include the total computational cost of tuning the small models as a proportion of the computational cost of training the large model once (with the small-model-identified parameters). This would give a much better sense of the practicality of applying such a technique in a LLM setting, where the computational burden of training is paramount. \n\nNits:\nIn Section 5.1, the architectures of LCBench and PD1 are mentioned, but for TaskSet no mention of architecture is made.\nIn the LLM section, total parameter count of the various models should be reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful review. We provide the following clarifications on the questions raised by the reviewer:\n\n- **Regarding “Significant work is done by line 319. We follow the common practice of conducting HPO with small transformers and then deploying the discovered optimal configuration on the full-scale transformers. Citations of prior work which discuss this convention or apply the ‘common practice’ should be included (perhaps the paper on Tensor Programs V (Yang et al.2022))“:**\n\n    \n    We agree with the reviewer. To improve clarity, we will refer again to the Tensor Programs V work from Line 74 (in the related work).\n\n- **Regarding \"Following the convention of that paper, it would be valuable to include the total computational cost of tuning the small models as a proportion of the computational cost of training the large model once (with the small-model-identified parameters).\"\":**\n\n    We agree with the reviewer. In our LLM experiment (Section 6, Hypothesis 4), it takes 3.66 hours to find the Oracle configuration for the largest model via HPO for the smallest model. In turn, it takes 21.52 amount of hours to train the largest model only once. As such, the proportion is 0.17. We will update our camera-ready version as suggested by the reviewer.\n\n- **Regarding: \"In Section 5.1, the architectures of LCBench and PD1 are mentioned, but for TaskSet no architecture is mentioned.\":**\n\n    We thank the reviewer for pointing out the missing information. The architectures for TaskSet are a Variational RNN, Identity RNN, GRU RNN, and LSTM RNN. We will make sure to update Section 5.1 for the camera-ready version with the details of the architectures included in TaskSet.\n\n- **Regarding: \"In the LLM section, the total parameter count of the various models should be reported.\"**\n    \n    We agree with the reviewer. The parameter counts for the various models are as follows:\n\n    | Embedding Size  | Total Parameters |\n    | :---------------------- | :---------------------: |\n    | 6                           | 0.3 M                   |\n    | 12                         | 0.6 M                   |\n    | 24                         | 1.2 M                   |\n    | 48                         | 2.6 M                   |\n    | 96                         | 5.5 M                   |\n    | 192                       | 12.4 M                 |\n    | 384                       | 30.1 M                 |\n\n    We will update the camera-ready version to include the parameter counts of the models.\n- **Regarding \"The results on PD1 seem weaker than those for LCBench and TaskSet in Figures 2,3,4, and PD1 is left out of Figure 5 and pushed to the appendix. Do you have any analysis or commentary on the relative performance of the method as a function of the architectures or tasks involved in each benchmark? From the description in Sec 5.1, it seems PD1 is a quite diverse benchmark. While the analysis presented quite reasonably focuses on the aggregated performance of each benchmark, was any analysis performed as to the relative performance on specific architectures, tasks, or hyper-parameter spaces?\"**\n\n    We thank the reviewer for the interesting question. We would like to point the reviewer to Section 6, Hypothesis 2 (Line 274) where we provide our comments on the lack of statistical significance for PD1. \n\n   \n\n    As a summary, we did investigate the specific tasks/search spaces where DPL does not perform as well as the other baselines, as the reviewer suggests. We notice that the tasks have a skewed distribution of hyperparameter configuration performances, where, a majority of the configurations achieve top performance. \n\n    The detailed analysis can be found in Appendix F.\n\n\nWe believe to have adequately addressed the questions from the reviewer. In case there are more questions, we are happy to answer them."}, {"review_id": "MSSt18dpZE", "reviewer": "Reviewer_wp1o", "summary": "AFTER REBUTTAL: I acknowledge reading the rebuttal. After the reviewer discussion, I will keep my score the same.\n\n----\n\nGray-box hyperparameter optimization involves optimizing neural network hyperparameters by evaluating performance at low budgets and terminating configurations if they seem unpromising. This paper proposes a gray-box scheme using Bayesian optimization over neural scaling laws to estimate future model performance. The key insight is to fit an ensemble of neural networks that can estimate the power law parameters and then apply BO to estimate new hyperparameter settings. Numerical results demonstrate the power law’s ability to forecast as well as state-of-the-art relative regret in HPO.", "questions": "Please see the weaknesses", "limitations": "Limitations are not meaningfully discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall the paper is well-written, clearly motivated, suggests an intuitive strategy, and rigorously experimented.", "weaknesses": "-\tThe function $f(\\lambda)$ gets overloaded with $f(\\lambda, b)$ or as $f(b)$ in various parts.\n-\tThe definition of the cost function is not present. It would be nice to have a slower explanation of the experiment setup including the cost function and budget, since the plots tend to just look at normalized budget metrics.\n-\tThe ideas of learning a meta model of power laws, of using Bayesian optimization over power laws, and using an ensemble of power laws have all been explored in recent works. Although these works do not focus on hyperparameter optimization, it would be nice to differentiate the current work from the previous in terms of methods.\n - Jain, Achin, et al. \"A meta-learning approach to predicting performance and data requirements.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n - Tejero, Javier Gamazo, et al. \"Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n - Mahmood, Rafid, et al. \"Optimizing data collection for machine learning.\" Advances in Neural Information Processing Systems 35 (2022): 29915-29928.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful review. We provide the following clarifications on the questions raised by the reviewer:\n\n- **Regarding “The function $f\\left(\\lambda\\right)$ gets overloaded with $f\\left(\\lambda, b\\right)$ or as $f\\left(b\\right)$ in various parts.”:**\n\n    We thank the reviewer for spotting the inconsistency in our formalism, to improve clarity, we will rephrase $f^{best}\\left(b\\right)$ and $f^{oracle}\\left(b\\right)$ as $f\\left(\\lambda^{best}, b\\right)$ and $f\\left(\\lambda^{oracle}, b\\right)$. This affects Equation 7 (bottom) and Equation 9 (as well as the text that refers to both terms in Line 126 and Lines 146-151). We will incorporate the aforementioned changes for the camera-ready version of our work.\n\n    \n    Regarding the preliminaries of Section 3, there is no notion of $b$ for the first 2 paragraphs as multi-fidelity has not been introduced yet. Additionally, in Section 6, when $f\\left(b\\right)$ is used, it refers to models that do not take the hyperparameter configuration as an input.\n- **Regarding \"The definition of the cost function is not present. It would be nice to have a slower explanation of the experiment setup including the cost function and budget, since the plots tend to just look at normalized budget metrics.\":**\n\n    We would like to point the reviewer to the **experimental protocol, Section 5, lines 137-144** where we describe the cost function and budget. **“One unit step of the HPO budget signifies training one hyperparameter configuration for one more step (1 epoch in LCBench, or 200 iterations in TaskSet)”**. We then describe in detail what step represents for all benchmarks and the benchmark-specific default evaluation metric. We summarize the information below:\n\n    |Benchmark | Learning curve step |Evaluation metric|\n    |:---------------|:---------------------------|:----------------------|\n    |LCBench    | 1 epoch                    | Bal. Accuracy      |\n    |TaskSet      | 200 batch iterations  | Loss                   |\n    |PD1            | 1 epoch                     | Accuracy           |\n\n- **Regarding \"The ideas of learning a meta-model of power laws, of using Bayesian optimization over power laws, and using an ensemble of power laws have all been explored in recent works. Although these works do not focus on hyperparameter optimization, it would be nice to differentiate the current work from the previous in terms of methods.\":**\n\n    We thank the reviewer for suggesting the parallel works at CVPR 2023, and the NeurIPS 2022 paper, which although do not address HPO, still elaborate on scaling laws for performance predictions. We will integrate the suggested works into our work for the camera-ready version. We agree with the comments on novelty, **except for the point “... using Bayesian optimization over power laws … has been explored in recent works”**. To the best of our awareness, we are the first to explore Bayesian optimization for HPO with power laws.\n\nWe believe to have adequately addressed the questions from the reviewer. In case there are more questions, we are happy to answer them\n\n"}, {"review_id": "TNFAM7isag", "reviewer": "Reviewer_ALT2", "summary": "The authors present a hyperparameter tuning scheme based on optimizing surrogate powerlaws. They claim substantial improvements over baselines on hyperparameter tuning benchmarks.", "questions": "On \"Unfortunately, HPO is not yet feasible for Deep Learning (DL) methods\" -- what does this mean? Hyperparameters are optimized all the time.\n\n", "limitations": "NA", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing is generally clear, and the method is well described. The authors report substantial improvements over baselines on what appear to be standard benchmarks. Real improvement here would be significant for the field.\n\nI should note here that I am unfamiliar with these HPO baselines and benchmarks. While the reported results seem potentially interesting, I place low confidence in my own assessment here. Input from other reviewers with more familiarity with the methods of assessment and the difficulties one encounters when using HPO in practice will be important here.", "weaknesses": "It is very hard to assess Hypothesis 1 from Figure 1. This figure seems like it ought to be a final summary figure after some more illustrative figures showing, for example, parameter vs. performance, with five proposed fits overlaid. More generally, the lack of figures explicitly showing powerlaws in this paper about powerlaws is quite odd and makes it hard to gauge the claims made. For example, I'm quite skeptical that *all* metrics one might want to track follow nice powerlaws as the proposed fitting function assumes -- which seems like it'd largely invalidate the proposed HP tuner -- but I could be convinced by lots and lots of plots showing that metric after metric is indeed forecasted nicely by a powerlaw. This is what I'd expect to see here.\n\nFor a paper using powerlaw scaling, there's a surprising lack of discussion of [Kaplan et al. (2020)](https://arxiv.org/abs/2001.08361) or the Chinchilla scaling laws, which are foundational results dealing with powerlaws + large models used in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful review. We provide the following clarifications on the questions raised by the reviewer:\n\n- **Regarding “It is very hard to assess Hypothesis 1 from Figure 1. This figure seems like it ought to be a final summary figure after some more illustrative figures showing, for example, parameter vs. performance, with five proposed fits overlaid. More generally, the lack of figures explicitly showing powerlaws in this paper about powerlaws is quite odd and makes it hard to gauge the claims made.”:**\n\n    We would like to initially clarify to the reviewer that our power law surrogate is a function of the number of epochs/iterations of optimization (as explained in the experimental protocol in Section 5) and not on the number of parameters.\n\n    Hypothesis 1 Figure 1 is directly addressing the reviewer's question on “How well do power laws model learning curves”. Our experiment is based on the rationale that “method A models learning curves better than method B if method A is able to forecast future unobserved values of the learning curve (given a partially observed curve) more accurately than method B”. We opted for Hypothesis 1 because we believe that reporting the aggregated forecasting accuracy across thousands of learning curves is a more principled assessment than visually interpreting a few learning curves.  Regarding the metric of evaluating the forecasting performance, we present both results in mean absolute relative error in Appendix J, Figure 18, as well as the rank correlation in Figure 1. \n\n\n    Nevertheless, we understand the reviewer’s concern that simpler visualizations will help readers understand the method more quickly. As a result, we provide additional figures (extra page, in the global response) that visually show how DPL fits learning curves from diverse datasets included in our experiments. DPL is given partial observations from **only the learning curve that is investigated** and infers the rest of the learning curve based on the observed points.\n\n    We will incorporate these visualizations into the camera-ready version.\n\n\n- **Regarding “For example, I'm quite skeptical that all metrics one might want to track follow nice powerlaws as the proposed fitting function assumes -- which seems like it'd largely invalidate the proposed HP tuner”:**\n\n    We would refer the reviewer to [1], additionally [2] (that the reviewer suggests) which show that well-behaved learning curves “generally” follow a power law pattern.\n\n    However, the reviewer is right in stating that not “all” learning curves follow a power law assumption. We fully agree with that statement, and also **stress this point in the paper in lines 247-249 “The findings indicate that even though not all learning curves are power laws, most of them are, therefore a power law surrogate is efficient in forecasting the final performance of a partially-observed learning curve.”**\n\n\n    At the end of the day, the quality of the HPO results is the ultimate metric of success, not the quality of learning curve modeling. HPO is a different process than learning curve forecasting, because it focuses on exploring and exploiting regions of performant hyperparameter configurations, in order to recommend the “best configuration to evaluate next”. Through ample experiments, we show that Bayesian optimization equipped with our novel power law surrogate and acquisition achieves better HPO results. \n\n    It is worth emphasizing that learning curves that deviate significantly from the power law assumption are usually diverging configurations (e.g. loss and error rate increasing suddenly, as in the case of very high learning rates). Modeling such learning curves suboptimally appears to not hurt the quality of HPO, because divergent learning curves usually represent bad hyperparameter configurations with high error rates. Since in HPO with Bayesian optimization the acquisition recommends only the best-estimated configuration (the one with the lowest estimated error rate), it is not essential if we optimally or suboptimally estimate the learning curve of the configurations with a high error rate, because the bad configurations are not recommended for evaluation by the HPO algorithm.\n\n\n    **For transparency, we would point the reviewer to Lines 239-245 where we provide a few ways on how to further tackle learning curves that have a divergent behavior. As pointed out by the aforementioned lines, the analysis is extended in Appendix C in more detail.**\n\n- **Regarding: For a paper using powerlaw scaling, there's a surprising lack of discussion of Kaplan et al. (2020) or the Chinchilla scaling laws, which are foundational results dealing with powerlaws + large models used in practice.**\n\n    We agree with the reviewer. We will add the suggested papers to the list of already-cited prior works on scaling laws in Section 3 for the camera-ready version.\n\n- **Regarding: On \"Unfortunately, HPO is not yet feasible for Deep Learning (DL) methods\" -- what does this mean? Hyperparameters are optimized all the time.**\n\n    By HPO we refer to principled and automated searching techniques for tuning hyperparameters, such as Bayesian optimization. rather than manual and ad-hoc parameter tuning. For modern Deep Learning, most researchers and practitioners follow suboptimal trial-and-error searching of hyperparameters based on a local search around an initial guess of hyperparameter values.\n\nIf the reviewer is pleased with the clarifications and proposed changes we would appreciate a reflection of the discussion to the score. In case there are more questions, we are happy to answer them.\n\n[1] Mohr, F., & van Rijn, J. (2022). Learning Curves for Decision Making in Supervised Machine Learning--A Survey. arXiv preprint arXiv:2201.12150.\n\n[2] Kaplan et al. Scaling laws for neural language models (2020)\n\n"}]

Author rebuttal:
We would like to thank all the reviewers for the thorough reviews and for helping us improve the quality of our work. Below, we summarize the main clarifications:

- **Time performance and ecological insights (Reviewer 7buU):** Hypothesis 2 of Section 6 provides a comparison between all methods for the total time it took random search (a model-free method) to evaluate 20 hyperparameter configurations, where our method achieves better or same results compared to our competitors with less computational resources.
- **Weaker results in the PD1 benchmark (Reviewer rbxf):** Hypothesis 2 of Section 6 provides insights on the lack of statistical significance in the PD1 benchmark. A detailed analysis is provided in Appendix F.
- **Missing definition of the cost function and budget (Reviewer wp1o):** The experimental protocol of Section 5.3 provides information in detail on the cost function of experiments (learning curve step or time). It additionally provides information on the default evaluation metric in every benchmark.
- **Explicit mention of the limitations:** We will add a new section to our work labeled “Limitations and Future Work”.

We believe to have answered all of the questions raised by the reviewers and we welcome any new questions during the discussion period.

Meta-review: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.

The reviewers and meta reviewer all acknowledge the original and sound contribution of exploiting a power-laws parametrization with deep ensembles for HPO. Similarly, the reviewers and meta reviewer all call out the overall good quality of the manuscript (writing, structure).

The rebuttal has considerably strengthened some aspects of the work (e.g., richer discussion of related work). However, there are still some remaining important concerns. As a result, the reviewers and the meta reviewer are weakly inclined to accept the paper.

In particular, the authors are urged to carefully update their final manuscript with the following points:

* Integrate in the final manuscript all the changes and discussions mentioned during the rebuttal phase
* Curically, the paper relies on deep ensembles for the surrogate model. This choice should be further motivated (e.g., it makes it possible to easily capture the uncertainty of the power-laws parametrization). Moreover, this choice brings up many important design decisions. In the current form, there are plenty of essential descriptions of those design decisions that are not present in the core manuscript (but rather hidden in the appendix or the code). _All those descriptions should instead be surfaced in the core manuscript_ (to that end, space can be gained by shortening/moving sections such as “preliminaries”). This includes (but it is not limited to)
    * Discussion about the architecture of ensemble members
    * Discussion about the retraining procedure when adding new observations (e.g., the code mentions some example-weighting and early-stopping schemes) 
    * Discussion of the robustness of the approach at the beginning when there are a handful of observations in H (e.g., how many random guesses, how to train the deep ensembles with so little data, ….)
    * Optimisation of the acquisition function (e.g., the appendix/code seem to indicate that this is not done in a gradient-based fashion—how? why?)
* Missing ablation studies:
    * The effect of the only strategy in Eq. (8) 
    * There is some previous research about deep ensembles for HPO, e.g.,  [A, B] (and [C], though more in RL-type settings). If the current submission makes design choices (see thread above) not supported by previous research, they need to be validated by some experiments/ablation studies. For instance, effect of the architecture, effect of the design of the retraining strategy, effect of the number of initial random guesses, etc. (in terms of organization, the discussion could happen in the core manuscript, as explained in the previous bullet point, and the ablation studies in the appendix)

If the paper was submitted to a journal, it would be accepted conditioned on those key changes, the meta reviewer thus expects all those changes to be carefully implemented.

[A] Salam Khazi et al., 2023, Deep Ranking Ensembles for Hyperparameter Optimization

[B] Kim et al., 2022, Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure

[C] Moberg et al., 2019, Bayesian Linear Regression on Deep Representations
Decision: Accept (poster)