SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 11: Distribution of GPU-hours required for training across different model fidelity values.
Figure 12: The incumbent performance of DPL and other baselines during the HPO budget of 6 full function evaluations for different values of the embedding size fidelity. Dashed lines indicate the point at which the oracle has been evaluated for every algorithm. Solid curves and shaded areas stand for mean value across runs and standard error.
proxy and target tasks is not always perfect. This can result in a proxy task incumbent that does not translate to the oracle in the target task, which can be observed particularly at lower fidelity levels.
Figure 13: The target task performance distribution for DPL and other methods over different HPO budgets ranging from 1 - 6 full function evaluations.
C Continuous Search Space
Figure 14: The incumbent error of DPL, as well as, the baselines on the CIFAR10 task over the HPO budget.
The primary objective of this study is to investigate the efficacy of Deep Power Laws (DPL) in trading off exploration vs exploitation in a continuous HPO search space. In this study, we do not make use of pre-computed tabular tables, but instead we optimize the hyperparameters of an EfficientNetV2 model online, by iteratively pausing unpromising configurations and moving forward only promising hyperparameter configurations during the HPO optimization procedure.
To benchmark our findings, we contrast the results against established baseline algorithms such as random search, BOHB, and SMAC.
Baseline: We employ EfficientNetV2 [43] as a benchmarking model and train it on the CIFAR10 dataset [24]. Specifically, we train the lightweight variant of EfficientNetV2-b0 from scratch for 50 epochs, using the RMSprop optimizer. The learning rate is initiated at  $10^{-6}$  and gradually increased over a span of five warmup epochs to reach the learning rate value of  $5 \cdot 10^{-4}$ . Following the warmup phase, we employ a cosine learning rate scheduler, with a decay factor of 0.97 applied every 10 epochs. The weight decay is set at  $10^{-5}$ , with no momentum used. Furthermore, the dropout rate is configured to be  $10^{-6}$  and the model's moving average exponential decay is set at 0.9996. During the training phase, the batch size is set to 64, while for the validation phase, it is reduced to 8. All experiments are performed using the timm library [46].
Search Space: In our experiment, we concentrate on optimizing the two most critical hyperparameters, learning rate, and weight decay, while keeping the remaining hyperparameters fixed as per the baseline model. We construct a search space for these two hyperparameters in accordance with common practices (Table 4).
HP	Values
LR	[10-5, 10-2]
weight decay	[0, 10-1]
Table 4: Search space of CIFAR10 task.
To emulate a continuous search space for the acquisition function, we generate 100 equally-sized steps on a logarithmic scale from the lower bound to the upper bound of each dimension. This process yields a search space comprising  $10^{4}$  potential configurations.
Our method demonstrates a substantial speedup in terms of anytime performance when compared to baseline algorithms (Figure 14). Acknowledging the practical constraints of evaluating large-scale models, we pragmatically allocated an HPO budget for a maximum of 3 full-function evaluations, equivalent to 150 epochs. The results of our exploration underline the compelling potential of the DPL algorithm to effectively manage HPO tasks within a continuous search space. Exhibiting significant speedup gains, DPL proves itself to be not only viable but an efficient method for identifying optimal hyperparameters. The findings further underscore the adaptability and efficacy of DPL in addressing complex HPO tasks, reinforcing its standing as a valuable tool in the machine learning toolbox.
D PD1 Investigation
Figure 15: Configuration test performance histograms for datasets where DPL does not outperform baselines.
We investigate the datasets: fashion_mnist_max_pooling_cnn_tanh_batch_size_256, CIFar100_width_resnet_batch_size_256, svhn_noExtra_width_resnet_batch_size_1024, where DPL does not outperform other methods for the PD1 benchmark as shown in Figure 23. We analyze the test performance of the individual hyperparameter configurations that belong to the aforementioned datasets. Figure 15 shows that the search spaces of the datasets have a skewed distribution of performances, where, there exist a large number of hyperparameter configurations that achieve top performance. In such datasets, even a non-model based technique will quickly find a well-performing configuration, since, there is a high chance for a randomly-sampled configuration to achieve the top performance. For this reason, there is little room for sophisticated HPO techniques in these datasets.
To further validate our hypothesis, we investigate two additional datasets  $dionis$  from the LCBench benchmark and uniref50_transformer_batch_size_128 from the PD1 benchmark, where DPL achieves a strong performance compared to baseline HPO methods. The results in Figure 16 show that DPL excels on tasks that are complex and that require optimizations to find hyperparameter configurations that achieve top performance.
Based on the results, we conclude that the lack of statistical significance in PD1 is not a specific failure mode of DPL, but a consequence of multiple PD1 datasets where the majority of configurations achieve the top performance.
Figure 16: Histograms of the distribution of performances for datasets where DPL performs strongly.
Figure 17: The percentage taken by the DPL overhead in the total time per iteration for the different embedding sizes in nanoGPT-Bench.
E DPL Overhead
To investigate the efficiency of DPL in terms of method runtime, we investigate the percentage of time that the DPL overhead contributes in the total time taken to perform one HPO iteration in the nanoGPT-Bench. The total time taken constitutes of the DPL overhead (training the DPL surrogate, calling the acquisition function to suggest the next hyperparameter configuration) and the time taken to run the target algorithm for one more step. Figure 17 shows that the impact of the DPL overhead is negligible in the total time taken. For the smallest embedding of size 6, DPL takes only  $10\%$  of the total time taken to perform one HPO iteration after spending half of the optimization budget. At the end, after circa 40 hours of HPO optimization, DPL has an impact of  $20\%$  in the total time taken to perform one HPO iteration. The impact is even smaller for the largest embedding of size 384, where DPL has an impact of only  $5\%$  in the total time taken per iteration after spending half of the optimization budget and it has an impact of only  $10\%$  in the total time per iteration after more than 120 hours of HPO optimization.
The findings validate our claim that DPL has a minor time overhead in performing hyperparameter optimization, which explains the strong any-time performance of our method.
F Details of Considered Benchmarks
LCBench: We use the official implementation as the interface for the LCBench benchmark  ${}^{2}$  . As suggested by the authors,we use the benchmark information starting from the second step and we skip the last step of the curve since it is a repeat of the preceding step.
TaskSet: The TaskSet benchmark features 1000 diverse tasks. We decide to focus on only 12 NLP tasks from the TaskSet benchmark to add variety to our entire collection of datasets. Our limitation on the number of included tasks is related to the limited compute power, as we are unable to run for the entire suite of tasks offered in TaskSet. TaskSet features a set of 8 hyperparameters, that consists of i) optimizer-specific hyperparameters, such as the learning rate, the exponential decay rate,  $\beta_{1}$  and  $\beta_{2}$ , and Adam's constant for numerical stability  $\varepsilon$ , ii) hyperparameters that control the linear and exponential decay schedulers for the learning rate decay, and lastly iii) hyperparameters that control the L1 and L2 regularization terms. Every hyperparameter in TaskSet except  $\beta_{1}$  and  $\beta_{2}$  is sampled logarithmically.
PD1: We use the synetune library [42] for our interface to the PD1 benchmark. From the benchmark, we only include datasets that have a learning curve of length greater than 10. We furthermore only include datasets that have a learning curve lower or equal to 50 to have a fair comparison between all benchmarks by having approximately 20 full-function evaluations. PD1 features 4 numerical hyperparameters, lr_initial_value, lr_power, lr Decay_steps_factor and one_MINUS_momentum, where lr_initial_value and one_MINUS_momentum are log sampled. The learning rate decay is applied based on a polynomial schedule, its hyperparameters taken from the search space.
G Baselines
Random Search: We implemented random search by randomly sampling hyperparameter configurations from the benchmarks with the maximal budget.
Hyperband, BOHB, LCNet: We use version 0.7.4 of the HpBandSter library as a common codebase for all 3 baselines  $^{3}$ . For the last approach mentioned, despite heavy hyperparameter tuning of the method, we could not get stable results across all the benchmarks and hence dropped the method from our comparison.
ASHA: For the implementation of ASHA we use the public implementation from the optuna library, version 2.10.0.
DEHB: We use the public implementation offered by the authors<sup>4</sup>.
MF-DNN: In our experiments we used the official implementation from the authors  $^{5}$ . However, the method crashes which does not allow for full results on all benchmarks.
SMAC: For our experiment with SMAC we used the official code base from the authors  $^{6}$ .
Dragonfly: We use version 0.1.6 of the publicly available Dragonfly library.
For all the multi-fidelity methods considered in the experiments, we use the same minimal and maximal fidelities. In more detail, for the LCBench, TaskSet and PD1 benchmarks we use a minimal fidelity lower bound of 1 and a maximal fidelity lower bound equal to the max budget.
H Plots
In Hypothesis 1, we prove that DPL achieves a better performance in comparison to other models in estimating the final performance for different hyperparameter configurations based on partial observations. Our analysis shows that DPL manages to retain the ranks of different hyperparameter
$^{2}$ https://github.com/automl/LCBench
<sup>3</sup>https://github.com/automl/HpBandSter
<sup>4</sup>https://github.com/automl/DEHB/
<sup>5</sup>https://github.com/shibOli/DNN-MFBO
<https://github.com/automl/SMAC3>
Figure 18: The absolute relative error distribution of DPL over the different learning curve fractions in the LCBench benchmark. The distribution is calculated from the ground truth and prediction values, averaged over all configurations of a dataset. Every distribution is over the datasets.
configurations. We complement Hypothesis 1 by additionally investigating the absolute relative error. We measure the difference between the DPL estimation of the final learning curve value for different fractions of available partial observations vs the actual end performance. Figure 18 shows that DPL does not only retain the ranks of the final performances of different hyperparameter configurations, but it also correctly estimates the final performance by attaining a small relative error, where the error is reduced the more partial observations we have from the learning curve.
In Hypothesis 3, we investigate the efficiency of DPL in exploring more promising configurations compared to other HPO methods. In Figure 19 we provide the same comparison with regards to the PD1 benchmark. Based on the results, we conclude that DPL explores more promising configurations compared to other HPO methods.
Figure 19: Post-hoc analysis to study DPL's efficiency. Left: Share of the best candidates selected during training. Middle: Average regret of configurations chosen to be trained at each budget. Right: Share of top third configurations at a given budget which were bottom two third configurations at a previous budget.
Lastly, we provide the per-dataset performances of all methods, where we present the mean regret of the incumbent trajectory and the standard error over 10 runs in LCBench (Figure 20 and 21), TaskSet (Figure 22), and PD1 (Figure 23). The results show that DPL consistently outperforms other methods in the majority of cases achieving strong any-time performance and not only a strong final performance.
Figure 20: Performance comparison over the number of epochs on a dataset level for LCBench. We plot the mean value over 10 runs and the standard error.
Figure 21: Performance comparison over the number of epochs on a dataset level for LCBench (cont.). We plot the mean value over 10 runs and the standard error.
Figure 22: Performance comparison over the number of steps on a dataset level for TaskSet. We plot the mean value over 10 runs and the standard error.
Figure 23: Performance comparison over the fraction of the total optimization iterations on a dataset level for PD1. We plot the mean value over 10 runs and the standard error.