SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Furthermore, when reporting the learning curve (LC) length fraction, we imply the fraction of the total learning curve length. LCBench and TaskSet have LCs of a fixed length for all datasets, corresponding to 51 epochs for LCBench and 50 epochs for TaskSet. In contrast, PD1 has varying LC lengths for different datasets.
In our experiments, all methods start with a history  $H^{(init)}$  of 1 randomly sampled hyperparameter configuration evaluated for 1 step/epoch in the case of multi-fidelity techniques (Hyperband, BOHB, DEHB, SMAC, ASHA, Dragonfly; descriptions in Section 5.2), or for the full budget for the black-box technique (Random Search). We ran experiments on a CPU cluster, where every node contains two Intel Xeon E5-2630v4 CPUs with 20 CPU cores running at 2.2 GHz. The total memory of every node is 120GB, and every experiment is limited to 2 cores which offer 12GB.
'https://github.com/releaunifreiburg/DPL
6 Research Hypotheses and Experiments
Hypothesis 1: The power law assumption improves the quality of learning curve forecasting.
In this experiment, we evaluate the predictive performance of forecasting models that given a fraction of the observed learning curve, estimate the remaining unobserved segment of the curve, on the LCBench benchmark. The results of Figure 1 compare three different forecasting models, concretely, neural networks (NN), Gaussian Processes (GP), and Power Law functions (PL). For the three variants (PL, NN, GP) we fitted one model on every learning curve of each hyperparameter configuration (i.e. given  $b$  in the  $x$ -axis estimate one  $\hat{f}(b)$  separately for every  $\lambda$ ). For the other two variants (DPL and Cond NN) we fit a single forecasting model (not an ensemble) for all configurations, by conditioning the
Figure 1: Rank correlations of learning curve forecasting models, which are given a fraction of the learning curve and estimate the remaining curve segment. DPL: Deep Power Law, Cond NN: Conditioned neural network, PL: Power Law, NN: Neural Network, GP: Gaussian processes.
surrogate on the configuration (i.e. given  $b$  and  $\lambda$ , estimate  $\hat{f}(\lambda, b)$ ). The purpose of the experiment is to assess whether a power law function regressor leads to superior predictive accuracy, compared to generic forecasting models, such as neural networks, or Gaussian processes. The evaluation metric of the experiment highlighted in Figure 1 is the rank correlation between the estimated performances at the end of the learning curve and the true performances.
We notice that although a Power Law regressor has significantly fewer parameters than a neural network (3 to 288 parameters), PL still achieves higher predictive performance than NN. Furthermore, our conditioning of the power law function to the hyperparameter configuration further improves the predictive quality, as the difference between DLP and PL demonstrates.
Lastly, we refer the reader to Appendix H, where we provide an analysis of the distributions for the absolute error rate between the DPL predictions and the ground truth values over the different LC length fractions, showing that DPL does not only retain the ranks but, it also accurately predicts the final performance. Based on the results, we consider Hypothesis 1 to be validated and that DPL is accurate in terms of learning curve forecasting.
What about learning curves that do not follow a power law pattern? Although the provided empirical evidence in this section strongly suggests that the presented power law model can accurately forecast learning curves, it is also true that some learning curves have divergent behaviour that does not follow the power law assumption. As a consequence, we investigate two different ways to handle curves that do not follow the power law assumption:  $i)$  recently-proposed power law functions that include breaking points [5], or shifts in the curve [8], and  $ii)$  min-smoothing the learning curves to transform them into monotonically decreasing time series. In Appendix A we provide ample empirical evidence showing that although the alternative formulations achieve comparable performances, still they do not outperform our simpler power law formulation. The findings indicate that even though not all learning curves are power laws, most of them are, therefore a power law surrogate is efficient in forecasting the final performance of a partially-observed learning curve. As a result, the forthcoming experiments will provide further empirical evidence that our power law surrogates lead to state-of-the-art HPO results when deployed in the proposed multi-fidelity Bayesian optimization setup.
Hypothesis 2: Our method DPL achieves state-of-the-art results in HPO.
In Figure 2, we show the performance of the considered methods over the HPO budget, where DPL manages to outperform all the rival baselines. In the case of LCBench, DPL quickly finds well-performing hyperparameter configurations compared to the competitor methods and continues to discover even better configurations until the HPO process ends. Furthermore, we observe the same trend with TaskSet and PD1, where after ca.  $25\%$  of the HPO budget, our method DPL converges to a better regret compared to the baselines and increases the lead until HPO ends. For a detailed overview of the performances of all methods on all individual datasets, we point the reader to Appendix H.
Figure 2: DPL discovers better hyperparameter configurations than all rival baselines in terms of regret (distance to oracle). Solid curves and shaded regions represent the mean and standard error of the averaged normalized regret.
Figure 3: The critical difference diagrams at  $50\%$  and  $100\%$  of the HPO budget. The ranks indicate the sorted position in terms of regret, aggregated across datasets (the lower the better). Thick horizontal lines highlight differences that are not statistically significant.
In addition, Figure 3 provides the critical difference diagrams of the per-dataset regret ranks at  $50\%$  and  $100\%$  of the HPO budget. Our method DPL outperforms all baselines in 5 out of 6 cases (in 4 of which with a statistically significant margin), while being second best only at the  $50\%$  of the HPO budget on the PD1 benchmark. We investigate the lack of statistical significance in PD1, by analyzing the individual dataset performances where DPL performs worse compared to other baselines. We notice that the datasets have a skewed distribution of hyperparameter configuration performances, where, a majority of the configurations achieve top performance. Based on the results, we conclude that a lack of statistical significance is the outcome of a search space that includes relatively simple optimization tasks and not a specific failure state of our method. We provide the detailed results of our analysis in Appendix D.
Lastly, we analyse the performance of DPL over time in Figure 4. As it can be observed, DPL manages to outperform the competitors even when the method's overhead time is included, showing that the overhead of DPL (i.e. fitting
Figure 4: The average normalized regret of DPL and rival methods over the normalized time for all the considered benchmarks. Solid curves and shaded regions represent the mean and standard average normalized regret.
surrogate and running the acquisition) is negligible in terms of the quality of the HPO results. For a more detailed information, regarding the DPL overhead time, we point to Appendix E. TaskSet is not included in Figure 4 since the benchmark does not offer runtimes. Given the results, we conclude that Hypothesis 2 is validated and that DPL achieves state-of-the-art performance in HPO.
Figure 5: Post-hoc analysis to study DPL's efficiency. Left: Share of the best candidates selected during training. Middle: Average regret of configurations chosen to be trained at each budget. Right: Share of top third configurations at a given budget which were bottom two third configurations at a previous budget.
Hypothesis 3: DPL explores the search space more efficiently compared to the baselines.
We conduct further analyses to understand the source of the efficiency of DPL versus the baselines. As a result, we analyze two important aspects, the quality of the evaluated configurations, as well as the exploration capability of our multi-fidelity HPO method. Initially, we measure what fraction of the top  $1\%$  configurations (ranked by accuracy) can our method discover. Figure 5 (left) shows that until convergence our method can discover significantly more top configurations compared to the baselines. The middle plots of Figure 5, show the average regret for each configuration promoted to the respective budget. According to the plot, DPL is more efficient and assigns the budget only to configurations with lower regret compared to the other methods. The precision and regret plots demonstrate that the quality of the evaluated configurations is largely better than all baselines, therefore, giving our method a significant lift in the performance rank. Last but not least, the right plot shows the percentage of configurations that were performing poorly in an earlier epoch (i.e. accuracy-wise in the bottom  $2/3$  of configurations up to the epoch indicated at the x-axis) but performed better at later epochs (i.e. at the top  $1/3$  of configurations). Furthermore, we added a line labeled with "Baseline", which represents the fraction of previously poor-performing configurations of all configurations. This behavior is observed often with learning curves, where for instance, strongly regularized networks converge slowly. For the same analysis regarding the PD1 benchmark, we point the reader to Appendix H.
The results indicate that our method explores well the unpromising early configurations, by considering them through the uncertainty estimation of our ensemble and the respective Bayesian optimization mechanism. The results validate Hypothesis 3 and confirm that DPL explores the search space more efficiently.
Hypothesis 4: Our method DPL offers an effective tool for HPO in Large Language Models.
In this experiment, we consider the case of tuning the hyperparameters of transformers in Large Language Models. To this end, we computed a tabular benchmark by training a smaller GPT-2 [39] model on the OpenWebText dataset [13] for a series of different hyperparameter configurations. We tune three learning rate hyperparameters: the fraction of warmup steps, the maximum learning rate at the end of warmup, and the minimum learning rate at the end of the decay. We repeat the experiments for seven model sizes ranging from 0.3M to 30M total parameters, ablating the embedding size of the multi-head attention layers (details in Appendix B).
We follow the common practice of conducting HPO with small transformers and then deploying the discovered optimal configuration on the full-scale transformers [50]. As a result, we search for the optimal hyperparameters of small transformers (embedding size of  $\{6,12,\dots ,96,192\}$ ) and then evaluate the discovered configurations at a full-scale transformer with an embedding size of 384.
Figure 6: HPO for Transformer architectures. Top: HPO on small-scale transformers in terms of the embedding size. Bottom: Error on the full-scale transformer, using the hyperparameter configuration discovered by conducting HPO using the small transformers. We present three analyses, ablating the HPO time on the small-scale transformer up to the HPO budget of 2 full function evaluations.
Figure 6 shows the HPO results of DPL against Random Search and BOHB (a rival multi-fidelity HPO baseline). In the top row of plots, we observe the performance of the discovered configurations at the small transformers for the indicated embedding size. We observe that our method finds better configurations than the baselines at any proxy space with small embedding sizes.
On the other hand, the bottom row of plots presents the performance of the discovered configurations in the small embedding space, by applying such hyperparameter configurations to the full-scale transformers. We observe that the configurations discovered by DPL on the small search space achieve very competitive results on the full-scale transformers, finding the oracle configuration of the full-scale transformers in the majority of cases. It takes DPL 3.6 hours to find the oracle configuration for the largest model via HPO for the smallest model. In turn, it takes 21.52 hours to train the largest model only once. For more details, we refer the reader to Appendix B. The results validate Hypothesis 4 and confirm that DPL is an efficient HPO technique for tuning the hyperparameters of large language models when the HPO is conducted using smaller transformer model sizes.
Summary. In this work, we introduce Deep Power Law (DPL), a probabilistic surrogate based on an ensemble of power law functions. The proposed surrogate is used within a novel multi-fidelity Hyperparameter Optimization (HPO) method based on Bayesian optimization. In contrast to the prior work, we exploit scaling laws for estimating the performance of Deep Learning (DL) models. Through extensive experiments comprising 7 baselines, 59 datasets, and search spaces of diverse deep learning architectures, we show that DPL outperforms strong HPO baselines for DL by a large margin. As an overarching contribution, we advance the state-of-the-art in the important field of HPO for DL.
Limitations and future work. Contrary to the common perception, we experienced that the uncertainty estimation arising from the Deep Ensemble approach [25] is suboptimal compared to standard BO surrogates such as Gaussian Processes. In addition, having to train an ensemble has additional computational costs, due to the necessity of training multiple power law models. In the future, we plan to investigate the combination of power laws with Gaussian Processes, as well as investigate additional fidelity types.
JG, AK and MJ would like to acknowledge the grant awarded by the Eva-Mayr-Stihl Stiftung. In addition, this research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, JG and AK acknowledge the support of the BrainLinks-BrainTools center of excellence.
[43] Mingxing Tan and Quoc Le. EfficientNetV2: Smaller models and faster training. github.com/ google/automl/tree/master/efficientnetv2, 2021.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[45] Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-training helps bayesian optimization too. arXiv preprint arXiv:2207.03084, 2022.
[46] Ross Wightman. PyTorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
[47] Martin Wistuba, Arlind Kadra, and Josif Grabocka. Supervising the multi-fidelity race of hyperparameter configurations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[48] Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 10303-10312, 2020.
[49] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[50] Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466, 2022.
[51] Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079 - 3090, 2021.
A Modeling
Figure 7: Left: The performance of different power law formulations, as well as, the baselines on the LCBench benchmark over the HPO budget. Right: The performance of the power law formulation when min-smoothing is applied to the learning curve to ensure a monotonically decreasing learning curve.
DPL	Dragonfly	Candidate 1
ASHA	Hyperband	Candidate 2
BOHB	Random	Broken Law
DEHB	SMAC
DPL	DEHB	Random
ASHA	Dragonfly	SMAC
BOHB	Hyperband	SPL
To inspect the modeling choices of the power law functions used as our surrogate, we consider different formulations for the ensemble members of our surrogate as shown in Table 1. Initially, we consider Candidate 1 which can handle shifts in the learning curve by introducing an additional parameter  $d$ . Furthermore, we consider a more complex version, Candidate 2, that allows us to additionally scale the budget, by introducing variable  $e$ . Lastly, we consider
Broken Laws [5] (BPL), which can handle breaking points in the learning curve. We use a version that can handle one breaking point since the authors of the method suggest it as a sufficient formulation to approximate the majority of cases. We run the DPL surrogate with every individual formulation on the LCBench benchmark to investigate their empirical performance.
Label	Formula
DPL	α + β · b-γ
Candidate 1	α - β · (b + d)-γ
Candidate 2	α - β · (e · b + d)-γ
Broken Law	α + β · b-γ · (1 + (b d1/7)) -c·f
Table 1: Alternative power law formulations for the DPL surrogate.
Figure 7 presents the results, where, our chosen surrogate formulation (DPL) despite being the simplest, outperforms all other formulations. The Candidate 2 formulation does not manage to outperform all competitor methods, the Candidate 1 formulation manages to outperform all rival methods, however, only with a marginal difference. The Broken Law formulation manages to outperform all the rival baselines by a considerable margin, however, it still performs worse than the simple power law formulation used for DPL.
We would like to point out that the alternative power law formulations are more difficult to optimize/run since they are prone to diverge and fall into a dead state given different combinations of parameter values. The most common is division by zero for e.g.  $d$  term in the Broken Law formulation, taking the root of a negative number  $b + d$  term in Candidate 1,  $e \cdot b + d$  for Candidate 2 and the  $d$  term in Broken Law.
We additionally consider min-smoothing the learning curve  $y$ , by taking at each step  $b$  of the learning curve the minimal value of the learning curve  $y_{b}^{smooth}$  where,  $y_{b}^{smooth} = \min(y_{0}, y_{1}, \dots, y_{b})$ . The min-smoothing allows diverging curves to be formulated as power laws since the diverging phase will be substituted with stagnation. Figure 7 shows that incorporating learning curve min-smoothing for our surrogate (SPL) performs comparably to DPL without learning curve smoothing and manages to beat the other HPO baselines.
B nanoGPT-Bench
In recent years, deep learning research has increasingly focused on large-scale models, particularly Large Language Models (LLMs) like the Generative Pre-trained Transformers (GPTs). To evaluate the effectiveness of search algorithms, we propose a benchmark based on the nanoGPT model [21], reproducing the performance of the small GPT-2 [39] model trained on the OpenText dataset [13].
Our experimental setup is designed with the practicalities of real-world hardware constraints in mind. The common practice in the field is to perform Hyperparameter Optimization (HPO) on an informative proxy task that adheres to model size scaling laws [37], and then to apply these optimized parameters to larger models. This pragmatic approach is necessitated by the fact that training larger models would require significantly more expensive computational resources and time. Within these constraints, we focus our experiments on NVIDIA RTX 2080 GPUs.
Figure 8: nanoGPT-Bench search space parametrization.
Baseline: We train a small nanoGPT model, a scaled-down variant of the small GPT-2, reducing the parameter count to approximately  $30\mathrm{M}$  from the original  $124\mathrm{M}$  parameters. The model architecture includes 6 transformer layers and 6 attention heads, and the embedding size is set to 384. The AdamW optimizer is utilized for training, with the first and second moment estimates configured to 0.9 and 0.98, respectively. The weight decay is set to  $10^{-1}$ , and we apply gradient clipping at a value of 1.0 to prevent large gradients from causing instability in the model training. For the training process, we optimize the cross-entropy loss for next-token prediction. The process involves 350 steps, with each step encompassing 1000 random samples, with a batch size of 12. Each data point has a context size of 512 tokens, encoded using OpenAI's token embeddings (sized of 50304). This procedure ensures that even the most resource-intensive experiments stay within the limits of a single GPU day.
Search Space: Our hyperparameter search space construction involves varying the number of warmup steps, along with the maximum and minimum learning rates for the cosine scheduler. The specific parametrization of the scheduler is illustrated in Figure 8. The discretized choices are presented in detail in Table 2.
HP	Values
Max LR	[10-5; 10-4; 10-3]
Min LR	[1%; 10%] of Max LR
Warmup Steps	[10%; 20%] of Budget
Table 2: Search space of nanoGPT-Bench.
Fidelity Space: To construct the fidelity space, we focus on two key dimensions: the number of training steps and the transformer's embedding size, serving as a proxy for
model size. In this exploration, the natural fidelity of the number of training steps is visualized by the validation curves during model training as depicted in Figure 9. On the other hand, the end performance correlation between the different fidelities is reflected by the Pearson correlation in Table 3. We establish proxy tasks by sampling embedding size from a log scale,  $\{6,12,\dots ,96,192\}$  with the maximum being 384. Consequently, each configuration has 6 proxy and one target task, leading to a total of 84 unique configurations in our full benchmark. Every configuration is trained for 350 steps.
Figure 9: Validation loss curves during model training for all nanoGPT-Bench configurations across model fidelity values.
Small Transformer Scale [embedding size]
Figure 10: Scaling of model size in relation to bytes, FLOPS, and runtime based on average values across all nanoGPT-Bench configurations.
Due to the GPU under-utilization of small model sizes, the runtime scales linearly as the model size scales exponentially. This relationship can be observed in Figure 10 and Figure 11. We expect the runtime to scale in a linear proportion to the model size when larger models are considered.
Figure 12 illustrates the effectiveness of DPL, particularly when the number of training steps is considered as a fidelity dimension. Vertical dotted lines denote the iteration at which an algorithm identifies the oracle value with an absolute tolerance of 0.01.
Figure 13 depicts the results over the different values of the embedding fidelity. We utilized DPL, BOHB, and random search in proxy tasks, incrementing the budget allocation over each run, up to a horizon of 6 full-function evalua
Embedding Size	Correlation
6	0.951
12	0.880
24	0.971
48	0.955
96	0.987
192	0.994
384	1.000
Table 3: Pearson correlation across 7 fidelities.
tions. From these proxy tasks, we extracted the incumbent hyperparameters and evaluated their performance on the target task, that corresponds to the maximum embedding size of 384.
Despite operating within short regimes, DPL consistently outperforms baselines in terms of the mean incumbent value and the explored regime, as evidenced by error bars indicating the range between best and worst incumbents across 10 seeds. It should be noted, however, that the correlation between