SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arlind Kadra
Representation Learning Lab
kadraa@cs.uni-freiburg.de
Martin Wistuba
Amazon Berlin
marwistu@amazon.com
Josif Grabocka
grabocka@cs.uni-freiburg.de
Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the issue of hyperparameter optimization, however, most of the methods do not exploit the dominant power law nature of learning curves for Bayesian optimization. In this work, we propose Deep Power Laws (DPL), an ensemble of neural network models conditioned to yield predictions that follow a power-law scaling pattern. Our method dynamically decides which configurations to pause and train incrementally by making use of gray-box evaluations. We compare our method against 7 state-of-the-art competitors on 3 benchmarks related to tabular, image, and NLP datasets covering 59 diverse tasks. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors.
Hyperparameter Optimization (HPO) is a major challenge for the Machine Learning community. Unfortunately, HPO is not yet feasible for Deep Learning (DL) methods due to the high cost of evaluating multiple configurations. Recently multi-fidelity HPO (a sub-problem of gray-box HPO) has emerged as a promising paradigm for HPO in DL, by discarding poorly-performing hyperparameter configurations after observing the validation error on the low-level fidelities of the optimization procedure [28, 9, 1, 29]. The advantage of multi-fidelity HPO compared to online HPO [7, 38], or meta-gradient HPO [32, 10, 31] is the ability to tune all types of hyperparameters.
In recent years, a stream of papers highlights the fact that the performance of DL methods is predictable [14], concretely, that the validation error rate is a power law function of the model size, or dataset size [41, 40]. Such a power law relationship has been subsequently validated in the domain of NLP, too [11]. In this paper, we demonstrate that the power-law principle has the potential to be a game-changer in HPO, because we can evaluate hyperparameter configurations in low-budget regimes (e.g. after a few epochs), then estimate the performance on the full budget using dataset-specific power law models. Concretely, we hypothesize and empirically demonstrate that optimization curves (training epochs versus accuracy, or loss) can be efficiently modeled as simple power law functions.
As a result, we introduce Deep Power Law (DPL) ensembles, a probabilistic surrogate for Bayesian optimization (BO) that estimates the performance of a hyperparameter configuration at future budgets using ensembles of deep power law functions. Subsequently, our novel formulation of BO dynamically decides which configurations to pause and train incrementally by relying on the performance
estimations of the surrogate. We demonstrate that our method achieves the new state-of-the-art in HPO for DL by comparing against 7 strong HPO baselines, and 59 datasets of three diverse modalities (tabular, image, and natural language processing). Our experimental protocol contains state-of-the-art deep learning architectures such as Transformers [44], XFormer [26], ResNeXt [49] and large language models like GPT-2 [39]. As a result, we believe the proposed method has the potential to finally make HPO for DL a feasible reality.
- We introduce a novel probabilistic surrogate for multi-fidelity HPO based on ensembles of deep power law functions.
- We derive a simple mechanism to combine our surrogate with Bayesian optimization.
- Finally, we demonstrate the empirical superiority of our method against the current state-of-the-art in HPO for Deep Learning, with a large-scale HPO experimental protocol.
Multi-fidelity HPO assumes a method has access to the learning curve of a hyperparameter configuration. Such a learning curve is the function that maps either training time or dataset size, to the validation performance. The early performance of configurations (i.e. first segment of the learning curve) is used to discard unpromising configurations, before waiting for full convergence. Successive halving [17] is a widely used multi-fidelity method that randomly samples hyperparameter configurations, starts evaluating them, and ends a fraction of them upon reaching a predefined budget. Afterwards, the budget is divided by the fraction of discarded hyperparameter configurations and the process continues until the maximum budget is reached. Although the method relies only on the last observed value of the learning curve, it is very efficient. In recent years, various flavors of successive halving have been suggested, including Hyperband [28], which effectively runs successive halving in parallel with different settings. A major improvement to Hyperband is replacing random search with a more efficient sampling strategy [1, 9]. A more efficient approach is to allocate the budget dynamically to the most promising configurations [47]. However, the only assumption these methods make about the learning curve is that it will improve over time, while recent work [4] exploits a power law assumption on the curves. Similarly, we fit surrogates that exploit a power law assumption, however, our method is able to estimate the performance of unobserved configurations through probabilistic surrogates learned jointly for all hyperparameter configurations.
Learning curve prediction is a related topic, where the performance of a configuration is predicted based on a partially observed learning curve [36]. Typically, the assumptions about the learning curve are much stronger than those described above. The prediction is often based on the assumption that the performance increases at the beginning and then flattened towards the end. One way to model this behavior is to define a weighted set of parametric functions [8, 23]. Then, the parameters of all functions are determined so that the resulting prediction best matches the observed learning curve. Another approach is to use learning curves from already evaluated configurations and to find an affine transformation that leads to a well-matched learning curve [6]. A more data-driven approach is to learn the typical learning curve behaviour directly from learning curves across different datasets [48]. Learning curve prediction algorithms can be combined with successive halving [2]. In contrast to this research line, we fit ensembles of power law surrogates for conducting multi-fidelity HPO with Bayesian optimization.
Scaling laws describe the relationship between the performance of deep learning models as a function of dataset size or model size as a power law [14, 16, 41, 40, 11, 20, 15, 33]. Another work tunes hyperparameters on a small-scale model and then transfers them to a large-scale version [50]. In contrast to these papers, we directly use the power law assumption for training surrogates in Bayesian optimization for HPO.
Hyperparameter Optimization (HPO) demands finding the configurations  $\lambda \in \Lambda$  of a Machine Learning method that achieve the lowest validation loss  $\mathcal{L}^{(\mathrm{Val})}$  of a model (e.g. a neural network), which is parameterized with  $\theta \in \Theta$  and learned to minimize the training loss  $\mathcal{L}^{(\mathrm{Train})}$  as:
$$
\lambda^ {*} := \underset {\lambda \in \Lambda} {\arg \min } \mathcal {L} ^ {(V a l)} (\lambda , \theta^ {*} (\lambda)), \quad \text {s . t .} \quad \theta^ {*} (\lambda) := \underset {\theta \in \Theta} {\arg \min } \mathcal {L} ^ {(T r a i n)} (\lambda , \theta) \tag {1}
$$
For simplicity, we denote the validation loss as our function of interest  $f(\lambda) \coloneqq \mathcal{L}^{(Val)}(\lambda, \theta^*(\lambda))$ . The optimal hyperparameter configurations  $\lambda^*$  of Equation 1 are found via an HPO policy  $\mathcal{A} \in \mathcal{P}$  (also called an HPO method) that given a history of  $N$  evaluated configurations  $H^{(N)} \coloneqq \{\lambda_i, f(\lambda_i)\}_{i=1}^N$  suggests the  $(N+1)$ -th configuration to evaluate as  $\lambda_{N+1} \coloneqq \mathcal{A}(H^{(N)})$  where  $\mathcal{A} : [\Lambda \times \mathbb{R}_+]^N \to \Lambda$ . The search for an optimal HPO policy is a bi-objective problem in itself, aiming at (i) finding a configuration out of  $N$  evaluations that achieves the smallest validation loss  $f(\lambda)$ , and (ii) ensuring that the costs of evaluating the  $N$  configurations do not exceed a total budget  $\Omega$ , as shown in Equation 2.
$$
\underset {\mathcal {A} \in \mathcal {P}} {\arg \min } \underset {i \in \{1, \dots , N \}} {\min } f (\lambda_ {i} := \mathcal {A} (H ^ {(i - 1)})), \tag {2}
$$
$$
\text {w h e r e :} \qquad H ^ {(i)} := \left\{ \begin{array}{l l} \{(\lambda_ {j}, f (\lambda_ {j})) \} _ {j = 1} ^ {i} & i > 0 \\ \emptyset & i = 0 \end{array} , \right.
$$
$$
\text {s u b j e c t} \Omega \geq \sum_ {i = 1} ^ {N} \operatorname {c o s t} (f (\lambda_ {i}))
$$
Bayesian optimization (BO) is the most popular type of policy for HPO, due to its ability to balance the exploration and exploitation aspects of minimizing the validation loss  $f$  in terms of hyperparameters  $\lambda \in \Lambda$ . Technically speaking, BO fits a surrogate  $\hat{f}(\lambda; \phi)$  parametrized with  $\phi \in \Phi$  to approximate the observed loss  $f(\lambda)$  using the history  $H^{(N)}$ , as  $\phi^* := \arg \max_{\phi \in \Phi} \mathbb{E}_{(\lambda, f(\lambda)) \sim p_{H^{(N)}}} p(f(\lambda)|\lambda, \phi)$ , where,  $p$  represents the probability. Afterwards, BO uses an acquisition/utility function  $a: \Lambda \to \mathbb{R}_+$  to recommend the next configuration as  $\lambda_{N+1} := \mathcal{A}\left(H^{(N)}\right) = \arg \max_{\lambda \in \Lambda} a\left(\hat{f}(\lambda; \phi^*)\right)$ .
Multi-fidelity HPO refers to the case where an approximation of the validation loss can be measured at a lower budget  $b \in B$ , where  $B := (0, b_{\max}]$ . For instance, in Deep Learning we can measure the validation loss after a few epochs ( $0 < b < \epsilon$ ), rather than wait for a full convergence ( $b = b_{\max}$ ). Throughout this paper, the term budget refers to a learning curve step. The evaluation of a configuration  $\lambda$  for a budget  $b$  is defined as  $f(\lambda, b): \Lambda \times B \to \mathbb{R}_+$ .
4 Power Law Surrogates for Bayesian Optimization
Prior work has demonstrated that the performance of Machine Learning methods as a function of budgets (i.e. dataset size, number of optimization epochs, model size, image resolution) follows a power law relationship [41, 40, 11, 20, 15]. In this work, we employ this power law dependence between the validation loss and the number of optimization epochs in Deep Learning. We propose a novel multi-fidelity Hyperparameter Optimization method which is based on power law surrogates. We assume that every learning curve  $f(\lambda, \cdot)$  can be described by a power law function defined by  $(\alpha, \beta, \gamma)$ . Concretely, we define a power law function for the validation loss of a configuration  $\lambda$  at a budget  $b$  (a.k.a. the number of epochs) as shown in Equation 3.
$$
\hat {f} (\lambda , b) := \alpha_ {\lambda} + \beta_ {\lambda} b ^ {- \gamma_ {\lambda}}, \alpha_ {\lambda}, \beta_ {\lambda}, \gamma_ {\lambda} \in \mathbb {R} \tag {3}
$$
Instead of fitting one separate power law function to each learning curve, we fit a single shared power law function across all configurations by conditioning the power law coefficients  $\alpha, \beta, \gamma$  on  $\lambda$  using a parametric neural network  $g$  that maps a configuration to the power law coefficients of its learning curve as  $g: \Lambda \to \mathbb{R}^3$ . The network  $g$  has three output nodes, corresponding to the power law coefficients, denoted as  $g(\lambda)_{\alpha}, g(\lambda)_{\beta}, g(\lambda)_{\gamma}$ , as defined in Equation 4.
$$
\hat {f} (\lambda , b) := g (\lambda) _ {\alpha} + g (\lambda) _ {\beta} b ^ {- g (\lambda) _ {\gamma}}, \quad g: \Lambda \rightarrow \mathbb {R} ^ {3} \tag {4}
$$
Using a history of learning curve evaluations  $H^{(N)} \coloneqq \{(\lambda_i, b_i, f(\lambda_i, b_i))\}_{i=1}^N$  we can train the power law surrogate to minimize the following loss function using stochastic gradient descent:
$$
\underset {\hat {f}} {\arg \min } \mathbb {E} _ {(\lambda , b, f (\lambda , b)) \sim p _ {H ^ {(N)}}} \left| f (\lambda_ {i}, b _ {i}) - \hat {f} (\lambda_ {i}, b _ {i}) \right| \tag {5}
$$
BO surrogates need to be probabilistic regression models because the acquisition functions require the posterior variance of the predictions. As a result, we train an ensemble of  $K$  diverse surrogates  $\hat{f}^{(1)}(\lambda ,b),\ldots ,\hat{f}^{(K)}(\lambda ,b)$  by initializing each surrogate with different weights and by training with a different sequence of mini-batches as in prior work [25, 22]. The posterior mean  $\mu$  and the posterior variance  $\sigma^2$  of the power law ensemble are trivially computed as:
$$
\mu_ {\hat {f}} (\lambda , b) := \frac {1}{K} \sum_ {k = 1} ^ {K} \hat {f} ^ {(k)} (\lambda , b), \quad \sigma_ {\hat {f}} ^ {2} (\lambda , b) := \frac {1}{K} \sum_ {k = 1} ^ {K} \left(\hat {f} ^ {(k)} (\lambda , b) - \mu_ {\hat {f}} (\lambda , b)\right) ^ {2} \tag {6}
$$
A commonly used acquisition function in the domain is Expected Improvement (EI) [35] which incorporates both the mean and uncertainty of predictions, applying a trade-off between exploration and exploitation. Consequently, in our work, we use the EI acquisition with the estimated full budget's  $(b_{\mathrm{max}})$  posterior mean and variance. We briefly define the acquisition function in Equation 7:
$$
\lambda_ {N + 1} := \underset {\lambda \in \Lambda} {\arg \max } \operatorname {E I} \left(\lambda , b _ {\max } \mid H ^ {(N)}\right), \tag {7}
$$
$$
\operatorname {E I} (\lambda , b _ {\max } | H) := \mathbb {E} _ {\hat {f} (\lambda , b _ {\max }) \sim \mathcal {N} \left(\mu_ {\hat {f}} (\lambda , b _ {\max }), \sigma_ {\hat {f}} ^ {2} (\lambda , b _ {\max })\right)} \left[ \max \left\{\hat {f} (\lambda , b _ {\max }) - f \left(\lambda_ {\mathrm {b e s t}}, b _ {\max }\right), 0 \right\} \right]
$$
where,  $f(\lambda_{\mathrm{best}}, b_{\mathrm{max}})$  corresponds to the best observed loss for any budget  $b' \leq b_{\mathrm{max}}$  from the history  $H^{(N)}$ . After selecting a configuration with our variant of the EI acquisition, we do not naively run it until convergence. Instead, we propose a novel multi-fidelity strategy that advances the selected  $\lambda_{N+1}$  of Equation 7 by a small budget of  $b_{\mathrm{step}}$ , e.g., 1 epoch of training. Therefore, the selected  $\lambda_{N+1}$  will be evaluated at  $b_{N+1}$  as defined in Equation 8. Notice, our proposed strategy also covers new configurations with no learning curve evaluations in  $H^{(N)}$ .
$$
b _ {N + 1} := \left\{ \begin{array}{l l} b _ {\text {s t e p}}, & \notin \lambda_ {N + 1}: (\lambda_ {N + 1}, \cdot , \cdot) \in H ^ {(N)} \\ b _ {\text {s t e p}} + \max  _ {(\lambda_ {N + 1}, b, \cdot) \in H ^ {(N)}} b, & \text {o t h e r w i s e} \end{array} \right. \tag {8}
$$
We provide the detailed pseudocode of our method at Algorithm 1.
5 Experimental Protocol
5.1 Benchmarks
LCBench: A benchmark that features 2,000 hyperparameter configurations that parametrize the architecture of simple feedforward neural networks, as well as, the training pipeline [51]. The benchmark features 7 numerical hyperparameters and 35 different datasets from the AutoML benchmark [12].
PD1: A deep learning benchmark [45] that consists of recent DL (including Transformers) architectures run on large vision datasets such as CIFAR-10, CIFAR-100, ImageNet, as well as statistical modeling corpora and protein sequence datasets from bioinformatics. Every search space includes varying learning curve lengths, ranging from 5 to 1414, and a different number of evaluated hyperparameter configurations ranging from 807 to 2807. The search space includes hyperparameter configurations that parametrize the learning rate, the learning rate scheduler, and the momentum.
TaskSet: A benchmark that features different optimization tasks evaluated in 5 different search spaces [34]. For our work, we focus on the Adam8p search space, which is among the largest search spaces
Algorithm 1: Multi-Fidelity HPO with Deep Power Laws
Input :Search space  $\Lambda$  , initial design  $H^{(\mathrm{init})}$  , budget increment  $b_{\mathrm{step}}$    
Output:Best hyperparameter configuration  $\lambda^*$    
1 Iteration  $i\gets 0$  ; Evaluate initial configurations and budgets  $H^{(i)}\coloneqq H^{(\mathrm{init})}$  .   
2 while still budget do   
3 Fit a DPL ensemble  $\hat{f}^{(1)}(\lambda ,b),\dots ,\hat{f}^{(K)}(\lambda ,b)$  from history  $H^{(i)}$  using Equation 5;   
4 Recommend the next configuration  $\lambda_{i + 1}$  and its budget  $b_{i + 1}$  using Equation 6, 7 and 8;   
5 Train  $\lambda_{i + 1}$  until  $b_{i + 1}$  and measure the validation loss  $f(\lambda_{i + 1},b_{i + 1})$  .   
6 Append to history  $H^{i + 1}\leftarrow H^i\cup \{(\lambda_{i + 1},b_{i + 1},f(\lambda_{i + 1},b_{i + 1}))\}$  .   
7  $i\gets i + 1$  ..   
8 end   
9 return Best configuration  $\lambda^*$  with the smallest validation loss min  $f(\lambda^{*},b)$  ..
in the benchmark with 1000 hyperparameter configurations. Every hyperparameter configuration features 8 continuous hyperparameters. The hyperparameters control the learning rate, the learning rate schedulers, and the optimizer. For variety among our benchmarks, we focus on 12 RNN text classification tasks that feature different RNN cells, embedding sizes, and batch sizes.
For a more detailed explanation of the benchmarks, we refer the reader to Appendix F.
Random Search stochastically samples hyperparameter configurations for the largest possible budget. Hyperband uses multiple brackets with different trade-offs of the initial budget and number of epochs to initially train [28]. It then applies Successive Halving (SH) [17] on every bracket. ASHA is an asynchronous version of SH [27] that does not wait for all configurations to finish in an SH bracket before starting the next one. Furthermore, BOHB is a follow-up of Hyperband that uses TPE [3] to sample the initial hyperparameter configurations of a bracket [9]. DEHB, on the other hand, modifies Hyperband by using evolutionary strategies to sample the initial hyperparameter configurations [1]. Similarly, multi-fidelity SMAC extends Hyperband but uses random forests to sample the initial hyperparameter configurations for a bracket [30]. We also use the Dragonfly Library [19] to compare against BOCA [18], a multi-fidelity method that uses Gaussian Processes to predict the next hyperparameter to evaluate, as well as the fidelity for which it should be evaluated. For all the baselines, we use their official public implementations. We provide additional details in Appendix G.
5.3 Architecture & Training
For our method, we use an ensemble of 5 models, where every model consists of a 2-layer feedforward neural network with 128 units per layer and Leaky ReLU for the non-linearity. The architecture of our method is motivated by prior work [25]. Our network has 3 output units, that are then combined with the budget  $b$  to yield the power law output. We apply the GLU non-linearity activation only on the  $\beta$  and  $\gamma$  output units, allowing  $\alpha$  to take any value.
We use the L1 loss to train our network, coupled with Adam featuring an initial learning rate of  $10^{-3}$ . For the first 10 iterations of our multi-fidelity HPO method in Algorithm 1 we train every network of our ensemble for 250 epochs with randomly sampled initial weights. This choice helps the convergence of the weights in the early stage of HPO. Next, we continuously refine the model for 20 epochs every HPO iteration. However, if the optimization stagnates (surrogate fitting loss does not improve) for more than the LC Length + a buffer of  $0.2 \times \mathrm{LC}$  Length, the training procedure is restarted with random weights, where every model is trained again for 250 epochs and then only refined. During the refining phase, the new data point at an HPO iteration (Line 9 at Algorithm 1) is sampled with repeat on every batch, to learn new data points equally compared to older data points. Since we are working with discrete search spaces, we evaluate the acquisition function exhaustively
on every hyperparameter configuration. When dealing with continuous search spaces, the acquisition function can either be evaluated exhaustively on a discretized version of the search space, or in a gradient-based way. Our implementation of DPL is publicly available. $^1$
5.4 Protocol
In our experiments, we standardize the hyperparameter values by performing min-max scaling for our method and every baseline. If a baseline has a specific preprocessing protocol, we do not apply min-max scaling but we apply the protocol as suggested by the authors.
The benchmarks do not support a common evaluation metric for configurations (i.e. the function  $f$ ). As a consequence, the evaluation metric for LCBench is the balanced accuracy, for TaskSet the log-likelihood loss, while for PD1 the accuracy. Moreover, the benchmarks do not offer learning curves with a common step size. For LCBench and PD1, one step size is equivalent to one epoch, while for TaskSet one step size is 200 batches. The HPO budget is defined as the maximum number of steps needed to fully evaluate 20 hyperparameter configurations. In that context, one unit step of the HPO budget signifies training a particular configuration for one more optimization step (e.g. 200 batches in TaskSet or 1 epoch in LCBench).
In the following experiments, for all methods, we report the regret of the best-found configuration as shown in Equation 9:
$$
R = f \left(\lambda_ {\text {b e s t}}, b _ {\max }\right) - f \left(\lambda_ {\text {o r a c l e}}, b _ {\max }\right) \tag {9}
$$
where the oracle is given as  $f(\lambda_{\mathrm{oracle}}, b_{\mathrm{max}}) \coloneqq \min \left\{f(\lambda, b) \mid (\lambda, b, f(\lambda, b)) \in H^{(D)} \wedge b \leq b^{\max} \right\}$ , and  $H^{(D)}$  corresponds to the set of all the exhaustively-evaluated hyperparameter configurations' performances on a dataset  $D$ . If the oracle configuration is not known in advance for the search space,  $H^{(D)}$  can be replaced with the history  $H^{(N)}$  at the end of the HPO procedure. The only difference between  $f(\lambda_{\mathrm{best}}, b_{\mathrm{max}})$  and  $f(\lambda_{\mathrm{oracle}}, b_{\mathrm{max}})$  is that the former only considers the history at the HPO step for which we are reporting the results.
In short, the regret is the difference in the evaluation metric performance from the best-found hyperparameter configuration during optimization to the best possible hyperparameter configuration (oracle) on the dataset (in a minimization setting). On a dataset level, we report the average regret across 10 repetitions with different seeds. To be able to aggregate results over datasets, we report the averaged normalized regret. As normalization, we divide the regret by the difference between the performances of the best and the worst hyperparameter configuration on a dataset. Then we compute the mean of the normalized regrets across all the datasets of a benchmark.
Moreover, in the experiments that report the average normalized regret over time, we provide results over normalized wall clock time. The wall clock time includes both the method's overhead (i.e. training the surrogate  $\hat{f}$  and selecting the next hyperparameter configuration to evaluate) and the time taken to evaluate the selected hyperparameter configuration (i.e. evaluating  $f$ ). Since the methods have different run times, we normalize the individual times by the time it took Random Search (the fastest non-model-based method) to complete the HPO optimization process. To provide a fair any-time comparison, we report results until the time it took Random Search to evaluate 20 hyperparameter configurations.