SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show experiments both for single and multi-worker cases (4 workers in our experiments). For the single workers, we report the mean validation error with standard error bars for 50 seeds; for multi-worker experiments, we use 10 seeds. The plots show average relative ranks achieved aggregated across all 12 benchmarks when comparing the anytime incumbent configuration at  $z_{\mathrm{max}}$ . We group the runs on benchmark under different qualities of an expert prior input. We also compare the average normalized regret per benchmark under good priors. The prior construction procedure follows the design from the work by Hvarfner et al. [28], and we describe our procedure in detail in Appendix D.3. In the main paper, we only evaluate the good and bad prior settings as we believe that this reflects a practical setting of prior qualities; in Appendix F we also evaluate the near-optimal prior settings and also evaluate all 3 prior settings over high and low-performance correlation problems. In Appendix F.5, we also report additional experiments for different budget horizons. Further experiment design details can be found in Appendices D.4 and D.5. As an example of potential post-hoc analysis possible with PriorBand, we show how the dynamic probabilities calculated for each of the sampling strategies in PriorBand can be visualized (Appendix E.3), revealing the quality of the prior used.
We now report and discuss the results of our experiments.
7.1 Robustness of PriorBand
Here, we demonstrate the robustness of PriorBand over a wide range of prior qualities by comparing them to the nearest non-prior-based algorithms: random search (RS) and HyperBand (HB). Figure 5(top) showcases PriorBand to be anytime equal or better than HB on each of our benchmarks under our good prior design. Note that the quality of a good prior varies per benchmark. The aggregation of Figure 5(top) is Figure 5(bottom-middle), which illustrates that PriorBand  $(\mathrm{HB} + \mathcal{E}_{\pi})$  can utilize good priors and gain strong anytime performance over HB. Moreover, Figure 5(bottom-right) clearly shows PriorBand's ability to recover from bad prior information and match vanilla-HB's performance in this adversarial case (the bad prior was intentionally chosen to be an extremely poor configuration – the worst of  $50k$  random samples). In most practical scenarios, the expert has better intuition than this setup and one can expect substantial speedups over HB, like in Figure 5(bottom-middle). Figure 5(bottom-left) demonstrates that when using an unknown quality of prior, even including the adversarial prior, PriorBand is still the best choice on average. We show a budget of 12 function evaluations here, which is approximately the maximal budget required for completing at least one HB iteration for the benchmarks chosen. In Appendix F.1 we show more comparisons to prior-based baselines and highlight PriorBand's robustness.
7.2 Generality of Ensemble Sampling Policy  $\mathcal{E}_{\pi}$
The ESP,  $\mathcal{E}_{\pi}$ , only needs access to an SH-like optimization state as input. In this section, we show that other popular multi-fidelity algorithms, such as ASHA or asynchronous-HB [36] that build on SH, can also support expert priors with the help of  $\mathcal{E}_{\pi}$ . In Figure 6, we compare the vanilla algorithms with their  $\mathcal{E}_{\pi}$ -augmented versions. Given that these algorithms were designed for parallel setups,
Figure 5: [Top] Comparing normalized regret in the good prior setting; The [Bottom] figure Good (middle) is a ranking aggregate view of the [Top] figure. [Bottom] Comparing average relative ranks of PriorBand to Random Search (RS) and HB, over single worker runs across benchmarks. Each benchmark-algorithm pair was run for 50 seeds where priors per benchmark are the same across a seed. We show mean rank and standard error. The All (left) plot averages the benchmark across Good and Bad priors.
we compare them on runs distributed among 4 workers, running for a total budget of 20 function evaluations. Similar to the previous section, the ESP-augmented (+ESP) algorithms can leverage good priors and recover under bad priors. Under bad priors, asynchronous-HB (+ESP) starts worst owing to bad prior higher fidelity evaluations at the start but shows strong recovery. In Appendix F.2 we compare these variants over different correlation settings of the benchmarks.
Figure 6: Comparing the average relative ranks of Asynchronous-SH (ASHA) and Asynchronous-HB (Async-HB) and their variants with the Ensemble Sampling Policy (+ESP) when distributed over 4 workers for a cumulative budget of 20 function evaluations.
7.3 Extensibility with models
Although our focus is the model-free low compute regime, we also show how PriorBand can be optionally extended with model-based surrogates to perform Bayesian Optimization, especially when longer training budgets are available. We compare BO,  $\pi$ BO, BOHB, and PriorBand with its model-extension, PriorBand+BO (Gaussian Processes as surrogates and Expected Improvement [37] for acquisition). All BO methods use an initial design of 10 function evaluations, except BOHB which sets this implicitly as  $N_{\mathrm{dim}} + 2$ . Compared to other prior-based algorithms (RS+Prior,  $\pi$ BO), PriorBand+BO is consistently robust on average (All) under all priors. The anytime performance gain of PriorBand+BO under good priors is evidence of ESP  $\mathcal{E}_{\pi}$ 's utility. We note that PriorBand+BO recovers quickly in the bad prior setting and even outperforms all other algorithms after as little a budget as 6 full trainings. The fact that, until  $10\times$  function evaluations, PriorBand+BO is actually just PriorBand and model search has not begun, highlights the effectiveness of ESP in adaptively trading off different sampling strategies and the initial strength of incumbent sampling. Appendices E.4, F.3 contain details on our modeling choices and more experimental results.
Figure 7: Comparing the average relative ranks of model-based algorithms and sampling from the prior (RS+Prior) under single-worker runs, aggregated over all benchmarks-per-prior quality.
7.4 Ablation studies
In Figure 8, we compare our choice of using density scores (Section 5.2.2) to trade-off prior-based and incumbent-based sampling against other similar or simpler heuristics, validating our decision. PriorBand(constant) employs a fixed heuristic to trade off prior and incumbent-based sampling as  $p_{\pi} = \eta \cdot p_{\hat{\lambda}}$ . In PriorBand(decay),  $p_{\pi}$  is decayed as a function of iterations completed. More precisely,  $p_{\hat{\lambda}} = 2^b \cdot p_{\pi}$ , subject to  $p_{\pi} + p_{\hat{\lambda}} + p_{\mathcal{U}} = 1$  (from Section 5.2), where  $b \in \mathbb{N}$  and indicates the index of the current SH being run in HB. Appendix E.2.2 contains more ablations.
Figure 8: A comparison of PriorBand with 3 different strategies for trading off incumbent vs. prior distribution sampling. PriorBand's default of using density scores shows a dominant performance in almost all scenarios. However, a bad prior induces a marked difference in performance. In general, the plot highlights that each variant reacts differently to bad priors. The adaptive nature of PriorBand clearly is robust to different scenarios comparatively.
While using expert priors and local search [38] for hyperparameter optimization has been explored previously, few works considered priors over the optimum [27, 28, 39], and they all target the
single-fidelity setting. The expert priors we consider should not be confused with the priors natively supported by BO, i.e., priors over the function structure determined by kernels [7, 40, 41].
In deep learning, training epochs and dataset subsets [17, 42, 43] are frequently used as fidelity variables to create cheap proxy tasks, with input resolution, network width, and network depth also occasionally used [35]. SH [29] and HB [30] are effective randomized policies for multi-fidelity HPO that use early stopping of configurations on a geometric spacing of the fidelity space and can also be extended to the model-based setting [44].
We acknowledge that PriorBand inherits pathologies of HB such as poor fidelity correlations and can be sensitive to the choice of fidelity bounds and early stopping rate. Despite that, our results across different correlation settings suggest a relatively strong performance of PriorBand (Appendix F). Though PriorBand supports any kind of prior distribution as input, in our experiments we only considered the Gaussian distribution (with a fixed standard deviation of 0.25), as it is a natural choice and was used previously in the literature [28]. However, the expert is free to represent the prior with other distributions. We expect PriorBand to show similar behaviors as we report when comparing to other prior-based algorithms under a similar distribution. We note that PriorBand is not entirely free of hyperparameters. In our experiments, we keep all PriorBand hyperparameters fixed to remove confounding factors while comparing different prior strengths and correlations. Moreover, our hyperparameter choices for the experiments, such as  $\eta = 3$  and Gaussian priors are largely borrowed from existing literature [28, 36, 44]. Depending on the context of the specific deep learning expert, the compute budgets ( $\sim 10 - 12$  full trainings) used in our experimental setting may not always be feasible. However, the key insight from the experiments is that PriorBand can interface informative priors, and provide strong anytime performance under short HPO budgets (also in less than 5 full trainings). Longer HPO budgets for PriorBand ensure recovery from potential bad, uninformative priors. In practice, deep learning experts often have good prior knowledge and thus PriorBand is expected to retain strong anytime performance under low compute.
We identify that existing HPO algorithms are misaligned with deep learning (DL) practice and make this explicit with six desiderata (Section 1). To overcome this misalignment, our solution, PriorBand, allows a DL expert to incorporate their intuition of well-performing configurations into multi-fidelity HPO and thereby satisfies all desiderata to be practically useful for DL. The key component in PriorBand, the ESP,  $\mathcal{E}_{\pi}$ , is modular, flexible, and can be applied to other multi-fidelity algorithms.
11 Acknowledgments and disclosure of funding
N. Mallik, D. Stoll and F. Hutter acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. E. Bergman was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. C. Hvarfner and L. Nardi were partially supported by the Wallenberg AI, Autonomous Sytems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. L. Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project — Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348. M. Lindauer was funded by the European Union (ERC, "ixAutoML", grant no.101041029). The research of Maciej Janowski was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, Maciej Janowski acknowledges the support of the BrainLinks-BrainTools Center of Excellence.
C.  ${\mathrm{H}}^{ + }$  -ATPase 位于保卫细胞质膜上,蓝光能够引起细胞内的  ${\mathrm{H}}^{ + }$  转运到细胞外
[64] J. Kiili, E. Laaksonen, R. Turner, D. Eriksson, S. Park, M. Mccourt, Z. Xu, and I. Guyon. Black box optimization challenge, 2020.
[65] A. Klein, L. Tiao, T. Lienart, C. Archambeau, and M. Seeger. Model-based Asynchronous Hyperparameter and Neural Architecture Search. arXiv:2003.10865 [cs.LG], 2020.
[66] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL http://arxiv.org/abs/1910.06403.
[67] B. Matern. Spatial Variation. Lecture Notes in Statistics. Springer New York, 2013. ISBN 9781461578925. URL https://books.google.se/books?id=HrbSBwAAQBAJ.
[68] Frank Hutter, Lin Xu, Holger H. Hoos, and Kevin Leyton-Brown. Algorithm runtime prediction: Methods & evaluation, 2013.