Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers

The performance of deep neural networks can be highly sensitive to the choice of a variety of meta-parameters , such as optimizer parameters and model hyperparameters . Tuning these well , however , often requires extensive and costly experimentation . Bayesian optimization ( BO ) is a principled approach to solve such expensive hyperparameter tuning problems efficiently . Key to the performance of BO is specifying and refining a distribution over functions , which is used to reason about the optima of the underlying function being optimized . In this work , we consider the scenario where we have data from similar functions that allows us to specify a tighter distribution a priori . Specifically , we focus on the common but potentially costly task of tuning optimizer parameters for training neural networks . Building on the meta BO method from Wang et al . ( 2018b ) , we develop practical improvements that ( a ) boost its performance by leveraging tuning results on multiple tasks without requiring observations for the same meta-parameter points across all tasks , and ( b ) retain its regret bound for a special case of our method . As a result , we provide a coherent BO solution for iterative optimization of continuous optimizer parameters . To verify our approach in realistic model training setups , we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets , as well as a protein sequence dataset . Our results show that on average , our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods . 1 INTRODUCTION . The careful tuning of a variety of meta-parameters , such as optimizer parameters and model hyperparameters has become a basic necessity for deep learning ( Bergstra et al. , 2011 ; Feurer et al. , 2015 ) . Such tuning requires extensive experimentation , retraining models repeatedly with different configurations , and can be challenging at realistic budgets because the tuning landscape is typically non-stationary , noisy and ill-behaved . Tuning has become sufficiently costly that finding more efficient and effective tuning procedures has the potential to save a substantial amount of resources or , alternatively , improve the accuracy of the final models at a given budget . Some hyperparameters might be common across a large number of tuning problems , such as those pertaining to an optimization algorithm . For example , Adam is used across many deep learning applications and has four parameters that require careful tuning ( Nado et al. , 2021 ) . Thus , if we have access to the performance of different optimizer-specific hyperparameters on different model training tasks , we may be able to transfer the knowledge among those tasks . This kind of meta-level learning is common among practitioners themselves : when faced with a new tuning problem , one might first try reusing hyperparameter settings that worked well on another problem . The underlying assumption is that hyperparameters should perform similarly across tasks . In this work , we aim to formalize this assumption and automate optimizer hyperparameter tuning by leveraging knowledge from previous experiments . Although our experiments consider optimizer parameter tuning as a practically important sub-problem of hyperparameter tuning for deep neural networks , our method applies to any hyperparameters that are common across multiple tasks . Bayesian optimization ( BO ) has become a popular methodology for optimizing the hyperparameters of machine learning models ( Snoek et al. , 2012 ; Bergstra et al. , 2011 ) and represents the state-ofthe-art ( Turner et al. , 2021 ) . BO involves specifying a probabilistic model over the function to be optimized and using this to reason about the location of the optimum . The optimization proceeds by iteratively updating the model with new data and using the posterior distribution to reason about where to next evaluate , trading off exploration and exploitation . The model is typically specified with only a-priori assumptions of smoothness , for example using a Gaussian process ( GP ) with a smooth covariance function . Even if the model is well-specified , BO can be slow to converge due to the generality of the prior assumptions . This would seem wasteful for problems that are repeated often or share considerable structure with previous experiments . One natural option is to cast our problem as meta Bayesian optimization , where the goal is to learn to optimize a function by generalizing from past experience with other similar functions . Indeed , several BO methods exist in the literature , but they are unsuitable for our scenario where we envision potentially thousands of related tasks within e.g . the context of a hyperparameter tuning service . Existing meta BO methods either scale cubically in the number of evaluations and tasks ( Swersky et al. , 2013 ; Bardenet et al. , 2013 ) ( See §4.3 for more details ) , impose a restrictive set of assumptions on the available data ( Wang et al. , 2018b ; Swersky et al. , 2013 ) to obtain efficient solutions , or make assumptions on the availability of GP parameters ( Volpp et al. , 2020 ) or descriptive task-level features ( Brazdil et al. , 1994 ; Bardenet et al. , 2013 ; Yogatama & Mann , 2014 ) . To address these issues , we introduce HyperBO : a meta BO method that builds upon Wang et al . ( 2018b ) with a relatively simple assumption : all the related functions being optimized are samples from the same GP prior distribution over functions . Concretely , HyperBO assumes the functions are conditionally independent given the hyperparameters , mean and covariance function of the GP . Compared to Wang et al . ( 2018b ) , HyperBO does not impose any strict conditions on data or model structures , and a special case of HyperBO retains strong regret bounds . From a computational perspective , HyperBO scales linearly in the number of tasks during training , and does not depend on the number of tasks when deployed . By not imposing assumptions about the data collection conditions , it can be used with large offline datasets or a few related optimization trajectories . To evaluate HyperBO , we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets , as well as on a protein sequence dataset . We compare HyperBO to several hyperparameter tuning baselines in the sequential BO setting . Our results show that optimizers that use hyperparameters suggested by our method are able to obtain better performing models requiring at least 3 times fewer function evaluations than other baselines . Our main contributions are two-fold : ( 1 ) a practical meta BO approach that makes minimal assumptions ; and ( 2 ) a large multi-task hyperparameter tuning dataset that not only benefits our method but also serves as an ideal benchmark to test future multi-task or meta-learning BO methods.1 Related work There is a rich literature of innovative methodologies to improve the efficiency of BO given related tasks or additional context . Here we discuss the most closely related work and explain why these don ’ t solve the specific scenario which we envision . Specifically , our goal is a methodology that is scalable enough to share information across thousands of tasks , each with potentially hundreds of observations , such as in the context of a large BO service or library . In this work we use the term meta-BO to refer to the class of BO methods that use data from existing tasks to optimize a new task . Multi-task BO ( Swersky et al. , 2013 ; Poloczek et al. , 2017 ; Yogatama & Mann , 2014 ) and transfer learning BO using contextual GPs ( Krause & Ong , 2011 ; Bardenet et al. , 2013 ; Poloczek et al. , 2016 ) are both meta BO approaches . Some meta BO methods have also been studied for hyperparameter tuning tasks in machine learning ( Feurer et al. , 2015 ) . HyperBO assumes all tasks are independent ( after conditioning on the GP ) , whereas both multi-task and contextual BO rely heavily on the assumption that tasks are related . Thus the latter approaches typically scale cubically in both the number of tasks and observations in each task , meaning that they can not gracefully scale across both without heavy approximations . When assuming that all inputs are equal across tasks , multi-task BO can be sped up using a Kronecker decomposition of the kernel 1We are working on open-sourcing the code base and dataset . The dataset is collected based on an opensourced code base ( Gilmer et al. , 2021 ) . to a task kernel and an input kernel which can be inverted separately ; a similar assumption is made by Wang et al . ( 2018b ) . In comparison , HyperBO scales linearly in the number of tasks ( see §B ) . End-to-end learning ( Chen et al. , 2017 ; Volpp et al. , 2020 ) is another popular meta BO approach for hyperparameter tuning that learns a strategy to suggest new query points based on past history of BO . One limitation of such approaches is that the total number of BO iterations must be determined a-priori . Furthermore , by nature of using a highly parameterized model to train the strategy , we lose the interpretability of intermediate steps that GPs and acquisition functions provide . HyperBO builds upon Wang et al . ( 2018b ) and Kim et al . ( 2017 ; 2019 ) . We resolve their issues with optimizing over a continuous space rather than a discrete set and limitations on using the same set of inputs across tasks . Kim et al . ( 2017 ; 2019 ) estimated a multivariate Gaussian that models values for search strategies in robot manipulation tasks , and thus only considered discrete inputs . Wang et al . ( 2018b ) provided regret bounds for Kim et al . ( 2017 ; 2019 ) , which was identified as meta BO without the knowledge of the GP prior . For both finite discrete search spaces and continuous ones , Wang et al . ( 2018b ) requires observations on the same set of inputs across tasks , which is an assumption that is not required for HyperBO ; HyperBO still inherits the same regret bound as Wang et al . ( 2018b ) for the special case where the same-inputs assumption is satisfied . Similar ideas also appeared in Perrone et al . ( 2018 ) , which can be viewed as a special case of HyperBO or Wang et al . ( 2018b ) . 2 PROBLEM FORMULATION . We consider the standard black-box function optimization scenario : given a real-valued function f defined over a compact , hyper-rectangular space X ⊂ Rd and given observations of similar functions f1 , · · · , fN , we seek an x ∈ X optimizing f . We inherit our problem formulation from Wang et al . ( 2018b ) , but we relax impractical assumptions on data availability ( we do not require all observations to be made on the same inputs across tasks ) and model restrictions . Assumptions and the goal . Concretely , we assume that there exists a Gaussian process GP ( µ , k ) with unknown mean function µ : X→ R and kernel k : X× X→ R. Let N be the number of tasks and let Mi be the number of observations we have for the ith task . Conditioned on independent function samples fi ∼ GP ( µ , k ) and inputs x ( i ) j ∈ X , i ∈ [ N ] , j ∈ [ Mi ] , we observe evaluations y ( i ) j ∼ N ( fi ( x ( i ) j ) , σ 2 ) perturbed by i.i.d . additive Gaussian noise N ( 0 , σ2 ) . Taken together , the collection of sub-datasets Dfi = { ( x ( i ) j , y ( i ) j ) } Mi j=1 define a dataset DN = { Dfi } Ni=1 . Finally , our goal is to maximize a new function independently sampled from the same GP , f ∼ GP ( µ , k ) ; that is , solve arg maxx∈X f ( x ) given dataset DN but unknown functions µ , k and unknown parameter σ 2 . An example . In our optimizer hyperparameter tuning application , a task corresponds to finding the best optimizer hyperparameters to train a given model on a particular dataset,2 e.g . training a ResNet ( He et al. , 2016 ) on ImageNet ( Russakovsky et al. , 2015 ) . Notice that we do not assume that the mean function µ , kernel k and noise variance σ2 are given . This is consistent with the reality of solving real-world black-box optimization problems including hyperparameter tuning . We must learn those unknown functions and parameters from data . However , in practice , searching in functional spaces to find the right mean µ or kernel k is a daunting task . Hence for practical concerns , a well defined search space for functions is required . More details on this can be found at §3.1 . Metrics . For simplicity , throughout this paper , we focus on the setting where the target function f can only be optimized by iteratively choosing where to evaluate , and defer batch evaluation setups to Sec . F. As we run BO on the target function f for T iterations , we accumulate a set of observations Df = { ( xt , yt ) } Tt=1 , yt ∼ N ( f ( xt ) , σ2 ) . We evaluate the quality of the optimization using the simple regret metric : RT = maxx∈X f ( x ) − f ( x̂ ) , where x̂ is the final recommendation at the end of the optimization process . There are various ways of setting x̂ based on the observations Df ; we use the input that achieved the best evaluation : x̂ = xτ ; τ = arg maxt∈ [ T ] yt . Bayesian viewpoint . As mentioned above , the observed functions f1 , · · · , fN and the evaluation target f are assumed to be independent draws from the same GP . This assumption is consistent with a hierarchical Bayes interpretation ( Fig . 1 ) , where all observed functions are independent conditioned on the GP . Notice that for BO , each selected input x ( i ) j depends on all previous observations . But we only describe the generative model of a hierarchical GP for simplicity . 2Technically , we also consider different batch sizes to be different tasks . More specifically , we assume that underlying functions of hyperparameter optimization tasks are defined by a parameter θ ∼ p ( θ ; α ) ; mean and kernel functions µ and k are defined by deterministic functions parameterized by θ . The independent function samples { fi } i∈ [ N ] are draws from GP ( µ , k ) . We then learn function f from observations DN on all other conditionally i.i.d . function samples f1 , · · · , fN . We forgo a fully Bayesian approach that samples from the posterior over θ at every BO iteration . Our method can be viewed as a type-II maximum likelihood approximation of such a Bayesian solution . Notations . Let [ n ] denote { 1 , · · · , n } , ∀n ∈ Z+ . For conciseness , we write the evaluation of a function f : X→ R on matrix x = [ xi ] n i=1 ∈ Rn×d as µ ( x ) : = [ µ ( xi ) ] ni=1 . Similarly , for two matrices ( i.e. , the concatenation of input vectors ) x , x′ , we write the corresponding kernel matrix as k ( x , x′ ) : = [ k ( xi , x′j ) ] i∈ [ n ] , j∈ [ n′ ] , and shorten k ( x ) : = k ( x , x ) . We denote a ( multivariate ) Gaussian distribution with mean u and variance Σ by N ( u , Σ ) , and a Gaussian process ( GP ) with mean function µ and covariance function k by GP ( µ , k ) . Let σ2 be the noise variance in observations . Given a set of observations D = { ( xt , yt ) } Tt=1 , yT = [ yt ] Tt=1 ∼ N ( f ( xT ) , σ2I ) , xT = [ xt ] Tt=1 and f ∼ GP ( µ , k ) , we denote the corresponding conditional GP distribution as GP ( µ , k | D ) . Let ψ ( x ) = k ( x , xT ) ( k ( xT ) + σ2I ) −1 . Recall that the conditional distribution , GP ( µ , k | D ) = GP ( µD , kD ) , is given for any x , x′ ∈ X as µD ( x ) = µ ( x ) + ψ ( x ) ( yT − µ ( xT ) ) , kD ( x , x′ ) = k ( x , x′ ) − ψ ( x ) k ( xT , x′ ) . ( 1 ) 3 OUR METHOD Algorithm 1 HyperBO with acquisition function α ( · ) . 1 : function HYPERBO ( f , DN ) 2 : GP ( µ̂ , k̂ ) ← TRAINGP ( DN ) 3 : Df ← ∅ 4 : for t = 1 , · · · , T do 5 : xt ← arg max x∈X α ( x ; GP ( µ̂ , k̂ | Df ) ) 6 : yt ← OBSERVE ( f ( xt ) ) 7 : Df ← Df ∪ { ( xt , yt ) } 8 : end for 9 : return Df 10 : end function As shown in Alg . 1 , our approach trains the GP hyperparameters on a representative set of datasets and fixes them for the duration of the optimization procedure ; we refer to this approach as HyperBO . HyperBO runs in two steps . First , we learn a GP model GP ( µ̂ , k̂ ) to approximate the ground-truth ( unknown ) GP that generated the dataset DN . Then , we do standard BO to optimize a new function f with the learned GP GP ( µ̂ , k̂ ) . The initial learning process ( line 2 ) is the critical difference between HyperBO and standard BO algorithms , as well as the key contribution of this paper . Based on the Bayesian graphical model interpretation ( Fig . 1 ) , our goal is to obtain a point estimate θ̂ for the parameter θ . Given this estimate , we can then estimate the mean function µ̂ and the kernel k̂ , which defines our learned model GP ( µ̂ , k̂ ) . During the BO iterations ( Alg . 1 , lines 4-8 ) , we update the conditional GP , but do not re-estimate the GP mean and kernel . By separating the data for the conditional GP update and GP parameter training , we minimize the computational cost while still maintaining good performance both theoretically and empirically . Moreover , we avoid the BO chicken-and-egg dilemma ( Wang et al. , 2018b ) where the search strategy is trained on data collected in the BO process and the data points are selected by the search strategy . Next , we introduce our GP training strategy based on two types of objectives : marginal data likelihood ( § 3.1 ) and distance between estimates and model predictions ( § 3.2 ) . In the appendix , §B reveals the complexity of HyperBO that is linear in the number of tasks and § 3.3 shows that a special case of HyperBO retains strong regret bounds ( Wang et al. , 2018b ) .

Summary:  HyperBO assumes the tasks are independent given the hyperparameters, unlike typical metalearning approaches which assume tasks are related. This allows for an efficient Kronecker decomposition of the kernel and thus linear, rather than cubic scaling, across tasks.   Using this model, HyperBO performs BO as usual; maximize the acquisition function to obtain the next point to evaluate. HyperBO also makes the critical assumption of an offline pre-training of hyperparameters on a representative set of completed tasks; during optimization itself the hyperparameters are fixed.