Batch Multi-Fidelity Bayesian Optimization with  Deep Auto-Regressive Networks

1 Introduction . Many applications demand we optimize a complex function of an unknown form that is expensive to evaluate . Bayesian optimization ( Mockus , 2012 ; Snoek et al. , 2012 ) is a powerful approach to optimize such functions . The key idea is to use a probabilistic surrogate model , typically Gaussian processes ( Rasmussen and Williams , 2006 ) , to iteratively approximate the target function , integrate the posterior information to compute and maximize an acquisition function so as to generate new inputs at which to query , update the model with new examples , and meanwhile approach the optimum . In practice , to enable a flexible trade-off between the computational cost and accuracy , many applications allow us to evaluate the target function at different fidelities . For example , to evaluate the performance of the hyperparameters for a machine learning model , we can train the model thoroughly , i.e. , with sufficient iterations/epochs , to obtain the accurate evaluation ( high-fidelity yet often costly ) or just run a few iterations/epochs to obtain a rough estimate ( low-fidelity but much cheaper ) . Many multi-fidelity BO algorithms ( Lam et al. , 2015 ; Kandasamy et al. , 2016 ; Zhang et al. , 2017 ; Song et al. , 2019 ; Takeno et al. , 2019 ) have therefore been proposed to identify both the fidelities and inputs at which to query , so as to reduce the cost and achieve a good benefit-cost balance . Notwithstanding their success , these methods often overlook the strong yet complex relationships between different fidelities or adopt an over-simplified assumption , ( partly ) for the sake of convenience in calculating/maximizing the acquisition function considering fidelities . This , however , can restrict the performance of the surrogate model , impair the optimization efficiency and increase the cost . For 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . example , Lam et al . ( 2015 ) ; Kandasamy et al . ( 2016 ) learned an independent GP for each fidelity , Zhang et al . ( 2017 ) used multitask GPs with a convolved kernel for multi-fidelity modeling and have to use a simple smoothing kernel ( e.g. , Gaussian ) for tractable convolutions . The recent work ( Takeno et al. , 2019 ) imposes a linear correlation across different fidelities . In addition , the standard one-byone querying strategy needs to sequentially run each query and can not utilize parallel computing resources to accelerate , e.g. , multi-core CPUs/GPUs and clusters . While incrementally absorbing more information , it does not explicitly account for the correlation between different queries , hence still has a risk to bring in highly correlated examples that includes redundant information . To address these issues , we propose BMBO-DARN , a novel batch multi-fidelity Bayesian optimization method . First , we develop a deep auto-regressive model to integrate training examples at various fidelities . Each fidelity is modeled by a Bayesian neural network ( NN ) , where the output predicts the objective function value at that fidelity and the input consists of the original inputs and the outputs of all the previous fidelities . In this way , our model is adequate to capture the complex , strong correlations ( e.g. , nonstationary , highly nonlinear ) across all the fidelities to enhance the surrogate learning . We use Hamiltonian Monte-Carlo ( HMC ) sampling for posterior inference . Next , to improve the quality of the queries , we develop a simple yet efficient method to jointly fetch a batch of inputs and fidelities . Specifically , we propose a batch acquisition function based on the state-of-the-art Max-value Entropy Search ( MES ) principle ( Wang and Jegelka , 2017 ) . The batch acquisition function explicitly penalizes highly correlated queries and encourages diversity . To efficiently compute the acquisition function , we use the posterior samples of the NN weights and moment matching to construct a multi-variate Gaussian posterior for all the fidelity outputs and the function optimum . To prevent a combinatorial search over multiple fidelities in maximizing the acquisition function , we develop an alternating optimization algorithm to cyclically update each pair of input and fidelity , which is much more efficient and guarantees an improvement at each step . For evaluation , we examined BMBO-DARN in both synthetic benchmarks and real-world applications . The synthetic benchmark tasks show that given a small number of training examples , our deep autoregressive model can learn a more accurate surrogate of the target function than other state-of-the-art multi-fidelity BO models . We then evaluated BMBO-DARN on four popular machine learning models ( CNN , online LDA , XGBoost and Physics informed NNs ) for hyperparameter optimization . BMBO-DARN can find more effective hyperparameters leading to superior predictive performance , and meanwhile spends smaller total evaluation costs , as compared with state-of-the-art multi-fidelity BO algorithms and other popular hyperparameter tuning methods . 2 Background . Bayesian Optimization ( BO ) ( Mockus et al. , 1978 ; Snoek et al. , 2012 ) is a popular approach for optimizing black-box functions that are often costly to evaluate and can not provide exact gradient information . BO learns a probabilistic surrogate model to predict the function value across the input space and quantifies the predictive uncertainty . At each step , we use this information to compute an acquisition function to measure the utility of querying at different inputs . By maximizing the acquisition function , we find the next input at which to query , which is supposed to be closer to the optimum . Then we add the new example into the training set to improve the accuracy of the surrogate model . The procedure is repeated until we find the optimal input or the maximum number of queries have been finished . There are a variety of acquisition functions , such as Expected Improvement ( EI ) ( Mockus et al. , 1978 ) and Upper Confidence Bound ( UCB ) ( Srinivas et al. , 2010 ) . The recent state-of-the-art addition is Maximum-value Entropy Search ( MES ) ( Wang and Jegelka , 2017 ) , a ( x ) = I ( f ( x ) , f∗|D ) , ( 1 ) where I ( · , · ) is the mutual information , f ( x ) is the objective function value at x , f∗ the minimum , and D the training data collected so far for the surrogate model . Note that both f ( x ) and f∗ are considered as generated by the posterior of the surrogate model given D ; they are random variables . The most commonly used class of surrogate models is Gaussian process ( GP ) ( Rasmussen and Williams , 2006 ) . Given the training dataset X = [ x > 1 , . . . , x > N ] > and y = [ y1 , . . . , yN ] > , a GP assumes the outputs y follow a multivariate Gaussian distribution , p ( y|X ) = N ( y|m , K + vI ) , where m is the mean function values at the inputs X , often set to 0 , v is the noise variance , and K is a kernel matrix on X . Each [ K ] ij = κ ( xi , xj ) , where κ ( · , · ) is a kernel function . For example , a popular one is the RBF kernel , κ ( xi , xj ) = exp ( −β−1‖xi − xj‖2 ) . An important advantage of GPs is their convenience in uncertainty quantification . Since GPs assume any finite set of function values follow a multi-variate Gaussian distribution , given a test input x̂ , we can compute the predictive ( or posterior ) distribution p ( f ( x̂ ) |x̂ , X , y ) via a conditional Gaussian distribution , which is simple and analytical . Multi-Fidelity BO . Since evaluating the exact value of the object function is often expensive , many practical applications provide multi-fidelity evaluations { f1 ( x ) , . . . , fM ( x ) } to allow us to choose a trade-off between the accuracy and cost . Accordingly , many multi-fidelity BO algorithms have been developed to select both the inputs and fidelities to reduce the cost and to achieve a good balance between the optimization progress and cost , i.e. , the benefit-cost ratio . For instance , MFGP-UCB ( Kandasamy et al. , 2016 ) sequentially queries at each fidelity ( from the lowest one , i.e. , m = 1 ) until the confidence band is over a given threshold . In spite of its great success and guarantees in theory , MF-GP-UCB uses a set of independent GPs to estimate the objective at each fidelity , and hence ignores the valuable correlations between different fidelities . MF-PES ( Zhang et al. , 2017 ) uses a multi-task GP surrogate where each task corresponds to one fidelity , and convolves a smoothing kernel with the kernel of a shared latent function to obtain the cross-covariance . The recent MF-MES ( Takeno et al. , 2019 ) also builds a multi-task GP surrogate , where the covariance function is κ ( fm ( x ) , fm′ ( x ′ ) ) = ∑d j=1 ( umjum′j + 1 ( m = m ′ ) · αmj ) ρj ( x1 , x2 ) , ( 2 ) where αmj > 0 , 1 ( · ) is the indicator function , { umj } dj=1 is d latent features for each fidelity m , and { ρj ( · , · ) } are d bases kernels , usually chosen as a commonly used stationary kernel , e.g. , RBF . 3 Deep Auto-Regressive Model for Multi-Fidelity Surrogate Learning . Notwithstanding the elegance and success of the existing multi-fidelity BO methods , they often ignore or oversimplify the complex , strong correlations between different fidelities , and hence can be inefficient for surrogate learning , which might further lower the optimization efficiency and incur more expenses . For example , the state-of-the-art methods MF-GP-UCB ( Kandasamy et al. , 2016 ) estimate a GP surrogate for each fidelity independently ; MF-PES ( Zhang et al. , 2017 ) has to adopt a simple form for both the smoothing and latent function kernel ( e.g. , Gaussian and delta ) to achieve an analytically tractable convolution , which might limit the expressivity in estimating the cross-fidelity covariance ; MF-MES ( Takeno et al. , 2019 ) essentially imposes a linear correlation structure between different fidelities — for any input x , κ ( fm ( x ) , fm′ ( x ) ) = u > m1um2 + αm where um = [ um1 , . . . , umd ] and α̃m = ∑d j=1 αmj if we use a RBF basis kernel ( see ( 2 ) ) . To overcome this limitation , we develop a deep auto-regressive model for multi-fidelity surrogate learning . Our model is expressive enough to capture the strong , possibly very complex ( e.g. , highly nonlinear , nonstationary ) relationships between all the fidelities to improve the prediction ( at the highest fidelity ) . As such , our model can more effectively integrate multi-fidelity training information to better estimate the objective function . Specifically , given M fidelities , we introduce a chain of M neural networks , each of which models one fidelity and predicts the target function at that fidelity . Denote by xm , Wm , and ψWm ( · ) the NN input , parameters and output mapping at each fidelity m. Our model is defined as follows , xm = [ x ; f1 ( x ) ; . . . ; fm−1 ( x ) ] , fm ( x ) = ψWm ( xm ) , ym ( x ) = fm ( x ) + m , ( 3 ) where x1 = x , fm ( x ) is the prediction ( i.e. , NN output ) at the m-th fidelity , ym ( x ) is the observed function value , and m is a random noise , m ∼ N ( m|0 , τ−1m ) . We can see that each input xm consists of not only the original input x of the objective function , but also the outputs from all the previous fidelities . Via a series of linear projection and nonlinear activation from the NN , we obtain the output at fidelity m. In this way , our model fully exploits the information from the lower fidelities and can flexibly capture arbitrarily complex relationships between the current and all the previous fidelities by learning an NN mapping , fm ( x ) = ψWm ( xm , f1 ( x ) , . . . , fm−1 ( x ) ) . We assign a standard Gaussian prior distribution over each element of the NN parameters W = { W1 , . . . , WM } , and a Gamma prior over each noise precision , p ( τm ) = Gam ( τm|a0 , b0 ) . Given the dataset D = { { ( xnm , ynm ) } Nmn=1 } Mm=1 , the joint probability of our model is given by p ( W , τ , Y , S|X ) = N ( vec ( W ) |0 , I ) M∏ m=1 Gam ( τm|a0 , b0 ) M∏ m=1 Nm∏ n=1 N ( ynm|fm ( xnm ) , τ−1m ) , ( 4 ) where τ = [ τ1 , . . . , τM ] , X = { xnm } , Y = { ynm } , and vec ( · ) is vectorization . The graphical representation of our model is given in Fig . 1 . We use Hamiltonian Monte Carlo ( HMC ) ( Neal et al. , 2011 ) sampling to perform posterior inference due to its unbiased , high-quality uncertainty quantification , which is critical to calculate the acquisition function . However , our method allows us to readily switch to other approximate inference approaches as needed ( see Sec . 4 ) , e.g. , stochastic gradient HMC ( Chen et al. , 2014 ) used in the excellent work of Springenberg et al . ( 2016 ) .

Summary: Multi-fidelity methods are prevalent in problem settings where high-quality (but difficult to acquire) observations are enhanced with lower-quality (but more easily obtained) samples of a function. In this paper, the authors focus on the application of multi-fidelity modelling to hyper parameter tuning, whereby different model settings are verified with varying fidelity constraints (such as epochs in a training loop) in order to achieve the best accuracy-cost trade-off. The model introduced here is based on chained neural networks, whereby the input at each fidelity layer combines the input in the original domain for the function to be estimated, along with the outputs from the preceding fidelities. In order to carry out BO, the authors develop a batch acquisition technique that encourages diversity across the samples being drawn. This is in contrast to other techniques where new samples are drawn one a time, thus incurring the risk of there not being sufficient diversity across successive iterations. In order to achieve this goal, the authors also propose an alternating optimisation scheme for iteratively selecting samples across different fidelities.