Batch Multi-Fidelity Bayesian Optimization with  Deep Auto-Regressive Networks

1 Introduction . Many applications demand we optimize a complex function of an unknown form that is expensive to evaluate . Bayesian optimization ( Mockus , 2012 ; Snoek et al. , 2012 ) is a powerful approach to optimize such functions . The key idea is to use a probabilistic surrogate model , typically Gaussian processes ( Rasmussen and Williams , 2006 ) , to iteratively approximate the target function , integrate the posterior information to compute and maximize an acquisition function so as to generate new inputs at which to query , update the model with new examples , and meanwhile approach the optimum . In practice , to enable a flexible trade-off between the computational cost and accuracy , many applications allow us to evaluate the target function at different fidelities . For example , to evaluate the performance of the hyperparameters for a machine learning model , we can train the model thoroughly , i.e. , with sufficient iterations/epochs , to obtain the accurate evaluation ( high-fidelity yet often costly ) or just run a few iterations/epochs to obtain a rough estimate ( low-fidelity but much cheaper ) . Many multi-fidelity BO algorithms ( Lam et al. , 2015 ; Kandasamy et al. , 2016 ; Zhang et al. , 2017 ; Song et al. , 2019 ; Takeno et al. , 2019 ) have therefore been proposed to identify both the fidelities and inputs at which to query , so as to reduce the cost and achieve a good benefit-cost balance . Notwithstanding their success , these methods often overlook the strong yet complex relationships between different fidelities or adopt an over-simplified assumption , ( partly ) for the sake of convenience in calculating/maximizing the acquisition function considering fidelities . This , however , can restrict the performance of the surrogate model , impair the optimization efficiency and increase the cost . For 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . example , Lam et al . ( 2015 ) ; Kandasamy et al . ( 2016 ) learned an independent GP for each fidelity , Zhang et al . ( 2017 ) used multitask GPs with a convolved kernel for multi-fidelity modeling and have to use a simple smoothing kernel ( e.g. , Gaussian ) for tractable convolutions . The recent work ( Takeno et al. , 2019 ) imposes a linear correlation across different fidelities . In addition , the standard one-byone querying strategy needs to sequentially run each query and can not utilize parallel computing resources to accelerate , e.g. , multi-core CPUs/GPUs and clusters . While incrementally absorbing more information , it does not explicitly account for the correlation between different queries , hence still has a risk to bring in highly correlated examples that includes redundant information . To address these issues , we propose BMBO-DARN , a novel batch multi-fidelity Bayesian optimization method . First , we develop a deep auto-regressive model to integrate training examples at various fidelities . Each fidelity is modeled by a Bayesian neural network ( NN ) , where the output predicts the objective function value at that fidelity and the input consists of the original inputs and the outputs of all the previous fidelities . In this way , our model is adequate to capture the complex , strong correlations ( e.g. , nonstationary , highly nonlinear ) across all the fidelities to enhance the surrogate learning . We use Hamiltonian Monte-Carlo ( HMC ) sampling for posterior inference . Next , to improve the quality of the queries , we develop a simple yet efficient method to jointly fetch a batch of inputs and fidelities . Specifically , we propose a batch acquisition function based on the state-of-the-art Max-value Entropy Search ( MES ) principle ( Wang and Jegelka , 2017 ) . The batch acquisition function explicitly penalizes highly correlated queries and encourages diversity . To efficiently compute the acquisition function , we use the posterior samples of the NN weights and moment matching to construct a multi-variate Gaussian posterior for all the fidelity outputs and the function optimum . To prevent a combinatorial search over multiple fidelities in maximizing the acquisition function , we develop an alternating optimization algorithm to cyclically update each pair of input and fidelity , which is much more efficient and guarantees an improvement at each step . For evaluation , we examined BMBO-DARN in both synthetic benchmarks and real-world applications . The synthetic benchmark tasks show that given a small number of training examples , our deep autoregressive model can learn a more accurate surrogate of the target function than other state-of-the-art multi-fidelity BO models . We then evaluated BMBO-DARN on four popular machine learning models ( CNN , online LDA , XGBoost and Physics informed NNs ) for hyperparameter optimization . BMBO-DARN can find more effective hyperparameters leading to superior predictive performance , and meanwhile spends smaller total evaluation costs , as compared with state-of-the-art multi-fidelity BO algorithms and other popular hyperparameter tuning methods . 2 Background . Bayesian Optimization ( BO ) ( Mockus et al. , 1978 ; Snoek et al. , 2012 ) is a popular approach for optimizing black-box functions that are often costly to evaluate and can not provide exact gradient information . BO learns a probabilistic surrogate model to predict the function value across the input space and quantifies the predictive uncertainty . At each step , we use this information to compute an acquisition function to measure the utility of querying at different inputs . By maximizing the acquisition function , we find the next input at which to query , which is supposed to be closer to the optimum . Then we add the new example into the training set to improve the accuracy of the surrogate model . The procedure is repeated until we find the optimal input or the maximum number of queries have been finished . There are a variety of acquisition functions , such as Expected Improvement ( EI ) ( Mockus et al. , 1978 ) and Upper Confidence Bound ( UCB ) ( Srinivas et al. , 2010 ) . The recent state-of-the-art addition is Maximum-value Entropy Search ( MES ) ( Wang and Jegelka , 2017 ) , a ( x ) = I ( f ( x ) , f∗|D ) , ( 1 ) where I ( · , · ) is the mutual information , f ( x ) is the objective function value at x , f∗ the minimum , and D the training data collected so far for the surrogate model . Note that both f ( x ) and f∗ are considered as generated by the posterior of the surrogate model given D ; they are random variables . The most commonly used class of surrogate models is Gaussian process ( GP ) ( Rasmussen and Williams , 2006 ) . Given the training dataset X = [ x > 1 , . . . , x > N ] > and y = [ y1 , . . . , yN ] > , a GP assumes the outputs y follow a multivariate Gaussian distribution , p ( y|X ) = N ( y|m , K + vI ) , where m is the mean function values at the inputs X , often set to 0 , v is the noise variance , and K is a kernel matrix on X . Each [ K ] ij = κ ( xi , xj ) , where κ ( · , · ) is a kernel function . For example , a popular one is the RBF kernel , κ ( xi , xj ) = exp ( −β−1‖xi − xj‖2 ) . An important advantage of GPs is their convenience in uncertainty quantification . Since GPs assume any finite set of function values follow a multi-variate Gaussian distribution , given a test input x̂ , we can compute the predictive ( or posterior ) distribution p ( f ( x̂ ) |x̂ , X , y ) via a conditional Gaussian distribution , which is simple and analytical . Multi-Fidelity BO . Since evaluating the exact value of the object function is often expensive , many practical applications provide multi-fidelity evaluations { f1 ( x ) , . . . , fM ( x ) } to allow us to choose a trade-off between the accuracy and cost . Accordingly , many multi-fidelity BO algorithms have been developed to select both the inputs and fidelities to reduce the cost and to achieve a good balance between the optimization progress and cost , i.e. , the benefit-cost ratio . For instance , MFGP-UCB ( Kandasamy et al. , 2016 ) sequentially queries at each fidelity ( from the lowest one , i.e. , m = 1 ) until the confidence band is over a given threshold . In spite of its great success and guarantees in theory , MF-GP-UCB uses a set of independent GPs to estimate the objective at each fidelity , and hence ignores the valuable correlations between different fidelities . MF-PES ( Zhang et al. , 2017 ) uses a multi-task GP surrogate where each task corresponds to one fidelity , and convolves a smoothing kernel with the kernel of a shared latent function to obtain the cross-covariance . The recent MF-MES ( Takeno et al. , 2019 ) also builds a multi-task GP surrogate , where the covariance function is κ ( fm ( x ) , fm′ ( x ′ ) ) = ∑d j=1 ( umjum′j + 1 ( m = m ′ ) · αmj ) ρj ( x1 , x2 ) , ( 2 ) where αmj > 0 , 1 ( · ) is the indicator function , { umj } dj=1 is d latent features for each fidelity m , and { ρj ( · , · ) } are d bases kernels , usually chosen as a commonly used stationary kernel , e.g. , RBF . 3 Deep Auto-Regressive Model for Multi-Fidelity Surrogate Learning . Notwithstanding the elegance and success of the existing multi-fidelity BO methods , they often ignore or oversimplify the complex , strong correlations between different fidelities , and hence can be inefficient for surrogate learning , which might further lower the optimization efficiency and incur more expenses . For example , the state-of-the-art methods MF-GP-UCB ( Kandasamy et al. , 2016 ) estimate a GP surrogate for each fidelity independently ; MF-PES ( Zhang et al. , 2017 ) has to adopt a simple form for both the smoothing and latent function kernel ( e.g. , Gaussian and delta ) to achieve an analytically tractable convolution , which might limit the expressivity in estimating the cross-fidelity covariance ; MF-MES ( Takeno et al. , 2019 ) essentially imposes a linear correlation structure between different fidelities — for any input x , κ ( fm ( x ) , fm′ ( x ) ) = u > m1um2 + αm where um = [ um1 , . . . , umd ] and α̃m = ∑d j=1 αmj if we use a RBF basis kernel ( see ( 2 ) ) . To overcome this limitation , we develop a deep auto-regressive model for multi-fidelity surrogate learning . Our model is expressive enough to capture the strong , possibly very complex ( e.g. , highly nonlinear , nonstationary ) relationships between all the fidelities to improve the prediction ( at the highest fidelity ) . As such , our model can more effectively integrate multi-fidelity training information to better estimate the objective function . Specifically , given M fidelities , we introduce a chain of M neural networks , each of which models one fidelity and predicts the target function at that fidelity . Denote by xm , Wm , and ψWm ( · ) the NN input , parameters and output mapping at each fidelity m. Our model is defined as follows , xm = [ x ; f1 ( x ) ; . . . ; fm−1 ( x ) ] , fm ( x ) = ψWm ( xm ) , ym ( x ) = fm ( x ) + m , ( 3 ) where x1 = x , fm ( x ) is the prediction ( i.e. , NN output ) at the m-th fidelity , ym ( x ) is the observed function value , and m is a random noise , m ∼ N ( m|0 , τ−1m ) . We can see that each input xm consists of not only the original input x of the objective function , but also the outputs from all the previous fidelities . Via a series of linear projection and nonlinear activation from the NN , we obtain the output at fidelity m. In this way , our model fully exploits the information from the lower fidelities and can flexibly capture arbitrarily complex relationships between the current and all the previous fidelities by learning an NN mapping , fm ( x ) = ψWm ( xm , f1 ( x ) , . . . , fm−1 ( x ) ) . We assign a standard Gaussian prior distribution over each element of the NN parameters W = { W1 , . . . , WM } , and a Gamma prior over each noise precision , p ( τm ) = Gam ( τm|a0 , b0 ) . Given the dataset D = { { ( xnm , ynm ) } Nmn=1 } Mm=1 , the joint probability of our model is given by p ( W , τ , Y , S|X ) = N ( vec ( W ) |0 , I ) M∏ m=1 Gam ( τm|a0 , b0 ) M∏ m=1 Nm∏ n=1 N ( ynm|fm ( xnm ) , τ−1m ) , ( 4 ) where τ = [ τ1 , . . . , τM ] , X = { xnm } , Y = { ynm } , and vec ( · ) is vectorization . The graphical representation of our model is given in Fig . 1 . We use Hamiltonian Monte Carlo ( HMC ) ( Neal et al. , 2011 ) sampling to perform posterior inference due to its unbiased , high-quality uncertainty quantification , which is critical to calculate the acquisition function . However , our method allows us to readily switch to other approximate inference approaches as needed ( see Sec . 4 ) , e.g. , stochastic gradient HMC ( Chen et al. , 2014 ) used in the excellent work of Springenberg et al . ( 2016 ) .

Summary: In multi-fidelity black-box optimization, f() can be queried at points x at different levels of precision. Higher levels of precision have higher costs. Prior work on Bayesopt for mfbbo used GP models for multi-fidelity observations that make simplistic assumptions about correlations across fidelities. In response, the authors propose instead using a flexible autoregressive model for modeling the sequence of fidelity scores for a given x. Bayesian inference on the model is performed using HMC.  The paper also introduces a batched form of max-value entropy search and an algorithm for optimizing this complex acquisition function that works well in practice.  The experimental results are impressive. The proposed method outperforms a large number of popular methods from the literature.