Learning Neural Surrogate Model for Warm-Starting Bayesian Optimization

1 INTRODUCTION . Bayesian optimization ( Mockus , 1975 ; Pelikan et al. , 1999 ; Gustavo Malkomes , 2018 ) is an optimization method to find global minimizer of a black-box function without knowing its convexity , differentiability or continuity . Bayesian optimization iteratively samples new point by an acquisition function , and uses Gaussian process ( Rasmussen , 2003 ; Gardner et al. , 2014 ) to estimate a surrogate model to fit f ( x ) based on the observed points . When using specific acquisition function , its regret bound has been proved as O ( √ n ) ( n is the number of sampled points ) in Srinivas et al . ( 2012 ) . Freitas et al . ( 2012 ) proposed a method that uses Bayesian optimization with dividing region , which successfully accelerates convergence rate . More algorithms have been developed in Wang et al . ( 2014 ) ; Kawaguchi et al . ( 2015 ) ; Springenberg et al . ( 2016 ) . In application , Bayesian optimization has been widely applied in machine learning for hyper-parameter tuning ( Snoek et al. , 2012 ) . A machine learning problem is formulated as follows . Given data set D , model M ( · , λ ) and loss function L ( · , · ) , it aims to find an optimal parameter vector w to minimize L ( M ( w ; λ ) , D ) . In practice , model M ( w ; λ ) is dependent on hyper-parameter λ , which may significantly influence the model . Thus choosing appropriate λ is a critical procedure of learning a model . For instance , M ( w ; λ ) is a neural network to solve image classification problem , then λ can be the number of hidden units or network layers . Hyper-parameter optimization problem can be formulated as min λ , w L ( M ( w ; λ ) , D ) = min λ L ( M ( w∗ ; λ ) , D ) , where w∗ = A ( L ( M ( w ; λ ) , D ) ) , ( 1 ) and A is an algorithm minimizing loss function w.r.t . w. It is challenging to optimize Eq . ( 1 ) since A ( L ( M ( w ; λ ) , D ) ) is a complicated function w.r.t λ , which might be non-differentiable . In real applications , hyper-parameter is commonly tuned by hand . Random search and grid search are typical methods to find plausible λ , but they are time-consuming to find the optimal hyper-parameter because they sample the hyper-parameter in a brute-force way . Bayesian optimization is an effective tool to minimize function for global optimization or hyperparameter tuning . Bayesian optimization iteratively samples new point and fits the function values by a surrogate model based on Gaussian process . It starts with several random points without using any prior knowledge on the task . In applications , we often solve several related machine learning problems , therefore , one question is that whether we can learn common knowledge from related problems and transfer them to new problem as an initialization for adaptation , commonly referred to as warm-starting Bayesian optimization . Meta-learning is a promising approach for learning optimizer for fast optimization or initial model for fast adaptation to novel task using a set of training problems as meta train set ( Andrychowicz et al. , 2016 ; Li & Malik , 2017 ; Wang et al. , 2019 ; Ravi & Larochelle , 2017 ; Chen et al. , 2017 ; Finn et al. , 2017 ; Nichol et al. , 2018 ) . This idea was also applied to Bayesian optimization in past years . Poloczek et al . ( 2016 ) built a joint statistical model of the collection of related objective functions . Feurer et al . ( 2015 ) initialized Bayesian optimization based on hand-crafted meta features for hyperparameter tuning . Given a new problem , its hyper-parameters are initialized by the most similar problem in meta train set with distance of hand-crafted meta features . However , it is challenging to define meta features artificially . Kim et al . ( 2017 ) learned meta features of problems by deep metric learning . Wistuba et al . ( 2016 ) ; Feurer et al . ( 2018 ) proposed to transfer surrogate model by linear combination of surrogate models of problems in meta train set with coefficients determined by ranking of function values or meta features . However , representation capacity of linear combination is limited , meaning that if surrogate model of a new problem can not be represented by linear combination of surrogate models of meta train problems , then this approach will fail . In this work , we propose a novel meta-learning approach to learn prior knowledge from meta train set as a warm-start of unseen related target problem . More specifically , we make the following two contributions to tackle the challenges in traditional Bayesian optimization . Firstly , as we know , traditional Bayesian optimization is data-driven and depends on a GP-based surrogate model fitting function values without learnable parameters , limiting its ability for adaptation to new problems . We propose to use neural network to model the surrogate model ( dubbed as neural surrogate model ) to substitute the mean of function values in GP . We further design a ranking loss for fitting surrogate model to sampled points to make surrogate model insensitive to the scale of function values . Secondly , we propose a novel meta-learning algorithm to learn an initial neural surrogate model from meta train set of related problems , which is taken as initial surrogate model for adaptation to new problem . To train the initial neural surrogate model , meta-learning methods such as MAML ( Finn et al. , 2017 ) are not applicable because Bayesian optimization is non-differentiable . Thus we propose a weighted Reptile algorithm inspired by Reptile algorithm ( Nichol et al. , 2018 ) using sampling strategy for exploration to train the initial neural surrogate model . We evaluate our method on tasks including optimization of synthetic functions , hyper-parameter tuning in evolutionary algorithm , neural network and SVM . These experiments show that our method outperforms traditional Gaussian process and related warm-starting methods in these tasks . 2 PRELIMINARIES . 2.1 BAYESIAN OPTIMIZATION . Bayesian optimization is to find a minimizer x∗ of a function f in a bounded space by iteratively querying the function at sampled points . At n-th iteration , we first construct a probabilistic model on D = { xi , f ( xi ) } ni=1 for estimating function values , then use an acquisition function a ( x ) to select next point . We estimate f by surrogate model based on Gaussian process ( GP ) . We assume f ( · ) ∼ GP ( m ( · ) , k ( · , · ) ) , where m ( · ) is mean function and k ( · , · ) is covariance function or kernel . Given D = { xi , f ( xi ) } ni=1 , GP is simply a joint Gaussian f ( x1 : n ) ∼ N ( m ( x1 : n ) , K ( x1 : n ) ) , where K ( x1 : n ) is a n×n covariance matrix with elements of k ( xi , xj ) defined as kernel function measuring relation between xi and xj . We choose RBF kernel k ( xi , xj ) = exp ( −ρ‖xi − xj‖2 ) in this work . Then for a new point xn+1 , we consider joint distribution of previously and newly sampled points : ( f ( x1 : n ) f ( xn+1 ) ) ∼ N ( ( m ( x1 : n ) m ( xn+1 ) ) , ( K ( x1 : n ) k ( xn+1 ; x1 : n ) k ( xn+1 ; x1 : n ) ᵀ k ( xn+1 , xn+1 ) ) ) , ( 2 ) where k ( xn+1 ; x1 : n ) = ( k ( x1 , xn+1 ) , k ( x2 , xn+1 ) , · · · , k ( xn , xn+1 ) ) . The posterior f ( xn+1|D ) satisfies Gaussian distribution ( Bishop , 2006 ) with mean and variance as µ ( xn+1|D ) = k ( xn+1 ; x1 : n ) ᵀK ( x1 : n ) −1f ( x1 : n ) , ( 3 ) σ2 ( xn+1|D ) = k ( xn+1 , xn+1 ) − k ( xn+1 ; x1 : n ) ᵀK ( x1 : n ) −1k ( xn+1 ; x1 : n ) , ( 4 ) which is the surrogate model of GP . Then we use acquisition function a ( x ) to choose which point will be evaluated next . Expected improvement ( EI ) ( Mockus , 1975 ) and upper confidence bound ( UCB ) ( Srinivas et al. , 2012 ) have been shown to be efficient . As we aim to find minimizer of f , we minimize acquisition function : arg min x a ( x ) = arg min x { µ ( x ) − κσ ( x ) } ( 5 ) to find next point . As we can see , GP estimates function value via given data D based on surrogate model of Eq . ( 3 ) and Eq . ( 4 ) . 2.2 META-LEARNING METHODS . MAML ( Finn et al. , 2017 ) and Reptile ( Nichol et al. , 2018 ) are two typical meta-learning methods to obtain an initialization of a model from meta train set to well adapt to new problems . Given data set D , modelM ( w ) and loss function L ( · , · ) , MAML learns initialization w∗ by minimizing L ( M ( w − α∇L ( M ( w ) , D ) ) , D ) . It implies that MAML learns a w∗ that can fast adapt by one gradient descent step . Optimization in MAML uses second-order derivatives of L ( w ) . In our application , Bayesian optimization procedures are non-differentiable . Therefore , MAML can not be applied to our task directly . Reptile is one of first-order meta-learning algorithms . As illustrated in Alg . 1 , to obtain an optimal initialization of model parameters w that can adapt well to new problem τ by k-steps of optimization denoted by Ukτ ( w ) , the parameter w is iteratively updated by vector ( w̃−w ) that enforces w to be close to the updated parameters w̃ of different tasks . In this work , we borrow idea from Reptile , but design a weighted Reptile algorithm for learning initial network-based surrogate model . Algorithm 1 Reptile 1 : Initialize w , i.e. , the initial parameter 2 : for i = 1 , 2 , · · · , n do 3 : Sample task τ 4 : Compute w̃ = Ukτ ( w ) , i.e. , k steps optimization by SGD or Adam 5 : Update w ← w + ( w̃ − w ) 6 : end for 3 METHOD . We aim to learn to warm-starting Bayesian optimization by substituting mean in Gaussian process by a network-based surrogate model φ ( w , x ) . Then we learn an initialization of the neural surrogate model from a meta train set of related problems using a weighted Reptile algorithm , for adaptation to new problem . We first present some key definitions . Neural surrogate model . The surrogate model of Bayesian optimization is Eqs . ( 3 ) and ( 4 ) , where mean in Eq . ( 3 ) is the expectation of predicted function value and variance in Eq . ( 4 ) represents uncertainty . Since related problems in meta train set may have related shapes of function surfaces , we substitute the mean by a learnable network φ ( w , x ) to learn initial surrogate model from meta train set . We still use the variance in Eq . ( 4 ) to measure uncertainty , because the estimation uncertainty should be independent in individual problems . Using network for modeling surrogate model , we learn an initial network φ ( w∗ , x ) from meta train set that can be adapted to new problem . Our neural surrogate model is just the combination of φ ( w , x ) and variance in Eq . ( 4 ) . Neural optimization episode . An optimization episode is the iterative process for optimizing the black-box function . For a problem Q with corresponding black-box function f , in ( t + 1 ) -th step , we already have surrogate network φ ( wt , x ) and observed points set D = { xj , f ( xj ) } tj=1 . First , we update neural surrogate model using LRank ( will be discussed in sect . 3.1 ) and Eq . ( 4 ) , and obtain φ ( wt+1 , x ) and σ2 ( x ) . Then we use acquisition function of Eq . ( 5 ) to choose next point xt+1 . After that , we update points set D = D ⋃ { xt+1 , f ( xt+1 ) } . We iterate the above process to obtain xNTbest = arg min { xn , n=1 , ... , NT } { f ( xn ) } . We called this procedure as Neural optimization episode ( NOE ) , as illustrated in the right part of Fig . 1 . In this work , we aim to learn initial φ ( w , x ) to initialize NOE of a new problem given a set of related problems in meta train set , as illustrated in Fig . 1 . Since the dynamics of NOE is complicated and might be non-differentiable , we propose a weighted Reptile Algorithm to learn the initial φ ( w , x ) from meta train set using reward defined based on xNTbest of problems . This learned initial model φ ( w , x ) is utilized to initialize the neural surrogate model of a new problem for its NOE .

Summary: Learning from past experience to quickly adapt to a new task has been an important and fast-growing issue in machine learning. Such technique facilitates Bayesian optimization as well, warm-starting Bayesian optimization. Recently a few methods have been developed along this direction, from designing handcrafted meta-features to learning meta-features. The current paper takes a similar step, learning neural surrogate model from related tasks to warm-start Bayesian optimization. The main idea is to replace the mean function of GP by neural surrogate model, so that parameterized models are used for meta-training, in the framework of RETILE. An idea of weighted REPTILE is another contribution in this paper, where parameter updates are done with weighs defined by rewards.