Title: LEARNING NEURAL SURROGATE MODEL FOR WARM-STARTING BAYESIAN OPTIMIZATION

Abstract: Bayesian optimization is an effective tool to optimize black-box functions and popular for hyper-parameter tuning in machine learning. Traditional Bayesian optimization methods are based on Gaussian process (GP), relying on a GP-based surrogate model for sampling points of the function of interest. In this work, we consider transferring knowledge from related problems to target problem by learning an initial surrogate model for warm-starting Bayesian optimization. We propose a neural network-based surrogate model to estimate the function mean value in GP. Then we design a novel weighted Reptile algorithm with sampling strategy to learn an initial surrogate model from meta train set. The initial surrogate model is learned to be able to well adapt to new tasks. Extensive experiments show that this warm-starting technique enables us to find better minimizer or hyper-parameters than traditional GP and previous warm-starting methods.

1. INTRODUCTION:

Bayesian optimization (Mockus, 1975; Pelikan et al., 1999; Gustavo Malkomes, 2018 ) is an optimization method to find global minimizer of a black-box function without knowing its convexity, differentiability or continuity. Bayesian optimization iteratively samples new point by an acquisition function, and uses Gaussian process (Rasmussen, 2003; Gardner et al., 2014) to estimate a surrogate model to fit f (x) based on the observed points. When using specific acquisition function, its regret bound has been proved as O( √ n) (n is the number of sampled points) in Srinivas et al. (2012) . Freitas et al. (2012) proposed a method that uses Bayesian optimization with dividing region, which successfully accelerates convergence rate. More algorithms have been developed in Wang et al. (2014) ; Kawaguchi et al. (2015) ; Springenberg et al. (2016) . In application, Bayesian optimization has been widely applied in machine learning for hyper-parameter tuning (Snoek et al., 2012) .

A machine learning problem is formulated as follows. Given data set D, model M(•, λ) and loss function L(•, •), it aims to find an optimal parameter vector w to minimize L(M(w; λ), D). In practice, model M(w; λ) is dependent on hyper-parameter λ, which may significantly influence the model. Thus choosing appropriate λ is a critical procedure of learning a model. For instance, M(w; λ) is a neural network to solve image classification problem, then λ can be the number of hidden units or network layers. Hyper-parameter optimization problem can be formulated as 

and A is an algorithm minimizing loss function w.r.t. w. It is challenging to optimize Eq. (1) since A(L(M(w; λ), D)) is a complicated function w.r.t λ, which might be non-differentiable. In real applications, hyper-parameter is commonly tuned by hand. Random search and grid search are typical methods to find plausible λ, but they are time-consuming to find the optimal hyper-parameter because they sample the hyper-parameter in a brute-force way.

Bayesian optimization is an effective tool to minimize function for global optimization or hyperparameter tuning. Bayesian optimization iteratively samples new point and fits the function values by a surrogate model based on Gaussian process. It starts with several random points without using any prior knowledge on the task. In applications, we often solve several related machine learning problems, therefore, one question is that whether we can learn common knowledge from related problems and transfer them to new problem as an initialization for adaptation, commonly referred to as warm-starting Bayesian optimization.

Meta-learning is a promising approach for learning optimizer for fast optimization or initial model for fast adaptation to novel task using a set of training problems as meta train set (Andrychowicz et al., 2016; Li & Malik, 2017; Wang et al., 2019; Ravi & Larochelle, 2017; Chen et al., 2017; Finn et al., 2017; Nichol et al., 2018) . This idea was also applied to Bayesian optimization in past years. Poloczek et al. (2016) built a joint statistical model of the collection of related objective functions. Feurer et al. (2015) initialized Bayesian optimization based on hand-crafted meta features for hyperparameter tuning. Given a new problem, its hyper-parameters are initialized by the most similar problem in meta train set with distance of hand-crafted meta features. However, it is challenging to define meta features artificially. Kim et al. (2017) learned meta features of problems by deep metric learning. Wistuba et al. (2016); Feurer et al. (2018) proposed to transfer surrogate model by linear combination of surrogate models of problems in meta train set with coefficients determined by ranking of function values or meta features. However, representation capacity of linear combination is limited, meaning that if surrogate model of a new problem cannot be represented by linear combination of surrogate models of meta train problems, then this approach will fail.

In this work, we propose a novel meta-learning approach to learn prior knowledge from meta train set as a warm-start of unseen related target problem. More specifically, we make the following two contributions to tackle the challenges in traditional Bayesian optimization.

Firstly, as we know, traditional Bayesian optimization is data-driven and depends Secondly, we propose a novel meta-learning algorithm to learn an initial neural surrogate model from meta train set of related problems, which is taken as initial surrogate model for adaptation to new problem. To train the initial neural surrogate model, meta-learning methods such as MAM-L (Finn et al., 2017) are not applicable because Bayesian optimization is non-differentiable. Thus we propose a weighted Reptile algorithm inspired by Reptile algorithm (Nichol et al., 2018) using sampling strategy for exploration to train the initial neural surrogate model.

We evaluate our method on tasks including optimization of synthetic functions, hyper-parameter tuning in evolutionary algorithm, neural network and SVM. These experiments show that our method outperforms traditional Gaussian process and related warm-starting methods in these tasks.

2. PRELIMINARIES:

2.1 BAYESIAN OPTIMIZATION Bayesian optimization is to find a minimizer x * of a function f in a bounded space by iteratively querying the function at sampled points. At n-th iteration, we first construct a probabilistic model on D = {x i , f (x i )} n i=1 for estimating function values, then use an acquisition function a(x) to select next point. We estimate f by surrogate model based on Gaussian process (GP). We assume

f (•) ∼ GP (m(•), k(•, •)), where m(•) is mean function and k(•, •) is covariance function or kernel. Given D = {x i , f (x i )} n i=1 , GP is simply a joint Gaussian f(x 1:n ) ∼ N (m(x 1:n ), K(x 1:n ))

, where K(x 1:n ) is a n×n covariance matrix with elements of k(x i , x j ) defined as kernel function measuring relation between x i and x j . We choose RBF kernel k(x i , x j ) = exp(-ρ x i -x j 2 ) in this work. Then for a new point x n+1 , we consider joint distribution of previously and newly sampled points:

EQUATION

where k(x n+1 ;

x 1:n ) = (k(x 1 , x n+1 ), k(x 2 , x n+1 ), • • • , k(x n , x n+1 )). The posterior f (x n+1 |D)

satisfies Gaussian distribution (Bishop, 2006) with mean and variance as

EQUATION

EQUATION

) which is the surrogate model of GP. Then we use acquisition function a(x) to choose which point will be evaluated next. Expected improvement (EI) (Mockus, 1975) and upper confidence bound (UCB) (Srinivas et al., 2012) have been shown to be efficient. As we aim to find minimizer of f , we minimize acquisition function: 3) and Eq. ( 4).

2.2. META-LEARNING METHODS:

MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) are two typical meta-learning methods to obtain an initialization of a model from meta train set to well adapt to new problems. Given data set D, model M(w) and loss function L(•, •), MAML learns initialization w * by minimizing L (M(w -α∇L(M(w), D)), D). It implies that MAML learns a w * that can fast adapt by one gradient descent step. Optimization in MAML uses second-order derivatives of L(w). In our application, Bayesian optimization procedures are non-differentiable. Therefore, MAML can not be applied to our task directly. Reptile is one of first-order meta-learning algorithms. As illustrated in Alg. 1, to obtain an optimal initialization of model parameters w that can adapt well to new problem τ by k-steps of optimization denoted by U k τ (w), the parameter w is iteratively updated by vector ( w -w) that enforces w to be close to the updated parameters w of different tasks. In this work, we borrow idea from Reptile, but design a weighted Reptile algorithm for learning initial network-based surrogate model. Update w ← w + ( w -w) 6: end for

3. METHOD:

We aim to learn to warm-starting Bayesian optimization by substituting mean in Gaussian process by a network-based surrogate model φ(w, x). Then we learn an initialization of the neural surrogate model from a meta train set of related problems using a weighted Reptile algorithm, for adaptation to new problem. We first present some key definitions.

Neural surrogate model. The surrogate model of Bayesian optimization is Eqs. ( 3) and ( 4), where mean in Eq. ( 3) is the expectation of predicted function value and variance in Eq. ( 4) represents uncertainty. Since related problems in meta train set may have related shapes of function surfaces, we substitute the mean by a learnable network φ(w, x) to learn initial surrogate model from meta train set. We still use the variance in Eq. ( 4) to measure uncertainty, because the estimation uncertainty should be independent in individual problems. Using network for modeling surrogate model, we learn an initial network φ(w * , x) from meta train set that can be adapted to new problem. Our neural surrogate model is just the combination of φ(w, x) and variance in Eq. ( 4).

Neural optimization episode. An optimization episode is the iterative process for optimizing the black-box function. For a problem Q with corresponding black-box function f , in (t + 1)-th step, we already have surrogate network φ(w t , x) and observed points set D = {x j , f (x j )} t j=1 . First, we update neural surrogate model using L Rank (will be discussed in sect. 3.1) and Eq. ( 4), and obtain φ(w t+1 , x) and σ 2 (x). Then we use acquisition function of Eq. ( 5) to choose next point x t+1 . After that, we update points set D = D {x t+1 , f (x t+1 )}. We iterate the above process to obtain x N T best = arg min {xn,n=1,...,N T } {f (x n )}. We called this procedure as Neural optimization episode (NOE), as illustrated in the right part of Fig. 1 .

In this work, we aim to learn initial φ(w, x) to initialize NOE of a new problem given a set of related problems in meta train set, as illustrated in Fig. 1 . Since the dynamics of NOE is complicated and might be non-differentiable, we propose a weighted Reptile Algorithm to learn the initial φ(w, x) from meta train set using reward defined based on x N T best of problems. This learned initial model φ(w, x) is utilized to initialize the neural surrogate model of a new problem for its NOE.

3.1. RANKING LOSS FOR LEARNING NEURAL SURROGATE MODEL:

We now focus on how to update neural surrogate model given observed points and their function values in neural optimization episode. As we know, different problems have different ranges of function values. To make surrogate model insensitive to the specific range of function values, our neural surrogate model is learned to predict ranking of function values of points instead of exactly fitting the function values. Describing data sets based on pairwise rankings has been investigated in a few machine learning approaches (Leite et al., 2012; Wistuba et al., 2015; Zheng et al., 2007) .

Given points set D = {x i , f (x i )} n i=1

, we update w of neural surrogate model φ(w, •) by L Rank . First we sort points in D by {f (x i )} n i=1 in descent order and get

D s = {x r(i) , f (x r(i) )} n i=1 , where r(i), i = 1 • • • n is a permutation of 1 • • • n.

Then we utilize L Rank as loss:

EQUATION

)

L Rank is based on hinge loss, and minimization of L Rank w.r.t. w enforces that φ(w, x r(i) ) is strictly larger than φ(w, x r(j) ) with margin of δ. Note that final layer of our surrogate model φ is a sigmoid function, therefore, the output of surrogate model is constrained in range of (0, 1) irrespective of different problems. We use gradient descent method to optimize L Rank , please see appendix A.1 for detail. We denote GD(w, L Rank ) as the learned surrogate model parameters by minimizing L Rank using gradient descent method. This above process is taken as the updating step of neural surrogate model in neural optimization episode.

3.2. WEIGHTED REPTILE ALGORITHM FOR LEARNING INITIAL SURROGATE MODEL:

We next focus on how to learn an initial neural surrogate model from a set of related problems in meta train set, taken as the initialization of neural surrogate model for NOE of a new problem. We propose a weighted Reptile algorithm to train the initial neural surrogate model.

Reptile in Alg. 1 is an effective meta-learning algorithm that can learn a good initialization of model parameter from a meta train set. Inspired by Reptile, our Weighted Reptile Algorithm samples a set of initializations of surrogate model parameters for NOE of each problem in the meta train set for exploration, and these sampled initializations are selected by reward and utilized for parameter updating with weights defined by rewards.

Reward. Given minimizer x 1 = arg min x φ(w 1 , x) 3: y 1 = f (x 1 ), D = D {x 1 , y 1 } 4: Update σ(x) by Eq. ( 4) 5: for t = 1 : N T -1 do 6:

x * i of i-th problem Q i (x) in meta train set, reward at step t of NOE of problem Q i (x) is defined as r t i . = -x t i,best -x * i 2 ,

x t+1 = arg min φ(w t , x) -κσ(x) 7:

y t+1 = f (x t+1 ), D = D {x t+1 , y t+1 } 8: w t+1 = GD(w t , L Rank ) 9:

Update σ(x) by Eq. ( 4) 10: end for 11:

x best = x t * , t * = arg min t {y t } N T i=t 12: return x best , w N T f i is the black-box function in problem Q i (x), x n is sampled point at n-th step, x t

i,best is the best point up to step t.

Weighted Reptile Algorithm. In Reptile, as in line 5 of Alg. 1, the parameter w is learned to be close to updated parameter w by adapting w in different problems. It implicitly assumes that updated parameter w should be better than initialized w for corresponding problem. In our approach, we aim to learn initial surrogate model φ(w, x) from problems in meta train set. Similar to Reptile, we learn initial network parameter w of φ(w, x) close to updated parameters after running NOE of problems in meta train set. To explore diverse paths of NOE, for each i-th problem in meta train set, we sample a few (M ) initial parameters {w p i } M p=1 ∼ N (w, σ 2 ) by disturbing currently estimated w. Then after running N T steps of NOE, we obtain points {x N T ,p i,best } M p=1 and updated parameters { wN T ,p i } M p=1 starting from surrogate models initialized with these sampled parameters. To learn a good initial parameter w of surrogate model φ, parameter is updated by w = w + β∆w, where

EQUATION

and reward r

N T i = -x N T i,best -x * i 2 , r N T i,p = -x N T ,p i,best -x * i 2 .

For i-th problem and p-th sampled parameter, weight max(r N T i,p -r N T i , 0) is always non-negative, and it is non-zero only when the sampled parameter w p i can produce better reward than the parameter w itself if taking them as initial surrogate model parameters for running NOE of problem Q i . This rule enforces that the parameter w of our learned initial surrogate model is close to better adapted parameters of different problems in meta train set. Algorithm 3 presents the pipeline of meta-train process using weighted Reptile algorithm, and it depends on the NOE as described in Alg. 2 .

4. EXPERIMENTS:

We evaluate our method for global optimization on synthetic functions set, and hyper-parameters tuning in evolutionary algorithm, neural network and SVM. In the following paragraphs, our method based on neural surrogate model and weighted Reptile algorithm will be dubbed as WRA-N. In all experiments, we set κ = 0.1, δ = 0.01, σ = 0.1, training epoch N E = 30, learning rate β = 0.1, number of samples M = 50. We use the mean squared error (MSE) between function values of our found best point and the global minimizer to measure the performance of optimization. We mainly compare with TST-R (Wistuba et al., 2016) and GP-UCB (Srinivas et al., 2012) . Moreover, we also compare with meta feature-based method TST-M (Wistuba et al., 2016) in SVM hyper-parameter tuning. But for the remaining tasks, we did not compare with it because it is hard to well define meta-features for these problems. The neural surrogate model is taken as a network with one hidden layer of 15 hidden units, and both hidden and output layers are with sigmoid nonlinearity. #Surrogate model adaptation loop.

5::

for i = 1 : |Q| do 6:

{x N T i,best , w N T i } = NOE(N T , κ, Q i , w l ) 7: r N T i = -x N T i,best -x * i 2 8:

for p = 1 : M do 9:

set w p i = w l + N (0, σ 2 I) 10:

{x N T ,p i,best , wN T ,p i } = NOE(N T , κ, Q i , w p i ) 11: r N T i,p = -x N T ,p i,best -x * i 2 , g p i = wN T ,p i -w l 12:

end for 13:

end for 14:

#Updating initial surrogate model parameter.

15: 

w l+1 = w l + β |Q| i=1 M p=1 max(r N T i,p -r N T i , 0)g p

4.1. SYNTHETIC FUNCTIONS:

We first evaluate on Alpine 1 function in Jamil & Yang (2013) without using absolute value: -10, 10] . We set functions with a = 4π 3 , 5π 4 , 6π 5 , 7π 6 as meta train set, and 18 functions with a = 3π 2 -iπ 24 , i ∈ {1, ..., 21} excluding {4, 6, 8} in meta train set for testing. Figure 2 shows logarithm of MSEs w.r.t. optimization iterations of WRA-N, TST-R and GP-UCB for a = 3π 2 , π, 2π 3 , and other results are in appendix. Optimization iteration means the iteration in optimization episode, e.g., the iterations in NOE of our WRA-N. Since TST-R needs base models for combination, we sample 20 points from uniform distribution in (-10, 10) to construct base models. TST-R and GP-UCB start with 3 random points, then run in 10 iterations. Our WRA-N starts with neural surrogate model learned by meta train set, and runs in 13 iterations, i.e., N T = 13. Optimization was repeated 20 times for TST-R and GP-UCB. For TST-R, each time is with differently sampled points for constructing base models. Figure 2 shows curves of average MSEs with variances in logarithm w.r.t. iterations. WRA-N starts with learned initial surrogate model, therefore curves are deterministic. WRA-N significantly improves baseline GP-UCB and outperforms TST-R, because test functions may not be well represented by base functions in TST-R.

f (x) = xsin(x + a) + 0.1x, x ∈ [

4.2. HYPER-PARAMETERS TUNING IN DIFFERENTIAL EVOLUTIONARY ALGORITHM:

Differential evolution (DE) (Galletly, 1996; Das & Suganthan, 2011) 

[x 0 1 , x 0 2 , • • • , x 0 Np ] ∈ R d×Np , and iterates u k+1 i = x k i + F (x k best -x k i ) + F (x k r(1) -x k r(2) )

, where x k r(i) (i = 1, 2) are randomly chosen from X k , x k best is the best point in k-th iteration, F is hyperparameter. Then we can use binomial crossover and selection methods to create X k+1 . We generalize this iteration formulation to be

u k+1 i = x k i + F (x k best -x k i ) + D(x k r(1) -x k r(2) )

using 2 hyper-parameters F and D, which is a general formulation covering most DE formulations. F and D influence optimization performance of DE (e.g., finding minimum of functions by DE) significantly, and thus we can utilize our method to tune these hyper-parameters. We use function set of CEC 2017 (Awad et al., 2016) for optimization by DE and use our method to find optimal hyperparameters. The neural surrogate model is to predict the function values w.r.t. hyper-parameters. We set function dimension as 10, construct meta train set containing basic functions of f 1 , f 2 , f 3 , f 4 , and learn initial surrogate model that can find optimal hyper-parameters well in f 5 -f 10 for testing. The formulas of these functions are shown in appendix. For TST-R, we sample 20 points uniformly in feasible domain and use these points to construct base models. TST-R and GP-UCB start with 3 randomly chosen points, and then proceed sequentially for 20 iterations. Our method starts with surrogate model learned by meta train set, and proceeds for a total of 23 iterations. The optimization was repeated 20 times similar to the experiments on synthetic function set. Figure 3 shows results on f 5 and f 6 , and results on the other functions are shown in appendix. Our method produces best hyper-parameter tuning performance. Note that the functions in meta train set are in different scales of function values. For TST-R, we follow its original implementation by normalizing point values of base models to normal distribution. However, in our approach, the neural surrogate model is updated using ranking-based loss, therefore it is not sensitive to the different scales of functions.

4.3. HYPER-PARAMETERS TUNING IN NEURAL NETWORK:

We will optimize hyper-parameters of neural networks for image classification. We use EMNIST data set (Cohen et al., 2017) , i.e., an extension of MNIST to handwritten letters. There are 62 classes of [0-9], [a-z] and [A-Z], and several classes that are similar between uppercase and lowercase are merged, and finally we have 47 classes in data set. The network is with one hidden layer, sigmoid as activation function and softmax for classification, followed by cross-entropy loss. We first randomly sample 10 classes in 3 times from EMNIST with 1000 images for each class, Note that we can also use other criterion for learning hyper-parameters. Figure 4 shows MSEs (between ERRs of best hyper-parameters and estimated hyper-parameters) w.r.t. optimization iterations by different methods. We can observe that our method performs best for this task.

4.4. HYPER-PARAMETERS TUNING IN SVM:

Support vector machine (SVM) dataset (Wistuba et al., 2016) consists of 50 classification datasets randomly chosen from UCI repository. The SVMs use Gaussian kernel with hyper-parameters of trade-off parameter C and width γ of Gaussian kernel. The candidate hyper-parameters are C ∈ {2 -5 , ..., 2 6 } and γ ∈ {10 -4 , 10 -3 , 10 -2 , 0.05, 0.1, 0.5, 1, 2, 5, 10, 20, 50, 10 2 , 10 3 }. The neural surrogate model is to predict misclassification rate w.r.t. to hyper-parameters C, γ. In experiments, we randomly choose 5 datasets as meta train sets, and the others as test sets. All the experiments are performed in 10 times. Figure 5 shows the average MSEs w.r.t. optimization iterations by different methods. Here we compare our WRA-N with TST-M (Wistuba et al., 2016) , which also relies on linear combination of surrogate models of meta train problems but using meta features. We can see that our WRA-N performs better than both TST-R and TST-M. Both our WRA-N and TST-R perform significantly better than baseline GP-UCB.

4.5. ABLATION STUDY:

We now justify effectiveness of weighted Reptile algorithm (WRA) and different network structures of surrogate model φ. All following networks are evaluated on synthetic set in section 4.1. In Fig. 6 (left), we compare our WRA with traditional Reptile algorithm without using sampling and weighting strategy. The average MSE curves show that our WRA performs better than Reptile. We further compare different structures of surrogate models including networks with one hidden layer of 10, 15, 25 units and 2 hidden layers with 15 units. We observe that they perform similarly for networks with one hidden layer, and adding more layers does not apparently improve performance.

5. CONCLUSION:

We have proposed a novel neural surrogate model and a weighted Reptile algorithm to learn an initial neural surrogate model for warm-starting Bayesian optimization. Extensive experiments on synthetic functions optimization and hyper-parameters tuning in multiple tasks have shown promising performance. In the future, we are interested to apply it to more global optimization or hyperparameters tuning tasks, and analyze its theoretical properties.

A.2 FUNCTIONS IN CEC 2017 FOR DE HYPER-PARAMETERS TUNING:

The ten functions are listed as follows.

f 1 (x) = x 2 1 + 10 6 H i=2 x i 2 , f 2 (x) = H i=1 |x i | i+1 f 3 (x) = H i=1 x i 2 + (0.5 H i=1 x i ) 4 + (0.5 H i=1 x i ) 4 f 4 (x) = H-1 i=1 (100(x i 2 -x i+1 ) 2 + (x i -1) 2 ) f 5 (x) = H i=1 (x i 2 -10cos(2πx i ) + 10) f 6 (x) = g(x 1 , x 2 ) + g(x 2 , x 3 ) + • • • + g(x H , x 1 ) f 7 (x) = min( H i=1 ( x i -µ 0 ) 2 , dH + s H i=1 ( x i -µ 1 ) 2 ) + 10(H - H i=1 cos(2π z i )) f 8 (x) = H i=1 (z i 2 -10cos(2πz i ) + 10) f 9 (x) = sin 2 (πw 1 ) + H-1 i=1 (w i -1) 2 [1 + 10sin 2 (πw i + 1)] + (w H -1) 2 [1 + sin 2 (2πw H )] f 10 (x) = 418.9829H - H i=1 g(z i )

g(x, y) in f 6 is defined as g(x, y) = 0.5 + sin 2 ( √ x 2 +y 2 )-0.5 (1+0.001(x 2 +y 2 )) 2 . Please refer to CEC 2017 (Awad et al., 2016) for details on parameters.

A.3 ADDITIONAL RESULTS IN EXPERIMENTS:

Figure 7 shows the function optimization results on 15 remaining test functions in synthetic function set. Figure 8 shows curves of average MSEs w.r.t. different optimization iterations for tuning hyper-parameters of number of hidden units and learning rate of SGD on EMNIST data sets of S 7 , S 8 , S 9 , S 10 , S 11 . Figure 9 shows these curves for tuning hyper-parameters in DE on functions of f 7 -f 10 in CEC 2017. 

A.4 AN ILLUSTRATION OF NEURAL SURROGATE MODEL BY MINIMIZING L Rank:



:



Figures and Tables:

w; λ), D) = min λ L(M(w * ; λ), D), where w * = A(L(M(w; λ), D)),

Figure 1: General pipeline of learning initial neural surrogate model using weighted Reptile algorithm from meta train set of functions f i , i = 1, • • • , N .

) = arg min x {µ(x) -κσ(x)} (5) to find next point. As we can see, GP estimates function value via given data D based on surrogate model of Eq. (

w = U k τ (w), i.e., k steps optimization by SGD or Adam 5:

Meta train by weighted Reptile algorithm. Require: Number of iterations N T in NOE, κ > 0, meta train set Q, number of epochs N E , learning rate β, number of samples M , standard deviation σ for sampling. Ensure: w * 1: Randomly initialize w 1 2: #Meta-train loop. 3: for l = 1 : N E do 4:

Figure 2: MSEs w.r.t. optimization iterations on synthetic functions with a = 3π 2 , π, 2π 3 .

Figure 3: MSEs w.r.t. iterations for f 5 , f 6 in CEC 2017 for hyper-parameters tuning of DE.

Figure 4: MSEs w.r.t. optimization iterations for hyper-parameters tuning of neural networks on data sets of S 4 , S 5 , S 6 .

Figure 5: MSEs w.r.t. iterations for hyper-parameters tuning of SVMs.

Figure 6: Ablation study.

Figure 10 shows the neural surrogate model by minimizing L Rank working on DE hyper-parameters tuning. In this task, we use neural surrogate model to predict the mapping from hyper-parameters D, F to function values of points. Figure 10(a) shows the surrogate model learned by neural network. Figure 10(b) shows the real function values w.r.t. hyper parameters. Figure 10(c) shows the logarithm of real function values w.r.t. hyper-parameters. Then we see that the neural surrogate model is within range of [0, 1] and it has similar local minimum with real function (by comparing Figs. 10(a) and (c)), but it is more smooth than real function surface. We further illustrate the distributions using histograms of function values of real function and neural surrogate model. As shown in Fig.10 (d),x-axis is the percentage (i.e. ratio) of maximal function value, y-axis is the percentage of number of data with function values falling in these ratios to maximal function values. We see that, for real function, most values are less than 0.25 of maximum function value, while our neural surrogate model is more uniform.

Figure 7: MSEs (in logarithm) w.r.t. different optimization iterations on 15 remaining test functions in synthetic function set.

Figure 8: MSEs (in logarithm) w.r.t. different optimization iterations for tuning number of hidden units and learning rate on EMNIST data sets of S 7 , S 8 , S 9 , S 10 , S 11 .

Figure 9: MSEs (in logarithm) w.r.t. different optimization iterations for tuning hyper-parameters of DE on f 7 -f 10 of CEC 2017.

Figure 10: An Illustration of neural surrogate model for hyper-parameter tuning of DE.

Figure 11: Training curves of rewards on meta train set respectively for our four different experiments in sections 4.1-4.4.

Neural optimization episode (NOE).Require: Maximum iterations N T , κ > 0, test problem Q with function f , initialized parameter w Ensure: x best , w N T 1: set w 1 = w, D = {} 2:

Table content: <table/>

Peer review (iclr2020):
Summary of the paper: The authors propose a meta-learning approach for BO. The method consists in using a NN as the predictive mean of the model used to guide the search in BO. This NN is initialized cleverly so that its solutions is close to the actual solution to the problem by using related optimization tasks. The proposed method is validated on several experiments. Detailed comments: The writing of the paper needs to be improved. It has awkward sentences like *Bayesian optimization iteratively samples new point by*. The intro on Bayesian optimization has to be improved, it explain very poorly this technique. The description of the MAML method is not clear. The same for the reptile algorithm. Eq. (3) is wrong. It does not take into account that the mean of the GP is different from 0. It seems the authors replace the mean of the GP predictive distribution with the output of a neural network. This is strange and not very well justified. I would have expected that they use the output of the NN as the GP prior mean, to then compute the GP posterior mean. In the related problems the actual objective is unknown. How is that difficulty addressed? In Section 3.1, why do not you standardize the output values to have zero mean and unit deviation instead of using the ranking? How are the hyper-parameters of the GP tuned? It seems GP based method will fail, essentially, because the authors do not consider the posterior distribution of the GP and change the mean of the predictive distribution to be the NN output. There are no error bars for WRA-N in the experiments. Does this means that only one realization has been carried out? If so, this is insufficient to extract any conclusion. The results can be simply obtained by chance. The authors should consider several repetitions with different random seeds or different problems. The experiments in section 4.1 are non realistic, since the actual shape of the objective function is known beforehand. In section 4.2 it is not clear what relation are between the train / test functions. Therefore, it is not possible to understand why the proposed approach works better. The experiments need another baseline to compare with. Namely, the same method in which the NN is randomly initialized. This will allow to check that the meta-learning procedure is useful. Currently, it can be the case that the improvements are simply due to using a different model for optimization. Summing up, I think that: (1) This paper needs further improvement in the writing. (2) The experimental section is questionable since there are missing methods in the comparison and no error bars in the experiments. Therefore I believe that this paper is still at an early stage and not ready for publication.