Title: LEARNING NEURAL SURROGATE MODEL FOR WARM-STARTING BAYESIAN OPTIMIZATION

Abstract: Bayesian optimization is an effective tool to optimize black-box functions and popular for hyper-parameter tuning in machine learning. Traditional Bayesian optimization methods are based on Gaussian process (GP), relying on a GP-based surrogate model for sampling points of the function of interest. In this work, we consider transferring knowledge from related problems to target problem by learning an initial surrogate model for warm-starting Bayesian optimization. We propose a neural network-based surrogate model to estimate the function mean value in GP. Then we design a novel weighted Reptile algorithm with sampling strategy to learn an initial surrogate model from meta train set. The initial surrogate model is learned to be able to well adapt to new tasks. Extensive experiments show that this warm-starting technique enables us to find better minimizer or hyper-parameters than traditional GP and previous warm-starting methods.

1. INTRODUCTION:

Bayesian optimization (Mockus, 1975; Pelikan et al., 1999; Gustavo Malkomes, 2018 ) is an optimization method to find global minimizer of a black-box function without knowing its convexity, differentiability or continuity. Bayesian optimization iteratively samples new point by an acquisition function, and uses Gaussian process (Rasmussen, 2003; Gardner et al., 2014) to estimate a surrogate model to fit f (x) based on the observed points. When using specific acquisition function, its regret bound has been proved as O( √ n) (n is the number of sampled points) in Srinivas et al. (2012) . Freitas et al. (2012) proposed a method that uses Bayesian optimization with dividing region, which successfully accelerates convergence rate. More algorithms have been developed in Wang et al. (2014) ; Kawaguchi et al. (2015) ; Springenberg et al. (2016) . In application, Bayesian optimization has been widely applied in machine learning for hyper-parameter tuning (Snoek et al., 2012) .

A machine learning problem is formulated as follows. Given data set D, model M(•, λ) and loss function L(•, •), it aims to find an optimal parameter vector w to minimize L(M(w; λ), D). In practice, model M(w; λ) is dependent on hyper-parameter λ, which may significantly influence the model. Thus choosing appropriate λ is a critical procedure of learning a model. For instance, M(w; λ) is a neural network to solve image classification problem, then λ can be the number of hidden units or network layers. Hyper-parameter optimization problem can be formulated as 

and A is an algorithm minimizing loss function w.r.t. w. It is challenging to optimize Eq. (1) since A(L(M(w; λ), D)) is a complicated function w.r.t λ, which might be non-differentiable. In real applications, hyper-parameter is commonly tuned by hand. Random search and grid search are typical methods to find plausible λ, but they are time-consuming to find the optimal hyper-parameter because they sample the hyper-parameter in a brute-force way.

Bayesian optimization is an effective tool to minimize function for global optimization or hyperparameter tuning. Bayesian optimization iteratively samples new point and fits the function values by a surrogate model based on Gaussian process. It starts with several random points without using any prior knowledge on the task. In applications, we often solve several related machine learning problems, therefore, one question is that whether we can learn common knowledge from related problems and transfer them to new problem as an initialization for adaptation, commonly referred to as warm-starting Bayesian optimization.

Meta-learning is a promising approach for learning optimizer for fast optimization or initial model for fast adaptation to novel task using a set of training problems as meta train set (Andrychowicz et al., 2016; Li & Malik, 2017; Wang et al., 2019; Ravi & Larochelle, 2017; Chen et al., 2017; Finn et al., 2017; Nichol et al., 2018) . This idea was also applied to Bayesian optimization in past years. Poloczek et al. (2016) built a joint statistical model of the collection of related objective functions. Feurer et al. (2015) initialized Bayesian optimization based on hand-crafted meta features for hyperparameter tuning. Given a new problem, its hyper-parameters are initialized by the most similar problem in meta train set with distance of hand-crafted meta features. However, it is challenging to define meta features artificially. Kim et al. (2017) learned meta features of problems by deep metric learning. Wistuba et al. (2016); Feurer et al. (2018) proposed to transfer surrogate model by linear combination of surrogate models of problems in meta train set with coefficients determined by ranking of function values or meta features. However, representation capacity of linear combination is limited, meaning that if surrogate model of a new problem cannot be represented by linear combination of surrogate models of meta train problems, then this approach will fail.

In this work, we propose a novel meta-learning approach to learn prior knowledge from meta train set as a warm-start of unseen related target problem. More specifically, we make the following two contributions to tackle the challenges in traditional Bayesian optimization.

Firstly, as we know, traditional Bayesian optimization is data-driven and depends Secondly, we propose a novel meta-learning algorithm to learn an initial neural surrogate model from meta train set of related problems, which is taken as initial surrogate model for adaptation to new problem. To train the initial neural surrogate model, meta-learning methods such as MAM-L (Finn et al., 2017) are not applicable because Bayesian optimization is non-differentiable. Thus we propose a weighted Reptile algorithm inspired by Reptile algorithm (Nichol et al., 2018) using sampling strategy for exploration to train the initial neural surrogate model.

We evaluate our method on tasks including optimization of synthetic functions, hyper-parameter tuning in evolutionary algorithm, neural network and SVM. These experiments show that our method outperforms traditional Gaussian process and related warm-starting methods in these tasks.

2. PRELIMINARIES:

2.1 BAYESIAN OPTIMIZATION Bayesian optimization is to find a minimizer x * of a function f in a bounded space by iteratively querying the function at sampled points. At n-th iteration, we first construct a probabilistic model on D = {x i , f (x i )} n i=1 for estimating function values, then use an acquisition function a(x) to select next point. We estimate f by surrogate model based on Gaussian process (GP). We assume

f (•) ∼ GP (m(•), k(•, •)), where m(•) is mean function and k(•, •) is covariance function or kernel. Given D = {x i , f (x i )} n i=1 , GP is simply a joint Gaussian f(x 1:n ) ∼ N (m(x 1:n ), K(x 1:n ))

, where K(x 1:n ) is a n×n covariance matrix with elements of k(x i , x j ) defined as kernel function measuring relation between x i and x j . We choose RBF kernel k(x i , x j ) = exp(-ρ x i -x j 2 ) in this work. Then for a new point x n+1 , we consider joint distribution of previously and newly sampled points:

EQUATION

where k(x n+1 ;

x 1:n ) = (k(x 1 , x n+1 ), k(x 2 , x n+1 ), • • • , k(x n , x n+1 )). The posterior f (x n+1 |D)

satisfies Gaussian distribution (Bishop, 2006) with mean and variance as

EQUATION

EQUATION

) which is the surrogate model of GP. Then we use acquisition function a(x) to choose which point will be evaluated next. Expected improvement (EI) (Mockus, 1975) and upper confidence bound (UCB) (Srinivas et al., 2012) have been shown to be efficient. As we aim to find minimizer of f , we minimize acquisition function: 3) and Eq. ( 4).

2.2. META-LEARNING METHODS:

MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) are two typical meta-learning methods to obtain an initialization of a model from meta train set to well adapt to new problems. Given data set D, model M(w) and loss function L(•, •), MAML learns initialization w * by minimizing L (M(w -α∇L(M(w), D)), D). It implies that MAML learns a w * that can fast adapt by one gradient descent step. Optimization in MAML uses second-order derivatives of L(w). In our application, Bayesian optimization procedures are non-differentiable. Therefore, MAML can not be applied to our task directly. Reptile is one of first-order meta-learning algorithms. As illustrated in Alg. 1, to obtain an optimal initialization of model parameters w that can adapt well to new problem τ by k-steps of optimization denoted by U k τ (w), the parameter w is iteratively updated by vector ( w -w) that enforces w to be close to the updated parameters w of different tasks. In this work, we borrow idea from Reptile, but design a weighted Reptile algorithm for learning initial network-based surrogate model. Update w ← w + ( w -w) 6: end for

3. METHOD:

We aim to learn to warm-starting Bayesian optimization by substituting mean in Gaussian process by a network-based surrogate model φ(w, x). Then we learn an initialization of the neural surrogate model from a meta train set of related problems using a weighted Reptile algorithm, for adaptation to new problem. We first present some key definitions.

Neural surrogate model. The surrogate model of Bayesian optimization is Eqs. ( 3) and ( 4), where mean in Eq. ( 3) is the expectation of predicted function value and variance in Eq. ( 4) represents uncertainty. Since related problems in meta train set may have related shapes of function surfaces, we substitute the mean by a learnable network φ(w, x) to learn initial surrogate model from meta train set. We still use the variance in Eq. ( 4) to measure uncertainty, because the estimation uncertainty should be independent in individual problems. Using network for modeling surrogate model, we learn an initial network φ(w * , x) from meta train set that can be adapted to new problem. Our neural surrogate model is just the combination of φ(w, x) and variance in Eq. ( 4).

Neural optimization episode. An optimization episode is the iterative process for optimizing the black-box function. For a problem Q with corresponding black-box function f , in (t + 1)-th step, we already have surrogate network φ(w t , x) and observed points set D = {x j , f (x j )} t j=1 . First, we update neural surrogate model using L Rank (will be discussed in sect. 3.1) and Eq. ( 4), and obtain φ(w t+1 , x) and σ 2 (x). Then we use acquisition function of Eq. ( 5) to choose next point x t+1 . After that, we update points set D = D {x t+1 , f (x t+1 )}. We iterate the above process to obtain x N T best = arg min {xn,n=1,...,N T } {f (x n )}. We called this procedure as Neural optimization episode (NOE), as illustrated in the right part of Fig. 1 .

In this work, we aim to learn initial φ(w, x) to initialize NOE of a new problem given a set of related problems in meta train set, as illustrated in Fig. 1 . Since the dynamics of NOE is complicated and might be non-differentiable, we propose a weighted Reptile Algorithm to learn the initial φ(w, x) from meta train set using reward defined based on x N T best of problems. This learned initial model φ(w, x) is utilized to initialize the neural surrogate model of a new problem for its NOE.

3.1. RANKING LOSS FOR LEARNING NEURAL SURROGATE MODEL:

We now focus on how to update neural surrogate model given observed points and their function values in neural optimization episode. As we know, different problems have different ranges of function values. To make surrogate model insensitive to the specific range of function values, our neural surrogate model is learned to predict ranking of function values of points instead of exactly fitting the function values. Describing data sets based on pairwise rankings has been investigated in a few machine learning approaches (Leite et al., 2012; Wistuba et al., 2015; Zheng et al., 2007) .

Given points set D = {x i , f (x i )} n i=1

, we update w of neural surrogate model φ(w, •) by L Rank . First we sort points in D by {f (x i )} n i=1 in descent order and get

D s = {x r(i) , f (x r(i) )} n i=1 , where r(i), i = 1 • • • n is a permutation of 1 • • • n.

Then we utilize L Rank as loss:

EQUATION

)

L Rank is based on hinge loss, and minimization of L Rank w.r.t. w enforces that φ(w, x r(i) ) is strictly larger than φ(w, x r(j) ) with margin of δ. Note that final layer of our surrogate model φ is a sigmoid function, therefore, the output of surrogate model is constrained in range of (0, 1) irrespective of different problems. We use gradient descent method to optimize L Rank , please see appendix A.1 for detail. We denote GD(w, L Rank ) as the learned surrogate model parameters by minimizing L Rank using gradient descent method. This above process is taken as the updating step of neural surrogate model in neural optimization episode.

3.2. WEIGHTED REPTILE ALGORITHM FOR LEARNING INITIAL SURROGATE MODEL:

We next focus on how to learn an initial neural surrogate model from a set of related problems in meta train set, taken as the initialization of neural surrogate model for NOE of a new problem. We propose a weighted Reptile algorithm to train the initial neural surrogate model.

Reptile in Alg. 1 is an effective meta-learning algorithm that can learn a good initialization of model parameter from a meta train set. Inspired by Reptile, our Weighted Reptile Algorithm samples a set of initializations of surrogate model parameters for NOE of each problem in the meta train set for exploration, and these sampled initializations are selected by reward and utilized for parameter updating with weights defined by rewards.

Reward. Given minimizer x 1 = arg min x φ(w 1 , x) 3: y 1 = f (x 1 ), D = D {x 1 , y 1 } 4: Update σ(x) by Eq. ( 4) 5: for t = 1 : N T -1 do 6:

x * i of i-th problem Q i (x) in meta train set, reward at step t of NOE of problem Q i (x) is defined as r t i . = -x t i,best -x * i 2 ,

x t+1 = arg min φ(w t , x) -κσ(x) 7:

y t+1 = f (x t+1 ), D = D {x t+1 , y t+1 } 8: w t+1 = GD(w t , L Rank ) 9:

Update σ(x) by Eq. ( 4) 10: end for 11:

x best = x t * , t * = arg min t {y t } N T i=t 12: return x best , w N T f i is the black-box function in problem Q i (x), x n is sampled point at n-th step, x t

i,best is the best point up to step t.

Weighted Reptile Algorithm. In Reptile, as in line 5 of Alg. 1, the parameter w is learned to be close to updated parameter w by adapting w in different problems. It implicitly assumes that updated parameter w should be better than initialized w for corresponding problem. In our approach, we aim to learn initial surrogate model φ(w, x) from problems in meta train set. Similar to Reptile, we learn initial network parameter w of φ(w, x) close to updated parameters after running NOE of problems in meta train set. To explore diverse paths of NOE, for each i-th problem in meta train set, we sample a few (M ) initial parameters {w p i } M p=1 ∼ N (w, σ 2 ) by disturbing currently estimated w. Then after running N T steps of NOE, we obtain points {x N T ,p i,best } M p=1 and updated parameters { wN T ,p i } M p=1 starting from surrogate models initialized with these sampled parameters. To learn a good initial parameter w of surrogate model φ, parameter is updated by w = w + β∆w, where

EQUATION

and reward r

N T i = -x N T i,best -x * i 2 , r N T i,p = -x N T ,p i,best -x * i 2 .

For i-th problem and p-th sampled parameter, weight max(r N T i,p -r N T i , 0) is always non-negative, and it is non-zero only when the sampled parameter w p i can produce better reward than the parameter w itself if taking them as initial surrogate model parameters for running NOE of problem Q i . This rule enforces that the parameter w of our learned initial surrogate model is close to better adapted parameters of different problems in meta train set. Algorithm 3 presents the pipeline of meta-train process using weighted Reptile algorithm, and it depends on the NOE as described in Alg. 2 .

4. EXPERIMENTS:

We evaluate our method for global optimization on synthetic functions set, and hyper-parameters tuning in evolutionary algorithm, neural network and SVM. In the following paragraphs, our method based on neural surrogate model and weighted Reptile algorithm will be dubbed as WRA-N. In all experiments, we set κ = 0.1, δ = 0.01, σ = 0.1, training epoch N E = 30, learning rate β = 0.1, number of samples M = 50. We use the mean squared error (MSE) between function values of our found best point and the global minimizer to measure the performance of optimization. We mainly compare with TST-R (Wistuba et al., 2016) and GP-UCB (Srinivas et al., 2012) . Moreover, we also compare with meta feature-based method TST-M (Wistuba et al., 2016) in SVM hyper-parameter tuning. But for the remaining tasks, we did not compare with it because it is hard to well define meta-features for these problems. The neural surrogate model is taken as a network with one hidden layer of 15 hidden units, and both hidden and output layers are with sigmoid nonlinearity. #Surrogate model adaptation loop.

5::

for i = 1 : |Q| do 6:

{x N T i,best , w N T i } = NOE(N T , κ, Q i , w l ) 7: r N T i = -x N T i,best -x * i 2 8:

for p = 1 : M do 9:

set w p i = w l + N (0, σ 2 I) 10:

{x N T ,p i,best , wN T ,p i } = NOE(N T , κ, Q i , w p i ) 11: r N T i,p = -x N T ,p i,best -x * i 2 , g p i = wN T ,p i -w l 12:

end for 13:

end for 14:

#Updating initial surrogate model parameter.

15: 

w l+1 = w l + β |Q| i=1 M p=1 max(r N T i,p -r N T i , 0)g p

4.1. SYNTHETIC FUNCTIONS:

We first evaluate on Alpine 1 function in Jamil & Yang (2013) without using absolute value: -10, 10] . We set functions with a = 4π 3 , 5π 4 , 6π 5 , 7π 6 as meta train set, and 18 functions with a = 3π 2 -iπ 24 , i ∈ {1, ..., 21} excluding {4, 6, 8} in meta train set for testing. Figure 2 shows logarithm of MSEs w.r.t. optimization iterations of WRA-N, TST-R and GP-UCB for a = 3π 2 , π, 2π 3 , and other results are in appendix. Optimization iteration means the iteration in optimization episode, e.g., the iterations in NOE of our WRA-N. Since TST-R needs base models for combination, we sample 20 points from uniform distribution in (-10, 10) to construct base models. TST-R and GP-UCB start with 3 random points, then run in 10 iterations. Our WRA-N starts with neural surrogate model learned by meta train set, and runs in 13 iterations, i.e., N T = 13. Optimization was repeated 20 times for TST-R and GP-UCB. For TST-R, each time is with differently sampled points for constructing base models. Figure 2 shows curves of average MSEs with variances in logarithm w.r.t. iterations. WRA-N starts with learned initial surrogate model, therefore curves are deterministic. WRA-N significantly improves baseline GP-UCB and outperforms TST-R, because test functions may not be well represented by base functions in TST-R.

f (x) = xsin(x + a) + 0.1x, x ∈ [

4.2. HYPER-PARAMETERS TUNING IN DIFFERENTIAL EVOLUTIONARY ALGORITHM:

Differential evolution (DE) (Galletly, 1996; Das & Suganthan, 2011) 

[x 0 1 , x 0 2 , • • • , x 0 Np ] ∈ R d×Np , and iterates u k+1 i = x k i + F (x k best -x k i ) + F (x k r(1) -x k r(2) )

, where x k r(i) (i = 1, 2) are randomly chosen from X k , x k best is the best point in k-th iteration, F is hyperparameter. Then we can use binomial crossover and selection methods to create X k+1 . We generalize this iteration formulation to be

u k+1 i = x k i + F (x k best -x k i ) + D(x k r(1) -x k r(2) )

using 2 hyper-parameters F and D, which is a general formulation covering most DE formulations. F and D influence optimization performance of DE (e.g., finding minimum of functions by DE) significantly, and thus we can utilize our method to tune these hyper-parameters. We use function set of CEC 2017 (Awad et al., 2016) for optimization by DE and use our method to find optimal hyperparameters. The neural surrogate model is to predict the function values w.r.t. hyper-parameters. We set function dimension as 10, construct meta train set containing basic functions of f 1 , f 2 , f 3 , f 4 , and learn initial surrogate model that can find optimal hyper-parameters well in f 5 -f 10 for testing. The formulas of these functions are shown in appendix. For TST-R, we sample 20 points uniformly in feasible domain and use these points to construct base models. TST-R and GP-UCB start with 3 randomly chosen points, and then proceed sequentially for 20 iterations. Our method starts with surrogate model learned by meta train set, and proceeds for a total of 23 iterations. The optimization was repeated 20 times similar to the experiments on synthetic function set. Figure 3 shows results on f 5 and f 6 , and results on the other functions are shown in appendix. Our method produces best hyper-parameter tuning performance. Note that the functions in meta train set are in different scales of function values. For TST-R, we follow its original implementation by normalizing point values of base models to normal distribution. However, in our approach, the neural surrogate model is updated using ranking-based loss, therefore it is not sensitive to the different scales of functions.

4.3. HYPER-PARAMETERS TUNING IN NEURAL NETWORK:

We will optimize hyper-parameters of neural networks for image classification. We use EMNIST data set (Cohen et al., 2017) , i.e., an extension of MNIST to handwritten letters. There are 62 classes of [0-9], [a-z] and [A-Z], and several classes that are similar between uppercase and lowercase are merged, and finally we have 47 classes in data set. The network is with one hidden layer, sigmoid as activation function and softmax for classification, followed by cross-entropy loss. We first randomly sample 10 classes in 3 times from EMNIST with 1000 images for each class, Note that we can also use other criterion for learning hyper-parameters. Figure 4 shows MSEs (between ERRs of best hyper-parameters and estimated hyper-parameters) w.r.t. optimization iterations by different methods. We can observe that our method performs best for this task.

4.4. HYPER-PARAMETERS TUNING IN SVM:

Support vector machine (SVM) dataset (Wistuba et al., 2016) consists of 50 classification datasets randomly chosen from UCI repository. The SVMs use Gaussian kernel with hyper-parameters of trade-off parameter C and width γ of Gaussian kernel. The candidate hyper-parameters are C ∈ {2 -5 , ..., 2 6 } and γ ∈ {10 -4 , 10 -3 , 10 -2 , 0.05, 0.1, 0.5, 1, 2, 5, 10, 20, 50, 10 2 , 10 3 }. The neural surrogate model is to predict misclassification rate w.r.t. to hyper-parameters C, γ. In experiments, we randomly choose 5 datasets as meta train sets, and the others as test sets. All the experiments are performed in 10 times. Figure 5 shows the average MSEs w.r.t. optimization iterations by different methods. Here we compare our WRA-N with TST-M (Wistuba et al., 2016) , which also relies on linear combination of surrogate models of meta train problems but using meta features. We can see that our WRA-N performs better than both TST-R and TST-M. Both our WRA-N and TST-R perform significantly better than baseline GP-UCB.

4.5. ABLATION STUDY:

We now justify effectiveness of weighted Reptile algorithm (WRA) and different network structures of surrogate model φ. All following networks are evaluated on synthetic set in section 4.1. In Fig. 6 (left), we compare our WRA with traditional Reptile algorithm without using sampling and weighting strategy. The average MSE curves show that our WRA performs better than Reptile. We further compare different structures of surrogate models including networks with one hidden layer of 10, 15, 25 units and 2 hidden layers with 15 units. We observe that they perform similarly for networks with one hidden layer, and adding more layers does not apparently improve performance.

5. CONCLUSION:

We have proposed a novel neural surrogate model and a weighted Reptile algorithm to learn an initial neural surrogate model for warm-starting Bayesian optimization. Extensive experiments on synthetic functions optimization and hyper-parameters tuning in multiple tasks have shown promising performance. In the future, we are interested to apply it to more global optimization or hyperparameters tuning tasks, and analyze its theoretical properties.

A.2 FUNCTIONS IN CEC 2017 FOR DE HYPER-PARAMETERS TUNING:

The ten functions are listed as follows.

f 1 (x) = x 2 1 + 10 6 H i=2 x i 2 , f 2 (x) = H i=1 |x i | i+1 f 3 (x) = H i=1 x i 2 + (0.5 H i=1 x i ) 4 + (0.5 H i=1 x i ) 4 f 4 (x) = H-1 i=1 (100(x i 2 -x i+1 ) 2 + (x i -1) 2 ) f 5 (x) = H i=1 (x i 2 -10cos(2πx i ) + 10) f 6 (x) = g(x 1 , x 2 ) + g(x 2 , x 3 ) + • • • + g(x H , x 1 ) f 7 (x) = min( H i=1 ( x i -µ 0 ) 2 , dH + s H i=1 ( x i -µ 1 ) 2 ) + 10(H - H i=1 cos(2π z i )) f 8 (x) = H i=1 (z i 2 -10cos(2πz i ) + 10) f 9 (x) = sin 2 (πw 1 ) + H-1 i=1 (w i -1) 2 [1 + 10sin 2 (πw i + 1)] + (w H -1) 2 [1 + sin 2 (2πw H )] f 10 (x) = 418.9829H - H i=1 g(z i )

g(x, y) in f 6 is defined as g(x, y) = 0.5 + sin 2 ( √ x 2 +y 2 )-0.5 (1+0.001(x 2 +y 2 )) 2 . Please refer to CEC 2017 (Awad et al., 2016) for details on parameters.

A.3 ADDITIONAL RESULTS IN EXPERIMENTS:

Figure 7 shows the function optimization results on 15 remaining test functions in synthetic function set. Figure 8 shows curves of average MSEs w.r.t. different optimization iterations for tuning hyper-parameters of number of hidden units and learning rate of SGD on EMNIST data sets of S 7 , S 8 , S 9 , S 10 , S 11 . Figure 9 shows these curves for tuning hyper-parameters in DE on functions of f 7 -f 10 in CEC 2017. 

A.4 AN ILLUSTRATION OF NEURAL SURROGATE MODEL BY MINIMIZING L Rank:



:



Figures and Tables:

w; λ), D) = min λ L(M(w * ; λ), D), where w * = A(L(M(w; λ), D)),

Figure 1: General pipeline of learning initial neural surrogate model using weighted Reptile algorithm from meta train set of functions f i , i = 1, • • • , N .

) = arg min x {µ(x) -κσ(x)} (5) to find next point. As we can see, GP estimates function value via given data D based on surrogate model of Eq. (

w = U k τ (w), i.e., k steps optimization by SGD or Adam 5:

Meta train by weighted Reptile algorithm. Require: Number of iterations N T in NOE, κ > 0, meta train set Q, number of epochs N E , learning rate β, number of samples M , standard deviation σ for sampling. Ensure: w * 1: Randomly initialize w 1 2: #Meta-train loop. 3: for l = 1 : N E do 4:

Figure 2: MSEs w.r.t. optimization iterations on synthetic functions with a = 3π 2 , π, 2π 3 .

Figure 3: MSEs w.r.t. iterations for f 5 , f 6 in CEC 2017 for hyper-parameters tuning of DE.

Figure 4: MSEs w.r.t. optimization iterations for hyper-parameters tuning of neural networks on data sets of S 4 , S 5 , S 6 .

Figure 5: MSEs w.r.t. iterations for hyper-parameters tuning of SVMs.

Figure 6: Ablation study.

Figure 10 shows the neural surrogate model by minimizing L Rank working on DE hyper-parameters tuning. In this task, we use neural surrogate model to predict the mapping from hyper-parameters D, F to function values of points. Figure 10(a) shows the surrogate model learned by neural network. Figure 10(b) shows the real function values w.r.t. hyper parameters. Figure 10(c) shows the logarithm of real function values w.r.t. hyper-parameters. Then we see that the neural surrogate model is within range of [0, 1] and it has similar local minimum with real function (by comparing Figs. 10(a) and (c)), but it is more smooth than real function surface. We further illustrate the distributions using histograms of function values of real function and neural surrogate model. As shown in Fig.10 (d),x-axis is the percentage (i.e. ratio) of maximal function value, y-axis is the percentage of number of data with function values falling in these ratios to maximal function values. We see that, for real function, most values are less than 0.25 of maximum function value, while our neural surrogate model is more uniform.

Figure 7: MSEs (in logarithm) w.r.t. different optimization iterations on 15 remaining test functions in synthetic function set.

Figure 8: MSEs (in logarithm) w.r.t. different optimization iterations for tuning number of hidden units and learning rate on EMNIST data sets of S 7 , S 8 , S 9 , S 10 , S 11 .

Figure 9: MSEs (in logarithm) w.r.t. different optimization iterations for tuning hyper-parameters of DE on f 7 -f 10 of CEC 2017.

Figure 10: An Illustration of neural surrogate model for hyper-parameter tuning of DE.

Figure 11: Training curves of rewards on meta train set respectively for our four different experiments in sections 4.1-4.4.

Neural optimization episode (NOE).Require: Maximum iterations N T , κ > 0, test problem Q with function f , initialized parameter w Ensure: x best , w N T 1: set w 1 = w, D = {} 2:

Table content: <table/>

Peer review (iclr2020):
POST-REBUTTAL FEEDBACK Thanks for your response. The justifications provided in the response have not convinced me to improve my score. They are at times hard to understand: For example, the authors have claimed that while their design choice is not reasonable, it is less unreasonable than the other. SUMMARY OF REVIEW The authors have proposed the use of a neural surrogate model in place of the GP posterior mean and a weighted Reptile algorithm to meta-learn the initial weights of the neural surrogate model. This approach appears interesting. However, there seems to be multiple highly restrictive (at times impractical) assumptions in this work that are atypical of the BO setting adopted by other meta BO algorithms and not discussed, as detailed below. Justifications are required. Clarifications are also needed with regards to how they exactly run their algorithm in the experiments and whether the prior/initial information from related problems/meta tasks provided to the tested algorithms is fair. DETAILED COMMENTS The authors say that *We still use the variance in Eq. (4) to measure uncertainty, because the estimation uncertainty should be independent in individual problems.* This does not seem to hold true. If a meta task or train set is indeed correlated (or provides information) to the new problem, the posterior variance/uncertainty at a point depends on the observations in the meta task or train set near to this point (see, for example, Feurer et al. (2018)). Can the authors discuss the implications of such an assumption in their work? Q and Q_i have always been referred to as problems. In Algorithm 3, Q is suddenly referred to as meta train set. On page 4, you have said that x^*_i is the minimizer of i-th problem Q_i(x) in meta train set. Based on these information, I assume that the authors consider x^*_i as the global minimizer and that x^*_i is known in order to compute the rewards. Can the authors discuss why is this a reasonable assumption? In their proposed weight Reptile algorithm (Algorithm 3), the authors have also assumed access to the black-box functions of the related problems or meta tasks, which is not typical of other meta BO works that only require the existing observations or datasets of the related problems/meta tasks. As a result, compared with the existing meta BO algorithms, their proposed weighted Reptile is considerably more expensive due to the need to additionally evaluate the black-box functions of the related problems or meta tasks many times during execution. Can the authors discuss the practical implications of such an assumption and how it affects the types of problems/applications that can be considered by this work? The authors have not provided any justification for their choice of reward on page 4. If the black-box function is indeed complex and highly varying, the distance between points may not work well at all. Can the authors provide a justification and discuss the practical implications and limitations with such a choice? Isn*t it more natural to consider a single Bayesian neural network instead of using a neural network for the mean and a GP for the variance? For the experiments, it would be good to see two other variants of the proposed algorithm to understand the individual contributions of the neural surrogate model and weighted Reptile algorithm: one without neural surrogate model and the other with simply the use of neural surrogate model. Can the authors explain in greater detail how they run their algorithms (Algorithms 2 and 3) in the experiments? For example, the authors say that *WRA-N starts with learned initial surrogate model*. I assume that WRA-N refers to Algorithm 3 based on its acronym. Isn*t the learned initial surrogate model the output of WRA-N in the first place? Also, the graphs in Fig. 2 seem to show iteration 1 to 13 in NOE (Algorithm 2). However, Algorithm 3 accepts N_T = 13 and executes NOE for N_T = 13 (and not 1, 2, or 3, ...) for each problem in each epoch. How do the authors generate the plot of WRA-N for iterations 1 to 12? What seems to make more sense to me is that the authors in fact run Algorithm 2 instead of Algorithm 3 for each experiment and they initialize w in Algorithm 2 to the output of Algorithm 3. In any case, a clarification is needed here. It is not clear to me whether the initial/prior information from related problems/meta tasks provided to WRA-N, TST-R, AND TSR-M is fair. Can the authors provide a justification? To clarify, for each related problem/function, only N_T number of datapoints are used to train a corresponding neural network with 1 hidden layer of 15 hidden units? The authors say that *Since TST-R needs base models for combination, we sample 20 points from uniform distribution in (?10, 10) to construct base models.* Is this sampling procedure the same as that in (Wistuba et al., 2016)? Can the authors explain the comparable performance of WRA-N and TST-R in Fig. 5? Why are the error bars missing? How does the proposed approach compare with that of Feurer et al. (2018)? Minor issues Page 1, 3: adapt well to new tasks. Page 2: The author says *depends on a GP-based surrogate model fitting function values without learnable parameters*. This is not true: The GP hyperparameters need to be learned and they adapt to new problems. Page 4: descent order? Pages 4, 5: Why is there an input x to Q_i? Algorithm 2: t^* should be at the superscript of x. Equation 7: What is N? Page 5: Does it make a difference in the performance when delta is set to 0? Page 5: well define meta-features?