Year: 2019

Abstract:
Bayesian optimization is an effective tool to optimize black-box functions and popular for hyper-parameter tuning in machine learning. Traditional Bayesian optimization methods are based on Gaussian process (GP), relying on a GP-based surrogate model for sampling points of the function of interest. In this work, we consider transferring knowledge from related problems to target problem by learning an initial surrogate model for warm-starting Bayesian optimization. We propose a neural network-based surrogate model to estimate the function mean value in GP. Then we design a novel weighted Reptile algorithm with sampling strategy to learn an initial surrogate model from meta train set. The initial surrogate model is learned to be able to well adapt to new tasks. Extensive experiments show that this warm-starting technique enables us to find better minimizer or hyper-parameters than traditional GP and previous warm-starting methods.

heading: 1 INTRODUCTION
text: Bayesian optimization (Mockus, 1975; Pelikan et al., 1999; Gustavo Malkomes, 2018) is an optimization method to find global minimizer of a black-box function without knowing its convexity, differentiability or continuity. Bayesian optimization iteratively samples new point by an acquisition function, and uses Gaussian process (Rasmussen, 2003; Gardner et al., 2014) to estimate a surrogate model to fit f(x) based on the observed points. When using specific acquisition function, its regret bound has been proved as O( √ n) (n is the number of sampled points) in Srinivas et al. (2012). Freitas et al. (2012) proposed a method that uses Bayesian optimization with dividing region, which successfully accelerates convergence rate. More algorithms have been developed in Wang et al. (2014); Kawaguchi et al. (2015); Springenberg et al. (2016). In application, Bayesian optimization has been widely applied in machine learning for hyper-parameter tuning (Snoek et al., 2012).
A machine learning problem is formulated as follows. Given data set D, model M(·, λ) and loss function L(·, ·), it aims to find an optimal parameter vector w to minimize L(M(w;λ),D). In practice, model M(w;λ) is dependent on hyper-parameter λ, which may significantly influence the model. Thus choosing appropriate λ is a critical procedure of learning a model. For instance, M(w;λ) is a neural network to solve image classification problem, then λ can be the number of hidden units or network layers. Hyper-parameter optimization problem can be formulated as
min λ,w L(M(w;λ),D) = min λ L(M(w∗;λ),D),where w∗ = A(L(M(w;λ),D)), (1)
and A is an algorithm minimizing loss function w.r.t. w. It is challenging to optimize Eq. (1) since A(L(M(w;λ),D)) is a complicated function w.r.t λ, which might be non-differentiable. In real applications, hyper-parameter is commonly tuned by hand. Random search and grid search are typical methods to find plausible λ, but they are time-consuming to find the optimal hyper-parameter because they sample the hyper-parameter in a brute-force way.
Bayesian optimization is an effective tool to minimize function for global optimization or hyperparameter tuning. Bayesian optimization iteratively samples new point and fits the function values by a surrogate model based on Gaussian process. It starts with several random points without using any prior knowledge on the task. In applications, we often solve several related machine learning problems, therefore, one question is that whether we can learn common knowledge from related problems and transfer them to new problem as an initialization for adaptation, commonly referred to as warm-starting Bayesian optimization.
Meta-learning is a promising approach for learning optimizer for fast optimization or initial model for fast adaptation to novel task using a set of training problems as meta train set (Andrychowicz et al., 2016; Li & Malik, 2017; Wang et al., 2019; Ravi & Larochelle, 2017; Chen et al., 2017; Finn et al., 2017; Nichol et al., 2018). This idea was also applied to Bayesian optimization in past years. Poloczek et al. (2016) built a joint statistical model of the collection of related objective functions. Feurer et al. (2015) initialized Bayesian optimization based on hand-crafted meta features for hyperparameter tuning. Given a new problem, its hyper-parameters are initialized by the most similar problem in meta train set with distance of hand-crafted meta features. However, it is challenging to define meta features artificially. Kim et al. (2017) learned meta features of problems by deep metric learning. Wistuba et al. (2016); Feurer et al. (2018) proposed to transfer surrogate model by linear combination of surrogate models of problems in meta train set with coefficients determined by ranking of function values or meta features. However, representation capacity of linear combination is limited, meaning that if surrogate model of a new problem cannot be represented by linear combination of surrogate models of meta train problems, then this approach will fail.
In this work, we propose a novel meta-learning approach to learn prior knowledge from meta train set as a warm-start of unseen related target problem. More specifically, we make the following two contributions to tackle the challenges in traditional Bayesian optimization.
Firstly, as we know, traditional Bayesian optimization is data-driven and depends on a GP-based surrogate model fitting function values without learnable parameters, limiting its ability for adaptation to new problems. We propose to use neural network to model the surrogate model (dubbed as neural surrogate model) to substitute the mean of function values in GP. We further design a ranking loss for fitting surrogate model to sampled points to make surrogate model insensitive to the scale of function values.
Secondly, we propose a novel meta-learning algorithm to learn an initial neural surrogate model from meta train set of related problems, which is taken as initial surrogate model for adaptation to new problem. To train the initial neural surrogate model, meta-learning methods such as MAML (Finn et al., 2017) are not applicable because Bayesian optimization is non-differentiable. Thus we propose a weighted Reptile algorithm inspired by Reptile algorithm (Nichol et al., 2018) using sampling strategy for exploration to train the initial neural surrogate model.
We evaluate our method on tasks including optimization of synthetic functions, hyper-parameter tuning in evolutionary algorithm, neural network and SVM. These experiments show that our method outperforms traditional Gaussian process and related warm-starting methods in these tasks.

heading: 2 PRELIMINARIES

heading: 2.1 BAYESIAN OPTIMIZATION
text: Bayesian optimization is to find a minimizer x∗ of a function f in a bounded space by iteratively querying the function at sampled points. At n-th iteration, we first construct a probabilistic model on D = {xi, f(xi)}ni=1 for estimating function values, then use an acquisition function a(x) to select next point. We estimate f by surrogate model based on Gaussian process (GP). We assume f(·) ∼ GP (m(·), k(·, ·)), where m(·) is mean function and k(·, ·) is covariance function or kernel. Given D = {xi, f(xi)}ni=1, GP is simply a joint Gaussian f(x1:n) ∼ N (m(x1:n),K(x1:n)), where K(x1:n) is a n×n covariance matrix with elements of k(xi, xj) defined as kernel function measuring relation between xi and xj . We choose RBF kernel k(xi, xj) = exp(−ρ‖xi − xj‖2) in this work. Then for a new point xn+1, we consider joint distribution of previously and newly sampled points:(
f(x1:n) f(xn+1)
) ∼ N (( m(x1:n) m(xn+1) ) , ( K(x1:n) k(xn+1;x1:n) k(xn+1;x1:n)ᵀ k(xn+1, xn+1) )) , (2)
where k(xn+1;x1:n) = (k(x1, xn+1), k(x2, xn+1), · · · , k(xn, xn+1)). The posterior f(xn+1|D) satisfies Gaussian distribution (Bishop, 2006) with mean and variance as
µ(xn+1|D) = k(xn+1;x1:n)ᵀK(x1:n)−1f(x1:n), (3)
σ2(xn+1|D) = k(xn+1, xn+1)− k(xn+1;x1:n)ᵀK(x1:n)−1k(xn+1;x1:n), (4) which is the surrogate model of GP. Then we use acquisition function a(x) to choose which point will be evaluated next. Expected improvement (EI) (Mockus, 1975) and upper confidence bound
(UCB) (Srinivas et al., 2012) have been shown to be efficient. As we aim to find minimizer of f , we minimize acquisition function:
arg min x a(x) = arg min x {µ(x)− κσ(x)} (5)
to find next point. As we can see, GP estimates function value via given data D based on surrogate model of Eq. (3) and Eq. (4).

heading: 2.2 META-LEARNING METHODS
text: MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) are two typical meta-learning methods to obtain an initialization of a model from meta train set to well adapt to new problems. Given data set D, modelM(w) and loss function L(·, ·), MAML learns initialization w∗ by minimizing L (M(w − α∇L(M(w),D)),D). It implies that MAML learns a w∗ that can fast adapt by one gradient descent step. Optimization in MAML uses second-order derivatives of L(w). In our application, Bayesian optimization procedures are non-differentiable. Therefore, MAML can not be applied to our task directly. Reptile is one of first-order meta-learning algorithms. As illustrated in Alg. 1, to obtain an optimal initialization of model parameters w that can adapt well to new problem τ by k-steps of optimization denoted by Ukτ (w), the parameter w is iteratively updated by vector (w̃−w) that enforces w to be close to the updated parameters w̃ of different tasks. In this work, we borrow idea from Reptile, but design a weighted Reptile algorithm for learning initial network-based surrogate model.
Algorithm 1 Reptile 1: Initialize w, i.e., the initial parameter 2: for i = 1, 2, · · · , n do 3: Sample task τ 4: Compute w̃ = Ukτ (w), i.e., k steps optimization by SGD or Adam 5: Update w ← w + (w̃ − w) 6: end for

heading: 3 METHOD
text: We aim to learn to warm-starting Bayesian optimization by substituting mean in Gaussian process by a network-based surrogate model φ(w, x). Then we learn an initialization of the neural surrogate model from a meta train set of related problems using a weighted Reptile algorithm, for adaptation to new problem. We first present some key definitions.
Neural surrogate model. The surrogate model of Bayesian optimization is Eqs. (3) and (4), where mean in Eq. (3) is the expectation of predicted function value and variance in Eq. (4) represents uncertainty. Since related problems in meta train set may have related shapes of function surfaces,
we substitute the mean by a learnable network φ(w, x) to learn initial surrogate model from meta train set. We still use the variance in Eq. (4) to measure uncertainty, because the estimation uncertainty should be independent in individual problems. Using network for modeling surrogate model, we learn an initial network φ(w∗, x) from meta train set that can be adapted to new problem. Our neural surrogate model is just the combination of φ(w, x) and variance in Eq. (4).
Neural optimization episode. An optimization episode is the iterative process for optimizing the black-box function. For a problem Q with corresponding black-box function f , in (t + 1)-th step, we already have surrogate network φ(wt, x) and observed points set D = {xj , f(xj)}tj=1. First, we update neural surrogate model using LRank (will be discussed in sect. 3.1) and Eq. (4), and obtain φ(wt+1, x) and σ2(x). Then we use acquisition function of Eq. (5) to choose next point xt+1. After that, we update points set D = D ⋃ {xt+1, f(xt+1)}. We iterate the above process to obtain xNTbest = arg min{xn,n=1,...,NT }{f(xn)}. We called this procedure as Neural optimization episode (NOE), as illustrated in the right part of Fig. 1.
In this work, we aim to learn initial φ(w, x) to initialize NOE of a new problem given a set of related problems in meta train set, as illustrated in Fig. 1. Since the dynamics of NOE is complicated and might be non-differentiable, we propose a weighted Reptile Algorithm to learn the initial φ(w, x) from meta train set using reward defined based on xNTbest of problems. This learned initial model φ(w, x) is utilized to initialize the neural surrogate model of a new problem for its NOE.

heading: 3.1 RANKING LOSS FOR LEARNING NEURAL SURROGATE MODEL
text: We now focus on how to update neural surrogate model given observed points and their function values in neural optimization episode. As we know, different problems have different ranges of function values. To make surrogate model insensitive to the specific range of function values, our neural surrogate model is learned to predict ranking of function values of points instead of exactly fitting the function values. Describing data sets based on pairwise rankings has been investigated in a few machine learning approaches (Leite et al., 2012; Wistuba et al., 2015; Zheng et al., 2007).
Given points set D = {xi, f(xi)}ni=1, we update w of neural surrogate model φ(w, ·) by LRank. First we sort points in D by {f(xi)}ni=1 in descent order and get Ds = {xr(i), f(xr(i))}ni=1, where r(i), i = 1 · · ·n is a permutation of 1 · · ·n. Then we utilize LRank as loss:
LRank(φ(w),D) = n∑ i=1 n∑ j=1,j>i [max(φ(w, xr(j))− φ(w, xr(i)) + δ, 0)]2, (6)
LRank is based on hinge loss, and minimization of LRank w.r.t. w enforces that φ(w, xr(i)) is strictly larger than φ(w, xr(j)) with margin of δ. Note that final layer of our surrogate model φ is a sigmoid function, therefore, the output of surrogate model is constrained in range of (0, 1) irrespective of different problems. We use gradient descent method to optimize LRank, please see appendix A.1 for detail. We denote GD(w,LRank) as the learned surrogate model parameters by minimizing LRank using gradient descent method. This above process is taken as the updating step of neural surrogate model in neural optimization episode.

heading: 3.2 WEIGHTED REPTILE ALGORITHM FOR LEARNING INITIAL SURROGATE MODEL
text: We next focus on how to learn an initial neural surrogate model from a set of related problems in meta train set, taken as the initialization of neural surrogate model for NOE of a new problem. We propose a weighted Reptile algorithm to train the initial neural surrogate model.
Reptile in Alg. 1 is an effective meta-learning algorithm that can learn a good initialization of model parameter from a meta train set. Inspired by Reptile, our Weighted Reptile Algorithm samples a set of initializations of surrogate model parameters for NOE of each problem in the meta train set for exploration, and these sampled initializations are selected by reward and utilized for parameter updating with weights defined by rewards.
Reward. Given minimizer x∗i of i-th problem Qi(x) in meta train set, reward at step t of NOE of problem Qi(x) is defined as rti . = −‖xti,best−x∗i ‖2, where xti,best = arg min{xn,n=1,...,t}{fi(xn)},
Algorithm 2 Neural optimization episode (NOE). Require: Maximum iterations NT , κ > 0, test problem Q with function f , initialized parameter w Ensure: xbest, wNT
1: set w1 = w, D = {} 2: x1 = arg minx φ(w1, x) 3: y1 = f(x1), D = D ⋃ {x1, y1} 4: Update σ(x) by Eq. (4) 5: for t = 1 : NT − 1 do 6: xt+1 = arg minφ(wt, x)− κσ(x) 7: yt+1 = f(xt+1), D = D ⋃ {xt+1, yt+1} 8: wt+1 = GD(wt,LRank) 9: Update σ(x) by Eq. (4)
10: end for 11: xbest = xt∗ , t∗ = arg mint{yt}NTi=t 12: return xbest, wNT
fi is the black-box function in problem Qi(x), xn is sampled point at n-th step, xti,best is the best point up to step t.
Weighted Reptile Algorithm. In Reptile, as in line 5 of Alg. 1, the parameter w is learned to be close to updated parameter w̃ by adaptingw in different problems. It implicitly assumes that updated parameter w̃ should be better than initialized w for corresponding problem. In our approach, we aim to learn initial surrogate model φ(w, x) from problems in meta train set. Similar to Reptile, we learn initial network parameter w of φ(w, x) close to updated parameters after running NOE of problems in meta train set. To explore diverse paths of NOE, for each i-th problem in meta train set, we sample a few (M ) initial parameters {wpi }Mp=1 ∼ N (w, σ2) by disturbing currently estimated w. Then after running NT steps of NOE, we obtain points {xNT ,pi,best}Mp=1 and updated parameters {w̃NT ,pi }Mp=1 starting from surrogate models initialized with these sampled parameters. To learn a good initial parameter w of surrogate model φ, parameter is updated by w = w + β∆w, where
∆w = M∑ p=1 N∑ i=1 (w̃NT ,pi − w) max(r NT i,p − r NT i , 0), (7)
and reward rNTi = −‖x NT i,best − x∗i ‖2, r NT i,p = −‖x NT ,p i,best − x∗i ‖2. For i-th problem and p-th sampled parameter, weight max(rNTi,p − r NT i , 0) is always non-negative, and it is non-zero only when the sampled parameter wpi can produce better reward than the parameter w itself if taking them as initial surrogate model parameters for running NOE of problem Qi. This rule enforces that the parameter w of our learned initial surrogate model is close to better adapted parameters of different problems in meta train set. Algorithm 3 presents the pipeline of meta-train process using weighted Reptile algorithm, and it depends on the NOE as described in Alg. 2 .

heading: 4 EXPERIMENTS
text: We evaluate our method for global optimization on synthetic functions set, and hyper-parameters tuning in evolutionary algorithm, neural network and SVM. In the following paragraphs, our method based on neural surrogate model and weighted Reptile algorithm will be dubbed as WRA-N. In all experiments, we set κ = 0.1, δ = 0.01, σ = 0.1, training epoch NE = 30, learning rate β = 0.1, number of samples M = 50. We use the mean squared error (MSE) between function values of our found best point and the global minimizer to measure the performance of optimization. We mainly compare with TST-R (Wistuba et al., 2016) and GP-UCB (Srinivas et al., 2012). Moreover, we also compare with meta feature-based method TST-M (Wistuba et al., 2016) in SVM hyper-parameter tuning. But for the remaining tasks, we did not compare with it because it is hard to well define meta-features for these problems. The neural surrogate model is taken as a network with one hidden layer of 15 hidden units, and both hidden and output layers are with sigmoid nonlinearity.
Algorithm 3 Meta train by weighted Reptile algorithm. Require: Number of iterations NT in NOE, κ > 0, meta train set Q, number of epochs NE ,
learning rate β, number of samples M , standard deviation σ for sampling. Ensure: w∗
1: Randomly initialize w1 2: #Meta-train loop. 3: for l = 1 : NE do 4: #Surrogate model adaptation loop. 5: for i = 1 : |Q| do 6: {xNTi,best, w NT i } = NOE(NT , κ,Qi, wl) 7: rNTi = −‖x NT i,best − x∗i ‖2 8: for p = 1 : M do 9: set ŵpi = w
l +N (0, σ2I) 10: {xNT ,pi,best, w̃ NT ,p i } = NOE(NT , κ,Qi, ŵ p i ) 11: rNTi,p = −‖x NT ,p i,best − x∗i ‖2, g p i = w̃ NT ,p i − wl 12: end for 13: end for 14: #Updating initial surrogate model parameter. 15: wl+1 = wl + β ∑|Q| i=1 ∑M p=1 max(r NT i,p − r NT i , 0)g p i 16: end for 17: return w∗ = wNE+1

heading: 4.1 SYNTHETIC FUNCTIONS
text: We first evaluate on Alpine 1 function in Jamil & Yang (2013) without using absolute value: f(x) = xsin(x + a) + 0.1x, x ∈ [−10, 10]. We set functions with a = 4π3 , 5π 4 , 6π 5 , 7π 6 as meta train set, and 18 functions with a = 3π2 − iπ 24 , i ∈ {1, ..., 21} excluding {4, 6, 8} in meta train set for testing. Figure 2 shows logarithm of MSEs w.r.t. optimization iterations of WRA-N, TST-R and GP-UCB for a = 3π2 , π, 2π 3 , and other results are in appendix. Optimization iteration means the iteration in optimization episode, e.g., the iterations in NOE of our WRA-N. Since TST-R needs base models for combination, we sample 20 points from uniform distribution in (−10, 10) to construct base models. TST-R and GP-UCB start with 3 random points, then run in 10 iterations. Our WRA-N starts with neural surrogate model learned by meta train set, and runs in 13 iterations, i.e., NT = 13. Optimization was repeated 20 times for TST-R and GP-UCB. For TST-R, each time is with differently sampled points for constructing base models. Figure 2 shows curves of average MSEs with variances in logarithm w.r.t. iterations. WRA-N starts with learned initial surrogate model, therefore curves are deterministic. WRA-N significantly improves baseline GP-UCB and outperforms TST-R, because test functions may not be well represented by base functions in TST-R.

heading: 4.2 HYPER-PARAMETERS TUNING IN DIFFERENTIAL EVOLUTIONARY ALGORITHM
text: Differential evolution (DE) (Galletly, 1996; Das & Suganthan, 2011) is a global optimization algorithm to minimize function f : Rd → R1. DE starts with initial population X0 =
[x01, x 0 2, · · · , x0Np ] ∈ R d×Np , and iterates uk+1i = x k i + F (x k best − xki ) + F (xkr(1) − x k r(2)), where xkr(i)(i = 1, 2) are randomly chosen from X k, xkbest is the best point in k-th iteration, F is hyperparameter. Then we can use binomial crossover and selection methods to create Xk+1. We generalize this iteration formulation to be
uk+1i = x k i + F (x k best − xki ) +D(xkr(1) − x k r(2))
using 2 hyper-parameters F and D, which is a general formulation covering most DE formulations. F and D influence optimization performance of DE (e.g., finding minimum of functions by DE) significantly, and thus we can utilize our method to tune these hyper-parameters. We use function set of CEC 2017 (Awad et al., 2016) for optimization by DE and use our method to find optimal hyperparameters. The neural surrogate model is to predict the function values w.r.t. hyper-parameters. We set function dimension as 10, construct meta train set containing basic functions of f1, f2, f3, f4, and learn initial surrogate model that can find optimal hyper-parameters well in f5−f10 for testing. The formulas of these functions are shown in appendix. For TST-R, we sample 20 points uniformly in feasible domain and use these points to construct base models. TST-R and GP-UCB start with 3 randomly chosen points, and then proceed sequentially for 20 iterations. Our method starts with surrogate model learned by meta train set, and proceeds for a total of 23 iterations. The optimization was repeated 20 times similar to the experiments on synthetic function set. Figure 3 shows results on f5 and f6, and results on the other functions are shown in appendix. Our method produces best hyper-parameter tuning performance. Note that the functions in meta train set are in different scales of function values. For TST-R, we follow its original implementation by normalizing point values of base models to normal distribution. However, in our approach, the neural surrogate model is updated using ranking-based loss, therefore it is not sensitive to the different scales of functions.

heading: 4.3 HYPER-PARAMETERS TUNING IN NEURAL NETWORK
text: We will optimize hyper-parameters of neural networks for image classification. We use EMNIST data set (Cohen et al., 2017), i.e., an extension of MNIST to handwritten letters. There are 62 classes of [0-9], [a-z] and [A-Z], and several classes that are similar between uppercase and lowercase are merged, and finally we have 47 classes in data set. The network is with one hidden layer, sigmoid as activation function and softmax for classification, followed by cross-entropy loss. We first randomly sample 10 classes in 3 times from EMNIST with 1000 images for each class,
and construct sets of Si, i = 1, 2, 3 as meta train sets. From the remaining classes, we construct test sets by randomly sampling 10 classes with 1000 images for each class in 8 times and produce S4 − S11. We aim to tune the number of hidden units Nh and learning rate lr of SGD as hyper-parameters. The candidate sets of Nh and lr are respectively {10, 15, · · · , 200} and {0.001, 0.002, · · · , 0.01, 0.02, 0.03, · · · , 0.1, 0.2, 0.3, · · · , 1, 2, 3, · · · , 10}. For this task, training accuracy is higher with increase of number of hidden units because more units produce higher capacity but may over-fit training data. Thus we define a criterion to balance accuracy (denoted by AC) and model capacity determined by Nh using ERR = (Nh + c)/AC where c is the trade-off coefficient and set to 1000. Then neural surrogate model is to predict ERR w.r.t. to hyper-parameters. Note that we can also use other criterion for learning hyper-parameters. Figure 4 shows MSEs (between ERRs of best hyper-parameters and estimated hyper-parameters) w.r.t. optimization iterations by different methods. We can observe that our method performs best for this task.

heading: 4.4 HYPER-PARAMETERS TUNING IN SVM
text: Support vector machine (SVM) dataset (Wistuba et al., 2016) consists of 50 classification datasets randomly chosen from UCI repository. The SVMs use Gaussian kernel with hyper-parameters of trade-off parameter C and width γ of Gaussian kernel. The candidate hyper-parameters are C ∈ {2−5, ..., 26} and γ ∈ {10−4, 10−3, 10−2, 0.05, 0.1, 0.5, 1, 2, 5, 10, 20, 50, 102, 103}. The neural surrogate model is to predict misclassification rate w.r.t. to hyper-parameters C, γ. In experiments, we randomly choose 5 datasets as meta train sets, and the others as test sets. All the experiments are performed in 10 times. Figure 5 shows the average MSEs w.r.t. optimization iterations by different methods. Here we compare our WRA-N with TST-M (Wistuba et al., 2016), which also relies on linear combination of surrogate models of meta train problems but using meta features. We can see that our WRA-N performs better than both TST-R and TST-M. Both our WRA-N and TST-R perform significantly better than baseline GP-UCB.

heading: 4.5 ABLATION STUDY
text: We now justify effectiveness of weighted Reptile algorithm (WRA) and different network structures of surrogate model φ. All following networks are evaluated on synthetic set in section 4.1. In Fig. 6 (left), we compare our WRA with traditional Reptile algorithm without using sampling and weighting strategy. The average MSE curves show that our WRA performs better than Reptile. We further compare different structures of surrogate models including networks with one hidden layer of 10, 15, 25 units and 2 hidden layers with 15 units. We observe that they perform similarly for networks with one hidden layer, and adding more layers does not apparently improve performance.

heading: 5 CONCLUSION
text: We have proposed a novel neural surrogate model and a weighted Reptile algorithm to learn an initial neural surrogate model for warm-starting Bayesian optimization. Extensive experiments on synthetic functions optimization and hyper-parameters tuning in multiple tasks have shown promising performance. In the future, we are interested to apply it to more global optimization or hyperparameters tuning tasks, and analyze its theoretical properties.

heading: A APPENDIX
text: In the appendix, we will first introduce how to optimize the ranking loss in A.1. Second, we will show the functions in CEC 2017 for DE hyper-parameters tuning in A.2. Third, in A.3, we will show more experimental results for synthetic function optimization, hyper-parameters tuning in DE and neural network. Fourth, we will show example to illustrate the neural surrogate model optimized by minimizing LRank in A.4. Fifth, we will show the training curves of our weighted Reptile algorithm for experiments in A.5. Finally, we will discuss on the relation between neural surrogate model and Gaussian process in A.6.
A.1 GRADIENT METHOD FOR MINIMIZING LRank
We first recall the definition of LRank in section 3.1. Given points set D = {xi, f(xi)}ni=1, we sort points in D by {f(xi)}ni=1 in descent order and get Ds = {xr(i), f(xr(i))}ni=1, where r(i), i = 1 · · ·n is a permutation of 1 · · ·n. Then the LRank is defined as
LRank(φ(w),D) = n∑ i=1 n∑ j=1,j>i ([φ(w, xr(j))− φ(w, xr(i)) + δ]+)2,
where [q]+ = max(q, 0). Then GD(w,LRank) is a gradient based method that iteratively runs the following procedure. Given each pair of sampled points (xr(i), xr(j)), we update parameter w of surrogate model by
wt+1 = wt − η[φ(wt, xr(j))− φ(wnt , xr(i)) + δ]+(∇φ(wnt , xr(j))−∇φ(wnt , xr(i))), where η is step size.
A.2 FUNCTIONS IN CEC 2017 FOR DE HYPER-PARAMETERS TUNING
The ten functions are listed as follows.
f1(x) = x 2 1 + 10 6 H∑ i=2 xi 2, f2(x) = H∑ i=1 |xi|i+1
f3(x) = H∑ i=1 xi 2 + (0.5 H∑ i=1 xi) 4 + (0.5 H∑ i=1 xi) 4
f4(x) = H−1∑ i=1 (100(xi 2 − xi+1)2 + (xi − 1)2)
f5(x) = H∑ i=1 (xi 2 − 10cos(2πxi) + 10)
f6(x) = g(x1, x2) + g(x2, x3) + · · ·+ g(xH , x1)
f7(x) = min( H∑ i=1 (x̂i − µ0)2, dH + s H∑ i=1 (x̂i − µ1)2) + 10(H − H∑ i=1 cos(2πẑi))
f8(x) = H∑ i=1 (zi 2 − 10cos(2πzi) + 10)
f9(x) = sin 2(πw1) + H−1∑ i=1 (wi − 1)2[1 + 10sin2(πwi + 1)] + (wH − 1)2[1 + sin2(2πwH)]
f10(x) = 418.9829H − H∑ i=1 g(zi)
g(x, y) in f6 is defined as g(x, y) = 0.5+ sin2( √ x2+y2)−0.5
(1+0.001(x2+y2))2 . Please refer to CEC 2017 (Awad et al., 2016) for details on parameters.
A.3 ADDITIONAL RESULTS IN EXPERIMENTS
Figure 7 shows the function optimization results on 15 remaining test functions in synthetic function set. Figure 8 shows curves of average MSEs w.r.t. different optimization iterations for tuning hyper-parameters of number of hidden units and learning rate of SGD on EMNIST data sets of S7,S8,S9,S10,S11. Figure 9 shows these curves for tuning hyper-parameters in DE on functions of f7 − f10 in CEC 2017.
A.4 AN ILLUSTRATION OF NEURAL SURROGATE MODEL BY MINIMIZING LRank
Figure 10 shows the neural surrogate model by minimizing LRank working on DE hyper-parameters tuning. In this task, we use neural surrogate model to predict the mapping from hyper-parameters D,F to function values of points. Figure 10(a) shows the surrogate model learned by neural network. Figure 10(b) shows the real function values w.r.t. hyper parameters. Figure 10(c) shows the logarithm of real function values w.r.t. hyper-parameters. Then we see that the neural surrogate model is within range of [0, 1] and it has similar local minimum with real function (by comparing Figs. 10(a) and (c)), but it is more smooth than real function surface. We further illustrate the distributions using histograms of function values of real function and neural surrogate model. As shown in Fig. 10 (d), x-axis is the percentage (i.e. ratio) of maximal function value, y-axis is the percentage of number of data with function values falling in these ratios to maximal function values. We see that, for real function, most values are less than 0.25 of maximum function value, while our neural surrogate model is more uniform.
A.5 THE TRAINING CURVES OF OUR WEIGHTED REPTILE ALGORITHM
In the four experiments in section 4, we use our weighted Reptile algorithm to learn the initial neural surrogate model. We now show the training curves when training initial surrogate model from meta train set. In Fig. 11, we show the curves of total rewards of rNTi on all problems in meta train set for four experiments. They show that weighted Reptile algorithm does increase the reward when training the initial surrogate model with increase of training epochs.
A.6 RELATION BETWEEN NEURAL SURROGATE MODEL AND GAUSSIAN PROCESS
Neal (2012) has proved that a network with random weights and one hidden layer converges to a Gaussian process when the number of hidden units increases. In this work, we directly model the surrogate model using neural network by substituting the formulation of function value prediction in Gaussian process. Different to Gaussian process, our network is with finite number of hidden units and learnable parameters, facilitating effective model learning for the task of learning to initialize surrogate model from a meta train set.

Review summaries:
Learning from past experience to quickly adapt to a new task has been an important and fast-growing issue in machine learning. Such technique facilitates Bayesian optimization as well, warm-starting Bayesian optimization. Recently a few methods have been developed along this direction, from designing handcrafted meta-features to learning meta-features. The current paper takes a similar step, learning neural surrogate model from related tasks to warm-start Bayesian optimization. The main idea is to replace the mean function of GP by neural surrogate model, so that parameterized models are used for meta-training, in the framework of RETILE. An idea of weighted REPTILE is another contribution in this paper, where parameter updates are done with weighs defined by rewards.