Learning Neural Surrogate Model for Warm-Starting Bayesian Optimization
Authors: Haotian Zhang
Jian Sun
Zongben Xu
Venue:  (2020)
Keywords: Bayesian optimization
meta learning
neural network
surrogate model
hyper-parameters tuning

Abstract: Bayesian optimization is an effective tool to optimize black-box functions and popular for hyper-parameter tuning in machine learning. Traditional Bayesian optimization methods are based on Gaussian process (GP), relying on a GP-based surrogate model for sampling points of  the function of interest.  In this work, we consider transferring knowledge from related problems to target problem by learning an initial surrogate model for warm-starting Bayesian optimization. We propose a neural network-based surrogate model to estimate the function mean value in GP.  Then we design a novel weighted Reptile algorithm with sampling strategy to learn an initial surrogate model from meta train set. The initial surrogate model is learned to be able to well adapt to new tasks. Extensive experiments show that this warm-starting technique enables us to find better minimizer or hyper-parameters than traditional GP and previous warm-starting methods.

Under review as a conference paper at ICLR 2020
LEARNING NEURAL SURROGATE MODEL FOR WARM- STARTING BAYESIAN OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian optimization is an effective tool to optimize black-box functions and popular for hyper-parameter tuning in machine learning. Traditional Bayesian op- timization methods are based on Gaussian process (GP), relying on GP-based surrogate model for sampling points of the function of interest. In this work, we consider transferring knowledge from related problems to target problem by learning an initial surrogate model for warm--tarting Bayesian optimization. We propose a neural network-based surrogate model to estimate the function mean value in GP. Then design novel weighted Reptile algorithm with sampling strategy to learn an initial surrogate model from meta train set. The initial sur- rogate model is learned to be able to well adapt to new tasks, Extensive experi- ments show that this warm-starting technique enables us to find better minimizer or pappeeerrrmeeerss than traditional GP and previous warm-starting methods.
INTRODUCTION
Bayesian optimization (Mockus 1975 Pelikan et al. 1999 Gustavo Malkomes 2018 is an opti-
mization method to find global minimizer of a black- black-box function without knowing its convexity, differentiability or continuity. Bayesian optimization iteratively samples new point by an acquisition function, and uses Gaussian process (Rasmussen) 2003) Gardner et al. (0144) to estimate a surrogate model to fit f' (ax) based on the observed points. When using specific acquisition function, its regret bound has been proved ((vn)) (n is the number of sampled points) in Srinivas et al. 2012) Freitas et al. (2012) proposed method that uses Bayesian optimization with dividing region. which successfully accelerates convergence rate. More algorithms have been developed in Waag et al. (2014); Kawaguchi et al. 2015); Springenberg et al. (2016). In application, Bayesian optimization has been widely applied machine learning for parae--t tuning (Snoek et al. (012).
A machine learning problem is formulated as follows. Given data set D, model M(. A) and loss function -(:-),, it aims to find an optimal parameter vector to minimize L(M(w; X), In practice, model M(w; X) is dependent on hyper-parameter X, which may significantly influence the model. Thus choosing appropriate À is critical procedure of learning model. For instance, Mw; A) is neural network to solve image classification problem, then À can be the number of hidden units or network layers. parameerr optimization problem can be formulated as
min L(M(w; D) min L(M(w*: A), D), where = A(C(M(w:; D)). X,30
X
and A is an algorithm minimizing loss function w.r.t. w. It is challenging to optimize Eq. I since A[C(M(w; X), is complicated function w.r.t X, which might be non-differentiable. In real applications, hyper- sapee--arameter commonly tuned by hand. Random search and grid search are typical methods to find plausible X, but they are time-consuming to find the optimal hyper-parameter because they sample the hyper-parameter in brute-force way.
Bayesian optimization is an effective tool to minimize function for global optimization or hyper- parameter tuning. Bayesian optimization iteratively samples new point and fits the function values by surrogate model based on Gaussian process. It starts with several random points without using any prior knowledge on the task. In applications, we often solve several related machine learning problems, therefore, question is that whether can learn common knowledge from related problems and transfer them to new problem as an initialization for adaptation, commonly referred to as warm-starting Bayesian optimization.

---

Under review as a conference paper at ICLR 2020
Meta-learning is promising approach for learning optimizer for fast optimization initial model for fast adaptation to novel task using a set of training problems as meta train set (Andrychowicz
2016 Li & Malik 2017 Wang al. 2019 Ravi & Larochelle, 2017 Chen et al. [017: Finn et al.
et al.
2017 Nichol et al. 2018). This idea was also applied to Bayesian optimization in past years.
Poloczek al. 62 built joint statistical model of the collection of related objective functions.
Feurer et. al. (2015) initialized Bayesian optimization based on hand-crafted meta features for hyper- parameter tuning. Given a new problem, its nyper-parameters are initialized by the most similar problem in meta train set with distance of hand-crafted meta features. However, it is challenging to define meta features artificially. Kim al. (2017) learned meta features of problems by deep metric learning. [Wistuba et al. (20T6); Feurer et al. (2018) 8) proposed to transfer surrogate model by linear combination of surrogate models of problems meta train set with coefficients determined by ranking of function values or meta features. However, representation capacity of linear combi- nation is limited, meaning that surrogate model of a new problem cannot be represented by linear combination of surrogate models of meta train problems, then this approach will fail.
In this work, we propose a novel meta-learning approach to learn prior knowledge from meta train set warm-start of unseen related target problem. More specifically, we make the following two contributions to tackle the challenges in traditional Bayesian optimization.
Firstly, as know, traditional Bayesian optimization is data-driven and depends surrogate model fitting function values without learnable parameters, limiting its ability for adapta- tion new problems. We propose use neural network model the surrogate model (dubbed as neural surrogate model) to substitute the mean of function values in GP. We further design a ranking loss for fitting surrogate model to sampled points to make surrogate model insensitive to the scale of function values.
GP-based
Secondly, we propose a novel meta-leaming algorithm to learn an initial neural surrogate model from meta train set of related problems, which is taken as initial surrogate model for adaptation to new problem. To train the initial neural surrogate model, meta-learning methods such as MAM- I. (Finn et al. 2017) are not applicable because Bayesian optimization is non non-differentiable. Thus we propose a weighted Reptile algorithm inspired by Reptile algorithm (Nichol et al.) 2018) using sampling strategy for exploration to train the initial neural surrogate model.
We evaluate our method on tasks including optimization of synthetic functions, prper-parameter tun- ing in evolutionary algorithm, neural network and SVM. These experiments show that our method outperforms traditional Gaussian process and related warm strm--tarting methods in these tasks.
2
PRELIMINARIES
2.1 BAYESIAN OPTIMIZATION
Bayesian optimization is to find a minimizer x of a function f in a bounded space by iteratively querying the function at sampled points. At 1-th iteration, we first construct probabilistic model on D- = {#;, (x;) ) 1= for estimating function values, then use an acquisition function a(x) to select next point. We estimate f by surrogate model based on Gaussian process (GP). We assume f()) ~ GP b(..)). where mi.) is mean function and k(.,:) is covariance function or kernel.
Given D = {xa, I { 1- GP is simply a joint Gaussian f(11=) ~ ' (m(11:n) ), K K(I1:n)) )), where ((ri.) is a nxn covariance matrix with elements of k(z,, xι) defined as kernel function measuring relation between x, and zy. We choose RBF kernel l( x;) =xx((- exp(-pllaz; - 2, (1²) in this work. Then for a new point In+1. we consider joint distribution previously and newly sampled points:
(
Marm)) ((er)) ) )~(((
m((2..) ) m((x=1))
k(Im411211.) k(Zn+1; k(E0+1, Rn+1)
K(21m)
(2)
where k(In+1; Fi:m) = (k(21, ([([1,,,,1] xn+1). k(22, 2n+1).... k(æns #n41)). The posterior f(#n+1|
satisfies Gaussian distribution (Bishopl 2006) with mean and variance
p( D) = k( #1m))) (((((..))) ((is)))
(3)
(z DD= k( - k(T Fi:) TK( (1)..) -
(4)
which is the surrogate model of GP. Then we use acquisition function a(x) to choose which point will be evaluated next. Expected improvement (EI) (Mockus 1975) and upper confidence bound
2

---

Under review as a conference paper at ICLR 2020
Meta train set
Acquiation farction RTA to choose nost poin)
bIstal sorreges moseo
Update
D= (i
opdriidten optidde for it. I....Ny)
Weighted Repille
Update regrt
Acquiiiton ton fureton
Using lewe
thosse nest patrt
Update
Braraloptiibationn
Fa
Figure 1:
General pipeline of learning initial neural surrogate model using weighted Reptile algo- rithm from meta train set of functions fú, = ,,
(UCB) (riinias al. 2012) have been shown be efficient. As aim to find minimizer of minimize acquisition function:
arg min a(x) = arg min(p(x) - ko)
(5)
to find next point. As we can see, GP estimates function value via given data D based on surrogate model of Eq. and Eq. (4.
2.2
META-LEARNING METHODS
MAML Finn et al. 2017 and Reptile (Nichol et al. 2018) are two typical meta-learning methods obtain an initialization of model from meta train set well adapt to new problems. Given data set D, model ((w) and loss function C(- -), MAML learns initialization w" by minimizing C (M VVL(M(MM) D)). It implies that MAML learns w" that can fast adapt by one gradient descent step. Optimization in MAML uses second-order derivatives of L(w). In our ap- plication, Bayesian optimization procedures are non-differentiable. Therefore, MAML can not be applied to task directly. Reptile is one of first-order meta-learning algorithms. As illustrated in Alg. D to obtain an optimal initialization of model parameters w that can adapt well to new problem by k-steps of optimization denoted by (² the parameter is iteratively updated by vector (w - w) that enforces w to close to the updated parameters w different tasks. In this work, we borrow idea from Reptile, but design weighted Reptile algorithm for learning initial network-based surrogate model.
Algorithm Reptile
1: Initialize w. i.e., the initial parameter
2: for = 2, ........' do
3: Sample task
4: Compute w - (w). i... steps optimization by SGD or Adam
5: Update +- t + c(wi w)
6: end for
3 METHOD
We aim to learn to warm-starting Bayesian optimization by substituting mean in Gaussian process by a network-based surrogate model o(w, x). Then we learn an initialization of the neural surrogate model from meta train set of related problems using weighted Reptile algorithm, for adaptation to new problem. We first present some key definitions.
Neural surrogate model. The surrogate model of Bayesian optimization is Eqs. and where mean in Eq. (3) is the expectation of predicted function value and variance in Eq. represents uncertainty. Since related problems in meta train may have related shapes of function surfaces,
3

---

Under review as a conference paper at ICLR 2020
substitute the mean by learnable network (w, to learn initial surrogate model from meta train set. We still use the variance in Eq. (4) to measure uncertainty, because the estimation uncer- tainty should be independent in individual problems. Using network for modeling surrogate model, we learn an initial network p(w", 2) from meta train set that can be adapted to new problem. Our neural surrogate model is just the combination of (ww, x) and variance in Eq. 4
Neural optimization episode. An optimization episode is the iterative process for optimizing the black-box function. For problem Q with corresponding black-box function f. in (1 + 1)-th step, we already have surrogate network 6(w", .) and observed points set D = {zg. J (x; ) ==1- First, we update neural surrogate model using CRank (will be discussed in sect. 3.1) and Eq. 4 and obtain @(w++1, .) and (x). Then we use acquisition function of Eq. (5) to choose next point It-1- After that, we update points set DD = =DU (1(41+ f(x++1)) We iterate the above process to obtain 'N = arg min m= } (xx)) We called this procedure as Neural optimization episode (NOE), as illustrated in the right part of Fig.
In this work, we aim to learn initial (w, a) to initialize NOE of new problem given set of related problems in meta train set, illustrated in Fig. [ Since the dynamics of NOE is complicated and might be non -in-differentiable, we propose a weighted Reptile Algorithm to learn the initial w, x) from meta train set using reward defined based on Thest of problems. This learned initial model eb( (w. x) is utilized to initialize the neural surrogate model of new problem for its NOE,
.N
3.1 RANKING LOSS FOR LEARNING NEURAL SURROGATE MODEL
We now focus on how to update neural surrogate model given observed points and their function values in neural optimization episode. As we know, different problems have different ranges of function values. To make surrogate model insensitive to the specific range of function values, our neural surrogate model is learned to predict ranking of function values of points instead of exactly fitting the function values. Describing data sets based on pairwise rankings has been investigated in a few machine learning approaches (Leite et al. 2012 Wistuba et al. 2015 Zheng et al.) (0007).
Given points set D = f(:e) we update of neural surrogate model (w,-) by RRnnk- First we sort points DD by (f(x;)) J= in descent order and get DD = (rr(0) (xr()) )= where r(i), 1...nnn. a permutation of n. Then we utilize Litank loss:
n.
(Ronk(o(w).DD =
[max(o(w, (onx(o((,,rooe Ir()) - ZT()) + 6, 0)
(6)
f=1,ン
Ronk is based on hinge loss, and minimization of LRank w.r.t w enforces that Φ(w, 2r()) is strictly larger than ((w, Xr())) with margin of 8. Note that final layer of our surrogate model is sigmoid function, therefore, the output of surrogate model is constrained in range of (0, 1) irrespective of different problems. We use gradient descent method to optimize Caank please see appendix A. for detail. We denote GD(w, LRnkk) ) as the learned surrogate model parameters by minimizing Lank using gradient descent method. This above process is taken as the updating step of neural surrogate model in neural optimization episode.
3.2 WEIGHTED REPTILE ALGORITHM FOR LEARNING INITIAL SURROGATE MODEL
We next focus on how to learn an initial neural surrogate model from a set of related problems in meta train set, taken the initialization of neural surrogate model for NOE of new problem. We propose a weighted Reptile algorithm to train the initial neural surrogate model.
Reptile in Alg. 1 is an effective meta-learning algorithm that can learn good initialization of model parameter from a meta train set. Inspired by Reptile, our Weighted Reptile Algorithm samples a set of initializations of surrogate model parameters for NOE of each problem in the meta train set for exploration, and these sampled initializations are selected by reward and utilized for parameter updating with weights defined by rewards.
Reward. Given minimizer xf of i-th problem (2 in meta train set, reward at step t of NOE of 2x; 112, where sf besst = arg min min (e. m=......... (f: (x.)).
problem Qi(x) is defined as r! ||
4

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Neural optimization episode (NOE).
Require: Maximum iterations Nr, test problem Q with function initialized parameter w
Ensure: Pbest, w
set w w. D = 0
2: x' arg min, p(w
3: y² =)==)>>), D D ,y
4: Update (xx) by Eq. 44
5: for t = :r
do
6: x arg min @(w' x) ko(
7: y'+1 = f(x(+1), D== (altl yet1}
8: w = GD(w", Roakk)
9: Update (xx) by Eq. (4)
end for
11: Tbest ; 1* = arg mink {y )
12: return Thest, Wr
fi is the black-box function in problem Qi(x), Tn is sampled point at n-th step, 2 ",est is the best point up to step t.
Weighted Reptile Algorithm. In Reptile, as in line 5 of Alg. [ the parameter w is learned to be close to updated parameter by adapting w in different problems. implicitly assumes that updated parameter w should be better than initialized w for corresponding problem. In our approach, we aim learn initial surrogate model p(w, 2) from problems in meta train set. Similar to Reptile, we learn initial network parameter of 6(w,x) close to updated parameters after running NOE of problems meta train set. To explore diverse paths of NOF, for each i-th problem in meta train set, sample few (MZ) initial parameters {w
J NNww, o²) by disturbing currently estimated w. Then after running Ny steps of NOE, we obtain points Nr.P M and updated parameters "iibbsttpp
(u.rr.MM p=1 starting from surrogate models initialized with these sampled parameters. To learn good initial parameter of surrogate model é, parameter is updated by w + 8Aw, where
M
N
Aw
Σ
Σ
New w) max(rip No .),
p=1 1=1
and reward r -110; x. r Nr = -1 - x1 For i-th problem and p-th sampled TTpp
parameter, weight max(r;) Nr r. Nr.r) is always non-negative, and is non-zero only when the sampled parameter w/ can produce better reward than the parameter w itself if taking them as initial surrogate model parameters for running NOE of problem Q, This rule enforces that the parameter w of our learned initial surrogate model is close to better adapted parameters of different problems in meta train set. Algorithm presents the pipeline of meta-train process using weighted Reptile algorithm, and it depends on the NOR as described in Alg.
EXPERIMENTS
We evaluate our method for global optimization on synthetic functions set, and hyper-parameters tuning evolutionary algorithm, neural network and SVM. In the following paragraphs, our method based on neural surrogate model and weighted Reptile algorithm will be dubbed as WRA-N. In all experiments, we set к 0.1, 5 0.01, a 0.1, training epoch N E 30, learning rate f 0.1, number of samples M = 50. We use the mean squared error (MSE) between function values of our found best point and the global minimizer to measure the performance of optimization. We mainly compare with TST-R (Wistuba et al., 2016) and GP-UCB (Srinivas al., 2012). Moreover, also compare with meta feature-based method TST-M (Wistuba et al. 2016) in SVM tuning. But for the remaining tasks, did not compare with it because it is hard well define meta-fcatures for these problems. The neural surrogate model is taken as a network with one hidden layer of 15 hidden units, and both hidden and output layers are with sigmoid nonlinearity.
5

---

Under review as a conference paper at ICLR 2020
Algorithm 3 Meta train by weighted Reptile algorithm.
Require: Number of iterations Nr in NOE, meta train set number of epochs Ng. learning rate 3, number of samples M, standard deviation o for sampling.
Ensure:
1: Randomly initialize w²
2: #Meta-train loop.
3: for 1 Nr do
4: #Surrogate model adaptation loop.
5: for = =::=1:10 |2| do
{arNNr Wi NT NOE(N K; Qi w²)
6:
r; .r = - j,estt 11
Nr
7:
8:
for p = M do
9:
w. w N000.011
10:
{ L.bet: 1i, ] = NOE(NOO Nr. K, Qi w.))
NT-P
Tip -1| -1111 z 112 J. = -ū;" N w
NY-P
11:
end for
13:
end for
#Updating initial surrogate model parameter.
15: wit1 = w' + | 2p=1 max(rig .rr 00)"
16: end for
return w Ne+1
Loganitnn
Legeeitm
12
10
12
10
Dirations
Dicrations
Bierations
Figure 2: MSEs w.r.t. optimization iterations on synthetic functions with a
3r 2
25 3
4.1 SYNTHETIC FUNCTIONS
We first evaluate on Alpine 1 function in Jamil & Yang (2013) without using absolute value: fi In as meta 3 4 5
f(z) = xsinn( + a) + 0.1xx x E [-10, 10]. train set, and 18 functions with = 3x for testing. Figure
We set functions with a 4n Su
2 27 E (1...., excluding {4, 6, 8} in meta train set 2 shows logarithm of MSES w.r.t optimization iterations of WRA-N, TST-R and 2x GP-UCB for 2 ,,, 3. and other results are in appendix. Optimization iteration means the it- eration in optimization episode, e.g., the iterations in NOE of our WRA-N. Since TST-R needs base models for combination, base models. TST-R and GP-UCB start with random points, then run in 10 iterations. Our WRA-N starts with neural surrogate model learned meta train set, and runs in 13 iterations, i.e., Ny 13. Optimization was repeated 20 times for TST-R and GP-UCB, For TST-R, each time is with differ- ently sampled points for constructing base models. Figure 2 shows curves of average MSEs with variances in logarithm w.r.t. iterations. WRA-N starts with learned initial surrogate model, therefore curves deterministic. WRA-N significantly improves baseline GP-UCB and outperforms TST-R, because test functions may not be well represented by base functions in TST-R.
sample 20 points from uniform distribution in (-10, 10) to construct
4.2 HYPER-PARAMETER TUNING IN DIFFERENTIAL EVOLUTIONARY ALGORITHM
Differential evolution (DE) Galletly
1996
Das Suganthan, 2011 is a global optimization
algorithm minimize function
Re
R
DE starts with initial population X

---

Under review as a conference paper at ICLR 2020
30
B8AX 6176
Logaribm MfE
Loguithm MSE
100
-40
-150
20
20
Bierations
Figure 3: MSES w.r.t. iterations for fs. fo in CEC 2017 for hyper-parameters tuning of DE.
KAAM TSTR
Loganiiin MASE
Logedtma MSSE
Logattin of MSE
Figure 4: MSES w.r.t optimization iterations for hyper-parameters tuning of neural networks on data sets of Ss Sp.
[ef. 222 ,,,,, RdxNN,, and iterates u. = x + F(x =x²) + F(a - x ) where 2 (()) 2) are randomly chosen from X ahest the best point k-th iteration, F is hyper- binomial crossover and selection methods to create X We general-
.x4,1
parameter. Then ize this iteration formulation to be
k+1 2. F(rbest - x1 D D(2 x11))) 1
using 2 syyer--arametten F' and D, which a general formulation covering most DE formulations. F and D influence optimization performance of DE (e.g., finding minimum of functions by DE) significantly, and thus we can utilize our method to tune these hyper-parameters. We use function set of CEC 2017 (Awad al. 2016) for optimization by DE and use our method find optimal hyper- parameters. The neural surrogate model is to predict the function values w.r.t. hyper-parameters We function dimension 10, construct meta train set containing basic functions of S1, J2, fa, f4, and learn initial surrogate model that can find optimal hyper-paramette formulas of these functions are shown in appendix. For TST-R, we sample 20 points uniformly in feasible domain and use these points to construct base models. TST-R and GP-UCB start with randomly chosen points, and then proceed sequentially for 20 iterations. Our method starts with surrogate model learned by meta train set, and proceeds for total of 23 iterations. The optimization was repeated 20 times similar to the experiments on synthetic function set. Figure shows results on Js and Ja. and results on the other functions are shown in appendix. Our method produces best hyper-parameter tuning performance. Note that the functions in meta train set are in different scales of function values. For TST-R. follow its original implementation by normalizing point values of base models to normal distribution. However, our approach, the neural surrogate model updated using ranking- based loss, therefore it is not sensitive to the different scales of functions.
well in fs - fio for testing. The
4.3 SYPER-PARAMETERR TUNING IN NEURAL NETWORK
We will optimize hyper-parameters of neural networks for image classification. We use EMNIST data set (Cohen et al.) 2017). i.e., extension of MNIST to handwritten letters. There are 62 classes of [0-9], [a-z] and [A-Z], and several classes that are similar between uppercase and low- merged, and finally have 47 classes in data set. The network is with one hidden layer, sigmoid as activation function and softmax for classification, followed by cross-entropy loss. We first randomly sample 10 classes in 3 times from EMNIST with 1000 images for each class,

---

Under review as a conference paper at ICLR 2020
WEAN 75T-R SPICB TST-M
No mdkes wito righ ueat.
GRARD (ORVI GEFO
Logaring or
Logenitie AOSI
22
Bcration
Thentions
Figure 5: MSEs w.r.t. iterations for hyper-parameters tuning of SVMs.
Figure 6: Ablation study.
and construct sets of S, 1,2.3 3 meta train sets. From the remaining classes, struct test sets by randomly sampling 10 classes with 1000 images for each class times and produce S4 - S11. We aim to tune the number of hidden units Nv. and learning rate Ir of S- GD as hyper-paramettes. The candidate sets of N₂, and Ir are respectively {10, 15, .... 200} and (0.001, 0.002..... 0.001 0.02, 0.03, 0,1, 0.2, 0.3,
1, 3, ..... 10]. For this task, training accuracy is higher with increase of number of hidden umits because more units produce higher ca- pacity but may over-fit training data. Thus we define criterion balance accuracy (denoted by AC) and model capacity determined by Nr using ERR = (N₂, c)/AC AC where is the trade-off co- efficient and set to 1000. Then neural surrogate model predict ERR w.r.t. to hyyer-parameen Note that we can also use other criterion for learning typer-parameters. Figure 4 shows MSEs (be- tween ERRs best paprmpprrreeters and estimated by different methods. We can observe that our method performs best for this task.
w.r.t. optimization iterations
4.4 HYPER-PARE PARAMETERS TUNING IN SVM
Support vector machine (SVM) dataset (Wistuba et al. 2016) consists of 50 classification datasets randomly chosen from UCI repository. The SVMs use Gaussian kernel with hyper-parameters of trade-off parameter C and width y of Gaussian kernel. The candidate hyper-parameters are E (2-5, ..... 26 and 7 E {10-4, 10 10-3, 1002 0.05,, 0. 0.5, 1, 2, 5, 10, 20, 50, 10 10a). The neural surrogate model is to predict misclassification rate w.r.t. to pppee-parameters C, In experiments, randomly choose datasets meta train sets, and the others test sets. All the experiments are performed in 10 times. Figure 5] shows the average MSEs w.r.t. optimization iterations by different methods. Here compare our WRA-N with TST-M (Wistuba et al., (016), which also relies linear combination of surrogate models of meta train problems but using meta features. We can that WRA-N performs better than both TST-R and TST-M. Both our WRA-N and TST-R perform significantly better than bascline GP-UCB
4.5 ABLATION STUDY
We now justifyy effectiveness of weighted Reptile algorithm (WRA) and different network structures of surrogate model Φ. All following networks are evaluated on synthetic set in section 4.1. In Fig. (left), compare WRA with traditional Reptile algorithm without using sampling and weighting strategy. The average MSE curves show that our WRA performs better than Reptile, We further compare different structures of surrogate models including networks with one hidden layer of 10, 15, 25 units and 2 hidden layers with 15 units. We observe that they perform similarly for networks with one hidden layer, and adding more layers does not apparently improve performance.
5 CONCLUSION
We have proposed novel neural surrogate model and a weighted Reptile algorithm to learn an initial neural surrogate model for warm-starting Bayesian optimization. Extensive experiments on synthetic functions optimization and hyper-parameee ing performance. In the future, we are interested to apply to more global optimization or hyper- parameters tuning tasks, and analyze its theoretical properties.
tuning in multiple tasks have shown promis-
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning learn by gradient descent by gradient descent. In NeurlPS, pp. 3981-3989, 2016.
N. H. Awad, M. 7. Ali, J. J. Liang, B. Y. Qu, and P. N. Suganthan. Problem definitions and evalu- ation criteria for the cec 2017 special session and competition on single objective real- callparameter numerical optimization. Tech. Rep., 2016.
Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
Yutian Chen, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matthew Botvinick, and Nando De Freitas. Learning to learn without gradient descent by gradient descent. JCML, pp. 748 0568756,200 2017.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters. arXiv preprint arXiv:1702.05373, 2017.
Swagatam Das and Ponnuthurai Nagaratnam Suganthan. Differential evolution: A survey of the state-of-the-art. IEEE Transactions on Evolutionary Computation, 15(1):4-31, 2011.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. AAAI, 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- mization. In ICML AutoML Workshop, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In ICML, pp. 1126-1 135, 2017.
Nando De Freitas, Alex Smola, and Masrour Zoghi. Exponential regret bounds for gaussian process bandits with deterministic observations. ICML, 2012.
John Galletly. Evolutionary algorithms in theory and practice. Complexity, 2(8):26-27, 1996.
Jacob Gardner, Matt Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John Cunningham. Bayesian optimization with inequality constraints. In ICMI,, pp. 937-945, 2014.
Roman Garnett Gustavo Malkomes. Automating bayesian optimization with bayesian optimization. In NeurlPS, 2018
Momin Jamil and Xin-She Yang. A literature survey of benchmark functions for global optimization problems. arXiv preprint 338 4008, 2013.
Kenji Kawaguchi, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Bayesian optimization with exponential convergence. In NeurlPS, pp. 8809-28111 2015.
Jungtack Kim. Sachoon Kim, and Seungjin Choi. Learning to warm-start bayesian hyperparameter optimization. arXiv preprint arXiv:1710.06111 2017.
Rui Leite, Pavel Brazdil, and Joaquin Vanschoren. Selecting classification algorithms with active testing. In International workshop on machine learning and data mining pattern recognition, pp. 117-131. Springer, 2012.
Ke Li and Jitendra Malik. Learning optimize. In ICLR, 2017.
Jonas Mockus. On bayesian methods for secking the extremum. In Optimization Techniques IFIP Technical Conference, pp. 4004404. Springer, 1975.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta- leta-learning algorithms. arXiv preprint arXiv: 1803. /803.02999 2018

---

Under review as a conference paper at ICLR 2020
Martin Pelikan, David E Goldberg, and Erick Cantupaz. Boa: the bayesian optimization algorithm. In Genetic and evolutionary computation conference, pp. 525-532, 1999.
Matthias Poloczek, Jialei Wang, and Peter Frazier. Warm starting bayesian optimization. In Winter simulation conference, pp- 770-781, 2016.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Lecture notes in computer science, pp. 63 13-71.30 Springer, 2003.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICIR, 2017.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In NewrIPS, 2012.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In NeurIPS, pp. 4141-4149, 2016.
N. Srinivas, A. Krause, S.M. Kakade, and M. Seeger. coormatioo-theoretti regret bounds for gaus- sian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5): 3250-3265, 2012.
Shipeng Wang, Jian Sun, and Zongben Xu. Hyperadam: A learablle task-adaptive adam for network training. In AAAI, 2019.
Ziyu Wang, Babak Shakibi, Lin Jin, and Nando De Freitas. Bayesian multi-scale optimistic opti- mization. In AISTAT, pp. 1005-1014, 2014.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt- Thinmde Sequential model-free hyperparam- eter tuning. In ICDM, pp. 1033-1038, 2015.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Two-stage transfer surrogate model for automatic hyperparameter optimization. In Joint European conference on machine learning and knowledge discovery in databases, pp. 199 199-214 214. Springer, 2016.
Zhaohui Zheng, Keke Chen, Gordon Sun, and Hongyuan Zha. A regression framework for learn- ing ranking functions using relative relevance judgments. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp- 287-294. ACM, 2007.
A APPENDIX
In the appendix, we will first introduce how to optimize the ranking loss in A.1. Second, we will show the functions in CEC 2017 for DE hyper-parameters tuning in A.2. Third, A.3, we will show more experimental results for synthetic function optimization, hyper-panee pprer-aarsmeters tuning in DE and neural network. Fourth, we will show example to illustrate the neural surrogate model optimized by minimizing Rank in A.4. Fifth, we will show the training curves of our weighted Reptile algorithm for experiments in A.5. Finally, we will discuss on the relation between neural surrogate model and Gaussian process in A.6.
A.1 GRADIENT METHOD FOR MINIMIZING LR
We first recall the definition of Rank in section 3.1. Given points set D = (5i, ((:))) 1- we sort points in D by (fxx)) _ in descent order and get D. = (Ir()): ((5r() 1=11 where r(1);
1...n n is permutation of 1- ... Then the Rann is defined as
CRank(Ó(u))
Σ
Σ
Σ
(lo(w, xx)) o(w, -fw, (r()) 6)
j=1
where [g]+ max(q, 0). Then GD(w, LRonk) is gradient based method that iteratively runs the following procedure. Given each pair of sampled points (Er(), Zr()), we update parameter w of surrogate model by w #+1 = = wt where 77 is step size.
, - -(w) =+(0)) + 5]+ (Vó(w,, - Vo(wF, zr(6)),
10

---

Under review as a conference paper at ICLR 2020
A.2 FUNCTIONS CEC 2017 FOR DE HYPER- PARAMEERSS TUNING
The ten functions are listed as follows.
H
H
fi(x)
= = + 10
Σ
f2(x)
= Σ
i+1
1=2
1=1
H
fa(x)
Σ
44 (0.5
Σ
v4)² + (0.5
1=1
1=1
i=1
H-1
1(2)
(100(2; - xx+11) (x; 1)2)
H Σ
fs(Ø)
(x,² 10cos(2Tx;) +0)
=1
f6(x)
g x2)
g 133 g(HH
H
fr(x)
min Σ ( ~0) dH + Σ
(3, 21)2) 10(H Σ cos (272.)
H Σ
fa(x)
(2,² 10c0s(2 + 10)
H1
fo(x) = sin "in?(Tw)) ) +
Σ
(w: 1) + 10sin + 1)] (WH 1 + +snn (2TT())
i=1
f10(2) 418.982911
g(2))
g( y) in So is defined as g(v, v) = 0.5 sin +x+ 2016) for details parameters.
0.5 -0.5 (1+0.001 +1
)
Please refer to CEC 2017 (Awad et al.
A.3 ADDITIONAL RESULTS IN EXPERIMENTS
Figure shows the function optimization results on remaining test functions in synthetic func- tion Figure shows curves of average MSEs w.r.t. different optimization iterations for tuning hyper-parameters
number of hidden units and learning rate of SGD on EMNIST data sets of ST, Sa, Sg, Sjo, S11. Figure shows these curves for tuning hyper-parameters in DE on functions of fr - fio in CEC 2017.
A.4 AN ILLUSTRATION OF NEURAL SURROGATE MODEL BY MINIMIZING LRank
Figure showwws the neural surrogate model by minimizing CRank working DE hyper-parameterr tuning. In this task, we use neural surrogate model to predict the mapping from hyper-parameters D. function values of points. Figure 10 shows the surrogate model learned by neural net- work. Figure b) shows the real function values w.r.t. hyper parameters. Figure [0(c) shows the logarithm of real function values w.r.t. hyper-pane syper-parameters. Then we see that the neural surrogate model is within range of [0. 1] and it has similar local minimum with real function (by comparing Figs. a) and (c)), but is more smooth than real function surface. We further illustrate the distri- butions using histograms of function values of real function and neural surrogate model. As shown in Fig. (d). x-axis is the percentage (i.e. ratio) maximal function value, y-axis is the percentage of number of data with function values falling in these ratios maximal function values. We that, for real function, most values are less than 0.25 of maximum function value, while our neural surrogate model is more uniform.
11

---

Under review as a conference paper at ICLR 2020
10
10
10
10
-10
-10
-10
10
10
10
15
10
Logarithm MSE
-5
10
-10
10
WRA-N
GP-UCB
TST-R
-10
-10
10
10
10
Iterations
Figure MSEs (in logarithm) w.r.t. different optimization iterations on 15 remaining test functions in synthetic function set.
Logarithm of MSE
Iterations
Iterations
Iterations
WRAN GP-ICR TST.
Iterations
Iterations
Figure 8: MSEs (in logarithm) w.r.t different optimization iterations for tuning number of hidden units and learning rate on EMNIST data sets of S7, Sz, Sg, S10,S11
A.5 THE TRAINING CURVES OUR WEIGHTED REPTILE ALGORITHM
In the four experiments in section 4, we use our weighted Reptile algorithm to learn the initial neural surrogate model. We now show the training curves when training initial surrogate model from meta train set. In Fig. 11 show the curves of total rewards of 7 all problems in meta train set for four experiments. They show that weighted Reptile algorithm does increase the reward when training the initial surrogate model with increase of training epochs.
12

---

Under review as a conference paper at ICLR 2020
40
-20
-42
-80-
-00
Logarithm of MSE
-00
122
10
WRA-N GP-UCB
20
10
TST-R
-20
-10
-0-
-120
10
15
-140
Iterations
Figure 9: MSEs (in logarithm) w.r.t different optimization iterations for tuning hyper-parameters of DE on f7 fio of CEC 2017.
Funatan value
Neural surrogate model
Funcrrion valle
10'
Real function
Percentage the number of data
Neural surrogate model
0.5
0.8
Real fimetion
0.6
F
0.4
Logarithm of real function
0.2
Funebien
100
Percentage maximum function value
D
(d)
Figure 10: An Illustration of neural surrogate model for hyper-parameter tuning of DE.
Meta traming reward
Meta training soward
Mata sramiing rowaad
WWV Mm
Mob
Meta training epochs
Meta training epochs
Meta training epochs
Meta training epochs
Figure 1I: Training curves of rewards on meta train set respectively for our four different experi- ments in sections 1-4.4.
13

---

Under review as a conference paper at ICLR 2020
A.6 RELATION BETWEEN NEURAL SURROGATE MODEL AND GAUSSIAN PROCESS
Neal (2012) has proved that a network with random weights and one hidden layer converges to Gaussian process when the number of hidden units increases. In this work, we directly model the surrogate model using neural network by substituting the formulation of function value prediction in Gaussian process. Different Gaussian process, our network is with finite number of hidden units and learnable parameters, facilitating effective model learning for the task of learning to initialize surrogate model from meta train set.
14


Reviews:
[{"review_id": "SJe6YTeoKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\n  \nThe authors propose a meta-learning approach for BO. The method consists in using a NN as the predictive mean of the model used to guide the search in BO. This NN is initialized cleverly so that its solutions is close to the actual solution to the problem by using related optimization tasks. The proposed method is validated on several experiments.\n\nDetailed comments:\n\nThe writing of the paper needs to be improved. It has awkward sentences like \"Bayesian optimization iteratively samples new point by\".\n\nThe intro on Bayesian optimization has to be improved, it explain very poorly this technique.\n\nThe description of the MAML method is not clear. The same for the reptile algorithm.\n\nEq. (3) is wrong. It does not take into account that the mean of the GP is different from 0.\n\nIt seems the authors replace the mean of the GP predictive distribution with the output of a neural network. This is strange and not very well justified. I would have expected that they use the output of the NN as the GP prior mean, to then compute the GP posterior mean.\n\nIn the related problems the actual objective is unknown. How is that difficulty addressed?\n\nIn Section 3.1, why do not you standardize the output values to have zero mean and unit deviation instead of using the ranking?\n\nHow are the hyper-parameters of the GP tuned? It seems GP based method will fail, essentially, because the authors do not consider the posterior distribution of the GP and change the mean of the predictive distribution to be the NN output.\n\nThere are no error bars for WRA-N in the experiments. Does this means that only one realization has been carried out? If so, this is insufficient to extract any conclusion. The results can be simply obtained by chance. The authors should consider several repetitions with different random seeds or different problems.\n\nThe experiments in section 4.1 are non realistic, since the actual shape of the objective function is known beforehand.\n\nIn section 4.2 it is not clear what relation are between the train / test functions. Therefore, it is not possible to understand why the proposed approach works better.\n\nThe experiments need another baseline to compare with. Namely, the same method in which the NN is randomly initialized. This will allow to check that the meta-learning procedure is useful. Currently, it can be the case that the improvements are simply due to using a different model for optimization.\n\nSumming up, I think that:\n\n(1) This paper needs further improvement in the writing.\n\n(2) The experimental section is questionable since there are missing methods in the comparison and no error bars in the experiments.\n\nTherefore I believe that this paper is still at an early stage and not ready for publication.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1exBwhaKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POST-REBUTTAL FEEDBACK\n\nThanks for your response. \n\nThe justifications provided in the response have not convinced me to improve my score. They are at times hard to understand: For example, the authors have claimed that while their design choice is not reasonable, it is less unreasonable than the other.\n\nSUMMARY OF REVIEW\n\nThe authors have proposed the use of a neural surrogate model in place of the GP posterior mean and a weighted Reptile algorithm to meta-learn the initial weights of the neural surrogate model. This approach appears interesting. However, there seems to be multiple highly restrictive (at times impractical) assumptions in this work that are atypical of the BO setting adopted by other meta BO algorithms and not discussed, as detailed below. Justifications are required.\n\nClarifications are also needed with regards to how they exactly run their algorithm in the experiments and whether the prior/initial information from related problems/meta tasks provided to the tested algorithms is fair.\n\n\n\nDETAILED COMMENTS\n\nThe authors say that \"We still use the variance in Eq. (4) to measure uncertainty, because the estimation uncertainty should be independent in individual problems.\" This does not seem to hold true. If a meta task or train set is indeed correlated (or provides information) to the new problem, the posterior variance/uncertainty at a point depends on the observations in the meta task or train set near to this point (see, for example, \nFeurer et al. (2018)). Can the authors discuss the implications of such an assumption in their work?\n\nQ and Q_i have always been referred to as problems. In Algorithm 3, Q is suddenly referred to as meta train set. On page 4, you have said that x^*_i is the minimizer of i-th problem Q_i(x) in meta train set. Based on these information, I assume that the authors consider x^*_i as the global minimizer and that x^*_i is known in order to compute the rewards. Can the authors discuss why is this a reasonable assumption?\n\nIn their proposed weight Reptile algorithm (Algorithm 3), the authors have also assumed access to the black-box functions of the related problems or meta tasks, which is not typical of other meta BO works that only require the existing observations or datasets of the related problems/meta tasks. As a result, compared with the existing meta BO algorithms, their proposed weighted Reptile is considerably more expensive due to the need to additionally evaluate the black-box functions of the related problems or meta tasks many times during execution. Can the authors discuss the practical implications of such an assumption and how it affects the types of problems/applications that can be considered by this work?\n\nThe authors have not provided any justification for their choice of reward on page 4. If the black-box function is indeed complex and highly varying, the distance between points may not work well at all. Can the authors provide a justification and discuss the practical implications and limitations with such a choice?\n\nIsn't it more natural to consider a single Bayesian neural network instead of using a neural network for the mean and a GP for the variance?\n\nFor the experiments, it would be good to see two other variants of the proposed algorithm to understand the individual contributions of the neural surrogate model and weighted Reptile algorithm: one without neural surrogate model and the other with simply the use of neural surrogate model.\n\nCan the authors explain in greater detail how they run their algorithms (Algorithms 2 and 3) in the experiments? For example, the authors say that \"WRA-N starts with learned initial surrogate model\". I assume that WRA-N refers to Algorithm 3 based on its acronym. Isn't the learned initial surrogate model the output of WRA-N in the first place? Also, the graphs in Fig. 2 seem to show iteration 1 to 13 in NOE (Algorithm 2). However, Algorithm 3 accepts N_T = 13 and executes NOE for N_T = 13 (and not 1, 2, or 3, ...) for each problem in each epoch. How do the authors generate the plot of WRA-N for iterations 1 to 12? \nWhat seems to make more sense to me is that the authors in fact run Algorithm 2 instead of Algorithm 3 for each experiment and they initialize w in Algorithm 2 to the output of Algorithm 3. In any case, a clarification is needed here.\n\nIt is not clear to me whether the initial/prior information from related problems/meta tasks provided to WRA-N, TST-R, AND TSR-M is fair. Can the authors provide a justification?\n\nTo clarify, for each related problem/function, only N_T number of datapoints are used to train a corresponding neural network with 1 hidden layer of 15 hidden units?\n\nThe authors say that \"Since TST-R needs base models for combination, we sample 20 points from uniform distribution in (−10, 10) to construct base models.\" Is this sampling procedure the same as that in (Wistuba et al., 2016)?\n\nCan the authors explain the comparable performance of WRA-N and TST-R in Fig. 5? Why are the error bars missing?\n\nHow does the proposed approach compare with that of Feurer et al. (2018)?\n\n\n\nMinor issues\n\nPage 1, 3: adapt well to new tasks.\nPage 2: The author says \"depends on a GP-based surrogate model fitting function values without learnable parameters\". This is not true: The GP hyperparameters need to be learned and they adapt to new problems.\nPage 4: descent order?\nPages 4, 5: Why is there an input x to Q_i?\nAlgorithm 2: t^* should be at the superscript of x.\nEquation 7: What is N?\nPage 5: Does it make a difference in the performance when delta is set to 0?\nPage 5: well define meta-features?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylBDJDFYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Learning from past experience to quickly adapt to a new task has been an important and fast-growing issue in machine learning. Such technique facilitates Bayesian optimization as well, warm-starting Bayesian optimization. Recently a few methods have been developed along this direction, from designing handcrafted meta-features to learning meta-features. The current paper takes a similar step, learning neural surrogate model from related tasks to warm-start Bayesian optimization. The main idea is to replace the mean function of GP by neural surrogate model, so that parameterized models are used for meta-training, in the framework of RETILE. An idea of weighted REPTILE is another contribution in this paper, where parameter updates are done with weighs defined by rewards.\n\n---Strength---\n\n- Learning initialization for a surrogate model to warm-start Bayesian optimization is a sound approach.\n\n---Weakness---\n\n- While mean function of GP is replaced by a neural surrogate model, the posterior variance of GP should be calculated. In other words, GP regression should be run in addition to updating the neural surrogate model. One can use the conditional neural process (instead of GP regression). Have a look at the ICML18 paper: Marta Garnelo et al. (2018), \"Conditional neural processes,\" ICML.\n\n---Comments---\n\n- You can also learn an initial mean function of GP. Any comparison?\n- There is also interesting work on meta Bayesian optimization: Zi Wang et al. (2018), \"Regret bounds for meta Bayesian optimization,\" NeurIPS.\n- Ranking loss is used to train neural surrogate models. It is not clear why minimizing ranking loss makes sense in this case. It will be different from the mean function of GP regression, so it is not clear what is the behavior of the acquisition function constructed by the neural surrogate model as the posterior variance of GP.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is concerned with warm-starting Bayesian optimization (i.e. starting with a better surrogate model) through transfer learning among related problems. 

While the key motivation for warm-starting BO is certainly important (although not novel), there are important shortcomings in the way the method is developed and demonstrated. Firstly, the reviewers questioned design decisions, such as why combine NNs and GPs in this particular way or why the posterior variance of the hybrid model is not calculated. Moreover, there are issues with the experimental methodology that do not allow extraction of confident conclusions (e.g. repeating the experiments for different initial points is highly desirable). Finally, there are presentation issues. The authors replied only to some of these concerns, but ultimately the shortcomings seem to persist and hint towards a paper that needs more work.

Decision: Reject