Impact of Invariant Objective for Order Preserving
Transformation in Bayesian Optimization
Shinichi Shirakawa
Faculty of Environment and Information Science
Yokohama National University
Yokohama, Kanagawa, JAPAN
Email: shirakawa-shinichi-bg@ynu.ac.jp

Abstract—Bayesian optimization is a black-box optimization
method, which maintains a surrogate model learned by using
previously evaluated solutions and selects the next solution to
be evaluated using the model. Since the Bayesian optimization
method models the objective function as it is, it is not invariant
under the order preserving transformation of the objectivefunction. On the other hand, many evolutionary algorithms
have that property only by using the ranking information ofsolutions. In this paper, we introduce two types of invariant
objective function: the ranking-based objective and the Lebesgue
measure-based objective, into the Bayesian optimization in orderto realize the invariance property. The impact of the invariant
objective function for the search performance is verified throughthe numerical experiment. The experimental result shows thatthe introduced objectives achieve the invariance for the orderpreserving transformation without the considerable performance
deterioration in the Bayesian optimization.

I. I NTRODUCTION
Bayesian optimization [1], [2] is a global optimization
method of black-box and noisy objective functions and is
known as an efficient optimization algorithm particularly in an
expensive scenario in which the cost of the function evaluation
is so high. A surrogate model that estimates the black-box
objective function is constructed and used for selecting apromising candidate solution to be evaluated. After the evaluation of the selected solution, the surrogate model is updated byadding the evaluated solution. A Gaussian process is usuallyadopted as the surrogate model, which can easily handle theuncertainty and noise of the objective function. Recently, theBayesian optimization has shown effectiveness in the hyperparameter tuning of machine learning algorithms [3]–[5], e.g.,
hyper-parameter tuning of the deep neural network.
In this paper, we focus on the order preserving transformation of objective function. Let us consider the two
maximization problems, maximizex∈X f (x) and the transformed problem by a monotonically increasing function g,
maximizex∈X g(f (x)), where X indicates the domain of thesolutions. The transformation by the function g preserves
the objective rankings of solutions on the original objectivefunction f (x), i.e., g(f (x)) < g(f (y)) for any x, y ∈ X s.t.
f (x) < f (y). This means that the optimum point and therankings of solutions do not change under such transformation.
From the viewpoint of black-box optimization, the performance of optimization algorithms should not be affected bysuch objective transformation1 . In some situations, the objective value is measured by logarithmic scale, log(f (x)), insteadof the original value f (x). If the optimization algorithm is notinvariant under the order preserving transformation, the performance of the optimization algorithm, such as the probability offinding the optimum and the convergence speed, must change
on log(f (x)) and f (x). Meanwhile, the experimental results ofa certain optimization algorithm in a certain optimization problem can carry over to the other order preserving transformed
problems if the optimization algorithm is invariant under theorder preserving transformation of objective function. In suchcase, we can generalize the experimental performance of the
optimization algorithm not only for the specific problem but
also for the set of problems. However, the Gaussian processmodel usually used in Bayesian optimization is not invariant
under the order preserving transformation as it attempts toestimate the objective function as it is.
The invariant property for such transformation is often
discussed in the evolutionary computation community. The
simple way to realize the invariance property is not using theobjective value as it is and only using the ranking (order)
information of solutions in the optimization algorithm. Suchranking or comparison-based black-box optimization algorithms are invariant under the order preserving transformationof the objective function. Many ranking- or comparison-based
methods are proposed in the context of evolutionary algorithm,
such as the genetic algorithm (GA) with tournament selection,
the covariance matrix evolution strategies (CMA-ES) [6], [7],and particle swarm optimization (PSO).
The ranking of a solution rk(x) directly links to the estimateof the quantile under the probability distribution p(x) of thesolutions {xi } [8], [9]. Let us consider the case of maximization problem, then the quantile of f (x) that represents theprobability of sampling a better point than x from p(x) is
defined by∫q(x) = I{f (x) ≤ f (y)}p(y)dy,(1)
where I{·} is the indicator function, and the small quantile
value corresponds to the better solution. We then get the
1 Such invariance property may not be required when we want to know thesensitivity of the objective values for the variables or estimate the concretelandscape of the objective function.

estimate of the quantile for the solution xi by the Monte-Carloapproximation using n samples drawn from p(x) as follows:1∑
rk(xi )
I{f (xi ) ≤ f (xj )} =
,
n j=1
n
n

q̂(xi ) =(2)

where rk(xi ) = #{j, f (xi ) ≤ f (xj )}. Given a non-increasing
function w, the original problem, maximizing f (x), is reduced
to the alternative problem, maximizing w(q(x)).
The preference weights of solutions used in the CMAES can be regarded as this type of objective transformation.
Generally, we can interpret that the ranking-based methods
measure the quality of the solutions based on the quantile
q(x) instead of the original objective f (x). As the quantile
q(x) obviously depends on the probability distribution p(x) of
samples, the quality measure of solutions changes according
to the sample distribution. Akimoto [9] has proposed a moregeneral quality measure of solutions using Lebesgue measure,which is also invariant under the order preserving transformation of the objective function.
The goal of this paper is to introduce the invariance property
for the order preserving transformation of objective functionin the Bayesian optimization. Specifically, we propose toincorporate the invariant objectives, the ranking-based objective, and the Lebesgue measure-based objective [9] into theBayesian optimization and verify their impact on the search
performance through the numerical experiments.
The next section of this paper presents an overview of theBayesian optimization. We then explain about the invarianceobjective functions for the order preserving transformation in
Section III. In Section IV, we describe the Bayesian optimization algorithm with invariant objective functions. Next,
in Section V, we verify the impact of the invariance objectivefunctions in the Bayesian optimization through the numericalexperiment. Finally, in Section VI, we describe our conclusionand future work.

covariance function k : Rd × Rd 7→ R, the predictive
2
distribution is given by N (µn+1 (x), σn+1(x)), andµn+1 (x) = µ0 + k(x)T (K + σ 2 I)−1 (y − µ0 1),

(3)

2
σn+1
(x) = k(x, x) − k(x)T (K + σ 2 I)−1 k(x),(4)

where K is n × n covariance matrix whose element
is the covariance function Kij = k(xi , xj ), k(x) =[k(x, x1 ), . . . , k(x, xn )]T is the vector of covariance between
x and the observed points, y = [y1 , . . . , yn ]T is the vector of
the observed objective values, and I and 1 indicate the identity
matrix and the vector of all 1, respectively. Additionally, a
constant bias µ0 corresponds to the prior mean, and σ 2 is thevariance of the observation noise.
The next promising solution to be evaluated is found by
optimizing the acquisition function α(x|Mt ) on the surrogate
model. The acquisition function generally controls the balance
between exploitation and exploration, and several acquisition
functions have been proposed., e.g., the expected improvement(EI) [1], the probability improvement (PI) [1], the upper
confidence-bound criteria (UCB) [11], and the mutual information (MI) [12]. The expected improvement is the mostcommonly used acquisition function and experimentally showsthe effectiveness, given by
αEI (x|M) = σ(x) (Z(x)Φ(Z(x)) + ϕ(Z)) ,(5)µ(x) − f (xbest )(6)
Z(x) =σ(x)where xbest is the current best solution, and Φ and ϕ denote thestandard normal cumulative distribution and density functions,
respectively.
III. I NVARIANT O BJECTIVE F UNCTION FOR THE O RDER
P RESERVING T RANSFORMATION
We describe two invariant objective functions for the orderpreserving transformation: the ranking-based objective and theLebesgue measure-based objective.

1) Optimize the acquisition function α(x|Mn ) on the surrogate model Mn and get the next promising solution,
xn+1 = argmaxx∈X α(x|Mn ).
2) Evaluate the selected solution, yn+1 = f (xn+1 ).
3) Add the new observation to the observed dataset,
Dn+1 = {Dn , (xn+1 , yn+1 )}, and update the surrogate
model to Mn+1 using Dn+1 .

A. Ranking-based Objective
As we described in Section I, the ranking-based method
which links to the quantile estimation is the most reasonable
way for obtaining the invariance property. By transforming theoriginal objective function into the quantile-based function, theblack-box optimization algorithm becomes invariant under theorder preserving transformation.
The quantile-based objective function to be maximized isgiven by Wfp (x) = w(q(x)), where q(x) is the quantile
function defined in Eq. (1), and w is a non-increasing function.
If we use w(z) = −z, the alternative objective function isrepresented by Wfp (x) = −q(x).
We can get the estimate of Wfp (xi ) by the Monte-Carloapproximation using n samples {x1 , . . . , xn } drawn from theprobability distribution p(x) as

A Gaussian process [10] model is often used as the surrogate
model of the objective function, in which the predictive
distribution is represented as the Gaussian distribution. Given
the n observed data Dn = {(x1 , y1 ), . . . , (xn , yn )} and therk(xi ).
(7)nNote that this transformed objective function changes depending on the sample distribution p(x).

II. BAYESIAN O PTIMIZATION
Bayesian optimization [1], [2] is one of the general frameworks of the black-box optimization, especially for expensive
and noisy objective functions. In the Bayesian optimization,
the relatively cheap surrogate model is maintained and updated
after the candidate solution is evaluated. Basically, the following steps are repeated in the Bayesian optimization algorithms:

Ŵfp (xi ) = −B. Lebesgue Measure-based Objective
Akimoto [9] has introduced an invariant objective function
for the order preserving transformation for continuous domain
and derived a theoretically attractive natural gradient method inwhich the parameters of the Gaussian distribution generating
the candidate solutions are updated (see [9] for detail).
The Akimoto’s invariant objective function to be maximized
is defined by
2/dVf (x) = −µLeb [y : f (x) ≤ f (y)]
(∫
)2/d
=−
I{f (x) ≤ f (y)}dy,(8)

where µLeb denotes the Lebesgue measure on Rd . The analytical value of Vf (x) cannot be obtained because we assume
that the original objective function is black-box.
However, we can estimate Vf (x) by the Monte-Carlo approximation using n samples {x1 , . . . , xn } drawn from theprobability distribution p(x).
)2/d( n
1 ∑ I{f (x) ≤ f (xi )}V̂f (x) = −
(9)
n i=1
p(xi )
Note that this transformed objective function does not dependon the sample distribution p(x), but we must know the
probability density function p(x) to estimate it.
IV. I NVARIANT O BJECTIVE F UNCTION IN BAYESIANO PTIMIZATION
Let f : Rd → R be a black-box objective function to be
maximized, defined on a bounded subset X ⊂ Rd . In order torealize the invariance property for the order preserving transformation of the objective function, we consider optimizing thealternative invariant objective instead of the original objectivefunction f . We can use either Wfp (x) or Vf (x), explained in
Section III, as the alternative objective function. In practice,
the estimated values of the invariant objectives, Ŵfp (x) andV̂f (x), are used.
Based on the standard Bayesian optimization manner, we
use the Gaussian process to model the alternative objectivefunction and find the next promising solution to be evaluatedby optimizing the acquisition function over the model. The
2
predictive distribution N (µn+1 (x), σn+1(x)) of the Gaussian
process model for the invariant objective function is given
by replacing f with ŵ = [Ŵfp (x1 ), . . . , Ŵfp (xn )]T or v̂ =[V̂f (x1 ), . . . , V̂f (xn )]T in Eq. (3). Note that the predictive
2
variance σn+1(x) is the same as in Eq. (4).
The ranking-based objective function Ŵfp (xi ) is easily
obtained by Eq. (7) using n collected samples Dn . On theother hand, we have to know the probability density functionof the collected samples, p(x), in order to calculate theLebesgue measure-based objective function V̂f (xi ). As thecurrent samples Dn are collected by the Bayesian optimization
procedure, the probability density of those samples is unknown. Therefore, we should estimate the probability density
function and use it for the estimation of V̂f (xi ). Let p̂(x) be theestimated probability density function, then the approximatedinvariant objective function V̂f (xi ) is expressed by replacingp(x) with p̂(x) in (9). In this paper, we use the simple kernel
density estimator with the Gaussian kernel. The estimateddensity function using n samples is represented as
)
(
n
∑
1
(x − xi )2p̂(x) = √
,
(10)
exp −
2h
2πnh i=1
where h is the bandwidth of the kernel density estimator.
In the experiment, the bandwidth is selected by the Scott’s
rule [13] for simplicity, i.e., h = n−1/(d+4) . More sophisticated bandwidth selection or probability density estimatorshould be considered in future work.
The approximated invariant objective values, Ŵfp (xi ) orV̂f (xi ), depend on the collected samples Dn and change when
new data are added. Therefore, the values Ŵfp (xi ) and V̂f (xi )for i = 1, . . . , n have to be recalculated at every iteration.
V. E XPERIMENTIn this section, we evaluate the impact of the invariant
objective functions in the Bayesian optimization using thesimple benchmark functions.
A. Experimental Setting
We implement the Bayesian optimization algorithm using
the pybo2 (version 0.1) framework [14], and the setting of theexperiment is based on the default setting of the framework.
The maximum number of the evaluations is 30d where dindicates the problem dimension, and the initial 3d points are
sampled according to a Latin hypercube sampling. Throughout
the experiment, we use the expected improvement (EI) as theacquisition function, which is commonly used and known as
the efficient acquisition function.
1) The Kernel Function: We test two types of kernel
functions for the covariance function in the Gaussian processmodel, the automatic relevance determination (ARD) squared
exponential kernel (SE) and the ARD Matérn 5/2 kernel,
which are commonly used in Bayesian optimization. The ARD
squared exponential kernel is defined by()
1
KSE (x, x′ ) = θ0 exp − r2 (x, x′ ) ,
(11)
2
d
∑
r2 (x, x′ ) =(xi − x′i )2 /θi2 ,
(12)
i=1

and the ARD Matérn 5/2 kernel is given by()√
5
KM52 (x, x′ ) =θ0 1 + 5r2 (x, x′ ) + r2 (x, x′ )
3
( √
)
2
′
· exp − 5r (x, x ) ,(13)
where both kernels have the d + 1 hyper-parameters[θ0 , . . . , θd ]T .
2 https://github.com/mwhoffman/pybo

2) The Hyper-parameter Setting of the Gaussian Process:
The hyper-parameter setting affects the modeling ability of theobjective function in the Gaussian process and the performanceof the Bayesian optimization. Here, we write the hyperparameters in the Gaussian process as θ consisting of thekernel amplitude θ0 , the length scales θ1 , . . . , θd , the priormean µ0 , and the variance of the observation noise σ 2 .
The hyper-parameters in the Gaussian process model are
marginalized by the Markov Chain Monte Carlo (MCMC)
method via slice sampling, implemented in the pybo framework. Namely,∫ we use the integrated acquisition function,α(x|Dn ) = αEI (x|Dn θ)p(θ|Dn )dθ. In order to computethe integrated acquisition function by MCMC, we need to
set the prior distributions for θ (hyperprior). Based on thedefault settings of the pybo framework, we use the uniform
distribution for θ1 , . . . , θd , the log normal distribution for θ0 ,the Gaussian distribution for µ0 , and the horseshoe distribution
for σ.
3) Test Problems: In order to evaluate the impact of the
proposed invariant objective, we use the Branin, Gramacy, andHartmann 6 functions, which are multimodal functions commonly used as the test functions in the Bayesian optimization
literatures. The definitions of the Branin fBranin and Gramacy
fGramacy functions are as follows:()2
5.1 2 5
fBranin (x) = − x2 − 2 x1 + x1 − 64ππ
1cos(x1 ) + 10),(14)
− 10(1 −8π
fGramacy (x) = −sin(10πx1 )− (x − 1)4 .
2x1

(15)

The dimensions of the Branin and Gramacy are 2 and 1,
respectively, and the domains of the functions are x1 ∈[−5, 10], x2 ∈ [0, 15] for fBranin , and x1 ∈ [0.5, 0.25] for
fGramacy . The Hartmann 6 function fHart6 is a six dimensional
problem, and its definition can be referred in http://www.sfu.
ca/∼ssurjano/hart6.html.
Additionally, we consider the following two monotonicallyincreasing functions to investigate the effect of the orderpreserving transformation of the objective function:1+ 10−5 y,
1 + exp(−10y)and g2 (y) = 0.05y + 0.15(⌊5y⌋).
g1 (y) =

(16)
(17)

Figure 1 illustrates the shapes of these functions. The function
g1 is a sigmoid-like function and g2 is a step-like function. We
can create the order preserved objective using these functions,e.g., g1 ◦ fBranin and g2 ◦ fGramacy .
B. Result and Discussion
The experiment was conducted with the same random seed
for all algorithms in each run, and the mean value of the bestsolutions is calculated over the 30 independent runs.
Figures 2 and 3 show the transitions of the best function value by the Bayesian optimization with the squared

g1(x)

1.0

6

g2(x)

4

0.8

2

0.6

0
0.4
−2
0.2
−4
0.0
−1.0

−0.5

0.0

0.5

1.0

−6
−6 −4 −2

0

2

4

6

Fig. 1: The monotonically increasing functions used in theexperiment, g1 (left) and g2 (right).
exponential kernel (KSE ) and the ARD Matérn 5/2 kernel(KM52 ), respectively. The results of the original objective f ,the transformed objectives g1 ◦ f and g2 ◦ f , and the invariant
objectives Wfp and Vf are plotted. The function values of theoriginal objective function f are plotted for the transformed
objectives and the invariant objectives for comparison.
Tables I and II show the function values for the best solutions obtained on each objective by the Bayesian optimization
with the kernel functions KSE and KM52 , respectively.
We note that the results of the invariant objective functions
under the order preserving transformation are exactly same
because the experiment was conducted with the same random
seed for all algorithm in each run. Namely, the results of Wfp ,Wgp1 ◦f , and Wgp2 ◦f (or Vf , Vg1 ◦f , and Vg2 ◦f ) are identical.
Therefore, we only show the results on the original objective,
Wfp and Vf , for the invariant objective functions in the figures
and tables.
1) Impact of the Order Preserving Transformation of the
Objective Function: In both Figs. 2 and 3, the performanceson the original objective function f and the transformed
objective functions g1 ◦ f and g2 ◦ f are different, i.e.,
the normal Bayesian optimization method is affected by theorder preserving transformation of the objective function. The
performance on g1 ◦ fBranin especially becomes worse thanother transformed objective functions. On the other hand, theperformances of the invariant objective functions, Wfp andVf , are invariant under the order preserving transformation off . By introducing the invariant objective, we do not need toworry about the performance change by the order preserving
transformation.
From the results in Tables I and II, the algorithm on theoriginal objective function can find better solutions than thaton other objectives for both the Branin and Gramacy functions.
In the Hartmann 6 function, the original objective and theinvariant objectives achieve almost the same quality of solution. Meanwhile, the best solutions obtained on the Lebesgue
measure-based invariant objective Vfp are competitive withthose on the original objective f .
2) Comparison of the Ranking and Lebesgue Measurebased Objective: We have conducted the experiment using
two invariant objective functions: the ranking-based objective and the Lebesgue measure-based objective. Interestingly,
we observe that the Bayesian optimization algorithm on theBO with KSE on Branin function

0.0

BO with KSE on Gramacy function

1.0

BO with KSE on Hartmann6 function

3.5

−0.1

−0.3
−0.4
−0.5

fBraning1 ◦ fBranin
g2 ◦ fBranin
WfpBranin

−0.6
−0.7
−0.8

Best function value on f

Best function value on f

Best function value on f

0.8
−0.2

0.6

0.4
fGramacyg1 ◦ fGramacy
g2 ◦ fGramacy

0.2

fHart6
g1 ◦ fHart6
g2 ◦ fHart6
WfpHart6

2.0

VfGramacy

VfHart6

0.0
10
20
30
40
50
Number of function evaluations

2.5

WfpGramacy

VfBranin

0

3.0

60

1.5
0

5

10
15
20
25
Number of function evaluations

30

0

20

40 60 80 100 120 140 160
Number of function evaluations

180

Fig. 2: Transitions of the best function value by the Bayesian optimization with the squared exponential kernel (KSE ) on theoriginal objective f , the transformed objectives g1 ◦ f and g2 ◦ f , and the invariant objectives Wfp and Vf . The results for
the Branin (left), Gramacy (center), and Hartmann 6 (right) functions are displayed. The mean values and standard errors areplotted over the 30 independent runs. For the transformed objectives and the invariant objectives, the function values of theoriginal objective function f are plotted for comparison.
BO with KM52 on Branin function

0.0

BO with KM52 on Gramacy function

1.0

BO with KM52 on Hartmann6 function

3.5

−0.1

−0.3
−0.4
−0.5

fBraning1 ◦ fBranin
g2 ◦ fBranin
WfpBranin

−0.6
−0.7
−0.8

Best function value on f

Best function value on f

Best function value on f

0.8
−0.2

0.6

0.4
fGramacyg1 ◦ fGramacy
g2 ◦ fGramacy

0.2

fHart6
g1 ◦ fHart6
g2 ◦ fHart6
WfpHart6

2.0

VfGramacy

VfHart6

0.0
10
20
30
40
50
Number of function evaluations

2.5

WfpGramacy

VfBranin

0

3.0

60

1.5
0

5

10
15
20
25
Number of function evaluations

30

0

20

40 60 80 100 120 140 160
Number of function evaluations

180

Fig. 3: Transitions of the best function value by the Bayesian optimization with the squared exponential kernel (KM52 ) on theoriginal objective f , the transformed objectives g1 ◦ f and g2 ◦ f , and the invariant objectives Wfp and Vf . The results for
the Branin (left), Gramacy (center), and Hartmann 6 (right) functions are displayed. The mean values and standard errors areplotted over the 30 independent runs. For the transformed objectives and the invariant objectives, the function values of theoriginal objective function f are plotted for comparison.

Lebesgue measure-based invariant objective can find the better
solution than that on the ranking-based invariant function
for both the Branin and Gramacy functions in Tables I and
II. Conversely, the ranking-based objective shows the fasterconvergence than the Lebesgue measure-based objective inthe Hartmann 6 function, but their qualities of solution arealmost the same at the final iteration. The ranking-based
invariant objective Wfp depends on the distribution of thesolutions, p(x), which changes from iteration to iteration. This
fact means that the objective function Wfp changes at every
iterations and may affect the performance decrement in theBranin and Gramacy functions.
VI. C ONCLUSION
In this paper, we have focussed on the invariance property
for the order preserving transformation of the objective function and introduced the invariant objectives into the Bayesian

optimization. We have tested two types of invariant objectives,
the ranking- and Lebesgue measure-based objectives, andevaluated their impact. As the analytical form of the invariant objective functions cannot be obtained, we approximate
them by the Monte-Carlo. In addition, the probability density
function of the current solutions is needed to compute theLebesgue measure-based objective. Therefore, we use the simple kernel density estimation to estimate the density function.
From the numerical experiment, we have confirmed that theperformance of the normal Bayesian optimization changes
by the order preserving transformation of the objective function, although the proposed invariant objectives do not affectthat transformation. The experimental result have shown thatthe introduced invariant objectives achieve the invariance forthe order preserving transformation without the considerable
performance deterioration in the Bayesian optimization. We

TABLE I: The function values on f of the best solutions obtained on each objective function when the kernel function KSE
is used in the Gaussian process. The mean values and standard deviations over the 30 independent runs are reported.
Objective function
Branin (fBranin )Gramacy (fGramacy )Hartmann 6 (fHart6 )# evals.
60
30
180

f
-0.0398 ± 3.41e-6
0.869 ± 3.94e-4
3.15 ± 3.22e-1

g1 ◦ f-0.267 ± 2.32e-1
0.756 ± 2.05e-1
2.68 ± 2.32e-1

g2 ◦ f-0.0626 ± 1.87e-2
0.759 ± 1.74e-1
2.83 ± 4.02e-1

Wfp-0.105 ± 9.37e-2
0.816 ± 9.94e-2
3.22 ± 2.04e-1

Vf-0.0490 ± 1.29e-2
0.849 ± 4.30e-2
3.18 ± 1.41e-1

TABLE II: The function values on f of the best solutions obtained on each objective function when the kernel function KM52
is used in the Gaussian process. The mean values and standard deviations over the 30 independent runs are reported.
Objective function
Branin (fBranin )Gramacy (fGramacy )Hartmann 6 (fHart6 )# evals.
60
30
180

f
-0.0424 ± 1.40e-2
0.869 ± 1.93e-3
3.27 ± 5.91e-2

g1 ◦ f-0.166 ± 2.05e-1
0.699 ± 1.99e-1
2.81 ± 1.37e-1

have observed that the Bayesian optimization algorithm on theLebesgue measure-based invariant objective can find the better
solution than that on the the ranking-based invariant function
for the Branin and Gramacy functions.
In the Lebesgue measure-based objective, the probability
density estimation is needed in order to estimate the invariant
objective function. In this paper, we use only the simple
kernel density estimator. Therefore, we should attempt toother probability density estimators and investigate their effect.
Moreover, the impact of the invariance property under a more
realistic optimization problem should be studied.
R EFERENCES[1] J. Mockus, Bayesian Approach to Global Optimization: Theory andApplications, ser. Mathematics and its Applications. Springer, 1989.
[2] E. Brochu, V. M. Cora, and N. de Freitas, “A Tutorial on Bayesian
Optimization of Expensive Cost Functions, with Application to Active
User Modeling and Hierarchical Reinforcement Learning,” arXiv
preprint, 2010. [Online]. Available: http://arxiv.org/abs/1012.2599[3] F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based
optimization for general algorithm configuration,” in Proceedings of the
5th International Conference on Learning and Intelligent Optimization(LION 5), ser. LNCS, vol. 6683. Springer, 2011, pp. 507–523.
[4] J. Bergstra, “Algorithms for Hyper-Parameter Optimization,” in Advances in Neural Information Processing Systems 24, J. Shawe-Taylor,
R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds., 2011, pp.
2546–2554.
[5] J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian Optimization of Machine Learning Algorithms,” in Advances in Neural
Information Processing Systems 25, 2012, pp. 2951–2959.

g2 ◦ f-0.0511 ± 1.07e-2
0.714 ± 1.74e-1
2.97 ± 2.92e-1

Wfp-0.0777 ± 6.45e-2
0.783 ± 1.18e-1
3.28 ± 5.63e-2

Vf-0.0426 ± 6.84e-3
0.857 ± 3.68e-2
3.25 ± 6.47e-2[6] N. Hansen, “The CMA Evolution Strategy: A Comparing Review,” in
Towards a New Evolutionary Computation, ser. Studies in Fuzziness andSoft Computing, J. A. Lozano, P. Larrañaga, I. Inza, and E. Bengoetxea,
Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, vol. 192,
pp. 75–102.
[7] N. Hansen and A. Auger, “Principled Design of Continuous Stochastic
Search: From Theory to Practice,” in Theory and Principled Methods for
Designing Metaheustics, ser. Natural Computing Series, Y. Borensteinand A. Moraglio, Eds. Springer Berlin Heidelberg, 2014, pp. 145–180.
[8] Y. Ollivier, L. Arnold, A. Auger, and N. Hansen, “InformationGeometric Optimization Algorithms: A Unifying Picture via Invariance
Principles,” arXiv preprint, 2011. [Online]. Available: http://arxiv.org/
abs/1106.3708v2[9] Y. Akimoto, “Analysis of a natural gradient algorithm on monotonic
convex-quadratic-composite functions,” in Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation (GECCO2012). ACM, 2012, pp. 1293–1300.
[10] C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine
Learning. MIT Press, 2006.
[11] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger, “Gaussian Process
Optimization in the Bandit Setting: No Regret and Experimental Design,” in Proceedings of the 27th International Conference on Machine
Learning (ICML 2010), 2010, pp. 1015–1022.
[12] E. Contal and N. Vayatis, “Gaussian Process Optimization with Mutual
Information,” in Proceedings of the 31st International Conference on
Machine Learning (ICML 2014), 2014.
[13] D. W. Scott, Multivariate Density Estimation: Theory, Practice, andVisualization. John Wiley & Sons, 1992.
[14] M. W. Hoffman and R. Shahriari, “Modular mechanisms for Bayesian
optimization,” in NIPS Workshop on Bayesian Optimization, 2014.