Ozaki et al. IPSJ Transactions on Computer Vision andApplications (2017) 9:20
DOI 10.1186/s41074-017-0030-7

IPSJ Transactions on Computer
Vision and Applications

R ES EA R CH PA PEROpen Access

Effective hyperparameter optimization
using Nelder-Mead method in deep learning
Yoshihiko Ozaki1,2 , Masaki Yano1,2 and Masaki Onishi1,2*
Abstract
In deep learning, deep neural network (DNN) hyperparameters can severely affect network performance. Currently,
such hyperparameters are frequently optimized by several methods, such as Bayesian optimization and thecovariance matrix adaptation evolution strategy. However, it is difficult for non-experts to employ these methods. In
this paper, we adapted the simpler coordinate-search and Nelder-Mead methods to optimize hyperparameters.
Several hyperparameter optimization methods were compared by configuring DNNs for character recognition andage/gender classification. Numerical results demonstrated that the Nelder-Mead method outperforms the othermethods and achieves state-of-the-art accuracy for age/gender classification.
Keywords: Hyperparameter optimization, Nelder-Mead method, Deep learning, Convolutional neural network

1 Introduction
The evolution of deep neural networks (DNNs) has dramatically improved the accuracy of character recognition[1], object recognition [2, 3], and other tasks. However,
the their increasing complexity increases the number ofhyperparameters, which makes tuning of hyperparameters an intractable task.
Traditionally, DNN hyperparameters are adjusted using
manual search, grid search, or random search [4]. However, search space expands exponentially relative to thenumber of hyperparameters; thus, such naive methods nolonger work well. Therefore, more sophisticated hyperparameter optimization methods are required.
In deep learning, a hyperparameter optimization problem can be formulated as a stochastic black box optimization problem to minimize a noisy black box objectivefunction f (x):Minimize f (x) (x ∈ χ).

(1)Here, using all information available about the objectivefunction, we can obtain its value at point x with noise  asfollows:iid
2
y = f (x) + ,  ∼ N 0, σnoise.
(2)*Correspondence: onishi@ni.aist.go.jp
Artificial Intelligence Research Center, AIST, 1-1-1 Umezono, Tsukuba, Ibaraki,Japan
2
University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, Japan
1

This means that no analytical properties of the objectivefunction, e.g., its derivatives, can be optimized. In addition, a loss function of the target DNN is typically chosen
as f (x), and its evaluation cost is so expensive that training and testing of the DNN is required. The search space
χ comprises combinations of multiple conditions such asreal numbers, integers, and categories.
Currently, Bayesian optimization [5] and the covariancematrix adaptation evolution strategy (CMA-ES) [6] areconsidered the most promising methods for DNN hyperparameter optimization, and their optimization ability has
been proven experimentally [7–10]. However, Bayesian
optimization has some hyperparameters that significantly
affect its optimization performance, e.g., choices of its
kernel and acquisition function. Moreover, maximizing a
non-convex acquisition function is required for each iteration of the optimization process. On the other hand,
CMA-ES requires several populations and generations for
sufficient performance. Although such calculations canbe parallelized easily, significant computing resources are
required.
It is evident that simple classical manual search, gridsearch, and random search remain common; thus, weconsider that most people are unwilling to adjust thehyperparameters of a difficult optimization method or
implement the method and do not have sufficient computing resources to optimize DNN hyperparameters.

© The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to theCreative Commons license, and indicate if changes were made.

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

In this paper, we describe simple substitutional methods, i.e., the coordinate-search and Nelder-Mead methods, for hyperparameter optimization in deep learning.
To the best of our knowledge, no report has examined
the application of these methods to hyperparameter optimization.
Our numerical results indicate that these methods are
more efficient than other well-known methods. In particular, the Nelder-Mead method is the most effective fordeep learning.

relevance determination (ARD) squared exponential (SE)kernel
1
KSE (x, x ) = θ0 exp − r2 (x, x ) ,
2
(4)
D



2

 2
2
where r (x, x ) =xd − xd /θd ,d=1

or ARD Matérn 5/2 kernel
KM52 (x, x ) 5
= θ0 1+ 5r2 (x, x )+ r2 (x, x ) exp − 5r2 (x, x ) ,3

2 Related work
2.1 Random search

Random search is one of the simplest ways to optimize
DNN hyperparameters. This method iteratively generates
hyperparameter settings and evaluates the objective function. Random search has excellent parallelization and canhandle integer and categorical hyperparameters naturally.
Bergstra and Bengio demonstrated that random searchoutperforms a manual search by a human expert and gridsearch [4].
2.2 Bayesian optimization

Bayesian optimization is one of the most remarkablehyperparameter optimization methods in recent years.
Its base concept was proposed in the 1970s; however, it
has been significantly improved since then due to theattention paid to DNN hyperparameter optimization.
There are several variations of Bayesian optimization,e.g., Gaussian process (GP)-based variation [11], Treestructured Parzen Estimator (TPE) [7], and Deep Networks for Global Optimization (DNGO) [12]. The moststandard one is the GP-based variation.
GP-based Bayesian optimization is shown in
Algorithm 1. In this method, we assume that an objective

Algorithm 1: Bayesian optimization [11]for t = 1, 2, . . . do
Find xt by optimizing the acquisition function over
GP: xt = argmaxx u(x|D1:t−1 );Sample the objective function: yt = f (xt ) + t ;Augment the data D1:t = {D1:t−1 , (xt , yt )} andupdate the GP;
endfunction follows the GP specified by its mean function mand kernel k:f (x) ∼ GP (m(x), k(x, x )).

Page 2 of 12(3)

For simplicity, we assume m(x) = 0. Then, we must
consider the kernel k(x, x ). For the kernel, an automatic(5)
is commonly used in Bayesian optimization [8]. Here, θ0 ,
θ1 , . . . , and θD are the kernel’s hyperparameters.
Once k(x, x ) is determined, we can predict information
about a new sample point xt+1 from previous observations
D1:t = {x1:t , y1:t }:2P(yt+1 |D1:t , xt+1 ) = N μt (xt+1 ), σt2 (xt+1 )+σnoise
,(6)
−1
T
2
(7)
μt (xt+1 ) = k K + σnoise I y ,
1:t

−1
2
σt2 (xt+1 ) = k(xt+1 , xt+1 ) − kT K + σnoise
I
k (8)
⎡
⎤
k(x1 , x1 ) · · · k(x1 , xt )
⎢
⎥
..
..
2
..
K=⎣
⎦ + σnoise I,
.
.
.
where
k(xt , x1 ) · · · k(xt , xt )k =[ k(xt+1 , x1 ) k(xt+1 , x2 ) · · · k(xt+1 , xt )] .
The remaining problem is how to determine new sample points iteratively. To determine new candidates fora sample, we generally employ an acquisition function.
Here, it is necessary to select an acquisition functionthat achieves a good balance between exploration andexploitation based on past observations. One well-known
acquisition function is expected improvement (EI):(μ(x)−f (x+ ))(Z)+σ (x)(Z)(σ (x) > 0)
EI(x) =(9)0
(σ (x) = 0)
where Z =μ(x) − f (x+ ) +, x = argmaxxi ∈x1:t f (xi ).
σ (x)The point that maximizes the acquisition function
becomes a new sample point. Although maximizing thenon-convex acquisition function is difficult, the evaluation cost of the function is considerably less than that of
the original objective function. Therefore, it is easier tohandle than the original problem.
Practically, Bayesian optimization is combined with random search to collect initial observation data.
Bergstra et al. and Snoek et al. performed several computational experiments. The results demonstrated that
Bayesian optimization outperforms manual search by ahuman expert and random search [7, 8].

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

2.3 Covariance matrix adaptation evolution strategy

While Bayesian optimization has been developed in themachine learning community, CMA-ES has been developed in the optimization community. CMA-ES is a typeof evolutionary computation that demonstrates outstanding performance in benchmarks as a state-of-the-art blackbox optimization method [13].
The (μW , λ)-CMA-ES [6] is shown in Algorithm 2. It
conducts a weighted recombination from the μ best outof λ individuals. The procedure is explained as follows.

Page 3 of 12

Here, cc and ccov are hyperparameters.
(g)(iv) Update the evolution path pσ and the step size σ (g) :
(g+1)

(g)

(g+1)

= (1 − cσ )pσ + cuσ cw B(g) zw , (13)
(g+1)
− χ̂n 
1 pσ
(g+1)
(g)
, (14)σ= σ exp
dσ
χ̂n

pσ

where

cuσ :=cσ (2 − cσ ),χ̂n = E[ N (0, I)] .

Here, cσ and dσ are hyperparameters.
Algorithm 2: The (μW , λ)-CMA-ES [6](0)Initialize the mean xw , and the standard deviation
σ (0) ;
(0)(0) = I;p(0)c = 0; pσ = 0; C
for g = 0, 1, . . . do
Generate g + 1 generation λ individuals;
Select μ best individuals;
Update the evolution path and the covariancematrix;Update the evolution path and the step size;
end(i) Initialize the mean x(0)w and the standard deviation
σ (0) of individuals. Set the evolution path(0)(0)
pc = pσ = 0 and the covariance matrix C(0) = I.
(ii) Generate g + 1 generation individuals(g+1)(k = 1, . . . , λ):
xk
(g+1)

xk

(g)

(g+1)

= xw + σ (g) B(g) D(g) zk
μ


1
(g)
xw := μi=1 wi i=1

where

B(g) D(g) B(g) D(g)
(g+1)

zk

T

,

(10)

(g)

wi xi:λ ,= C(g) ,∼ N (0, I).

Here, w1 , . . . , wμ are weights, and i : λ denotes the i th
best individual.
(g)(iii) Update the evolution path pc and the covariance(g)matrix C :
(g+1)

pc

(g)

(g+1)

= (1−cc )pc + cuc cw B(g) D(g) zw
(g+1)

C(g+1) = (1−ccov )C(g) + ccov pc

,

(11)

(g+1) Tpc

cuc :=cc (2 − cc ),μwi,
cw := i=1
μ
2
w
where
i=1 i
μ

1
(g+1)
(g+1)
zw
:= μ
wi zi:λ .
w
i=1 i i=1

, (12)

Details about the hyperparameters used to update CMAES parameters are provided in the literature [6].
Since evaluations of each individual for each generation
can be calculated simultaneously, CMA-ES can be parallelized easily. Watanabe and Le Roux and Loshchilov et al.
demonstrated that CMA-ES outperforms manual search
by a human expert and Bayesian optimization in certain
cases [9, 10].

3 Coordinate-search and Nelder-Mead methods
In the previous section, we introduced the random search,
Bayesian optimization, and CMA-ES methods. Note thatthe achievements of these methods have already been
proven experimentally, and the results indicate that thelatter two methods are very promising and considered
superior to random search. However, both Bayesian
optimization and CMA-ES have many hyperparameters
related to their optimization performance. To set these
hyperparameters appropriately, it is necessary to have sufficient knowledge about the given method. In addition,
Bayesian optimization must maximize its non-convex
acquisition function, and CMA-ES requires significant
computing resources to exploit its advantages. These factors make it difficult for non-experts to utilize thesemethods.
In this section, we introduce two optimization methods, coordinate-search and Nelder-Mead, that are easy to
implement. These methods have fewer hyperparameters
to adjust and are practically usable with fewer computing
resources.
3.1 Mathematical concepts

Before introducing the methods, we define some requiredmathematical concepts here.
Definition 1 The positive span of a set of vectors[ v1 · · · vr ] ∈ Rn is the convex cone:{v ∈ Rn : v = α1 v1 + · · · + αr vr , αi ≥ 0, i = 1, . . . , r}.
Definition 2 A positive spanning set in Rn is a set ofvectors whose positive span is Rn .

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Page 4 of 12

Definition 3 The set [v1 · · · vr ] ∈ Rn is considered positively dependent if one of the vectors is in the positive spanof the remaining vectors; otherwise, the set is considered
positively independent.
Definition 4 A positive basis for Rn is a positively independent set whose positive span is Rn . A positive basis for
Rn that has n + 1 vectors is considered a minimal positive
basis and a positive basis that has 2n vectors is considereda maximal positive basis (Fig. 1). Here, a maximal positive
basis is denoted as D⊕ .
Definition 5 A simplex of dimension m is a convex hull of an affinely independent set of points Y ={y0 , y1 , . . . , ym }.
3.2 Coordinate-search method

The coordinate-search method [14] (Algorithm 3, Fig. 2) is
one of the simplest direct search methods. It minimizes its
objective function iteratively using the maximal positive
basis D⊕ =[ I − I] =[ e1 · · · en − e1 · · · − en ].
Algorithm 3: Coordinate-search method [14]Initialization: Choose x0 and α0 > 0;for k = 0, 1, . . . doPoll step;
Parameter update;endThis method performs a poll step iteratively to searcha better solution and updates parameters to adjust its
learning rate.
(i) Poll step: Order the poll set Pk = {xk + αk d :d ∈ D⊕ }. Evaluate f at the poll points in order. If apoll point xk + αk dk that satisfies the conditionf (xk + αk dk ) < f (xk ) is found, then stop polling, set
xk+1 = xk + αk dk , and declare the iteration and pollstep successful.
(ii) Parameter update: If iteration k succeeds, then set
αk+1 = αk (or αk+1 = 2αk ). Otherwise, set αk+1 = αk /2.

Fig. 2 Coordinate-search method

When the step size becomes sufficiently small, the searchis terminated. Note that the evaluation of functions in thepoll step can be parallelized.
This method deteriorates the performance for searchranges with different scales; thus, in this study, we normalize parameters to [0, 1] in our computational experiments.
In addition, we adopt the updating rule αk+1 = 2αk on
iteration success and order the vectors of the poll set
randomly for each iteration.
3.3 Nelder-Mead method

The Nelder-Mead method [14, 15] (Algorithm 4, Fig. 3)
is an optimization method that uses a simplex proposed
by Nelder and Mead. Gilles et al. applied this method
for the hyperparameter tuning problem in support vector
machine modeling. They demonstrated that the method
can find very good hyperparameter settings reliably forsupport vector machines [16]. Currently, the Nelder-Meadmethod is not considered in DNN research; however,it has a long history and many achievements in otherresearch areas [14]. Thus, we think it is worth considering

Algorithm 4: Nelder-Mead method [15]Initialization:
Choose an initial simplex of vertices
Y0 = y00 , y10 , . . . , yn0 . Evaluate f at the points in Y0 .
Choose constants:
0 < γ s < 1,

Fig. 1 Maximal positive basis (D1 ) and minimal positive basis (D2 ) in R2−1 < δ ic < 0 < δ oc < δ r < δ e .

for k = 0, 1, . . . do
Set Y = Yk ;Order;
Reflect;
Expand;Contract;
Shrink;end

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Page 5 of 12

Fig. 5 Shrinking a simplex by the Nelder-Mead method. y1 and y2 areshrunk to ys1 and ys2 , respectively

Fig. 3 Nelder-Mead method

the Nelder-Mead method for DNN hyperparameter optimization. In the study by Gilles et al., their SVM has only
two hyperparameters. On the other hand, DNNs often
have more than 10 times number of hyperparameters. So,
our task is more challenging.
The Nelder-Mead method minimizes the objectivefunction by repeating its evaluation at each vertex of thesimplex and by replacing points according to the following
procedure (Figs. 4 and 5).
(i) Order: Order the n + 1 vertices Y = y0 , y1 , . . . , yn
as follows:
f 0 = f (y0 ) ≤ f 1 = f (y1 ) ≤ · · · ≤ f n = f (yn ).
(ii) Reflect:
Reflect the worst vertex yn over the centroidn−1cy = i=0 yi /n of the remaining n vertices:yn with thereflected point yr and terminate iterationk : Yk+1 = y0 , y1 , . . . , yn−1 , yr .
(iv) Contract: If f r ≥ f n−1 , then a contraction is
performed between the best of yr and yn .
(a) Outside contraction: If f r < f n , perform an
outside contraction:
yoc = yc + δ oc (yc − yn )and evaluate f oc = f (yoc ). If f oc ≤ f r , thenreplace yn with the outside contraction point
yoc
iteration k :
k and terminate
Yk+1 = y0 , y1 , . . . , yn−1 , yoc . Otherwise,
perform a shrink.
(b) Inside contraction: If f r ≥ f n , perform an
inside contraction:
yic = yc + δ ic (yc − yn )yr = yc + δ r (yc − yn ).

and evaluate f ic = f (yic ). If f ic < f n , thenreplace yn with the inside contraction point
yic and terminate iteration k :Yk+1 = {y0 , y1 , . . . , yn−1 , yic }. Otherwise,
perform a shrink.

Evaluate f r = f (yr ). If f 0 ≤ f r < f n−1 , then replaceyn with thereflected point yr and terminate iterationk : Yk+1 = y0 , y1 , . . . , yn−1 , yr .
(iii) Expand: If f r < f 0 , calculate:ye = yc + δ e (yc − yn )and evaluate f e = f (ye ). If f e ≤ f r , then replace ynwith the expansion
ye and 0 1 pointn−1
 terminate iterationk : Yk+1 = y , y , . . . , y , ye . Otherwise, replace(v) Shrink: Evaluate f at the n points
y0 + γ s (yi − y0 ), where i = 1, . . . , n, replacey1 , . . . , yn with these points, and terminate iterationk : Yk+1 = {y0 + γ s (yi − y0 ), i = 0, . . . , n}.
Here, γ s , δ ic , δ oc , δ r , and δ e are constant hyperparameters usually taking the following values:
γs =

1 ic
1
1
, δ = − , δ oc = , δ r = 1 and δ e = 2.
2
2
2

(15)

Note that each step of an iteration, e.g., initialization andshrink operations, can be parallelized easily.

4 Poor hyperparameter setting detection
Fig. 4 Reflection, expansion, outside contraction, and insidecontraction of a simplex by the Nelder-Mead method

DNNs are very sensitive to hyperparameter settings. As
a result, training can fail simply because some hyperparameters, e.g., the learning rate, are slightly inappropriate.

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

When appropriate hyperparameter values are given, training loss is reduced in each iteration (Fig. 6, top graph);otherwise, regardless of how many iterations have been
executed, training loss is not reduced (Fig. 6, bottom
graph).
The advantage of human experts is that they can detect
training failures and terminate them at an early stage.
Domhan et al. proposed a method that accelerates hyperparameter optimization methods by detecting and terminating such training failures using learning curve prediction [17]. In addition, Klein et al. proposed a specialized
Bayesian neural network to model DNN learning curves[18, 19]. We apply Algorithm 5 to detect training failuresat an early stage.

Page 6 of 12

Note that this method does not optimize hyperparameters directly, but accelerates a hyperparameter optimization method. If a large number of training iterations withpoor hyperparameter settings appear in the optimization
process, this detection process improves the execution
time of the optimization method.
In our experiments, we apply this method to all hyperparameter optimization methods with n equaling 10% of
the training iterations and t equaling 0.8. These values arechosen based on experience. As can be seen in Fig. 6, thelearning curve of poor hyperparameter settings is distinctive and easy to detect; thus, there is no need to be too
careful to decide n and t.

5 Numerical results
Algorithm 5: Poor hyperparameter setting detection
Initialize n and t. Calculate initial loss l0 ;After n learning iterations, calculate current loss ln ;if t < ln /l0 thenTerminate the training;end

We perform computational experiments to optimize realand integer hyperparameters in combination with various datasets, tasks, and convolutional neural networks(CNNs) to compare the performance of the random
search, Bayesian optimization, CMA-ES, coordinatesearch, and Nelder-Mead methods.
The experimental settings for each method are given in
Table 1. We use the first 100 random search evaluations
to initialize the Bayesian optimization and coordinatesearch methods. The number of evaluations and initialization parameters of CMA-ES and Bayesian optimization
are determined with reference to the literature [10]. We
implement CMA-ES using Distributed Evolutionary Algorithms in Python (DEAP) [20], which is an evolutionary
computation framework. In addition, for optimization
methods that cannot handle integer values directly, integer hyperparameters are handled as continuous values
and rounding is performed when evaluating the objectivefunction.
Table 1 Experimental setting for each method
MethodFig. 6 Learning curve when training is successful (top) andunsuccessful (bottom)

DetailRandom search

Perform 600 random evaluations.

Bayesian optimization

Initialize the observation data with the first
100 evaluations of the random search, thenperform the optimization with exactly 500
evaluations. The kernel is the ARD Matérn 5/2
and the acquisition function is the EI [8, 10].

CMA-ES

Perform 600 evaluations with 20 generations
where each generation consists of 30(0)
individuals. xw = 0.5, σ (0) = 0.2. Allvariables are scaled to [ 0, 1] [10].

Coordinate-searchmethodInitialize x0 as the best point of the first 100
random search evaluations, then perform
optimization for up to 500 evaluations.
α = 0.5. All variables are scaled to [ 0, 1].

Nelder-Mead method

Generate an initial simplex randomly, thenperform optimization for up to 600
evaluations (including initialization).
γ s = 12 , δ ic = − 12 , δ oc = 12 , δ r = 1 and δ e = 2.

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Page 7 of 12

Table 5 Network architecture of Batch-Normalized Maxout
Network in Network [25]Fig. 7 MNIST database of handwritten digits [21]Conv 1

Kernel size: 5, stride: 1, pad: 2, BN

MMLP 1-1

Kernel size: 1, stride: 1, pad: 0, k = 5, BN

MMLP 1-2

Kernel size: 1, stride: 1, pad: 0, k = 5, BN

Pool 1 (AVE pooling)

Kernel size: 3, stride: 2, pad: 0, dropout

Conv 2Kernel size: 5, stride: 1, pad: 2, BN

MMLP 2-1

Kernel size: 1, stride: 1, pad: 0, k = 5, BN

5.1 MNISTMMLP 2-2

Kernel size: 1, stride: 1, pad: 0, k = 5, BN

The LeNet [1] hyperparameters are optimized by the fivemethods to measure their performance. This network performs a 10-class classification of the MNIST handwritten
digit database [21] (Fig. 7). Here, we use Caffe’s tutorial
implementation [22, 23]. This implementation uses a rectified linear unit [24] as its activation function rather than
the sigmoid used in the original LeNet.
These methods are also applied to the optimization of
hyperparameters of the Batch-Normalized Maxout Network in Network proposed by Chang et al. [25]. Note that
this network is deeper and has many more hyperparameters to optimize than LeNet.
Tables 2, 3, 4, 5, 6, and 7 show the details of each
network, the fixed hyperparameters, the optimized hyperparameters, and the search ranges. Note that preprocessing and augmentation of the training data are not
performed.

Pool 2 (AVE pooling)

Kernel size: 3, stride: 2, pad: 0, dropout

Conv 3

Kernel size: 3, stride: 1, pad: 1, BN

MMLP 3-1

Kernel size: 1, stride: 1, pad: 0, k = 5, BN

MMLP 3-2

Kernel size: 1, stride: 1, pad: 0, k = 5, BN

Table 2 LeNet network architecture [1]Conv 1

Kernel size: 2, stride: 1, pad: 0

Pool 1 (MAX pooling)

Kernel size: 2, stride: 2, pad: 2

Conv 2Kernel size: 5, stride: 1, pad: 0

Pool 2 (MAX pooling)

Kernel size: 2, stride: 2, pad: 0

FC 1

Table 3 LeNet fixed parameters
Name

Description

Iteration

10,000Batch size

64

Learning rate decay policy

inv (gamma = 0.01, power = 0.75) [29]Pool 3 (AVE pooling)

5.2 Age and gender classification

Gil and Tal proposed a CNN for age/gender classification [26]. In these experiments, the hyperparameters of
this CNN are optimized by the five methods. This network consists of three convolution layers and two fully
connected layers, receives an image, and outputs a genderlabel or an age group label. We use the implementation
available on the project’s web page [27]. We test DNNs
using the Adience DB [28] for the age/gender classification benchmark used in the literature [26] (Fig. 8). We
divide the dataset into five sets, train the network withfour sets, and test it with one set. Note that these processes require significant calculation time; thus, in the
optimization process, cross validations are not performed.
We perform cross validation for only the optimal solution
among the optimal solutions of all methods and calculate
the cross-validated accuracy for comparison with resultsin the literature [26].
Tables 8, 9, and 10 show the details of each network, thefixed hyperparameters, the optimized hyperparameters,
and the search ranges. Note that, for this experiment, data
augmentation is conducted in a single-crop manner [26].
5.3 Results

Table 4 LeNet hyperparameters

The experiments are executed for one month using 32
modern GPUs. The experimental results are given in

Name

Description

Range

Table 6 Fixed parameters of Batch-Normalized Maxout Network
in Network

x1

Learning rate (= 0.1x1 )[1, 4]

Name

Description

x2

Momentum (= 1 − 0.1x2 )[0.5, 2]Iteration

20,000x3

L2 weight decay[0.001, 0.01]

Batch size

100

x4∗

FC1 units[256, 1024]

Learning rate decay policy

Multistep (gamma = 0.1,
step value = {15, 000, 18, 000}) [29]Integer parameters are marked with ∗Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Page 8 of 12

Table 7 Batch-Normalized Maxout Network in Network
hyperparameters

Table 9 Fixed parameters of the age/gender classification CNN
Name

Description

Name

Description

Range

Iteration

20,000x1

Learning rate (= 0.1x1 )[0.5, 2]Batch size

50

x2

Momentum (= 1 − 0.1x2 )[0.5, 2]

Learning rate decay policy

Step (gamma = 0.1, step size = 10000) [29]x3

L2 weight decay[0.001, 0.01]

x4

Dropout 1[0.4, 0.6]x5

Dropout 2[0.4, 0.6]

x6

Conv 1 initialization deviation[0.01, 0.05]

x7

Conv 2 initialization deviation[0.01, 0.05]

x8

Conv 3 initialization deviation[0.01, 0.05]

x9

MMLP 1-1 initialization deviation[0.01, 0.05]

x10

MMLP 1-2 initialization deviation[0.01, 0.05]

x11

MMLP 2-1 initialization deviation[0.01, 0.05]

x12

MMLP 2-2 initialization deviation[0.01, 0.05]

x13

MMLP 3-1 initialization deviation[0.01, 0.05]

x14

MMLP 3-2 initialization deviation[0.01, 0.05]

age classification is 51.25% (±5.461970). These values are higher than previous state-of-the-art results(86.8% (±1.4) and 50.7% (±5.1)) reported in the literature[26]. The stability and search performance of this method
are magnificent.
The coordinate-search method also achieves goodresults with LeNet and Batch-Normalized Maxout
Network in Network. However, the coordinate-searchmethod searches points using each vector of the positive basis; thus, convergence speed is reduced as thenumber of dimensions increases. This appears to be thereason why the coordinate-search method does not workfor the age/gender classification CNN, which has more

Table 10 Hyperparameters of the age/gender classification CNN

Fig. 8 Faces from the Adience benchmark for age/genderclassification [28]Tables 11, 12, 13, and 14. In all experiments, the NelderMead method achieves both minimal loss and variance.
The small variance suggests that the initial values ofthe method do not significantly affect the results. Furthermore, the accuracy of the cross validation with thebest solution found by the Nelder-Mead method in gender classification is 87.20% (±1.328024) and that for

Name

Description

Range

x1

Learning rate (= 0.1x1 )[1, 4]

x2

Momentum (= 1 − 0.1x2 )[0.5, 2]x3

L2 weight decay[0.001, 0.01]

x4

Dropout 1[0.4, 0.6]x5

Dropout 2[0.4, 0.6]

x6∗

FC 1 units[512, 1024]

x7∗

FC 2 units[256, 512]x8

Conv 1 initialization deviation[0.01, 0.05]

x9

Conv 2 initialization deviation[0.01, 0.05]

x10

Conv 3 initialization deviation[0.01, 0.05]

x11

FC 1 initialization deviation[0.001, 0.01]

x12

FC 2 initialization deviation[0.001, 0.01]

x13

FC 3 initialization deviation[0.001, 0.01]

x14

Conv 1 bias

[0, 1]

Table 8 Network architecture of the age/gender classification
CNN [26]

x15

Conv 2 bias

[0, 1]

x16

Conv 3 bias

[0, 1]

Conv 1

Kernel size: 7, stride: 4, pad: 0

x17

FC 1 bias[0, 1]

Pool 1 (MAX pooling)

Kernel size: 3, stride: 2, pad: 0

x18

FC 2 bias[0, 1]

Conv 2Kernel size: 5, stride: 1, pad: 2∗
x19

Normalization 1 localsize (= 2x19 + 3)

[0, 2]

Pool 2 (MAX pooling)

Kernel size: 3, stride: 2, pad: 0∗x20

Normalization 2 localsize (= 2x20 + 3)

[0, 2]

Conv 3

Kernel size: 3, stride: 1, pad: 1x21

Normalization 1 alpha[0.0001, 0.0002]

Pool 3 (MAX pooling)

Kernel size: 3, stride: 2, pad: 0x22

Normalization 2 alpha[0.0001, 0.0002]

FC 1

Dropoutx23

Normalization 1 beta

[0.5, 0.95]

FC 2

Dropoutx24

Normalization 2 beta

[0.5, 0.95]

FC 3

Integer parameters are marked with ∗Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Table 11 MNIST results (LeNet)

Page 9 of 12

Table 13 Gender classification results

MethodMean loss

Min loss

MethodMean loss

Min loss

Random search

0.005411 (±0.001413)

0.002781

Random search

0.001732 (±0.000540)

0.000984

Bayesian optimization

0.004217 (±0.002242)

0.000089

Bayesian optimization

0.00183 (±0.000547)

0.001097

CMA-ES

0.000926 (±0.001420)

0.000047

CMA-ES

0.001804 (±0.000480)

0.001249

Coordinate-search method

0.000052 (±0.000094)

0.000002

Coordinate-search method

0.002240 (±0.001448)

0.000378

Nelder-Mead method

0.000029 (±0.000029) 0.000004

Nelder-Mead method

0.000395 (±0.000129)

0.000245

MethodMean accuracy (%)

Accuracy withmin loss (%)

MethodMean accuracy (%)

Accuracy withmin loss (%)

Random search

98.98 (±0.08)

99.06

Random search

87.93 (±0.24)

88.21

Bayesian optimization

99.07 (±0.02)

99.25

Bayesian optimization

88.07 (±0.27)

87.85

CMA-ES

99.20 (±0.08)

99.30

CMA-ES

88.20 (±0.38)

88.55

The coordinate-search method 99.26 (±0.05)

99.35

Coordinate-search method

87.04 (±0.52)

87.72

99.24 (±0.04)

99.28

Nelder-Mead method

88.38 (±0.47)

88.83

The Nelder-Mead method

The smallest loss for each experiment is indicated by bold-faced font

The smallest loss for each experiment is indicated by bold-faced font

hyperparameters. Thus, we should use the Nelder-Meadmethod rather than the coordinate-search method. As
demonstrated in the literature [9], CMA-ES is superior
to random search because it finds better parameters earlier. Despite using the same hyperparameters for Bayesian
optimization, the method works well for age estimation
but not for other tasks. This indicates that, for Bayesian
optimization, hyperparameters should be adjusted carefully depending on the given task.
The mean loss graphs (Figs. 9, 10, 11 and 12) showthat the Nelder-Mead method rapidly finds a good solution and converges faster than the other methods. We
anticipate that the objective function for hyperparameter optimization is multimodal, and many local optima
that achieve similar results exist. We confirmed thisproperty via additional experiments that optimized thehyperparameters of the gender classification CNN, thenetwork which has the largest search space, using the
Nelder-Mead method. The optimized hyperparametersettings after 600 evaluations of each experiment are
shown using the parallel coordinates plot in Fig. 13. In
the figure, points in the search space are represented aspolylines with vertices on parallel axes. The position of
the vertex on the ith axis corresponds to the value of thehyperparameter xi . The polylines exhibiting small losses
are shown in dark colors.
Experimental results showed that the Nelder-Meadmethod converged to different points every time andthe objective function was almost multimodal. Different hyperparameters settings achieved similar losses.
From Table 13 and Fig. 13, we deduce that many local
optima that achieve similar results exist. In such cases,

Table 12 MNIST Results (Batch-Normalized Maxout Network in
Network)

Table 14 Age classification results

MethodMean loss

Min loss

MethodMean loss

Min loss

Random search

0.045438 (±0.002142)

0.042694

Random search

0.035694 (±0.006958)

0.026563

Bayesian optimization

0.045636 (±0.001197)

0.044447

Bayesian optimization

0.024792 (±0.003076)

0.020466

CMA-ES

0.045248 (±0.002537)

0.042250

CMA-ES

0.031244 (±0.010834)

0.016952

Coordinate-search method0.045131 (±0.001088)

0.043639

Coordinate-search method

0.032244 (±0.006109)

0.024637

Nelder-Mead method

0.044549 (±0.001079)

0.043238

Nelder-Mead method

0.015492 (±0.002276)

0.013556

MethodMean accuracy (%)

Accuracy withmin loss (%)

MethodMean accuracy (%)

Accuracy withmin loss (%)

Random search

99.56 (±0.02)

99.58

Random search

57.18 (±0.96)

57.90

Bayesian optimization

99.47 (±0.05)

99.59

Bayesian optimization

56.28 (±1.68)

57.19

CMA-ES

99.49 (±0.14)

99.59

CMA-ES57.17 (±0.80)

58.19

Coordinate-search method

99.48 (±0.04)

99.53

Coordinate-search method

55.06 (±2.31)

56.98

Nelder-Mead method

99.53 (±0.00)

99.54

Nelder-Mead method

56.72 (±0.50)

57.42

The smallest loss for each experiment is indicated by bold-faced font

The smallest loss for each experiment is indicated by bold-faced font

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Fig. 9 Mean loss of all executions for each method per iteration(LeNet)the Nelder-Mead method tends to directly converge toa close local optimum without being influenced by theobjective function values of distant points. In contrast,
other methods perform a global search, e.g., Bayesian
optimization and CMA-ES try to find potential candidates of global optima and require more iterations tofind a local optimum in comparison to the Nelder-Meadmethod.
According to the poor hyperparameter setting detection
rates (Tables 15, 16, 17, and 18), on average, approximately 8, 1, 33, and 26% of executions in each experimentare detected as having poor hyperparameter settings andoptimization is accelerated in proportion to the detection rate. In particular, the CNN for age/gender classification tends to be very sensitive to hyperparametersettings.
Note that the Nelder-Mead method rarely generates
poor hyperparameter settings because of its strategy, e.g.,reflection moves the simplex in a direction away from thepoint of poor hyperparameter settings.

Fig. 10 Mean loss of all executions for each method per iteration(Batch-Normalized Maxout Network in Network)

Page 10 of 12

Fig. 11 Mean loss of all executions for each method per iteration(gender classification CNN)

From the above results, we conclude that the NelderMead method is the best choice for DNN hyperparameter
optimization.

6 Conclusions
In this study, we tested methods for DNN hyperparameter
optimization. We showed that the Nelder-Mead method
achieved good results in all experiments. Moreover,
we achieved state-of-the-art accuracy with age/genderclassification using the Adience DB by optimizing the
CNN hyperparameters proposed in [26].
Complicated hyperparameter optimization methods are
difficult to implement and have sensitive hyperparameters, which affects their performance. Therefore, it isdifficult for non-experts to use these methods. In contrast,
the Nelder-Mead method is easy to use and outperforms
such complicated methods in many cases.
In our experiments, we optimized the hyperparametersof DNNs for character recognition and age/gender classification. These tasks are important and have been wellFig. 12 Mean loss of all executions for each method per iteration (ageclassification CNN)

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

Page 11 of 12

Fig. 13 Parallel coordinates plot of the optimized hyperparameters of the gender classification CNN

Table 15 Poor hyperparameter setting detection rate for eachmethod (LeNet)

Table 18 Poor hyperparameter setting detection rate for eachmethod (age classification CNN)

MethodDetection rate

MethodDetection rate

Random search

0.144722 (±0.010690)

Random search

0.444667 (±0.039355)

Bayesian optimization

0.066041 (±0.013354)

Bayesian optimization

0.355933 (±0.008577)

Coordinate-search method

0.080674 (±0.041299)

Coordinate-search method

0.147418 (±0.019866)

CMA-ES

0.111304 (±0.098500)

CMA-ES

0.317533 (±0.207479)

Nelder-Mead method0.003151 (±0.003056)

Nelder-Mead method0.040334 (±0.004082)

Table 16 Poor hyperparameter setting detection rate for eachmethod (Batch-Normalized Maxout Network in Network)

optimization of categorical variables will be also the focus
of future work.

MethodDetection rate

Random search

0 (±0)

Bayesian optimization

0.004943 (±0.001991)

Acknowledgements
This paper is based on results obtained from a project commissioned by the
New Energy and Industrial Technology Development Organization (NEDO).

Coordinate-search method

0.048941 (±0.054884)

CMA-ES0 (±0)

Nelder-Mead method

0 (±0)

known for a long time. However, it is desirable to evaluate
the proposed method using the generic object recognition data set. Therefore, in future, we plan to evaluate
the proposed method using other data sets. A detailed
analysis of the dependency on initial parameters and the

Authors’ contributions
YO implemented the hyperparameter optimization methods for DNNs,
performed the experiments, and drafted the manuscript. MY implemented thehyperparameter optimization methods for DNNs and helped perform theexperiments and draft the manuscript. MO guided the work, supervised theexperimental design, and helped draft the manuscript. All authors have readand approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.
Received: 9 January 2017 Accepted: 4 September 2017

Table 17 Poor hyperparameter setting detection rate for eachmethod (gender classification CNN)
MethodDetection rate

Random search

0.506177 (±0.015931)

Bayesian optimization

0.445244 (±0.005658)

Coordinate-search method

0.274671 (±0.198506)

CMA-ES

0.360734 (±0.091648)

Nelder-Mead method

0.051413 (±0.006113)

References
1. LeCun Y, Bottou L, Bengio Y, Patrick H (1998) Gradient-based learning
applied to document recognition. Proc IEEE 86(11):2278–2324
2. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification withdeep convolutional neural networks. Adv Neural Inf Process Syst (NIPS)
25:1097–1105
3. Christian S, Wei L, Yangqing J, Pierre S, Scott R, Dragomir A, Dumitru E,
Vincent V, Andrew R (2015) Going deeper with convolutions. Comput Vis
Pattern Recognit (CVPR):1–9. http://ieeexplore.ieee.org/document/
7298594/

Ozaki et al. IPSJ Transactions on Computer Vision and Applications (2017) 9:20

4.
5.
6.
7.

8.

9.

10.

11.

12.

13.

14.

15.
16.

17.

18.

19.

20.

21.
22.

23.

24.

25.

26.

Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization.
J Mach Learn Res 13:281–305
Mockus J (1974) On Bayesian Methods for Seeking the Extremum. In:
Optimization Techniques IFIP Technical Conference. pp 400–404
Hansen N, Ostermeier A (2001) Completely derandomized self-adaptationin evolution strategies. Evol Comput 9:159–195
Bergstra J, Bardenet R, Bengio Y, Kégl B (2011) Algorithms for
hyper-parameter optimization. Adv Neural Inf Process Syst (NIPS)
24:2546–2554
Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization ofmachine learning algorithms. Adv Neural Inf Process Syst (NIPS)
25:2951–2959
Watanabe S, Le Roux J (2014) A black box optimization for automatic
speech recognition. In: International Conference on Acoustics, Speech,
and Signal Processing. (ICASSP). pp 3256–3260
Loshchilov I, Hutter F (2016) CMA-ES for hyperparameter optimization ofdeep neural networks. https://arxiv.org/abs/1604.07269.
Accessed 20 Sept 2017
Brochu E, Cora VM, De Freitas N (2010) A tutorial on Bayesian optimizationof expensive cost functions, with application to active user modeling andhierarchical reinforcement learning. https://arxiv.org/abs/1012.2599.
Accessed 20 Sept 2017
Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary M,
Prabhat M, Adams R (2015) Scalable Bayesian optimization using deepneural networks. In: International Conference on Machine Learning.
(ICML). pp 2171–2180
Hansen N, Auger A, Ros R, Finck S, Pošík P (2010) Comparing results of 31
algorithms from the black-box optimization benchmarking BBOB-2009.
In: Proceedings of the 12th Annual Conference Companion on Genetic
and Evolutionary Computation. pp 1689–1696
Conn AR, Scheinberg K, Vicente LN (2009) Introduction to derivative-free
optimization. MPS-SIAM Ser Optim. http://epubs.siam.org/doi/book/10.
1137/1.9780898718768
Nelder JA, Mead RA (1965) Simplex method for function minimization.
Comput J 7:308–313
Gilles C, Patrick R, Mélanie H (2005) Model selection for support vector
classifiers via direct simplex search. The Florida Artificial Intelligence
Research Society (FLAIRS) Conference:431–435
Domhan T, Springenberg JT, Hutter F (2015) Speeding up automatic
hyperparameter optimization of deep neural networks by extrapolationof learning curves. In: Proceedings of the 24th International Joint
Conference on Artificial Intelligence. (IJCAI). pp 3460–3468
Klein A, Falkner S, Springenberg JT, Hutter F (2016) Bayesian neuralnetworks for predicting learning curves. Workshop on Bayesian Deep
Learning, NIPS. http://bayesiandeeplearning.org/2016/papers/BDL_38.
pdf
Klein A, Falkner S, Springenberg JT, Hutter F (2017) Learning curve
prediction with bayesian neural networks. International Conference on
Learning Representations (ICLR). http://www.iclr.cc/doku.php?id=
iclr2017:conference_posters#tuesday_morning
De Rainville FM, Fortin FA, Gardner MA, Parizeau M, Gagné C (2012) “DEAP:
A Python Framework for Evolutionary Algorithms”. In: EvoSoft Workshop,
Companion proc. of the Genetic and Evolutionary Computation
Conference (GECCO). https://dl.acm.org/citation.cfm?id=2330799
LeCun Y, Cortes C (2010) MNIST handwritten digit database. http://yann.
lecun.com/exdb/mnist/. Accessed 20 Sept 2017
Yangqing J, Evan S, Jeff D, Sergey K, Jonathan L, Ross G, Sergio G, Trevor D(2014) Caffe: Convolutional architecture for fast feature embedding.
https://arxiv.org/abs/1408.5093. Accessed 20 Sept 2017
Evan S (2014) Training LeNet on MNIST with Caffe. http://caffe.
berkeleyvision.org/gathered/examples/mnist.html. Accessed 20 Sept
2017
Nair V, Hinton GE (2010) Rectified linear units improve restricted
Boltzmann machines. International Conference on Machine Learning.
https://dl.acm.org/citation.cfm?id=3104425
Chang JR, Chen YS (2015) Batch-Normalized Maxout Network in Network.
In: Proceedings of the 33rd International Conference on Machine
Learning. https://arxiv.org/abs/1511.02583. Accessed 20 Sept 2017
Gil L, Tal H (2015) Age and gender classification using convolutional
neural networks. Computer Vision and Pattern Recognition Workshops(CVPRW). http://ieeexplore.ieee.org/document/7301352/

Page 12 of 12

27. Gil L, Tal H (2015) Age and gender classification using convolutional
neural networks. http://www.openu.ac.il/home/hassner/projects/cnn_
agegender. Accessed 20 Sept 201728. Eran E, Roee E, Tal E (2014) Age and gender estimation of unfiltered faces.
IEEE Trans Inf Forensic Secur 9(12):2170–2179
29. Yangjin J (2013) The learning rate decay policy. https://github.com/BVLC/
caffe/blob/master/src/caffe/proto/caffe.proto. Accessed 20 Sept 2017