Investigation of Simply Coded Evolutionary Artificial Neural
Networks on Robot Control Problems
Yoshiaki Katada and Jun Nakazawa

Abstract— One of the advantages of evolutionary robotics
over other approaches in embodied cognitive science would be
its parallel population search. Due to the population search, ittakes a long time to evaluate all robot in a real environment.
Thus, such techniques as to shorten the time are required
for real robots to evolve in a real environment. This paper
proposes to use simply coded evolutionary artificial neuralnetworks for robot control to make genetic search space assmall as possible and investigates the performance of them using
simulated robots. Two types of genetic algorithm (GAs) are
employed, one is the standard GA and the other is an extended
GA, to achieve higher final fitnesses as well as achieve highfitnesses faster. The results suggest the benefits of the proposed
method.

I. I NTRODUCTION
Embodied cognitive science[1] have attracted much research interest in recent years, where a robot must interact
with a real world environment for obtaining successful androbust behaviors. One of approaches in embodied cognitive
science is evolutionary robotics[2], where robot control systems are designed by using evolutionary techniques.
It has been pointed out that evolutionary approaches arepotentially advantageous over other approaches due to theirparallel population search. However, they make difficult touse physical robots without any consideration, especially inthe case of tasks with human intervention because they takea long time to evaluate all robots in a real environment.
One approach to overcoming this problem would be theuse of simulations. Computer simulations may be helpful to
reduce the amount of experimental time to evaluate individuals in a population. However, the controllers evolved in asimulated environment do not always work well in the real
one because of uncertain effects, e.g., noise and differencesin the electronics and mechanics of robots [3]. Therefore,
the validity of simulations is a particularly relevant problem.
Some researchers claimed that the problem described above
can be overcome by carefully designing simulators [4][5].
Others proposed to calibrate the model of robot/environment
interaction dynamics during evolution [6][7]. However, theseapproaches are limited to the tasks where interaction dynamics between robot and environment is properly modeled.
Therefore, we focus on conducting the evolutionary run
entirely in a real environment where an evolved controller
must work well, while we consider to shorten the total
amount of experimental time to evolve all robots. Our
Yoshiaki Katada and Jun Nakazawa are with the Department of Electricaland Electronic Engineering, Setsunan University, 17-8 Ikeda-nakamachi,
Neyagawa, Osaka 572-8508, JAPAN (phone/fax: +81 728 39 9148; email:katada@ele.setsunan.ac.jp).

approach is to make genetic search space as small as possible.
In other words, if the length of the genotype becomes short, it
would be possible to shorten the experimental time whatever
evolutionary algorithms are employed. Moreover, two typesof GAs are applied, which are the standard GA and an
extended GA, in order to achieve higher final fitnesses aswell as achieve high fitnesses faster.
In this paper, we propose simply coded evolutionary
artificial neural networks in order to shorten the time to
evolve robots and investigates the performance of themfor the GAs on evolutionary robotics tasks using simulated
robots. The reasons why our work conducts computer simulations in advance although we argue against simulationbased approaches in evolutionary robotics are as follows; (1)Good performances of the proposed approach in computer
simulations can be considered as the precondition for good
performances in real environments that the one is expected to
be able to show. (2) It takes extraordinary time to investigate
the ability of the proposed approach in real environments forall experimental conditions, which would be tough works.
The paper is organized as follows. Section II describes how
to simply code artificial neural networks using a binary string
for GAs. Section III and IV define the robot control problems
where the evolved neural networks are evaluated, and givethe results of our experiments. Conclusions are given in thelast section.
II. S IMPLY C ODED E VOLUTIONARY A RTIFICIAL N EURALN ETWORKS (SCEANN)This section describes how to simply code artificial neuralnetworks for GAs.
A. Artificial Neural Networks
Artificial neural networks (ANN) is used with Ns sensory
neurons, No fully interconnected motor neurons and Nh fullyinterconnected hidden neurons for a robot’s controller.
The output of the i-th neuron at time t is given by:ωij xj (t − 1))(1)
xi (t) = f (jwhere ωij is the connection weight from the neuron j to theneuron i, and f (x) is the output function of neurons, givenby the sigmoid function shown in Figure 1. Namely, their
outputs are given by:
1(2)
f (x) =
1 + exp(−x/T )where T is a positive parameter to control the slope ofthe sigmoid function. The output range is [0, 1]. The output

1

sign

synapse presence

f(x)

0.8

+ s s s s h h h o o
㧙

0.6
0.4

(a) SCEANN1+ s+ s+ h+ o+ o
㧙
㧙
㧙
㧙
㧙

0.2
0
-8

-6

-4

Fig. 1.

-2

0
x

2

4

6

8

(b) SCEANN2+ + + + + + + + + +㧙 㧙 㧙 㧙 㧙 㧙 㧙 㧙 㧙 㧙

Sigmoid functionof neurons given by the sigmoid function becomes easily
saturated by getting successive inputs whether excitatory or
inhibitory. Therefore, refractoriness[8] is introduced to reset
the state of a neuron to 0 when the total amount of inputs
exceeds its threshold, ±xc . In this work, |xc | is set at 8. In
a behavioral level, these resets prevent a robot from being
stagnated in its environment.
In general EANN, network’s connection weights, firing
thresholds for each neuron (the slope when the sigmoid
function employed), the architecture of networks and learning rules are evolved [9]. In our approach, only connection
weights are evolved as variables of GAs in order to shorten
the length of the genotype. Therefore, the slope of thefunction, T , is set at 1, which is usually employed in patternclassification.
We proposed to use three kinds of architecture. Eacharchitecture is genetically represented by a binary string. The
details are described as follows.

Fig. 3.

(c) SCEANN3
Genetic representation of one block{1, 0} of a connection from each neuron (Figure 3(a)). The
synaptic strengths of all existing connections are set at 1.
Therefore, the total length of the string is L = (Nh +No )(1+
Ns + Nh + No ) bits, where Ns , Nh and No are the numberof sensory, hidden and motor neurons, respectively.
C. SCEANN 2
The second one is an extended form of SCEANN1, which
we propose. The signs of all the connection weights fromsensory, hidden and motor neurons to a corresponding neuron
are encoded in order to increase variety in the architecture.
Thus, each block is composed of the signs {+, −} andthe presence/absence {1, 0} of all the connection weights(Figure 3(b)). Therefore, the total length of the string is
L = 2(Nh + No )(Ns + Nh + No ) bits.

B. SCEANN 1

D. SCEANN 3

The first one is originally proposed by Floreano [10][11],which was used for evolving real robots. A string is composed of a series of blocks, each block defined for a neuron
in hidden and motor neurons (Figure 2). The first bit of a
block encodes the sign {+, −} of all the connection weightsfrom sensory, hidden and motor neurons to a corresponding
neuron and the remaining bits encode the presence/absence

Sensory, hidden and motor neurons are fully connectedwithout coding the presence/absence of a connection fromeach neuron like SCEANN1 and SCEANN2. Thus, eachblock is composed of only signs of all the connection weightsfrom sensory, hidden and motor neurons (Figure 3(c)). The
synaptic strengths of all connections are set at 1. The total
length of the string is L = (Nh + No )(Ns + Nh + No ) bits.
In the following sections, we apply these SCEANNs toevolutionary robotics tasks using simulated robots in order
to investigate performances of SCEANNs.
III. G OAL R EACH P ROBLEM FOR A M OBILE ROBOT
A. The Task and the Fitness Function

Sensor Neuron
Fig. 2.

Hidden Neuron

Motor Neuron

Architecture of ANNs for 1 block of a string

A two-wheeled robot was used in this experiment (Figure 4). The environment of the robot was a rectangular arena
surrounded by walls with a target placed at the upper rightcorner. The control task used in this section was a goal reachproblem where a robot approaches to a target (goal). At the
beginning of each trial, a robot was always placed at the same
initial position, the bottom left corner, at random orientations

the interval between the wheels and ω[·]t is the system error.
It has been known that the system error in Equation (4) ismodeled as the normal distribution N (0, σ) with mean zeroand standard deviation, σ [12]. For this experiment, σ wasset at the 10 percent of the displacement at the current step.

GOAL

B. Simulation Conditions

Fig. 4.

Experimental setup for a goal reach problem(Figure 4). One trial ends either when the robot reaches thegoal or when 200 steps are performed without the goal. The
performance measure to be maximized was as follows:
1
N umT rials

F itness =N umT
rials

(1 −

i=1

Stepi
) (3)
M axStep

where N umT rials is the number of trials for a robot (8trials for each initial orientation of a robot) and M axStep isset at 200. The fitness function increases as the robot reaches
the goal more quickly.
The robot was provided with Ns infrared proximity sensors which have a maximum detection range in the environment. If a target intersects a proximity sensor, the sensor
outputs a value inversely proportional to the distance between
the target and the sensor. Employing a mathematical modelof a mobile robot (Figure 5), the displacement of the robot
was computed as follows:
VR + VLcos θt + ωxt+1
2
VR + VLsin θt + ωyt+1
yt+1 = yt +(4)2VR − VL+ ωθt+1 ,θt+1 = θt +2R
where VR and VL are the velocities applied to the right andleft wheel respectively, R is the radius of a robot, 2R is= xt +

xt+1

For this experiment, a robot’s controllers were SCEANNs
with Ns sensory neurons, 2 fully interconnected motor
neurons and Nh fully interconnected hidden neurons. The
performances of the controllers were investigated over the
relatively large Ns range, where N s ∈ {1, 2, 3, 5, 7}, because
the number of sensors generally depends on the settingof such an experiment that EANNs are applied to. The
performances over the relatively small Nh range were also
investigated, where Nh ∈ {1, 2, · · · , 5}, because SCEANNs
aim to shorten the length of the genotype. Such results as
to vary Ns and Nh respectively would be useful for EC
practitioners.
Computer simulations were conducted using populationsof size 25. The standard GA (SGA) and the operon-GA(OGA)[13] were employed to evolve the string of SCEANNs.
The OGA uses standard bit mutation and five additional
genetic operators: connection, division, duplication, deletion
and inversion. The probabilities for genetic operations wereset at 0.3 for connection and division, 0.6 for duplicationand 0.3 for deletion and inversion, based on our previous
results in [14][15]. The genetic operation for the SGA wasstandard bit mutation. For both GAs, the per-bit mutation
rate, q, was set at 1/L. Crossover was not used for either
GA [14]. Tournament selection was adopted. Elitism1 wasapplied. The tournament size was set at 2 for the SGA
and 6 for the OGA because the SGA prefers low selection
pressure while the OGA prefers high selection pressure[14]. A generational model was used. Each run lasted 50
generations. We conducted 50 independent runs. All results
were averaged over 50 runs. In this work, the performancesin 50 generations are compared for both GAs to examine
whether the evolving time of robots can be reduced. Thus,
acquisitions of a feasible solution in that generations are
considered to be the validity of the proposed approach.
C. Simulation Results
Figure 6 shows the maximum fitness at the final generation
for the SGA and the OGA for controllers with Ns ∈{1, · · · , 7} and Nh ∈ {1, · · · , 5}. Figure 6(a), 6(b), 6(c)and 6(d) show the results for SCEANN1 and SCEANN2,
respectively. The higher final fitnesses were achieved with
the increase of Ns and with the decrease of Nh for both
GAs. The OGAs performed better than the SGAs for eachNs and Nh .
Figure 6(e) and 6(f) show the results for SCEANN3. Also,
the higher fitnesses were achieved with the increase of Ns .
The final fitnesses became higher in order of Nh = 1 →

y

VL
VR

2R

P(x,y)

0Fig. 5.

x
Simulated model for a mobile robot

1 The fittest individual of each generation was passed un-mutated to thenext generation (if several individuals had the highest fitness, one was
randomly chosen.)Fitness

0.4
0.3

0.5

Nh=1
Nh=2
Nh=3
Nh=4
Nh=5

0.4
Fitness

0.5

0.3

0.2

0.2

0.1

0.1
1

2

3
Ns

5

7

Nh=1
Nh=2
Nh=3
Nh=4
Nh=5
1

0.5

0.5

0.4

0.4

0.3
Nh=1
Nh=2
Nh=3
Nh=4
Nh=5

0.2
0.1
1

2

3

5

0.1
7

1

2

0.4
Fitness

Fitness

0.4
0.3
Nh=1
Nh=2
Nh=3
Nh=4
Nh=5
3
Ns

(e) SCEANN3 by the SGA
Fig. 6.

57(d) SCEANN2 by the OGA

0.5

2

3
Ns

0.5

1

7

Nh=1
Nh=2
Nh=3
Nh=4
Nh=5

0.2

(c) SCEANN2 by the SGA

0.1

5

0.3

Ns

0.2

3
Ns

(b) SCEANN1 by the OGA

Fitness

Fitness(a) SCEANN1 by the SGA

2

5

0.3
Nh=1
Nh=2
Nh=3
Nh=4
Nh=5

0.2
0.1
7

1

2

3
Ns

(f) SCEANN3 by the OGA

Maximum fitness at the final generation for each coding by the SGA and the OGA on a goal reach problem

57

5 → 3 → 4 → 2 for both GAs. Unlike SCEANN1 and
SCEANN2, both GAs performed worst with Nh = 1 in all
Nh . The OGAs performed better than the SGAs for each Ns
and Nh .
For the SGA with Nh = 1, 2, 3, SCEANN1 performed better than SCEANN2 and SCEANN3. With
Nh = 4, 5, SCEANN3 performed better than SCEANN1and SCEANN2. Whereas, for the OGA with Nh =1, 3, SCEANN1 performed better than SCEANN2 and
SCEANN3. With Nh = 2, 4, 5, SCEANN3 performed better
than SCEANN1 and SCEANN2.
Figure 7 shows the typical behavior of the best evolved
controller (Figure 8) for SCEANN3 with (Ns , Nh ) = (7, 2)
by the OGA. The robot turned right at the initial position to
direct toward the goal, then approached to it. The trajectoryof the robot was almost like a straight line.
On these results, it seems likely that the control task usedin this section was easy because the task was achieved inalmost all runs for each condition. In the next section, anothercontrol task was used, where performances of the codings
would be more distinguishable than the task used in thissection.

Fig. 7. Behavior of the genetically determined controller for SCEANN3
with (Ns , Nh ) = (7, 2) by the OGA

IV. P OLE - BALANCING P ROBLEM
A. The Task and the Fitness Function
The control task used in this section was a pole-balancing
problem[16][17], which is a classical test problem not onlyin evolutionary robotics but also in control theory. A single
pole is centered on a cart, which may move left or righthorizontally along the track (Figure 9). The cart must moveleft or right in order to keep the pole balanced and avoid thetrack boundaries. In this experiment, the length of the pole is
1.0 m. The mass of the pole is 0.1 kg and the one of the cart
is 1.0 kg. The maximum force to push the cart horizontallyis 10 N. The cart and pole system were simulated for 3, 000
time steps using Euler’s method with the time step, 0.02
seconds. The performance measure to be maximized was thenumber of time steps in which the system keeps balanced.
One trial ends either when the pole falls past 36 degrees orwhen the cart reaches the boundary of the 4.8 meter track.
The state of this system is defined by the following statevariables: the position of the cart on the track, x, the angleof the pole from vertical, θ, the velocity of the cart, ẋ, andthe angular velocity of the pole, θ̇.
B. Simulation Conditions
For this experiment, controllers were SCEANNs with N ssensory neurons, where N s ∈ {2, 4}, 1 output neuron
and Nh fully interconnected hidden neurons, where Nh ∈{1, 2, · · · , 5}. For N s = 4, the state information, x, θ, ẋ andθ̇, is given to the controller, and for N s = 2, the amount ofstate information is restricted, that is, only providing x andθ. The state variables were scaled to [0.0, 1.0] before being
input to the controller. Every time step, the controller outputs
a force value in the range [−10.0, 10.0] N. The initial angle
for the pole was set at 1◦ so that the pole naturally falls
without any control.
Computer simulations were conducted using populationsof size 25. The SGA and the OGA were employed to evolvethe string of SCEANNs with the same settings as those in
the previous section. Each run lasted 50 generations. We
conducted 50 independent runs. All results were averaged
over 50 runs.
C. Simulation Results
Figure 10 shows the maximum fitness at the final generation for the SGA and the OGA for controllers with

Fig. 8.
A genetically determined controller for SCEANN3 with
(Ns , Nh ) = (7, 2) by the OGA on a goal reach problem: the solid lines
indicate excitatory connection weights and the dashed lines inhibitory.

x
Fig. 9.

Experimental setup for a pole-balancing problem

2000

2000

Ns=2
Ns=4

1500
Fitness

Fitness

1500

Ns=2
Ns=4

1000
500

1000
500

0

0
1

2

3
Nh

4

5

1

(a) SCEANN1 by the SGA

2000

2000

Ns=2
Ns=4
Fitness

500

5

4

5

4

5

Ns=2
Ns=4

1000
500

0

0
1

2

3

4

5

1

2

Nh

(d) SCEANN2 by the OGA

2000

Ns=2
Ns=4

Ns=2
Ns=4

1500
Fitness

1500
1000
500

1000
500

0

0
1

2

3
Nh

(e) SCEANN3 by the SGA
Fig. 10.

3Nh(c) SCEANN2 by the SGA

Fitness

4

1500

1000

2000

3
Nh

(b) SCEANN1 by the OGA

1500
Fitness

2

4

5

1

2

3
Nh

(f) SCEANN3 by the OGA

Maximum fitness at the final generation for each coding by the SGA and the OGA on pole-balancing problems

0.1

x
θ

TABLE IS UCCESS RATE (%) OF SCEANN3 FOR THE SGA IN 50 RUNS

30

PP
PPNh
1
Ns
P
P

20

0.05
0
-10
-0.05
-20
-0.1
0

1000

2000

0
0

2

3

4

5

0
0

34
12

26
4

20
0

o

0

θ[ ]

x [m]

10

2
4

-30
3000

TABLE IIS UCCESS RATE (%) OF SCEANN3 FOR THE OGA IN 50 RUNS

PP
PPNh
1
Ns
P
P
2
4

0
0

2

3

4

5

0
0

44
26

36
8

26
6

step
Fig. 11.

Position of the cart and the angle of the pole at each time step

those results cannot be considered fully satisfactory because
the success rates of SCEANN3 achieving the task (balancing
the pole and the cart for the maximum time steps) were notso high (Table I and II).
V. C ONCLUSIONS

Fig. 12.
A genetically determined controller for SCEANN3 with
(Ns , Nh ) = (2, 3) by the OGA on a pole-balancing problem: the solidlines indicate excitatory connection weights and the dashed lines inhibitory.

Ns ∈ {2, 4} and Nh ∈ {1, · · · , 5}. Figure 10(a) and 10(b)show the results for SCEANN1. For this coding, the polelost its balance in the early time steps under all conditions.
Figure 10(c) and 10(d) show the results for SCEANN2.
The lower fitnesses were achieved with the increase of Nhfor both GAs. The OGAs performed better than the SGAs
for each Ns and Nh .
Figure 10(e) and 10(f) show the results for SCEANN3.
The higher fitnesses were achieved with the increase of Nhfor both GAs but they fall sharply when Nh exceeds 4. The
differences between for N s = 2 and for N s = 4 were much
more pronounced than for SCEANN2, that is, the fitnesses
were higher for N s = 2 than those for N s = 4 for both
GAs. This result is different from those obtained in generalcases because it is known [17] that pole-balancing problems
without velocity information are more difficult than thosewith it. As with SCEANN2, the OGAs performed better thanthe SGAs for each Ns and Nh .
Figure 11 shows the behavior of the best evolved controller(Figure 12) for SCEANN3 with (Ns , Nh ) = (2, 3) by the
OGA. The controller was able to balance the pole and the cartby keeping them swinging left and right. The amplitudes ofthe oscillation were so small as not to reach the boundaries.
In this experiment, SCEANN3 with (Ns , Nh ) = (2, 3) for
the OGA shows best performance in all conditions. However,In this work, we proposed simply coded evolutionary
artificial neural networks in order to shorten the time to
evolve robots and investigated the performance of them on
evolutionary robotics tasks using simulated robots. Two typesof GA were employed to achieve higher final fitnesses aswell as achieve high fitnesses faster. Our results can besummarized as follows:• In a goal reach problem, increasing Ns improved theperformance of SCEANNs for both GAs. SCEANN1
shows good performances when Nh is small, and sodoes SCEANN3 when Nh is relatively large. The controllers reaching the goal were obtained in almost allruns.
• In a pole-balancing problem, SCEANN3 for Nh =3 shows the best performances for both GAs. Also,
SCEANN3 achieved higher final fitnesses without velocity information than those with velocity information.
The controllers keeping the pole balanced for the maximum time steps were obtained in many runs.
• The OGAs performed better than the SGAs for each Ns
and Nh on both tasks.
These results suggest that SCEANNs are applicable to relatively easy control tasks, such as the navigation of mobile
robots. From the analytical point of view, the simplicity ofthe obtained architectures would make behavioral analyses
in a neuron level easier.
Establishing the generality of these results will require
investigating the performance of SCEANNs on real robot
problems.
R EFERENCES[1] R. Pfeifer. and C. Scheier, Understanding Intelligence, MIT
Press, Cambridge, 1999.
[2] S. Nolfi and D. Floreano, Evolutionary Robotics: The Biology,
Intelligence, and Technology of Self-Organizing Machines,
MIT Press, 2000.

[3] R. A. Brooks, Artificial Life and Real Robots, In Proceedingsof the First European Conference on Artificial Life, pp.3-10,
1992.
[4] N. Jakobi, Half-baked Ad-hoc and Noisy: Minimal Simulation for Evolutionary Robotics, In Proceedings of the Fourth
European Conference on Artificial Life, pp.348-357, 1997.
[5] O. Miglino, H. H. Lund and D. Nolfi, Evolving Mobile Robots
in Simulated and Real Environments, Artificial Life 2, pp.417434, 1995.
[6] D. Keymeulen, M. Iwata, K. Konaka, R. Suzuki, Y. Kuniyoshi
and T. Higuchi, Off-line Mode-free and On-line Model-based
Evolution for Tracking Navigation Using Evolvable Hardware,
In Proceedings of the First European Workshop on Evolutionary Robotics, Springer-Verlag, Paris, 1998.
[7] Y. Katada and K. Ohkura, An Update Method of Computer
Simulation for Evolutionary Robotics, Intelligent Autonomous
Systems 9 (IAS-9), pp.357-364, 2006.
[8] W. Maass and C.M. Bishop, Pulsed Neural Networks, MITpress (1998)[9] X. Yao, Evolving Artificial Neural Networks, In Proceedingsof the IEEE, 87(9):1423-1447, 1999.
[10] D. Floreano, C. Mattiussi, Evolution of Spiking Neural Controllers, In Gomi, T. (ed.): Evolutionary Robotics: From Intelligent Robots to Artificial Life (ER’01), AAI Books, SpringerVerlag, pp. 38-61, 2001.
[11] J.C. Zufferey, D. Floreano, M. van Leeuwen and T. Merenda,
Evolving Vision-based Flying Robots, 2nd International

Workshop on Biologically Motivated Computer Vision(BMCV’2002), Lecture Notes in Computer Science, pp. 592–
600, 2002.
[12] K. Komoriya, E. Oyama and K. Tani, Planning of Landmark
Measurement for the Navigation of a Mobile Robot, Journalof the Robotics Society of Japan, Vol. 11, No. 4, pp.533-540,
1993 (in Japanese).
[13] K. Ohkura, K, K. Ueda, Adaptation in Dynamic Environment
by Using GA with Neutral Mutations, International Journal of
Smart Engineering System Design, 2, pp.17-31, 1999.
[14] Y. Katada, K. Ohkura, K. Ueda, Tuning Genetic Algorithms
for Problems Including Neutral Networks -A More Complex
Case: The Terraced NK Problem-, In Proceedings of the
7th Joint Conference on Information Sciences, pp.1661-1664,
2003.
[15] Y. Katada, K. Ohkura, K. Ueda, An Approach to Evolutionary
Robotics Using the Genetic Algorithm with Variable Mutation
Rate Strategy, In Proceedings of the 8th Parallel Problem
Solving from Nature (PPSN VIII), pp.952-961, 2004.
[16] D. E. Moriarty and R. Miikkulainen, Efficient Reinforcement
Learning through Symbiotic Evolution, Machine Learning,
Kluwer Academic Publishers, Vol. 22, pp. 11–33, 1996[17] F. J. Gomez and R. Miikkulainen, Solving Non-Markovian
Control Tasks with Neuro-Evolution, In Proceedings of The
International Joint Conference on Artificial Intelligence (IJCAI99), pp. 1356-1361, 1999