Book Title
Book Editors
IOS Press, 2003

1

An Update Method of Computer
Simulation for Evolutionary Robotics
Yoshiaki KATADA a,1 and Kazuhiro OHKURA ba
Faculty of Engineering, Setsunan Universityb
Faculty of Engineering, Kobe University
Abstract. One of advantages of evolutionary robotics over other approaches is its
parallel population search. However, it generally takes an unrealistically long timeto evaluate all candidate solutions by using a real robot. Thus, a technique of computer simulation is considered to be one of the most important topics in evolutionary robotics. Although it is quite difficult to provide a precise computer simulation model of a physical experiment, simulated robot/environment interaction dynamics must be synchronously consistent with the physical one during its evolution for reducing the gap between the simulated and physical one. In this paper,in order to overcome this problem, we propose an update method of simulated
robot/environment interaction dynamics based on a statistical test during evolution,and then investigate the consistency between the performances of a robot in thesimulated and physical environment. A series of experiments with a mobile robot
illustrate the effectiveness of the proposed method in practice.
Keywords., Simulation, Real robot, Evolutionary robotics, Embodied cognitive
science

1. Introduction
Embodied cognitive science[1] have attracted much research interest in recent years,
where a robot must interact with a real world environment for obtaining successful
and robust behaviors. One of approaches in embodied cognitive science is evolutionary
robotics (ER)[2], where robot control systems are designed by using evolutionary techniques.
It has been pointed out that evolutionary approaches are potentially advantageousover other approaches due to their parallel population search. However, a number oftrials needed to evaluate all individual in a real environment make difficult the use ofphysical robots during evolution, especially in the case of tasks with human intervention.
One approach to overcoming this problem would be multiplying the number of the same
real robots for the evaluation of individuals [3]. However, a controller should be robust
enough to work well for all robots because it may perform differently due to slight differences in the electronics and mechanics of robots, even if they are apparently identical. Addition to this, the approach may result significantly expensive due to preparing a
number of robots.
1 Correspondence to: Yoshiaki KATADA, 17-8 Ikeda-nakamachi, Neyagawa, Osaka 572-8508, JAPAN.
Tel./FAX: +81 728 39 9148; E-mail: katada@ele.setsunan.ac.jp

2

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

The other approach would be the use of simulations. Computer simulations may be
helpful to reduce the amount of experimental time to evaluate individuals in a population.
However, the controllers evolved in a simulated environment do not always work wellin the real one because of uncertain effects, e.g., noise and differences in the electronics
and mechanics of robots, as Brooks pointed out [4]. Therefore, the validity of simulations is particularly relevant problem. Some researchers claimed that the problem described above can be overcome by carefully designing simulators. Jakobi et al. claimed
that the addition of noise to relevant parts for a control task in a simulated environment
can reduce the discrepancy between the simulated and real environment. However, such
relevant parts are different on tasks and the detections are crucially dependent on priorknowledges of an experimenter. Miglino et al. [6] developed a simulator which requires
experimental and multiple samplings for constructing a detailed model of sensors andmotors, and claimed that such a method can reduce the performance gap between simulated and real environments. The authors also claimed that the decrease in performanceafter the transfer of a system from a simulated environment to a real one can be recovered by continuing the evolutionary process in the real environment. However, themethod can not adapt to unpredictable changes in a real environment during evolution
due to fixed conditions of the simulation. Moreover, performances would not be always
improved after the transfer of a system from a simulated environment to the real one
without any consideration. An approach to overcoming this problem is to calibrate themodel of robot/environment interaction dynamics during robot’s life time. Such an approach is found in [7], where a world model is learned during robots’ life time. But it isdeterministic and composed of only the most recent data set of sensory inputs and motor
outputs.
In this paper, we propose an update method of a probabilistic model of robot/environment
interaction dynamics for a mobile robot during its evolution employing a statistical test
in order to maintain the validity of the model in simulations. The paper is organized asfollows. Section 2 describes the procedure of the proposed method. Section 3 defines
the robot control problem where the evolved neural networks are evaluated, and representation of robot/environment interaction dynamics. Section 4 gives the results of ourexperiments. Section 5 discusses the effect of the update method on performances of arobot after the transfer to a real environment. Conclusions are given in the last section.

2. An Update Method of a Probabilistic Model of Robot/Environment Interaction
Simulated and real robot/environment interaction dynamics are denoted M (s M , aM ) andR(sR , aR ) respectively, where s is sensory inputs and a is actions. The procedure ofevolutionary algorithms with an update method can be summarized as follows:
0. Create an initial population and initial model, M 0 (sM , aM ).
1. Divide individuals in a population into ones to be evaluated in a real environment
and the others in a simulated one.
2. Gather data, R(s R , aR ), while robots are evaluated in a real environment.
3. Update M (sM , aM ) based on R(sR , aR ).
4. Evaluate the individuals in the simulated environment, M (s M , aM ).

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

3

5. Select the individuals according to the fitness values evaluated in the simulated
and real environment, then apply genetic operations to create a new population of
the same size.
6. Repeat from 1. on until the terminal conditions are satisfied.
In the step 3, the Student t-test is employed in order to examine the validity of amodel in a simulation; R(s R , aR ) and M (sM , aM ) are defined to be statistically identical if we fail to reject the null hypothesis that they are not significantly different. Otherwise, M (sM , aM ) is corrected repeatedly until we fail to reject it. Such a definition
seems appropriate for a probabilistic model like the proposed method. The details aredescribed in the experimental setup.

3. Experimental Setup
3.1. The Task and the Fitness Function
A two-wheeled robot was used in the experiment (Figure 1). The robot’s source of sensory input comes from an omnidirectional camera. The environment of the robot was arectangular arena 2400 × 2200 mm surrounded by walls colored black with a red target
placed at the upper right corner (Figure 2). The control task used in this paper was a goalreach problem where a robot approaches to a target (goal).
Each individual of a population was tested on the robot 4 times for around 40 seconds
each (40 sensory-motor steps). At the beginning of each trial, a robot was always placedat the same initial position, the bottom left corner, at either an upward orientation or arightward orientation (Figure 2). One trial ends either when the robot reaches the goalor when 40 steps are performed without the goal. The performance measure to be maxN umT rials(1 − Step/M axStep)
imized was as follows: F itness = 1/N umT rials i=1
where N umT rials is the number of trials for a robot (2 trials for each initial orientationof a robot) and M axStep is set at 40. The fitness function increases as the robot reaches
the goal more quickly.
3.2. Representation of Robot/Environment Interaction Dynamics
3.2.1. Simulating Motors
The number of the combination of motor commands used for operating the wheels of therobot is 400 (20 motor commands are available for each wheel). Thus, a R is composed of
the set of displacements of a robot for each axis in figure 3, (X R (i, j), YR (i, j), ΘR (i, j)),Omni-directional Camera

Goal
2400mmUpwardRightward
ightward

Note PC

Figure 1. A mobile robot

2200mmMobile Robot

Figure 2. Experimental setup for a goal reach problem

4

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

Figure 3. Displacement of a mobile robot in a simulated environment

for a motor command, (i, j) (i, j = 1, · · · , 20 for the left and right wheel). Basedon these data, the set of displacements for each axis in a simulated environment,(XM (i, j), YM (i, j), ΘM (i, j)), is constructed as aM . In a probabilistic form, the displacements of a two-wheeled robot for each motor command (i, j) are represented as follows [8]: x(i, j) = N (µ(XM (i, j)), σ(XM (i, j))), y(i, j) = N (µ(YM (i, j)), σ(YM (i, j))),θ(i, j) = N (µ(ΘM (i, j)), σ(ΘM (i, j))), where N (µ, σ) is the normal distributions with mean, µ, and standard deviation, σ, derived from the sampled data set,
(XM (i, j), YM (i, j), ΘM (i, j)). Thus, a position of a simulated robot at the t-th stepfor each axis is computed as follows: x t+1 = xt + x(i, j), yt+1 = yt + y(i, j),θt+1 = θt + θ(i, j).
3.2.2. Simulating Sensors
For this experiment, an omni-directional visual sensor was used for the sensory inputsof the robot (Fig. 4). The sensory inputs of an omni-directional vision was computed asfollows. The binarized image is taken from an omni-directional image, where the centerof the image is identified with the center of a robot (Figure 4(a)). In the image, thedirection of the center of gravity of the target to a robot and the distance between a robot
and the nearest point of the target are first calculated by using the pixel informations.
Next, the image is divided into I individual cells in a circumferential direction and J
individual cells in a radial direction (Figure 4(b)) to detect a target by using the above twovalues (Figure 4(c)) and walls by using the pixel informations with a threshold (Figure4(d)) on each cell (q ij = {0, 1}, i = 1, . . . , I;j = 1, . . . , J), where qij signifies thepresence or absence of the object. If an object whose height is almost the same as theone of the robot is detected in the j-th cell, the one is also detected in the (j + 1)-th cell
due to the shape of an omni-directional mirror. The walls are also detected for each cell
(Figure 4(d)).
In a simulation, omni-directional visual sensors were computed as well as the real
ones, assuming binarized images taken from the simulated image. In this experiment, we
kept sM constant during evolution, that is, the model of sensory inputs was not updated
due to the reliability of omni-directional visual sensors and the precision of the model
based on the characteristic of them.
3.3. An Update Method of Displacements of a Simulated Robot
According to the procedure described in section 2, we explain the details to update andcorrect a model of displacements of a robot, a M .

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

2
3

1
4
3
2
1

16
15

Edge

ObjectInput Cell

5

Input Cell

14
13

4

12

5
11

6

Wall

Wall

(a) Camera image(b) Binarized image

10

7
8

9

(c) Target detection(d) Wall detection

Figure 4. Omni-directional image plane

For measuring the positions of a real robot during evaluations of it, a CCD camera was positioned on the top of the testing environment. Every 0.5 seconds, the displacements are computed as follows: x R (i, j) = xt+1 (i, j) − xt (i, j), yR (i, j) =yt+1 (i, j) − yt (i, j), θR (i, j) = θt+1 (i, j) − θt (i, j). After the evaluations, all the data
set in (XR (i, j), YR (i, j), ΘR (i, j)) are add to (XM (i, j), YM (i, j), ΘM (i, j)) as well as
the data set before this 500 generations in it are deleted to keep the data set fresh.
In order to examine the validity of a M , the Student t-test was conducted with
the null hypotheses, µ(X M (i, j)) = µ(XR (i, j)), µ(YM (i, j)) = µ(YR (i, j)) andµ(ΘM (i, j)) = µ(ΘR (i, j)). If the hypotheses are rejected, that is, there are discrepancies between aM and aR , then XM (i, j), YM (i, j), ΘM (i, j) are corrected, respectively.
For instance, if µ(XM (i, j)) < µ(XR (i, j)) and the hypothesis is rejected with a significance level, α, then the smallest data in X M (i, j) is removed to reduce the discrepancy between them. These corrections are repeated until when all the hypotheses are ac
cepted. Based on the corrected data set, (X M (i, j), YM (i, j), ΘM (i, j)), displacements
of a simulated robot are computed as follows: x(i, j) = N (µ(X M (i, j)), σ(XM (i, j))),
y(i, j) = N (µ(YM (i, j)), σ(YM (i, j))), θ(i, j) = N (µ(ΘM (i, j)), σ(ΘM (i, j))).
In the experiments carried out in the next section, we take α = 0.05. For constructing
initial aM0 , 10 data set are sampled for each motor command in advance. a M is updated
after evaluations of real robots equipped with the best and second controller every 50
generations.
3.4. Simulation Conditions
For this experiment, an agent’s controller was pulsed neural networks (PNN) with 128
sensory neurons corresponding to the number of cells I × J in Figure 4(c) and 4(d)for detecting a target and walls, 2 fully interconnected motor neurons and a fully interconnected hidden neuron. The network’s connection weights and the firing thresholdfor each neuron were genetically encoded and evolved. The total number of parameters
was 396. The parameters were mapped linearly onto the following ranges: connection
weights, ω ∈ [−1.0, 1.0], and thresholds, θ ∈ [0.0, 3.9]. The parameters of the neurons
and synapses were set as follows: τ m = 4, τs = 10, ∆ax = 2 for all neurons and allsynapses in the network following the recommendations given in [9]. The experiment
was conducted using populations of size 50. Each individual was encoded as a binary
string with 10 bits for each parameter. Therefore, the total length of the genotype was
L = 3960. An extended GA, which is called the operon-GA (OGA)[10], were employed6

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

to evolve the PNN parameters 1 . The OGA uses standard bit mutation and five additional genetic operators: connection, division, duplication, deletion and inversion. The
probabilities for genetic operations were set at 0.3 for connection and division, 0.6 forduplication and 0.3 for deletion and inversion, based on our previous results in [11,12].
The length of the value list in a locus was 6. The per-bit mutation rate, q, was set at
1/L = 0.00025. Crossover was not used for the OGA, following Nimwegen’s suggestion[13]. Tournament selection was adopted. Elitism was optionally applied. The tournament
size was set at 6 because the OGA prefers high selection pressure. A generational model
was used. Each run lasted 2, 000 generations.
The individuals in a population were divided into two groups before evaluations in areal environment as described in section 2; Two individual (the best and second one) of
the population to be evaluated in a real environment and the others in a simulated one.
4. Experimental Results
Figure 5 shows the maximum and average fitness at each generation for the OGA. The
maximum fitness increased in the early generations without being trapped on local optima although the fluctuations are large in the middle generations.
In order to verify what happens at the level of behaviors, we compared the trajectories of the best evolved individual of 2, 000 generations in the simulated and in the realenvironment. Figure 6 and 7 show the typical behaviors from the initial orientation of thebest evolved robot. In the simulated environment (Figure 6), the robot at the initial rightward orientation approached to the goal turning left to direct toward it. This trajectory
matched almost perfectly the one in the real environment (Figure 7). It demonstrates thatthe proposed update method reduces very significantly the discrepancy between behaviors in the simulated and real environment. Due to space restrictions we only show results
for the behavior at the initial rightward orientation. However, the discrepancy between
behaviors in the simulated and real environment was also reduced at the initial upwardorientation.
5. Discussion
In order to investigate the validity of the proposed update method, we compared the performance of the robot controller evolved in both a simulated and real environment with
1 In the preliminary runs, it has been confirmed that the OGA outperformed the standard GA in this problem.

Fitness

1.0

Max
Average

Goal
Mobile Robot

0.5

0.0
0

500

1000
1500
Generation

2000

Figure 5. Maximum and average fitness at each gen- Figure 6. Behavior of the best evolved robot in thesimulated environment
eration for the OGA

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

7

Goal

Start

1

2

3

4

Figure 7. Behavior of the best evolved robot in the real environment
Table 1. Success Rate(%) for Each Method
Initial orientation

UpwardRightward

Without update

10

0

Total
5

With update

40

100

70

the updates, the method with update, with the performance of the evolved one only ina simulated environment without any update, the method without update. An additional
experiment was therefore conducted. We evolved robot controllers only in a simulated
environment without any update in 2, 000 generations and then transfered the obtained
control system to the real environment. The initial data set, a M0 , were used until the lastgeneration in the method without update. The representation of robot/environment interaction dynamics in the method without update was the same as the one in the methodwith update. It is likely that the controller evolved only in a simulated environment is
not robust enough to work well when transfered to the real environment because it may
have discrepancies between the simulated and real environment. Table 1 shows the rates
achieving the task in the 10 trials for the controllers evolved by the method with and without update. As predicted, the degradations in performance after the transfer of the systemfrom the simulated environment to the real one are observed in the controllers evolved
only in the simulated environment. Moreover, the controller evolved by the method performed better with update than without it for both initial orientation of a robot. This result
indicates the effectiveness of the proposed method.

6. Conclusions
In this work, we proposed an update method of simulated robot/environment interaction
dynamics for an evolutionary mobile robot during its evolution employing a statistical
test in order to maintain the validity of a model in the simulation. Our results can besummarized as follows:• An update method of a probabilistic model of robot/environment interaction dynamics was designed to reduce the discrepancy between the simulated and realenvironment.
• The proposed method successfully reduced the gap between performances observed in a simulated environment and performances obtained in the real environment.
• In order to investigate the benefits of the proposed method, the performance of thebest evolved robot after the transfer to the real environment is compared with the

8

Y. Katada and K. Ohkura / An Update Method of Simulation for ER

one by the method employing only simulation. The proposed method performed
better than the method employing only simulation.
These results suggest some guidelines for reducing the gap between performances
observed in simulated environment and performances obtained in the real environment
for evolved robots. Future work will investigate how well these updating guidelines apply
to more complex experiments to be performed, e.g., updating a model of sensory inputs
and humanoid robots.

References[1] R. Pfeifer. and C. Scheier, Understanding Intelligence, MIT Press, Cambridge, 1999.
[2] S. Nolfi and D. Floreano, Evolutionary Robotics: The Biology, Intelligence, and Technology
of Self-Organizing Machines, MIT Press, 2000.
[3] R. A. Watson, S. G. Ficici and J. B. Pollack, Embodied Evolution: Embodying an Evolutionary Algorithm in a Population of Robots, In Proceedings of Congress on Evolutionary
Computation, pp.335-342, 1999.
[4] R. A. Brooks, Artificial Life and Real Robots, In Proceedings of the First European Conference on Artificial Life, pp.3-10, 1992.
[5] N. Jakobi, Half-baked Ad-hoc and Noisy: Minimal Simulation for Evolutionary Robotics, In
Proceedings of the Fourth European Conference on Artificial Life, pp.348-357, 1997.
[6] O. Miglino, H. H. Lund and D. Nolfi, Evolving Mobile Robots in Simulated and Real Environments, Artificial Life 2, pp.417-434, 1995.
[7] D. Keymeulen, M. Iwata, K. Konaka, R. Suzuki, Y. Kuniyoshi and T. Higuchi, Off-line Modefree and On-line Model-based Evolution for Tracking Navigation Using Evolvable Hardware,
In Proceedings of the First European Workshop on Evolutionary Robotics, Springer-Verlag,
Paris, 1998.
[8] K. Komoriya, E. Oyama and K. Tani, Planning of Landmark Measurement for the Navigationof a Mobile Robot, Journal of the Robotics Society of Japan, Vol. 11, No. 4, pp.533-540,
1993 (in Japanese).
[9] D. Floreano, C. Mattiussi, Evolution of Spiking Neural Controllers, In Gomi, T. (ed.): Evolutionary Robotics: From Intelligent Robots to Artificial Life (ER’01), AAI Books, SpringerVerlag, pp. 38-61, 2001.
[10] K. Ohkura, K, K. Ueda, Adaptation in Dynamic Environment by Using GA with Neutral
Mutations, International Journal of Smart Engineering System Design, 2, pp.17-31, 1999.
[11] Y. Katada, K. Ohkura, K. Ueda, Tuning Genetic Algorithms for Problems Including Neutral
Networks -A More Complex Case: The Terraced NK Problem-, In Proceedings of the 7th
Joint Conference on Information Sciences, pp.1661-1664, 2003.
[12] Y. Katada, K. Ohkura, K. Ueda, An Approach to Evolutionary Robotics Using the Genetic
Algorithm with Variable Mutation Rate Strategy, In Proceedings of The 8th Parallel Problem
Solving from Nature (PPSN VIII), pp.952-961, 2004.
[13] E. Nimwegen,J. Crutchfield, M. Mitchell, Statistical Dynamics of the Royal Road Genetic
Algorithm, Theoretical Computer Science, Vol. 229, No. 1, pp.41-102, 1999