Evolutionary Approachfor Designing the Behavior Generator of Communication Robot
Yuki SUGA1 , Tetsuya OGATA2 and Shigeki SUGANO1
1Department of Mechanical Engineering, Waseda University, Tokyo, Japan(Tel : +81-3-5286-3264; E-mail: ysuga, sugano@sugano.mech.waseda.ac.jp)
2
Graduate School of Informatics, Kyoto University, Kyoto, Japan(Tel : +81-75-753-5977; E-mail: ogata@i.kyoto-u.ac.jp)

Abstract: Our goal is to create the robot system which interacts with human users keeping their interest during along period. We focus on the Interactive Evolutionary Computation (IEC) technique to achieve this goal. Although theIEC enables users to design various systems which reflect their subjective preferences, it forces users to evaluate a huge
number of individuals in the genetic pool during the evolution period. To solve this problem, we propose a refined IEC
technique, named Human-Machine Hybrid Evaluation (HMHE), which selects the representative genes for user evaluationand estimates the evaluation results of the other genes. It can increase the population size without increasing the users’
evaluation processes. We carried out some simulations where a humanoid robot with our method interacted with a user.
The experimental results demonstrated that the HMHE could continue to generate the various robot behaviors by adapting
to the transition of user’s subjective preferences.
Keywords: Evolutionary Computation, Communication Robot, Human-Robot Interaction

1. INTRODUCTION
In recent years, robots are expected to contribute to ourlives as helpful staffs. They offer potential uses in many
areas, especially in entertainment, healthcare, housekeeping and nursing.Those robots that work around people
should be capable of communication with people not only
to receive their commands but also to entertain or to takecare of them [1]. However, most of the human-robot
communication is based on scenarios (models) designed
by developers. Though at present, the model-based communication is certainly the most practical and fastest way
to achieve verbal and context-sensitive communication,
the robot becomes quite boring, once users take noticesof the scenarios.
On the other hand, to create more flexible and adaptive robot, the behavior based technique is focused on by
many researchers [2]. In the technique, the behaviors of
the robot are generated through the mutual interaction between the robot and the environment including human beings. Nevertheless, those behaviors are much influenced
by the developers’ own preference, so they sometimes appear questionable for users.
In this case, it is preferable for the robot to be readjustable, but the users usually do not have any programming techniques. Therefore the robots should have a machine learning function to configure the behavior automatically, such as artificial neural network, reinforcement
learning, genetic algorithm.Moreover, the learning function generates the diversity of behaviors, which preventsusers from boredom. So our goal is to create a communication robot which can adapt to the users preference inorder to keep the users interested in the robot for a long
period.
However, in order to learn the preference of a user, it
is indispensable to evaluate how successful the interaction between the user and the robot was. Ishiguro et al.

conducted some experiments of behavior adaptation using Policy Gradient Reinforcement Learning [3]. In thisexperiment, they hypothised that a human’s gaze, motion
speed, and distance between a human and a robot indicate how well the interaction was. However, the human’s
evaluation and interaction are quite complicated and easily varied, so the subjects sometimes behaved in the ways
the experimenters did not expected.
To address the problem, in this paper, we propose atechnique of applying the Interactive Evolutionary Computation (IEC) into a human-robot communication system. The IEC is a type of Evolutionary Computation like
Genetic Algorithm (GA), but its fitness function is performed by users. When using the IEC, it is unnecessary
to define the fitness function of the evolutionary computation. We also propose the human-machine hybrid evaluation (HMHE) to solve a problem of human fatigue.
The structure of the rest of this paper is as follows:
In the next section, the IEC with HMHE is described in
more detail. In section 3, we show the experimental settings. In section 4, experimental results are shown and
we discuss them in section 5. Finally, we summarize thispaper, and we show our future works.

2. IEC WITH HMHE
2.1 Interactive Evolutionary Computation
In the conventional Evolutionary Computation (EC),each individual is evaluated using a given fitness function. In the IEC, its fitness function is performed by users,
that is, all of the pheno-types are evaluated by the users.
It therefore applies the EC techniques to problems of subjective optimization without an explicit modeling of human subjective evaluation [4]. The IEC has previously
been applied to aesthetic areas, such as art, design, andmusic [5] [6]. In these areas, fitness functions can not always be defined explicitly. With the IEC, however, it isunnecessary to define them.
The IEC has also been applied to optimizing behaviors of robots. For example, in a study entitled “A Children’s Game” [7], Lund et al. confirmed that the children
could develop the robot’s controller by repeatedly selecting one robot from nine robots shown in the CRT display.
This study’s aim was developing a robot controller without programming.
Since our goal is to develop more communicative andinteractive behaviors, however, our experiment requires
mutual interaction between a human and a robot. In suchcase, the human must interact with and evaluate the robot
one by one. Thus, the human fatigue becomes a morecrusial problem of this study.
2.2 Human Machine Hybrid Evaluation
In this study, the problem of the human fatigue can not
be ignored. The longer the experiment is, the more boreda human gets. Therefore, we need to reduce the numberof genes and generations to minimize the human fatigue,
for these limitations make the EC’s learning capability
poorer and slower.
To solve this problem, we developed a human machine
hybrid evaluation (HMHE). The HMHE is an evaluation
method that reduces the human fatigue by decreacing
the number of mannually evaluated pheno-types. Using
the HMHE, the human only needs to test some individuals that are automatically selected from the genetic pool.
Then, the fitness values of the other genes are estimatedon the basis of the distances from the data of all selected
genes that the human gave his/her subjective fitness values before.
Our proposed algorithm is shown in Figure 1. First,
the IEC process generates an initial genetic pool at random. Next, the process analiyzes the newest genes using
the SOM algorithm [8].The SOM is an algorithm that issuitable for analyzing large multi-dimensional dataset. In
the SOM, similar dataset are placed nearby and dissimilar ones are placed further from one another. The SOM
is trained by the dataset of genes in the genetic pool (seeFig.2a), and then, seven genes that have the best matchingdataset with seven neurons placed at the position shownin Fig.2b, are selected. Thus, we can select the individuals that have distant dataset of genes from one another.
This is efficient for the HMHE because, if the selected
genes are similar to one another, the fitness values of theother genes can not be estimated.
Then the selected genes are translated into phenotypes (parameters of controller) and installed into a robot.
The robot interacts with a human and is evaluated byhim/her. This process continues until all the selected
genes are evaluated.
After that, the genes that weren’t selected are automatically evaluated on the basis of the fitness values ofthe selected genes. The fitness values are estimated asfollows:
Eiauto =n
k=1

αik Ekmaunal(1)

Newestgenetic pool
Self Organization
Map
The othergenes

Selectedgenes
MA Generator
MA network
WAMOEBA-3

Human Evaluation
EstimationSystem
Interaction
Human Subjective
Evaluation

Genetic operators(Selection)(Crossover)(Mutation)

If all genes areevaluated

No

Yes

Fig. 1 Proposed Algorithm
Self-Organizing Map

Genetic Pool

Genetic Pool

(a)

(b)

Fig. 2 The SOM algorithms in the HMHE: (a) Training
period. The SOM is trained with all genes. (b) Selecting period. All vertice of angles and the center of
the trained hexagon map are used to select the genes
to be evaluated mannually.
r manual − rk 
αik = 1 − n i manual− rk 
l=1 rl(2)Here, Eiauto is the fitness value of gene i, and Ekmanualis the fitness value of gene k evaluated manually. n is
the number of the genes evaluated by the user. αik is
the weight, which is calculated according to the distance
between genes i and k using equation (2). r is a datasetof a gene.
When all the genes evaluated both automatically andmanually, the process applies genetic operators (selection, crossover, and mutation) to the genetic pool andgenerates more appropriate genes.

3. EXPERIMENTAL SETTINGSIn this section, we show the settings of the experiment that we made in order to confirm the efficiency of
the HMHE. The experiment was carried out by using thesimulator. Experimental subjects controlled a simulated
robot and interacted with a robot controlled by a motion
generator.
We also carried out an experiment without using the
HMHE. In the plain IEC system (without the HMHE), thegenetic pool has only 7 individuals in the genetic pool, inorder to make the physical work load equal to the IEC
with HMHE. The subjects were 8 members of our laboratory staff. They have high knowledge of the computersand robots.
3.1 Interaction Simulator
We developed a simulator to investigate for the humanrobot interaction (see Figure 3). In this experiment, tworobots are placed in the virtual environment. The experimental environment is shown in Fig.3a. There were no
objects, obstacles, and textures that could influence subjects. There were only two robots. One was controlled by
the experimental subject, the other was controlled by the
program.

Fig. 4 WAMOEBA-3(a)(b)
Fig. 3 Interaction environment: The impression of
the environment can influence the human mind. We
didn’t prepare any textures or background images inthis experiment.
To control the simulated robot manually, subjects useda joystick with force-feedback function (Fig.3b). On thesimulator screen, the vision captured by the camera of
the human-controlled robot was displayed.The subjects
interacted freely with a program-controlled robot, and ifthey were satisfied, they evaluated the robot’s behaviors
by using the same joystick. The score of the robot wasfrom 0 to 1.
3.2 WAMOEBA-3
The robot which was displayed in the simulator was
WAMOEBA-3 which is a wheeled, independent robot
with inbuilt batteries and control systems (see Fig.4). It is656-mm long, 825-mm wide, 1316-mm tall, and weighsapproximately 105 kg. Its upperbody has two arms (6 degrees of freedom) and a head (4 degrees of freedom). Twoeyes can move independently, which is useful for communication [11], and its vehicle is an omni-directional
vehicle capable of moving in any directions without actually turning at any stage. It has microphones, ultrasonicrange sensors, two CCD cameras, bumper switches.
3.3 Motor-Agent Network
In this study, as a motion generator, we used a MotorAgent (MA) model we previously proposed [11]. In the
MA model, all actuators are regarded as autonomous
agents that determine their own actions autonomously
and independently according to the information collected
through the informational network.
Equation (3) shows the output decision of the actuators.
θ˙i= λi σi (xi ) − δi θi(3)

σi (xi )

xi

=

=



2
exp −γi (xi − c)2
− exp −γi (xi + c)j

ωji sj

(4)
(5)

m=0

Here θ˙i is the rotation velocity of the motor agent i. The
first term of quation (3) is the calculation result from sensor inputs, and the second term is for the stabilization. σi
is a sigmoid function defined by equation (4). Here, theγ affects the linearity of the function. If the γ is larger,
the linearity is smaller. x is the summation of the inputs
from the sensor network, defined by equation (5). ω is theconnection weight between the agents, and s is the sensor
input like the encoders or switches.
In this experiment, we prepared a very simple MA network, as shown in Figure 5. Agents with actuation capabilities included the eye and neck agents, and the omnidirectional vehicle agent. The eye agent determined its
rotation direction and speed autonomously according to
the angle of the neck joint and data from the ear sensors
and bump sensors. The neck agent determined its action
according to the angles of both eye joints and the actionof the omni-directional vehicle. The omni-directional vehicle determined its translational motion on the basis of
input from the bumper switches and ultrasonic range sensors, and its rotational motion on the basis of input fromthe angle of the neck joint as well as the input of the sensors. The image captured by the cameras were not in use
in this experiment.
The examples of the interactions between the humancontrolled robot and the program-controlled robot are as
follows:
1. When physical contact occurred between the humancontrolled robot and the program-controlled one in a simulated environment, the simulator generated force feedback effects for the subject. Then the bumper agents in-Eye Eye
Ear

Ear
Neck

Sonar

Bumper

Sonar

VehicleBumperFig. 5 Simple MA network: The actual MA network
is constructed with all of the possible connections.
However, in this study, we used very simple network
for simplicity.

fluenced the action of the vehicle agent, and the robot
moves.
2. When the human-controlled robot made sounds (whenthe subject pressed a button on the joystick), the ear sensors (microphones) captured the sound and the ear agents
influenced the eye motor agents.
3. When the ultrasonic range sensors of the programcontrolled robot captured the human-controlled robot or
the walls of environment, the sonar agents influenced thevehicle agents.
4. The program-controlled robot stopped moving whenits battery level was too low. The human-controlled robot
was equipped with an energy-charging gun which worked
by pressing a button on the joystick. If the humancontrolled robot with the gun was close enough to theprogram-controlled one, it could recharge its battery.
3.4 Genetic Settings
To evolve the robot’s controller, we must encode its
parameters into geno-type. In this experiment, the ω values in Equation 5 were encoded into the geno-type represented by numerical data, which are easy to analyze.
Each connection in the MA network has ω, and the number of components of dataset of a gene was 28.
We used the roulette wheel method and elitism where
the 40 % genes which had high fitness values were preserved.
The probability of mutation was 1%, and multipointcrossover was used. If a mutation occurred, the value ofa selected location was incremented by a small number
randomly extracted from a distribution centered aroundzero (biased mutation).
3.5 Questionnaire Surveys
During the experiments, we suveyed some questionnaires to confirm the subjects’ workload. The surveys
were carried out every three generations. The questionnaire items we used were developed by Haga et al.
[12]. Actually, the questionnaire was made to confirm
the workload of motormans, but the circumstance of thisexperiment is very similar to that of motormans’ (seeFig.3b), so we thought that it is applicable to confirm themental and physical workload of these experiments. The
questionnaire is made of 14 items;
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

You needed to concentrate.
You were satisfied with your work.
You did not have incentive to work.
You wanted to rest.
You felt heavy-eyed.
You did not have any consolable time.
You got bored.
You were nervous.
You did your best.
You could not concentrate.
You wanted to stop working.
You were almost asleep.
You were pressed for time.
You wanted to evade from work.

Experimental subjects score [0, 100] on each items.
After suveying them, we categorized them into sevensubdivisions, such as “difficulty”, “busyness”, “difficultyof concentration”, “boredom”, “sleepiness”, “tiredness”,
and “sense of accomplishment”. Each category has twoquestionnaire items and the sum of the scores of the twoitems was the total score of the category.

4. RESULTS
The behavior acquisition experiments took approx.
120 minutes on average. One evaluation of an individual took approx. 100 seconds.
The acquired behaviors could be categorized into two
types, one to evade from the subject, the other to get close
to him/her. The subjects preferred dynamic behaviors
that seemed to make sense.
Figure 6 shows the standard deviation of dataset of thegenes which were evaluated by the subjects. The verticalaxis indicates the standard deviation and the horizontal
axis indicates the generation. The line graph of ‘IEC’
represents the standard deviation of the prain IEC system, and the line graph of ‘IEC with HMHE’ represents
that of the IEC with HMHE. As shown in the graph, thestandard deviation of the genes in the plain IEC system
dropped. In the same way, that of IEC with HMHE also
dropped but kept higher level than that of the plain IEC.
This shows that diversity of the genes in the genetic pool
could be maintained at higher level, using the HMHE.
Figure 7 shows the evolution processes of experiments. During the experiments, the fitness values had
been increasing. However, that of the IEC with HMHE
had been higher than the plain IEC.
Figure 8 shows the result of the questionnaire surveys. We could observe the significant differences in‘tiredness’, ‘boredom’, and ‘difficulty of concentration’.
The IEC with HMHE showed higher performance thanthe prain IEC. We couldn’t observe the significant differences between them in the other categories.

160

160

*

140

*

140
120

120

sse 100
dne 80
ir 60
T

m
o 100
edr 80
o
B 60

40

40
20

20

0

0
0

3

6

0

9

3

(a)
140

no
tia 120
trn 100
ec
no 80
C
fo 60
ylt
uc 40
fifi 20
D

100
80
ltyu
icf 60
if
D 40
20
0

3

6

0

9

6

9

6

9

Generation

(c)

(d)

120

160

tn
e 140
hm
120
lisp
100
m
coc 80
A 60
fo
es 40
ne
S 20

100

ss 80
en
pie 60
el
S
40
20

0

0
0

3

6

0

9

3
Generation

Generation(e)

In such experiment, to keep the fitness values increasing is quite difficult, because, as the experiments were
continued, they had changed their criteria for their evaluation and interaction due to their boredom, and robots
that had received favorable evaluations in previous generations were evaluated differently. Using the HMHE,
though we could gain only the diversity of the genetic
pool, the fitness values increased more quickly. This was
becauase the genetic pool contained various individuals
to adapt to the changes in interactions and evaluations,even if the criteria was changed.
In the usual EC, if we want to gain the diversity of
the genetic pool, the mutation ratio is highly set a priori. However, high mutation ratio make the human impression worse because the IEC can not reflect the resultsof the human’s selection. On the other hand, since the
HMHE can increase the number of the genes, it can not
only gain the diversity but also reflect the assessor’s preference.
The effectiveness of the HMHE could also be confirmed in the questionnaire results. We could not observe
any significant differences in ‘difficulty of the work’,‘busyness’, and ‘sense of accomplishment’. Since the
HMHE can work only for the diversity of the genetic
pool, it was not concerned to the difficulty of the work.
On the contrary, the HMHE showed the advantagesin ‘boredom’, ‘tiredness’, and ‘difficulty of concentra-*

3

Generation

5. DISCUSSION

*

0

0

Fig. 7 Comparison of fitness value with or without the
HMHE: Opened circles and opened triangles respectively represent average and max fitness values of theIEC without HMHE. Closed circles and closed triangles represent average and max fitness values of theIEC with HMHE.

9

(b)

120

Fig. 6 Standard deviation of the displayed genes

6
Generation

Generation

(f)

90
80
70
sse 60
50
yns
u 40
B30
20
10
0
0

3

6

9

Generation

(g)
Fig.
8 Questionnaire results: Opened circles andclosed ones respectively represented the plain IEC
and IEC with HMHE. (a)Tiredness. (b)Boredom.
(c)Difficulty of work. (d)Difficulty of concentration. (e)Sleepiness. (f)Sense of accomplishment.
(g)Busyness.

tion’. If the communication robot can provide various
behaviors, it will prevent the humans from their boredom.
Therefore they could concentrate on the experiments.

6. CONCLUSION AND FUTURE WORKIn this paper, we proposed to use the IEC to acquirevarious behaviors of communication-robot. As the result,the IEC was capable of learning the behavior that the subjects preferred, though we did not prepare the criteria forinteractions and evaluations. The IEC with HMHE increased the number of individuals in the genetic pool, andeffectively overcame the difficulty of continue the experiment. Therefore, it is thought that to keep the varietyof genes in the genetic pool is very important in order to
evolve communicative behaviors.
We have already applied the IEC into the real robot,
WAMOEBA-3 (Fig.4) [15]. In the real world, the behaviors both of the robot and of humans are quite complicated. As the results, the fitness values were gradually
increased. However, to realize longer-term communication, it is thought that more dynamic behavior adaptation
is favourable. The Evolutionary Computation can also

provide some degrees of adaptability, but it is very slow.
So, the learning function that enable the robot itself toadapt to the user’s preference is required now. We are
now focusing on the reinforcement learning whose reward system evolves by the IEC.

ACKNOWLEDGEMENT
This research was supported in part by a Grant-in-Aid
for the WABOT-HOUSE Project by Gifu Prefecture.

REFERENCES[1]

Dautenhahn, K., Werry, I.: Towards interactive
robots in autism therapy; Pragmatics and Cognition
12(1), pp. 1-35.
[2] Pfeifer, R., Scheier, C.: Understanding Intelligence;
MIT Press (1999).
[3] Mitsunaga, N., Smith, C., Kanda, T., Ishiguro,
H., Hagita, N.: Robot Behavior Adaptation for
Human-Robot Interaction based on Policy Gradient Reinforcement Learning; Proc. of the 2005
IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS2005), pp.1594-1601, (2005).
[4] H. Takagi: Interactive Evolutionary Computation :Fusion of the Capabilities of EC Optimization and
Human Evaluation, in Proc. of the IEEE, Supecial
Issue on Industrial Innovations Using Soft Computing, Vol. 89, No. 9, September, 2001.
[5] R. Dawkins:
The Blind Watchmaker, Essex:Longman, 1986.
[6] J. A. Bildes: GenJam: A Genetic Algorithm for
Generating Jazz Solos, Proceedings of International
Computer Music Conference (ICMC94), pp. 131137, 1994.
[7] H. H. Lund, O. Miglino, L. Pagliarini, A. Billard,and A. IjspeertF Evolutionary Robotics - A Children’s Game, in IEEE Int’l Conf. on Evolutionary
Computation (ICEC ’98), pp.154-158 ,1998.
[8] T. Kohonen: Self-Organizing Maps, SpringerVerlag, Berlin Heidelberg, 1995.
[9] J. H. Holland: Adaptation in Natural and Artificial
System, The MIT Press, 1992.
[10] T. Ogata, and S. Sugano: Emotional Communication Between Humans and the Autonomous Robot
which has the Emotion Model, in Proc. of IEEE
Int. Conf. on Robotics and Automation (ICRA’99),
pp.3177-3182, 1999.
[11] T. Ogata, T. Komiya, K. Noda and S. Sugano: Influence of the Eye Motions in Human-Robot Communication and Motion Generation based on the
Robot Body Structure, in Proc. of IEEE/RAS International Conference on Humanoid Robots (Humanoid 2001), 00. 83-89, Nov. 2001.
[12] S. Haga: Mental Workload: Its Theory and Measurement, Japan publishing service, 2001.
[13] Nolfi, S., Floreano, D.: Evolutionary Robotics; MIT
Press (2000).
[14] Suga, Y., Ogata, T., Sugano, S.: Aquisition of Reactive Motion for Communication Robots Using Inter-

active EC; Proc. of IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS 2004), pp.11981203, Sept. 2004.
[15] Suga, Y., Ikuma, Y., Nagao, D., Ogata, T., Sugano, S.; Interactive Evolution of Human-Robot
Communication in Real World; Proc. of IEEE/RSJ
Int’l Conf. on Intelligent Robots and Systems(IROS2005), August, 2005