Proceedings of 2004 IEEE/RSJ International Conference onIntelligent Robots and Systems
September 28 - October 2, 2004, Sendai, Japan

Acquisition of Reactive Motion forCommunication Robots Using Interactive EC
Yuki SUGA , Tetsuya OGATA† and Shigeki SUGANO Humanoid Robotics Institute (HRI), Waseda University

3-4-1 Ohkubo Shinjuku-ku Tokyo 169-8555, Japan† Graduate School of Informatics, Kyoto University
Yoshida-Honmachi Sakyo-ku Kyoto 606-8501, Japan
E-mail: ysuga@suou.waseda.jp, ogata@i.kyoto-u.ac.jp, sugano@paradise.mech.waseda.ac.jp

Abstract— We’ve developed an emotional communication
robot, WAMOEBA, using behavior-based techniques. We also
proposed motor-agent (MA) model, which is an autonomous
distributed-control algorithm constructed of simple sensormotor coordination. Though it enables WAMOEBA to behave
in various ways, the weight of the combinations between
different motor agents is influenced by the preferences ofthe developer. We usually use machine-learning algorithms toautomatically configure these parameters for communication
robots. However, this makes it difficult to define the quantitative evaluation required for communication. We therefore
used the method of interactive evolutionary computation(IEC), which can be applied to problems involving quantitative evaluation. IEC does not require to define a fitness
function; this task is performed by users. But the biggest
problem with using IEC is human fatigue, which causes
insufficiency of individuals and generations for convergenceof EC. To fix this problem, we use the prediction function that
automatically calculates the fitness values of genes from somesamples that have received the human subjective evaluation.
Then we carried out the behavior acquisition experimentusing the IEC simulation system with the prediction function.
As the results of experiments, it is confirmed that diversifying
the genetic pool is an efficient way for generating a variety
of behavior.

condition [8]. Currently, we are developing WAMOEBA-3(see Fig.1) that is designed to behave in a greater variety
of ways than the WAMOEBA-2 series. It is a wheeltype independence robot with inbuilt batteries and control
systems. It is 656-mm long, 825-mm wide, 1316-mm tall,and weighs approximately 105 kg. It is equipped withan omni-directional vehicle that is capable of moving inany direction without actually turning at any stage. It has
two arms (6 degrees of freedom) and a head (4 degrees
of freedom). Its two eyes move independently, which isuseful for communication [9]. It has enough sensors of
sufficient sensitivity to enable it to behave in various ways.
The sensors used to acquire external information include
microphones, ultrasonic range sensors, two CCD cameras,
bumper switches, and so on. But two cameras are not usedin the current experiment shown in this paper.

I. I NTRODUCTION
In the 21st century, robots have potential uses in many
areas, especially entertainment, healthcare, and nursing.
Robots that work around human beings must be equippedwith capability for human-robot communication. Most ofthe ‘communication’ provided by robots is based on scenarios designed by their developers. Certainly, model-based
communication is at present the most practical and fastest
way of achieving verbal and context-sensitive communication, but it is boring for users once they are familiar withthe scenarios. Though robots may behave in various ways
to keep users interested, the number of their behaviors is
physically limited. As an autonomous generator of variety,emotion has attracted the attention of numerous researchers[1], but emotional models are still based on the subjective
preferences of developers.
To create a more interactive robot, we have developed
an emotional-communication robot WAMOEBA (Waseda
Artificial Mind On Emotional BAse) whose emotional
behaviors are based on evaluations of its own hardware

0-7803-8463-6/04/$20.00 ©2004 IEEE

Fig. 1.

WAMOEBA-3

We have also proposed a motion-generation algorithm
called motor-agent (MA) model. MA model is an autonomous distributed-control algorithm. Each motor is regarded as an autonomous agent that connects with neighboring agents and collects information from sensors and1198

other motors through a network in the robot. Each agent
then determines its action autonomously according to thecollected data. Though this algorithm enables WAMOEBA
to behave in various ways, the weights of the combinations
between agents are configured by the developer. Therefore,
its behavior is influenced by the developer’s preferences.
In this paper, we describe the method of using theevolutionary computation (EC) as a basis for installing
behavior in a communication robot to automatically configure the weights of the MA network. First, we pointout the problems for the human-robot communication
learning and focus attention on a problem of evaluating
the communication. As a prescription for it, we show
the interactive evolutionary computation (IEC). Next, we
describe some contrivances to acquire the variety of behaviors. In the communication learning system, there must be
a mutual informational and physical interaction. Therefore
we developed a simulator to investigate the interaction
between a robot and a human being. According to theresults of our experiment, we show that it is important
for the communication learning of IEC to diversify theindividuals in the genetic pool.

III. I NTERACTIVE E VOLUTIONARY C OMPUTATION
In conventional EC, each individual is evaluated using agiven fitness function. IEC is an evolutionary computation;
its fitness function is provided by users (see Fig. 2). It
therefore enables EC techniques to be applied to problemsof subjective optimization without an explicit model ofhuman subjective evaluation. IEC has previously been
applied to aesthetic areas, such as art, design, and music.
In these areas, fitness functions cannot always be defined
explicitly. With IEC, however, it is unnecessary to define
them.
Framework of EC• SELECTION• CROSSOVER• MUTATION

Human
Subjective
Evaluation

Human-Interface

Fig. 2.

II. L EARNING PROBLEMS FORCOMMUNICATION - ROBOTS
There are numerous learning algorithms, such as artificial neural networks (ANN), reinforcement learning, evolutionary computation (EC), and so on. We have to select themost appropriate algorithm for providing communication
robots with various behaviors. We decided on EC because
it is suitable for exploring a large search area, and cangenerate a number of possible solutions. An additional
advantage of using EC is that it does not require modelsof the system and the environment.
When EC is used to optimize the behavior of acommunication-robot, it is first necessary to define thefitness function. However, it is difficult to evaluate communicative behavior quantitatively, rather than qualitatively.
No matter which learning method is selected, it is stillnecessary to explicitly define a method for evaluating
communication.
Two methods have already been proposed to fix thisproblem. One involves using an evaluation model [2], andthe other is an interactive learning system. The former
method is based on building an evaluation model of ahuman being from trends shown by subjects. However,the phenomenon of communication is too difficult to build
an explicit model of subjective human evaluation because
there are mutual informational and physical interactions
between people in communicative situations. The lattermethod incorporates a human being into the learning system instead of an evaluation function. This approach avoids
the problem of complexity. Therefore, we decided to useinteractive evolutionary computation (IEC), which combines evolutionary computation and an interactive learning
system [5].

Gene Pool

Framework of IEC

IEC has also been applied to optimizing robot behavior.
For example, in a study entitled “A Children’s Game” [4],a robot controller was developed using an ANN; the connection weights in this ANN were developed according tochildren’s preferences for robot locomotion. By repeating
the selected locomotion on a simulator screen, IEC wasable to generate the behavior that the children preferred.
This study was aimed at developing a robot controller
without programming. IEC has also been used to optimize
the emotional expression of autonomous robots. Assessors
observed the different movements of autonomous robots
and chose the movements that seemingly mimicked theemotions, such as happiness, anger or sadness, that they
were looking for [10].
Both of these studies used IEC to advantage in optimizing robot behavior. Our aim however, is to develop morecommunicative and interactive behavior. Therefore, ourexperiment required physical and informational interaction
between human beings and robots.
IV. I NTRODUCTION OF IEC INTO H UMAN -ROBOTC OMMUNICATION
A. Motor-Agent Network
We used MA model as the motion-generation algorithm
for an robot. In this experiment, we used a very simple MA
network, as shown in Fig. 3. Agents with actuation capabilities include the eye agents, neck agents, and the omnidirectional vehicle agent. Each agent determines its own
actions autonomously and independently. The eye agent
determines its rotation direction and speed autonomously
according to the angle of the neck joint and data fromthe ear sensors. The neck agent determines its action
according to the angles of both eye joints and the action of1199

the omni-directional vehicle. The omni-directional vehicle
determines its translational motion on the basis of input
from the bumper switches and ultrasonic range sensors,and its rotational motion on the basis of input from theangle of the neck joint as well as the input of the sensors.

Eye Eye
Ear

Ear
Neck

Sonar

Sonar

BumperFig. 3.

VehicleBumperMotor-Agent network

The output of the motors is described in terms of angularvelocity, and movement of the whole body is generated byintegrating the reflection motions caused by sensor input.

θ̇

n

m

i 0

j 0

 ∑ αi θ  ∑ β js  δ θ(1)Here, α is a weight coefficient that shows the ratio fromwhich the reflection motion is selected. α is given by thefollowing function which changes according to an angle ofanother joint.
αi θ   γi  exp ωi θi cconst 2
γi  exp ωi θi  cconst 2(2)

θ is an angle of joint, and c const is a threshold. γ is thedegree of influence of θ , and ω is the degree of sensitivityof θ . If the value of ω is large, the change in the sensor
input influences α sensitively. β is also defined according
to the sensor signal.
βi s ∑ γi  si

C. Evaluation Prediction
The biggest problem with applying IEC is human fatigue. Since assessors cooperate with a computer to evaluate individuals, the IEC process spends huge length of time.
To minimize human fatigue, we need to reduce the numberof individuals and generations. This results in poorer andslower EC convergence.
To solve this problem, we used a method for predicting
the fitness values of genes [6].With this method, the assessor tests only some individuals that are selected from thegenetic pool. And then the system predicts fitness values
for the other genes. Fitness values of them are predictedon the basis of the Euclidian distance from the datasetsof genes that an assessor gives his/her subjective fitness
values before.
In usual prediction methods, the individuals evaluatedby an assessor are selected at random. However, as amore efficient method to select individuals, we used a selforganization map (SOM) [3]. The SOM is an algorithm thatis suitable for analyzing large multi-dimensional datasets.
In SOM, similar vectors are placed close to each other anddissimilar vectors are placed further from each other.
The individuals are selected using the SOM as follows:
first, the SOM is trained by the datasets of genes in thegenetic pool (see Fig.4(a)). The number of neurons of
SOM network is 37, and of training cycles is 1000. Next,
seven genes are selected; each selected gene has the best
matching dataset with each of seven neurons placed at theposition shown in Fig.4(b). In this way, we can select theindividuals that have distant datasets of genes from eachother, which is efficient for the prediction system. Using
this prediction system enables to increase the number ofindividuals in the genetic pool to 50.
The system also displays the best individual from theprevious generation, which makes sure the progression oflearning and eases an assessor’s mind.
Self-Organizing Map(3)i

s is the parameter changed according to the sensor inputs(e.g. the bumper switched).
Genetic Pool

B. Gene Expression
The γ values in Equation 2 and 3 are encoded to genes
represented by numerical data, which are easy to analyze.
Each connection in the motor-agent network has γ , and thedimension of datasets of a gene is 28. The probability ofmutation is 0.5% for each element in datasets of a gene.
If a mutation occurs in an element, it is updated by arandom number generator. If the parent genes are close
to each other, the probability of mutation increases to 50%
to ensure that the genetic pool contains a variety of genes.

(a)Training Period
Fig. 4.

Genetic Pool(b)Selecting Period

Positions of selected genes on SOM

Our proposed algorithm is shown in Figure 5. The
newest genes are analyzed using the SOM algorithm.
Then, selected genes are translated into individuals (motoragent network) and interact with assessors, who thenevaluat them. Genes that are not selected are automaticallyevaluated by the prediction system. When all the genes

1200

have been evaluated both automatically and manually, theEC applies genetic operators (selection, crossover, andmutation) to the gene pool and renewed the pool.
Newestgenetic pool
Self Organization
Map

Selectedgenes

The othergenes

MA Generator
MA network
WAMOEBA-3

Human Evaluation
Prediction System
Interaction
Human Subjective
Evaluation

Genetic operators(Selection)(Crossover)(Mutation)

If all genes areevaluated

No

Yes

Fig. 5.

Proposed Algorithm

the ear sensors (microphones) capture the sound andthe ear agents influence the eye motor agents.
3) When the ultrasonic range sensors of the programcontrolled robot capture the human-controlled robot
or the walls of environment, the sonar agents influence the vehicle agents.
4) The program-controlled robot stops moving whenits battery level is too low. The human-controlled
robot is equipped with an energy-charging gun whichworks by pressing a button on the joystick. If thehuman-controlled robot with the gun is close enoughto the program-controlled one, it can recharge its
battery also.
We used this experiment to examine the following aspectsof applying IEC to the acquisition of various behaviors bycommunication robots.
1) Interaction Task
2) Presentation of Individuals
3) Evaluation
4) Dealing with the disadvantages of IEC
B. Interaction Task

V. E XPERIMENT
A. Interaction Simulator
We developed a simulator of WAMOEBA-3 in preliminary experiments to investigate the interaction between
robots and human beings (see Fig. 6). In this paper, we
discuss the problems, and techniques required to apply IEC
to communication robots.

Usually, a communication has its own goal (task). Assessors can evaluate how the task was achieved. In thisexperiment, however, since we wanted to acquire a variety
of behaviors, we did not prepare communication tasks inadvance.

Fig. 7.
Fig. 6.

Interaction environment

Interaction simulator

In this experiment, two robots placed in a simulated
environment interacted with each other. One was controlled
by an assessor, and the other by a program (MA). To control the simulated robot, the assessor used a joystick withforce-feedback capability. This enabled bilateral physicalinteraction.
The main interactions between the human-controlled
robot and a program-controlled robot were as follows:
1) When physical contact occurs between a humancontrolled robot and a program-controlled one ina simulated environment, the simulator generates
force feedback effects for the assessor and sends
information on the collision to the bumper agentsof the MA. Then the bumper agents influence theaction of the vehicle agent.
2) When the human-controlled robot makes sounds(when the assessor presses a button on the joystick),

The environment is shown in Fig. 7. There are no
objects, obstacles, or textures that could influence the assessor. There are only the two robots. The assessors interact
freely with the program-controlled robot, and evaluate its
behavior.
C. Presentation of Individuals
The IEC is usually applied to aesthetic areas such as
design or art. In such cases, when an examinee evaluates
individuals in the genetic pool, the static images of individuals are represented on the CRT screen at the sametime. On the other hand, we use the IEC to acquire thebehavior of the communication-robot. The communication
is dynamic phenomenon that has duration of time, and itis spatially impossible to display. Therefore we have todisplay one by one. The examinee interacts with only one
individual. On the simulator screen, the vision captured by
the camera of the human-controlled robot is displayed, andan examinee evaluates the behavior of the robot.

1201

2.5

D. Evaluation
In normal IEC, several individuals are shown on thescreen at the same time, and the assessor then evaluates
individuals by selecting or ranking them. In this experiment, robot behavior was evaluated using a scoring method
because the assessors evaluated individuals one by one.
We did not supply any adjectives for the evaluation dialogue that took place at the end of each interaction because
this would have provided the assessors with evaluation
criteria that were influenced by the developer. Assessors
were simply informed of the abilities of the robot andallowed to evaluate the behavior freely.
E. Behavior Acquisition Experiment with IEC
First, we made the behavior acquisition experiment using
the IEC with the prediction function described in the section IV-C. It was carried out using the simulator describedin the section V-A. We also carried out an experiment
without using the prediction system. In the non-prediction
system, IEC has only 7 individuals in the genetic pool.
F. Reevaluation Experiment with Paired Comparisons
After the experiment in using the IEC with the prediction
function, we did a further experiment to check the successfulness of IEC learning. The experiment was carried out
as follows:
1) Five individuals who had experienced learning wereselected from the genetic pool.
2) The assessors evaluated the individuals using a paired
comparisons method.
3) Priorities were calculated from the results of thecomparisons.
Each assessor reevaluated the individuals in the genetic
pool which was trained by him/her self. The paired comparisons method is one of the most famous decision making
methods. The decision maker makes a comparison between
two samples from the alternatives, and then he/she rates thepriority of one of two. This process is carried out for eachof the possible combinations. Finally, the priorities of thealternatives are calculated from the results of comparisons.
The paired comparisons method is expected to produce
more accurate evaluations than the scoring method usedin the first experiment. However it increases the humanfatigue because the required number of the interactions andevaluations is as large as the number of the combinations
between individuals that is to be evaluated. Therefore thenumber of individuals was more severely restricted andgot down to five. The priorities were calculated from thegeometric average of the judgments made by the assessor.
VI. R ESULT
A. Behavior Acquisition Experiment
The behavior acquisition experiments took about 90
minutes on average. One evaluation of an individual took
about 90 seconds (The assessors were five members of ourlaboratory staff).
Figure 8 shows the standard deviation of dataset of thegenes. The vertical axis indicates the standard deviation

no 2
it
ai
ve 1.5
D
rda
dn 1
at
S0.5
0
0

Fig. 8.

1

2

3 4 5
Generation

6

7

8

9

10

Non-Prediction
Prediction

Standard deviation of the genes represented(evaluated)

and the horizontal axis indicates the generation. The line
graph of ‘non-prediction’ represents the standard deviationof the IEC system without the prediction function, and theline graph of ‘prediction’ represents that of the IEC systemwith the prediction function. The standard deviation of thegenes in the non-prediction system drops, but that of theprediction system remains high. This shows that variationin the genetic pool is maintained at a constant level withthe prediction system.
Fitness Value of IEC
without the prediction function

Fitness Value of IEC
with the prediction function

.6
0

6
.
0

.5
0
e
lu
a
V
ss
e
nt
i
F

.5
0

4.
0

e
u
al
V
ss
e
tn
i
F

.3
0
.2
0
1.
0

.3
0
.2
0
.1
0

0

0
0

1

2

3

4

5

6

Generation

Fig. 9.
function

.4
0

7

8

9

10

Maximum
Average

0

1

2

3

4

5

6

Generation

7

8

9

10

Maximum
Average

Comparison of fitness value with or without the prediction

Figure 9 shows an evolution process of experiments.
Circle plots represent maximum fitness values in the population of each generations, and square plots represent
population averages. In the early generations in the plotsof the non-prediction system, the fitness values tended toincrease, but from the fifth generation, the fitness value
tended to drop.
In contrast, the values in the prediction system remained
almost at an even level or increased.
B. Reevaluation Experiment
Figure 10 shows a graph of the priority values for individuals. The vertical axis indicates the priority value, andthe horizontal one represents the order in which selectedindividuals were generated. The line graph represents theaverage priority; the variance in priority is also shown.
In early generations, the priorities of individuals increased. In later generations, the average priority stayed
the same, but its variance increased.

1202

4

yit 3
ro
ir 2
P
1
0
1

2

3

4

5

Generation Order

Fig. 10.

Priority of individualsVII. D ISCUSSION
In the early generations, though we had not set a task forthe interaction or provided criteria for the evaluation, theassessors were able to interact with the robots and evaluate
them. The learning capability of IEC was demonstrated in
the reevaluation experiment. With the later generations, it
was difficult to maintain or increase the fitness values.
Values dropped because the assessors changed theircriteria for the evaluation and interaction due to theirboring. When the criteria were changed, an assessor observed different aspects of the robots, and robots that hadreceived favorable evaluations in previous generations were
evaluated more critically. The change in the criteria was a
result of the unstructured format of this experiment. We
did not provide standards for the evaluation or a purpose
for the interaction. In the reevaluation experiment, theincrement in the variance of priorities might have been
due to the changed criteria. In the later generations in the
behavior acquisition experiment, the assessors evaluatedthe robots differently. For example, in the early generations,an assessor evaluated individuals looking at the movementof whole body vaguely, and in the latter generations, he/she
paid attention to the head movement. This was why it
was difficult to duplicate the evaluation in the reevaluation
experiment.
However, using the prediction system we enabled thefitness values to be maintained at a constant level or to be
increased. This system was effective in maintaining variety
in the genetic pool. Even if the criteria was changed, thegenetic pool contained various individuals that could produce individuals that could adapt to changes in interactions
and evaluations. Therefore, even though the fitness values
dropped, the genetic pool was able to adapt to the changed
criteria and the fitness values recovered.
VIII. C ONCLUSION AND F UTURE WORKIn this paper, we showed the property of using IEC toacquire various behaviors of communication-robot. And weapplied IEC to configure the weights between agents in the
WAMOEBA-3 MA network. Then we discussed the conditions to diversify the generated individuals (behaviors).
Though we did not prepare the criteria for interactions andevaluations, IEC was capable of learning the behavior thatthe assessors preferred. In the latter half of the experiment,
though it was difficult to keep increasing the fitness values

because of changes to the criteria, the IEC prediction
system effectively overcame this difficulty and increased
the number of individuals in the genetic pool. This was why
it is important for acquisition of various communicative
behaviors to keep the variety of genetic pool.
We have two future works. First, to reduce humanfatigue, we will try to refine the system. For example,
to reduce assessors’ effort in having to mentally compare
current and past behavior, the system will enable assessors
to replay the behavior of elite individuals whenever theychoose.
Secondly, we will introduce greater diversity and complexity into the motion-generation function (MA) andemotional-behavior generator (endocrine system) we havealready proposed [8].
We also need to do further work on the techniques
required to apply IEC in the real world.
ACKNOWLEDGEMENT

This research was supported in part by a Grant-in-Aid
for the WABOT-HOUSE Project by Gifu Prefecture.
R EFERENCES[1] M. Fujita: Emotional Expressions of a Pet-type Robot, in Journalof Robotics Society in Japan, Vol.17, No.7, pp.947-951, 1999. (inJapanese)[2] T. Kanda, H. Ishiguro, M. Imai and T. Ono: Body Movement
Analysis of Human-Robot Interaction, Proceedings of International
Joint Conference on Artificial Intelligence (IJCAI2003), pp. 177182, 2003.
[3] T.Kohonen: Self-Organizing Maps, Springer-Verlag, Berlin Heidelberg, 1995.
[4] H. H. Lund, O. Miglino, L. Pagliarini, A. Billard, and A. IjspeertF
Evolutionary Robotics - A Children’s Game, in IEEE Int’l Conf. on
Evolutionary Computation (ICEC ’98), pp.154-158 ,1998.
[5] H. Takagi: Interactive Evolutionary Computation : Fusion of the
Capabilities of EC Optimization and Human Evaluation, in Proc.
of the IEEE, Supecial Issue on Industrial Innovations Using Soft
Computing, Vol. 89, No. 9, September, 2001.
[6] Y. Nakanishi: Control and Evaluation of Interactive Evolutionary
System with a Preference Function: Applying Genetic Programming
to Design Aid System, in Journal of Personal Technologies, Vol.4,
No.4, pp.221-224, 2000.
[7] R. Dawkins: The Blind Watchmaker, Essex:Longman, 1986.
[8] T. Ogata, and S. Sugano: Emotional Communication Between
Humans and the Autonomous Robot which has the Emotion Model,
in Proc. of IEEE Int. Conf. on Robotics and Automation (ICRA’99),
pp.3177-3182, 1999.
[9] T. Ogata, T. Komiya, K. Noda and S. Sugano: Influence of the Eye
Motions in Human-Robot Communication and Motion Generation
based on the Robot Body Structure, in Proc. of IEEE/RAS International Conference on Humanoid Robots (Humanoid 2001), 00.
83-89, Nov. 2001.
[10] Y. Katurada and Y. Maeda: Support system for automaticallyemotional generation using interactive EC, in 16th Fuzzy System
Symposium (Akita, Japan), pp. 297-300, September, 2000. (inJapanese).
[11] Y. Todoroki and H. Takagi: User Interface of an Interactive Evolutionary Computation for Speech Processing, in 6th Int. Conf. on
Soft Computing (IIZUKA2000), pp.112-118, October, 2000.
[12] T. Shimokawa and T. Sawaragi: Acquiring Communicative Motor
Acts of Social Robot Using Interactive Evolutionary Computation,
in Proc. of 2001 IEEE SMC Conference, Tucson, pp.1396-1401,
2001.
[13] T. Saaty: Multicreteria Decision Making: The Analytic Hierarchy
Process, RWS Publications, 1996.

1203