Human-Adaptive Robot Interaction Using IEC

Paper: Rb20-4-3498:2008/7/31

Human-Adaptive Robot Interaction
Using Interactive EC with Human-Machine Hybrid Evaluation
Yuki Suga , Tetsuya Ogata , and Shigeki SuganoDepartment of Creative Science and Engineering, School of Modern Mechanical Engineering, Waseda University#59-325, 3-4-1 Okubo Shinjuku, Tokyo 169-8555, Japan
E-mail: ysuga@suou.waseda.jp
 Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University
Yoshida-honmachi, Sakyo-ku, Kyoto 606-8501, Japan[Received February 19, 2008; accepted June 5, 2008]

Using interactive evolutionary computation (IEC), we
created human-robot interaction system that maintains user interest over time. Although IEC enables
users to design systems reflecting subjective preferences, it forces them to evaluate a large number ofindividuals. The refined IEC techniques, we propose in this regard, human-machine hybrid evaluation(HMHE), lets users manually evaluate only representative genes, after which HMHE automatically estimates the fitness of other genes, thereby increasing a
population without increasing user evaluation process.
Experimental results showed that preferences easilychange in interaction. We confirmed that HMHE
maintains high diversity, while maintaining user interest.
Keywords: interactive evolutionary computation, humanrobot interaction

1. Preface
The increasing use of robotics in fields such as entertainment, medicine, nursing care, and psychotherapy,
makes it necessary for such symbiotic robots to be able
to communicate. Specifically, in interaction that requires
maintaining user interest, particularly with entertainment
robots, such robots must be able to automatically learn
and adapt their behaviors to user preferences. In humanrobot interaction learning algorithms such as neural networks are not suited for communication depending primarily on linguistic contexts, so it may be more practical to apply such learning algorithms to reactive communications. Introducing behavior-generating algorithms insearches, for example, could help keep users from gettingbored by adapting to individual user preferences and continually changing robot behavior.
In attempting to realize such adaptive interaction, we
focused on three issues:

To address these issues, we used interactive evolutionary computation (IEC) in learning for communication robots, enabling them to learn directly from subjective user evaluation represented by genetic algorithms inwhich the fitness function is replaced by subjective humanevaluation [1].
IEC requires deliberate user evaluation in learning, so
gene pool populations cannot be increased too much,meaning that when IEC is applied to learning by communication robots, users must interact with all geneticphenotype robot and provide them with evaluations. If a
population is too large, it will overly burden users, but if
it is too small, genetic data within a gene pool cannot bemade diverse enough, preventing progress in learning.
As a trade-off among gene pool diversity, learning efficiency, and the burden on users, we propose humanmachine hybrid evaluation (HMHE), which uses automatic evaluation referencing user evaluation in parallelwith IEC and enabling population size to grow without
adding to user fatigue.
In this paper, Section 2 details the three issues we arefocusing on problems - diversity of interaction, changes
over time in subjective preferences, and evaluation of interactions, and features in applying IEC to human-robot
interaction. Section 3 discusses HMHE including an
HMHE performances test using a mathematical model
conducted prior to interactive experiments with users.
Section 4 tells how to install HMHE in the interaction
system. Section 5 describes experiment setup. Section
6 discusses experimental result. Section 7 mentions considerations. Section 8 Presents conclusions and projectedwork.

2. Problems in Communication Learning
Problems in learning with human-robot interaction involve diversity of interaction, changes over time in subjective preferences, and evaluation of interactions.

¯ Diversity of interaction¯ Changes over time in subjective preferences¯ Evaluation of interactions

Journal of Robotics and Mechatronics Vol.20 No.4, 2008

2.1. Problems Related to Diversity of Interaction
Maintaining the diversity of robot behavior to realizeinteraction keeping users interested requires multiobjec1

Suga, Y., Ogata, T., and Sugano, S

tive optimization of learning through dynamic selectionof a solution among candidates rather than single-solution
searching, since multiple types of behavior coincide withuser preferences. In multiobjective optimization with ordinary GAs, gene pool clustering is first worked out, thenparallel searching conducted for individual fitness functions. Since we do not know, however, how many evaluation items each fitness function consists of, it may be difficult to optimize GAs by changing evaluations for genepools [2].
Regarding learning through dialogues with users, Inamura et al. used a Bayesian network in searching toadapt robot behavior to user instructions [3]. Althoughtechniques using user instructions may be effective to
avoid uncertainties in the real world, such a Bayesianbased technique requires behavior units designed prior
to setting network nodes. For the technique we propose,robot behavior is described in terms of weight, etc., rather
than behavior units, because behavior units, not yet differentiated, can continue to change variously by learning
through interaction.

2.2. Changes in User Preference over Time
Regarding changes in user preference over time,changes in the evaluation axis caused by user boredom
present a problem in learning in open-ended human-robot
interactions in this study. If an interaction is long, interactions previously highly evaluated may be reevaluatedlower as the evaluation axis changes over time, meaningthat robots must learn continuously.
To cope with a dynamically changing fitness function, conventional GAs propose conducting a search while
maintaining diversity or temporarily increasing a mutation ratio by detecting changes in fitness functions [4, 5].
HMHE as we propose it maintains diversity and enables
well more effective adaptation by manipulating mutation
ratios upon detecting worsening evaluations.
Ogata et al. used consolidated learning in interaction using recurrent neural networks (RNNs) [6]. Consolidated learning achieves adaptation without sacrificing currently acquired dynamics by learning instructor
data together with current acquired dynamics as instructor data. Ogata et al. applied this to a navigation task inwhich humans and robots interacted and confirmed thatsuch navigation learning was more than mere learning andwas, in fact, highly varied creative process that generated
mutual interactions between learning while maintaining
consistency in output behavior, it may yet be difficult toapply RNNs to interactions with a high degree of freedom(DOF). The evolutionary technique we used is applicable to interactions with a high DOF even though searching time increase. Conservation of genes in a gene poolmakes it possible to ensure some consistency in behavior.

behavior success or failure if learning is through behavior
intensification or restraint, regardless of the learning algorithm used. In practice, it is difficult to determine whether
a robot has succeeded or failed autonomously with user
preference on its own account. Since user preference depends strongly on user subjectivity and on context information such as the relationship to the robot, the environment involved and past behavior, it is difficult to constructa model that quantitatively evaluates interactions regardless of circumstances.
Mitsunaga et al. used the perspective of personal spaceand optimized behavior parameters through intensified
learning with human-robot distance and meeting of linesof sight set as the evaluation axis [7]. When psychological knowledge is relied on, learning is conducted based ona criterion of interaction success or failure based in turnon the user’s unconscious feelings of pleasure or displeasure. Some users interacted completely differently, however, from how researchers expected, so a learning modelwith a much wider scope of application is needed.

2.4. IEC Application to Communication Robots

2.3. Problems Related to Interaction Evaluation
Regarding how to evaluate success or failure of interactions in adaptation to user preferences, whatever learning algorithm might be used, it is necessary to evaluate

To solve our three issues, we propose applying IEC, as
has been proposed in response to subjective human evaluation or individual differences, in fields such as computer
graphics (CG), fashion design, and music [1, 8]. IEC directly using subjective user evaluation is also expected toaddress individual user differences, dynamic changes in
evaluation indexes, etc. The fact that IEC generates different genes gives robots variety in their reactions to user
behavior, which in turn lets users notice new interactions,leading to variations in interaction. Being able to retain avariety of genes at one time should enable robots to adapt
to changing user subjectivity without disrupting learningand to acquire “intellectual” behavior with both of diversity and rationality, thus reducing user boredom.
IEC has a variety of advantages in optimizing robot behavior involving human interaction. Lund et al. proposed
applying IEC to optimizing the controller for a dolly robot
using neural networks [9], presenting plural obstacles andone robot in nine simulator domains so that users could
select robots that would undertake behavior preferable tosubjects. Their experiments confirmed that promotion of
IEC with such selected individual robots as parents for
subsequent generations could produce children with noprogramming capability who undertook desired behavior. Although this study coincides objectively with ours
in robot adapting themselves to psychological user preferences, simultaneously presenting nine robots makes it
difficult to introduce interactions with users.
We targeted one-to-one interaction between a subject
and a robot, enabling robots complete with IEC to receive evaluations through interactions with users and toproceed with long-term interactions with users, responding to changing demands without boring subjects.
IEC involving one-to-one interaction is required to individually evaluate robots with different genes, which istiring to users. The population size in a gene pool must

2

Journal of Robotics and Mechatronics Vol.20 No.4, 2008

Human-Adaptive Robot Interaction Using IEC

thus be limited and the time required for users to evaluate
robot behavior must be reduced. Populations that are too
small, however, tend to cause early convergence, lower
learning efficiency, deprive robots of diversified behavior,and, as a result, increase user boredom.
These issues could be dealt with using large mutation
ratios to maintain diversity within a gene pool, but suchratios are the result of random searches, so mutation ratiosthat are too large cannot fruitfully reflect user intentions.
Applying IEC to human-robot interaction thus involves
a significant trade-off in learning efficiency, burden on theuser, and gene diversity.

3. Proposal

Step 1

Newestgenetic pool

Process
Step 2 Selection
using SOM
The othergenes

Translator
Pheno-Type

Step 3

Human Evaluation

Step 4 Estimation System

Interaction
Human Subjective
Evaluation

Step 5

Genetic operators(Selection)(Crossover)(Mutation)

If all genes areevaluated

No

Yes

Fig. 1. Proposed Algorithm.

We start by detailing our proposed HMHE and thendiscuss HMHE performance tests using a mathematics
model.

3.1. HMHE
To increase total population size without increasing the
population size for user evaluation and fatiguing users, wedeveloped human-machine hybrid evaluation (HMHE)
with IEC in which users only evaluate the phenotype ofrepresentative genes in a gene pool, and evaluation ofother genes is automatically calculated based on user evaluation of preselected representative genes. This enlarges
the population size without burdening users.
Nagao et al., in a proposal lightening the burden on
users in IEC, made genetic evaluations based on the distance from facial images of user choices stored in abuffer [10]. We targeted open-ended interactions and theneed to respond to ever-changing subjectivity. As user
subjectivity changes, previous evaluation results may prevent current interactions from being evaluated correctly,
so we decided not to use records of evaluations but, instead, to use the evaluations of representative genes selected from current gene pools.
Our proposed algorithm is divided into five steps(Fig. 1).
Step 1 Generation of Initial Individual Genes
As in ordinary EC, initial genetic groups are generated
at random (Step 1, Fig. 1). Information on each gene isexpressed in real numbers.

Selected
genes

Self-Organizing Map

Genetic Pool(a) Learning Period

Genetic Pool(b) Selecting Period

Fig. 2. Selection of Representative Genes Using an SOM.

layer of the SOM and thus self-organizing [11]. Despite
its simple algorithm, the SOM analyzes multidimensional
data and is visualized to help adjust parameters used foranalysis, which is why we used the SOM to analyze genes
in this algorithm.
Figure 2 shows the selection of representative genes
using the SOM. In HMHE, SOM learning is first conducted for all genetic data in a gene pool. Element dataon the recognition layer of the SOM is self-organized andnear data arranged at a near location and distant data at a
distant location. Elements at plural distant points on the
SOM are selected and genes with data closest to such selected elements are selected as representative genes. The
selection of elements on the SOM for selection of representative genes may vary with the SOM used, but visualization enables developers to make interactive adjustments and facilitates design. Phenotypes (robot behavior)presented to users are diversified to avoid “boredom.”

Step 2 Selection of representative Genes
Representative genes are selected in gene pools out of
the genetic groups generated (Step 2, Fig. 1). Any random
selection of representative genes may result in the selection of genes with very similar data, lack of availability forproperly evaluating other genes, and possible induction ofboredom in users. We used a self-organizing map (SOM)
as a tool to analyze genetic data at selection. The SOM
analyzes multidimensional data. Numerous elements arranged in the SOM correspond to data on the same dimension with sample data – in this case, vectors of genes– distant data is arranged at a distant location and data
at a short distance, at a near location, on the recognition

Step 4 Automatic Evaluation Estimation
After all selected genes are evaluated, the fitness ofother genes is automatically estimated (Step 4, Fig. 1). In

Journal of Robotics and Mechatronics Vol.20 No.4, 2008

3

Step 3 Interaction
Only genes selected above are translated as a behaviorgenerating algorithm representing the phenotype and introduced to a robot. Users interact with robots and maketheir own subjective evaluations (Step 3, Fig. 1).

Suga, Y., Ogata, T., and Sugano, S

automatic evaluation, the fitness of representative genes
obtained by manual evaluation is weighted based on the
distance between data on genes evaluated and data on representative genes. Evaluation expressions are as follows:n∑

Eiauto

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
400

400
200

200

αik Ekmaunal. . . . . . . . . . (1)

-400

0
-200

0

-400

-200
200

400

0
-200

-400

0

-200
200

400

-400

k 1

1

 . . . . . . . (2)r i r manualknmanual∑ j 1 r i r j

where Eiauto , a calculated evaluation value of gene i, theweighted sum of Ekmanual , evaluation values of representative genes as obtained by users, and n indicates the number of individual genes evaluated by users. Weight α ik ,the ratio of distance between evaluated gene i and representative gene k to the sum of distances between evaluated gene i and all representative genes, is calculated using Eq. (2), where r denotes data on individual genes: rr i
refers to evaluated gene i and rrmanualand r manual
to reprejksentative genes k and j.
Step 5 Generation of New Gene Pools
Once evaluation is completed, next-generation genegroups are generated by applying genetic operators suchas selection, crossover, and mutation (Step 5, Fig. 1).
In HMHE, where the majority of evaluation processes
can be programmed and population size enlarged without
fatiguing users so as to ensure diversity in a gene pool,user boredom is avoided by increasing diversity in robot
behavior.

3.2. Performance Tests
To determine searching capabilities of the IEC using
HMHE in application to human-robot interaction, we conducted performance tests using a mathematics model, detailed below.
3.2.1. Tests
We used a mathematics model in the performance testof HMHE for simplification. Mathematics modeling of
human preferences is not really possible. Noting that several behaviors, not just one behavior, of robots coincidewith user preferences, we used the polymodal Schwefel
function, together with a single-modal sphere function, tocompare searching efficiency.
fsphere x n∑xi 2. . . . . . . . . . (3)Schwefel in n dimensions is a polymodal function expressed as follows:
fschwe f el x n∑ sin xi  . . . . . . . . (4)where x denotes an n-dimensional vector and x i the ith element of vector x (Fig. 3). The ordinate is inverted throughdivision by 1 after being normalized to [0, 1] to establish
uniform criteria for subsequent tests so that the higher theevaluation, the higher the degree of adaptability to the environment.
4(a) Sphere Function(b) Schwefel Function

Fig. 3. Surf Plots of Functions used in Tests.

0.96
fitness value

αik

0.95
0.94
0.93
0.92
0.91
0.9

IEC (size=8)
IEC (size=100)
HMHE (size=100)
020

40 60 80 100
generation
Fig. 4. Results of Sphere Function (Two-Dimensional).

We tested these two functions when HMHE is used
(population size: 100) and not used (population size: 8
and 100). For the population size in a gene pool, HMHE
would allow unlimited use of all population size in any
test with HMHE including this performance test, but toolarge a population would take too long for calculation in
SOM learning and further induce user fatigue. Since ourpreliminary test showed that HMHE performance could
be evaluated even for a 100-population size, we used a
population of 100 for EC using HMHE and also for EC
not using HMHE for comparison. In this performance
test and subsequent test evaluation with HMHE on a population size of eight (8), evaluations without HMHE weredone on the same population size for comparison. Sinceboth functions were two-dimensional, the gene type wasexpressed as a sequence of real values, and phenotype,also by two dimensional vectors (x 1 , x2 ).
Next-generation genetic groups were generated basedon elite conservation strategy: 40% of genes survive
into the next generation and the remaining 60 % is generated from parent genes selected from the surviving
40 % through roulette by fitness. Searching was conducted without using crossover by applying mutation toa gene locus (array element) within the chromosome. For
HMHE, a hexagon map with sides of four elements (totalelements: 37) was used for the SOM. With the distance
between genes being Euclidean, eight genes consisting ofgenes near seven points, i.e., the center of the map and six
vertices of the hexagon, and a representative gene of theprevious generation were selected as representative.
3.2.2. Results
Figure 4 shows the transition of fitness values in a
sphere function (abscissa: generation; ordinate: fitness),
Journal of Robotics and Mechatronics Vol.20 No.4, 2008

Human-Adaptive Robot Interaction Using IEC

fitness value

1
0.99
0.98
0.97
0.96
0.95
0.94
0.93
IEC (size=8)0.92
IEC (size=100)
0.91 HMHE (size=100)0.9
020 40 60 80 100
generation
Fig. 5. Results of Schwefel Function (Two-Dimensional).
Fig. 7. WAMOEBA-3.
600

600

400

400

200

x1
-600

-400

200

0
-200
0
-200
-400

x1
200

400

600

-600

Unselected
Selected

0
-400

-200
0
-200

200

400

600

-400

-600

-600

x2

x2

(a) IEC (size = 8)(b) IEC+HMHE (size = 100)
Fig. 6. Genetic Data by Two-Dimensional Schwefel Function.

representing average maximum fitness in ten trials undereach condition. For the sphere function, HMHE was inferior in performance in the initial search for IEC (size =100) and IEC (size = 8).
For the polymodal Schwefel function, HMHE showed
performance equivalent to IEC with size = 8, although far
inferior to IEC with size =100 (Fig. 5).
Figure 6 shows genetic data of generation 100 th obtained by optimizing genes as two-dimensional real number vectors using the Schwefel function ( abscissa: genetic data x1 ; ordinate: genetic data x 2 ), and round andtriangular plots in Fig. 6(b), HMHE-selected and unselected genes. This confirms that increased population sizein using HMHE extends the search range and that gene
selection by the SOM presents to users genes distant fromeach other (Fig. 6(b)).

For the polymodal Schwefel function, IEC using HMHE
closely matches conventional IEC in performance. Using
HMHE enables much wider searching than conventional
IEC and ensures dispersion of genes presented to users.
While IEC using HMHE ensures diversity in solutions
and a very wide search range, uncertainties in the estimation of fitness for unselected genes may lead to poorer
learning convergence. In interaction with users, however,learning convergence will not depend on evaluation axis
transition, so it is more important to maintain the diversity of genes that will enable continuous learning during
interaction.

4. Mounting in Interaction System
We implemented our proposed HMHE into interaction
system using a simulator to validate the effectiveness of
our proposal in user-robot interactions. For experiments,
we used a simulator to verify behavior of the Waseda Artificial Mind on an Emotion Base (WAMOEBA-3), arranging two robots in virtual space inside the simulator and auser manipulating one of the two robots interacting withthe other robot complete with a behavior-generation algorithm.

3.2.3. Discussion
Test results show that HMHE is inferior in performance
to an ordinary IEC with the same population size. WhenIEC is applied to human-robot interaction, however, whatmatters is search capabilities in limited evaluation time inview of user fatigue. For us, it is therefore more important
to compare the case of the same number of evaluations
rather than that of the same population size.
A comparison of the two cases under the same condition, i.e., HMHE and conventional IEC with population size = 8, shows that conventional IEC is superior
to HMHE in the single-modal sphere function. Ordinary
IEC increases fitness monotonously, but HMHE uses representative genes for evaluation to calculate fitness basedon similarity to representative genes and cannot searchingunless representative genes are extreme.
Behaviors that users prefer in interaction between users
and robots is so diversified that it presents polymodality.

4.1. WAMOEBA-3
We developed WAMOEBA-3 (Fig. 7), a platform for experiments in human-robot communications.
WAMOEBA-3 is 1316 mm high, 825 mm wide, and
656 mm deep, i.e., about the height of an elementary
school student. We consider it advantageous in communication for a robot to be humanoid, so WAMOEBA-3
has two arms with 6 DOF and a head with 8 DOF, plus
several sensors. The head has two CCD color cameras,
two microphones, and one speaker. Each camera and auricles (ears) turn left and right independently and represent a direction of attention in each mode. We used anomnidirectional vehicle to move WAMOEBA-3, enablingsmall slewing radii and omnidirectional movements to enhance safety in interaction with users. The vehicle has ultrasonic sensors, contact switching sensors, etc., enabling
the robot to move, measure distances to users, and evadewalls and other obstacles.

Journal of Robotics and Mechatronics Vol.20 No.4, 2008

5

Suga, Y., Ogata, T., and Sugano, S

Eye

Eye
Ear

Neck

Sonar

Ear

Sonar
Vehicle(a) Subject’s View

Bumper(b) Environment

BumperFig. 8. Simulator and Environments in Virtual Space.

Fig. 9. Motor Agent Network for Experiments.

4.2. Interaction Simulator
In experiments, we used a simulator to verify
WAMOEBA-3 activity (Fig. 8). Inside the simulator,
there were no solid textures or obstacles that would psychologically affect users, apart from the two robots. One
robot has a behavior-generation algorithm and users manipulate the other robot using a joystick with force feedback to interact in virtual space. Users can get the robot tomove around or turn its head by manipulating the joystick,and can have the robot speak by pushing a button. Potential contact with a wall or robot is presented via force
feedback.

tion from sensors (Fig. 9).
Below are examples of interaction in the motor agent
network.

4.3. Behavior Generation Algorithm
The behavior generation algorithm for WAMOEBA3 used motor agents, an autonomous distributed behavior generation algorithm[12]. This algorithm generates
reflexive behavior in a communication robot such as
WAMOEBA-3. Each motor in the motor agent determines its own direction of turning and velocity, based on
the weighted sum of sensor information collected fromthe network.
Equation (5) determines the output of each motor:¯ Following the movement of the camera and head, theλi σi xi σ x  exp γ x c

θ̇i

i



i

δi θi . . . . . . . . .

i

i

2

exp

γi xi  c2



(5)

. .

(6)

∑ ω jis j . . . . . . . . . . .

(7)

j

xi

m 0

where θ̇i indicates the output angular velocity of the i th
motor agent, the first member commands calculated basedon information from the sensor network, δ i in the second
member a constant for stabilization, σ i a sigmoid function as defined by Eq. (6) having two gentle steps wherea negative value is given to x  0 and a positive value tox  0, γi is the more steps a function has and the wider
the dead zone near x 0; x i is the sum of values obtainedby multiplying sensor network information by a coupling
coefficient as determined by Eq. (7), ω ji coupling strengthwith the ith motor agent and the j th sensor, and s j indicatesnormalized input of the j th sensor such as an encoder orswitch.
Motor agents must define a network of motors and sensors connected by weighted coupling to collect informa6¯ Touching touch sensors around the vehicle turns therobot toward an object or a user, or turns it inversely
while making the vehicle move back and forth or left
and right. The camera also turns in each direction.
¯ If a user speaks, the camera and head move.
¯ A user standing near the robot triggers the ultrasonicsensor and makes the vehicle move back and forth orleft and right.
vehicle turns in the same or opposite direction.

4.4. Genetic Description and Evolutionary Computation Parameters
To optimize a controller consisting of the above motor agents by IEC, ω in Eq. (7) corresponding to the network’s coupling strength was input to gene, while parameters such as γ , functional-linearity values, were madeconstant. Each parameter was described using real numbers, and genes were arrayed using real values with 27
dimensions consisting of coupling strength of 27 connections: from a microphone to a camera, from a camera to
the head for turning, from a camera to the head for updown movement, 12 connections between four ultrasonicsensors and three vehicle movements, and 12 connections
between four touch sensors and three vehicle movements.
Experiments used crossover realized by random exchanges of array elements so that robot behavior parameters on genes, concentrated in the head, vehicle, etc.,
would be propagated all together within each group. If
mutation occurred whose ratio was set at 1 % for each array element, random values were added. Of parent genes
selected by roulette, individuals having 40 % epistatic
genes are inherited by the next generation in what is called
elitism.
4.5. HMHE Setup
The number of individuals was set at 50 so that learning
time taken by the SOM during experiments was about 10
seconds in view of preliminary experiment results.
For the SOM, for which quadrangular and hexagonalneighborhood functions are generally used, we used the
Journal of Robotics and Mechatronics Vol.20 No.4, 2008

Human-Adaptive Robot Interaction Using IEC

hexagonal neighborhood function setting the number ofneurons on the cognitive layer to 37 (4 elements on one
side of a hexagonal map) based on results of preliminary
experiments on learning time and convergence on the cognitive layer.
We used a Euclidean distance to define the distance between genes used for learning by the SOM and estimationof fitness values.

5. Evaluation Experiments
Using the interaction system above, we conducted evaluation experiments among users, who were nine of ouruniversity students. In experiments, which are based on(1) conventional IEC (population size = 8) alone and (2)IEC using HMHE (population size = 50, 8 evaluations),
10 attempts at evaluation were conducted from generation 0 of an initially generated gene pool to generation 9.
Since sequential effects cannot be ignored in these experiments, the user sequence was changed for each experimental condition. Once experiments were completed, we
conducted a t-test for each sequential condition with nosignificant difference between the two condition.
We also surveyed users during and after experiments todetermine their work loads, as detailed in the sections that
follow.

5.1. Experimental Setup
Information given to users before experiments was asfollows:
1. The robot has a microphone fitted in its ears and canreact to sound.
2. The counterpart robot does not use a camera.
3. The robot’s arms do not move.
4. Ultrasonic sensors on the robot recognize distance toobjects or walls.

Table 1. Survey Questions (after Haga et al.).

Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14

Details
Concentration was needed.
Work was done as desired.
Motivation was lacking.
I wanted to take a break.
I became sleepy.
I was too busy to rest.
The work was boring.
I felt tense.
I worked very hard.
I could not concentrate on the work.
I wanted to stop the work before it was finished.
I felt like going to sleep.
I was pressed for time.
I wanted to get away from the work.

given the burden on users determining priorities. Users
were allowed to input evaluation values at any timing.

5.2. Questionnaire
To determine user work loads, we surveyed evaluations
for every three generations after completion of evaluationof generation 0. For questions, we used the burden survey
proposed by Haga et al. [13] (Table 1).
Users gave each question (14 in total) to 5 points. The
14 items were classified into the following seven pairs:
items 1 and 8 (difficulty); items 6 and 13 (busyness);
items 3 and 10 (difficulty in concentration); items 7 and
14 (boredom); items 5 and 12 (sleepiness); items 4 and
11 (tiredness); and items 2 and 9 (accomplishment). The
points given to each pair yielded a score for each category.
In these experiments, we used the above burden survey,
initially developed to measure the work load of a train
operator, because it presented an environment similar tosimulator operation.

5. Switches on the vehicle detect contact.
6. Score the robot presenting interesting behavior outof a total of 100 points (perfect score).
Users then did a trial experiment for one generation toensure that they understood the experimental flow, andpracticed operating the simulator. After a break, users
conducted experiments.
When we apply IEC to a survey on impressions madeby robots, questions should be prioritized based on a single fitness value based in turn on a weighted sum, but it
is difficult given individual differences among users, toset priorities in advance. Based on the task load index(NASA-TLX), users should determine their own priorities of items such as mental demand, physical demand,time demand, self-performance, effort, and frustration. In
this study, involving a large number of experiments, however, evaluations were based on a single evaluation axis
Journal of Robotics and Mechatronics Vol.20 No.4, 2008

6. Experimental Results
Evaluation by a single user took about 100 seconds andone user spent about two hours in evaluation.

6.1. Fitness Value Transition
Fitness values by IEC alone tended to stagnate in
growth in generation 5 and after, while HMHE-combined
IEC maintained growth, meaning combined use of
HMHE in IEC showed higher fitness than conventional
IEC later on in experiments (Fig. 10).
6.2. Diversity in Interaction
The interaction system we used in experiments dealt
with the following interactions: a user walking around arobot observes whether the counterpart robot turns toward7

Suga, Y., Ogata, T., and Sugano, S

A

80

Fitness

70
60

B

50
40

6th generation

30
0

1

2

3

4 5 6
Generation

HMHE (AVG)
HMHE (MAX)(P  0 05

(

7

8

9

IEC (AVG)
IEC (MAX)P  0 01 )Standard Deviation

Fig. 10. Fitness Value Transition.
1.2

IEC
HMHE

1
0.8
0.6
0.4
1

2

3 4 5 6
Generation

7

8

9

Fig. 11. Standard Deviation of Genetic Data Presented to Users.

the user; the user observes robot reactions to the user’s
speech; the user observes how the robot reacts when theuser touches the robot.

6.3. Maintenance of Gene Diversity by HMHE
We analyzed the maintenance of gene diversity by
HMHE (Fig. 11). Standard deviation (S D ) was calculatedusing the following equation:

∑ni 0 xinx̄i 2. . . . . . . . (8)where xi indicates the ith element of vector x , part of genetic data, and x̄ i the average of all x i for all genes in thegene pool.
Compared to IEC alone, we confirmed that use of
HMHE with IEC ensured that genetic data remained dispersed during experiments.

6.4. Changes in User Evaluation Axes
In an example of robot movement output by the simulator (Fig. 12), robot A is manipulated by the user androbot B is programmed. Two types of movement indicate
interactions for individuals with the same gene in different generations. For generation 6, the user observed robot
behavior at a constant distance from the robots approaching the user. For generation 7, the user walked aroundrobots from the counterpart robot’s right side as soon as
the experiments started and observed robot reactions touser speech. As a result, the robot won 90 points in evaluation in generation 6 but scores dropped to 50 in generation 7.
8

50
40
30
20
10
0

0.2
0

SD x Fig. 12. Changes in Interaction between Generation: A indicates the movement path of the robot manipulated by theuser and B, the movement path of the preprogrammed robot.
Difference of Fitness Value

20

7th generation

0

1

2

3
4
5
Generation

6

7

8

Fig. 13. Variations in Fitness Values of Most Elite Individual between Generations.

We analyzed changes in fitness values by users. In
experiments, users were presented with seven individual
robots with genes selected by the SOM for HMHE and thebest individual robot in the previous generation to determine how the best individual robot in the previous generation was evaluated in the next generation (Fig. 13).
This represents an average absolute value of differencesin fitness values for the most elite individual between preceding and subsequent generations, excluding the value
marked by one of the nine users that noticeably differedfrom others. This represents whether interactions by themost elite individual in each generation were evaluatedhighly or lower in the next generation, and thus is agood indicator of changes in user evaluations. Results forchanges in fitness values were widely dispersed, particularly between generation 3-4 and 6-7.

6.5. Survey Findings
In survey findings by generation for combined use with
HMHE, scores for tiredness in generation 3, for boredomand difficulty in concentration in generation 6, and fordifficulty in concentration in generation 9 were significantly lower (Fig. 14). No significant differences wereseen, however, in categories such accomplishment, difficulty, busyness, and sleepiness.

7. Discussion
7.1. Maintenance of Diversity in Interaction
HMHE was developed for and implemented in a communication robot to avoid user boredom by maintaining
Journal of Robotics and Mechatronics Vol.20 No.4, 2008

Human-Adaptive Robot Interaction Using IEC
150 Difficulty
Accomplishment
100

Busyness

Difficulty 150
Accomplishment 100

50
Tiredness

0

Busyness

50
Difficulty ofConcentration0*

Difficulty ofConcentration

Tiredness
Sleepiness

Boredom

initial generation
150 Difficulty
Accomplishment
Busyness

3rd generation
150 Difficulty
Accomplishment
Busyness50
0

Tiredness

Sleepiness

50

Difficulty ofConcentration**

Boredom

6th generation

IEC

Boredom

Sleepiness

0

Difficulty ofConcentration*

Tiredness
Sleepiness

Boredom

9th generation

HMHE(P  0 05 )Fig. 14. Comparison of Survey Findings.

diversity in the gene pool. In comparison to conventional IEC by a simulator, use of HMHE significantly
increased fitness values in the later part of experiments(Fig. 10). Differences in fitness values thus appeared toreflect changes in evaluations by users, i.e., their degreeof boredom.
Effects of HMHE are due to the maintenance of diversity in the gene pool (Fig. 11). The ability to retain different genes in a gene pool at all times thus appears to haveenabled a variety of behaviors to be presented to users.

7.2. Changes in Evaluation Axes of Users
A detailed look at the transition of fitness values among
generations (Fig. 10) showed that timing when fitness values dropped coincided with changes in evaluation axes
between generations 3-4 and 6-7 (Fig. 13), apparently indicating that users were so bored with monotonous experiments that they used different evaluation axes between
generations 3-4 and 6-7, resulting in decrease in overall
fitness values. Changes in evaluation axes are inevitablein an interaction experiments.
HMHE appears to cope with changes in evaluation
axes of users by increasing the population size, meaningthat even if evaluation guidelines of users are changed, it
is possible to produce genes closely approximating new
evaluation guidelines as long as diversity within a genepool is maintained. This is why fitness values did not decrease in the later part of experiments.
7.3. Calculating Distances between Genes in
HMHE
Survey findings confirmed the effectiveness of HMHE
in handling psychological burdens such as boredom, so
no differences were recognized in items such as difficulty
and accomplishment in learning effects. Accomplishment
used in these experiments refers to accomplishment of a
Journal of Robotics and Mechatronics Vol.20 No.4, 2008

robot in learning, i.e., whether a robot comes to behave
as desired by a user. Effects of HMHE are limited to the
avoidance of user boredom and do not directly affect accomplishment in learning.
Why an increased number of genes did not improve
accomplishment lies in the method for calculating distance between genes. In HMHE, fitness values are automatically calculated by the program assuming that genes
closer to each other will receive similar fitness values,
whereas in interaction between robot and users, distance
between genes as parameters for a robot does not always
coincide with distance between user evaluation of interaction. In these experiments, we used genes expressed
as real numbers and Euclidean distance, which, however,requires further consideration.
According to user opinions of interaction, some coupling strength in the motor agent network, e.g., touch sensors on the back, used in these experiments does not directly affect overall impression, but others do affect suchconsiderations significantly, e.g., those related to vehicle
and head movement. This suggests that effects can be improved by calculating distance using weighted elements
rather than simple Euclidean distance, with weights acquired during learning.

7.4. HMHE Effects and Scope of Application
Although the interaction system using IEC with HMHE
does not greatly improve learning efficiency, it requires
only a small number of evaluations while retaining thediversity of genes in individuals presented to users. Results of evaluation experiments showed significant improvement in boredom and tiredness, but not in accomplishment of learning by interaction. It also could copewith changes in evaluation axes during long-term experiments.
This suggests that in adaptive interaction using IEC, thetrilateral trade-off of learning efficiency, number of evaluations, and diversity should be addressed by prioritizing the diversity of behavior and lightening the burden on
users rather than learning efficiency so that user do not
become bored in long-term interaction.
This interaction, which places less emphasis on learning efficiency, can be better applied to areas that requirelong-term interaction, rather than to tasks whose accomplishment is related to serious problems such as user
safety. The system used in these experiments cannot output behavior that takes contexts into consideration and arelimited in application to communication based on reactive
behavior. We consider it possible, however, to develop
communication that adapts to time series interactions by
combining the system with a behavior generator such as arecurrent neural network that deals with time series information.

8. Conclusions and Projected Work
We have proposed HMHE in which subjective user
evaluations are used together with automatic calculation
9

Suga, Y., Ogata, T., and Sugano, Sof fitness values for coping with problem of user boredom
and tiredness arising when IEC is applied for acquisitionof behavior by a communication robot. Application of
IEC to human-robot interaction involves the problem ofbalancing learning efficiency, user boredom due to early
convergence, and tiredness due to increased population
size. To cope, HMHE is programmed to automaticallycalculate fitness values of remaining genes by referencing subjective user evaluations. Results of performance
tests and simulation showed that HMHE effectively reduces user tiredness and boredom, although no significant
improvements were seen in learning efficiency.
Our results suggest that the trilateral trade-off of learning efficiency, gene diversity, and burden on users canbe better addressed by emphasizing the maintenance ofdiversity and the reduction of burden on the user rather
than pursuing learning efficiency. HMHE, which generates a variety of genes with a small number of evaluations,proved effective in enabling the above balance.
In repeated experiments using actual robots with IEC,
we confirmed that IEC with HMHE proceeds with learning by adapting itself to users’ changing preferences [14].
We are planning to conduct experiments over a longer
term.
The system we used has a number of shortcomings:
Since no dynamic changes are incorporated in genes ofphenotype robots, users will be bored quickly and shortened time of evaluations by users may prevent users
from evaluating robots appropriately. To overcome such
shortcomings, we have introduced intensive learning into
robot behavior generation algorithms and optimized reward functions by IEC. This is expected to create adaptability of IEC on a group level, and learning efficiency
of IEC and diversity of behavior on an individual level,
thereby lengthening user interaction time.
The behavior generation algorithm we used is a simple
reactive-behavior generator that cannot deal with time series information. We are therefore planning to introduce
a behavior generator that deals with context information,such as a recurrent neural network.

[5] H. G. Cobb, “An investigation into the use of hypermutation as an
adaptive operator in genetic algorithms having continuous, timedependent nonstationary environments,” NRL Memorandum Report 6760, 1990.
[6] T. Ogata, S. Sugano, and J. Tani, “Open-end Human-Robot Interaction from the Dynamical Systems Perspective -Mutual Adaptation
and Incremental Learning, Advanced Robotics,” VSP and Robotics
Society of Japan, Vol.19, No.6, pp. 651-670, July, 2005.
[7] N. Mitsunaga, C. Smith, T. Kanda, H. Ishiguro, and N.
Hagita, “Robot Behavior Adaptation for Human-Robot Interaction based on Policy Gradient Reinforcement Learning,” Proc. ofthe 2005 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems(IROS2005), pp. 1594-1601, 2005.
[8] J. A. Bildes, “GenJam: A Genetic Algorithm for Generating
Jazz Solos, Proc. of Int. Computer Music Conference (ICMC94),
pp. 131-137, 1994.
[9] H. Lund, O. Miglino, L. Pagliarini, A. Billard, and A. Ijspeert,“Evolutionary Robotics - A Children’s Game,” Proc. of IEEE Int.
Conf. on Evolutionary Computation (ICEC ’98), pp. 154-158, 1998.
[10] M. Nagao, M. Yamamoto, K. Suzuki, and A. Ohuchi, “Evaluation of
the Image Retrieval System Using Interactive Genetic Algorithm,”
Journal of Japanese Society for Artificial Intelligence, Vol.13, No.5,
pp. 720-727, 1998 (in Japanese).
[11] T. Kohonen, “Self-Organizing Maps,” Springer-Verlag, Berlin Heidelberg, 1995.
[12] T. Ogata and S. Sugano, “Emotional Communication Between Humans and the Autonomous Robot which has the Emotion Model,”Proc. of IEEE Int. Conf. on Robotics and Automation (ICRA’99),
pp. 3177-3182, 1999.
[13] S. Haga, “Theory and Measurement of Mental Workload,” Japan
Publishing Service, 2001 (in Japanese).
[14] Y. Suga, Y. Ikuma, D. Nagao, T. Ogata, and S. Sugano, “Interactive Evolution of Human-Robot Communication in Real World,”
Proc. of IEEE/RSJ Int. Conf. on Intelligent Robots and Systems(IROS2005), August, 2005.

Name:
Yuki Suga

Acknowledgements
This work was partly supported by the WABOT-HOUSE Projectcommissioned by Gifu Prefecture, Waseda University Faculty of
Science and Engineering, and the 21st Century COE Programin Japan Society for the Promotion of Science “Innovative Research on Symbiosis Technologies for Human and Robots in Elderly Dominated Society.”

Affiliation:
Research Associate, Department of Creative Science and Engineering, School of Modern Mechanical Engineering, Waseda University

Address:#59-325, 3-4-1 Okubo Shinjuku, Tokyo 169-8555 , Japan

References:[1] H. Takagi, “Interactive Evolutionary Computation:Fusion of the Capabilities of EC Optimization and Human Evaluation,” Proc. of theIEEE Special Issue on Industrial Innovations Using Soft Computing, Vol.89, No.9, 2001.
[2] H. Kitano, “Genetic Algorithm 2,” Sangyou-Tosho, pp. 72-87, 1995
(in Japanese).
[3] T. Inamura, M. Inaba, and H. Inoue, “Acquisition of Probabilistic Behavior Decision Model based on the Interactive Teaching
Method,” Proc. of the Ninth Int. Conf. on Advanced Robotics(ICAR’99), pp. 523-528, 1999.
[4] J. J. Grefenstette, “Genetic algorithms for changing environments,”
Proc. of 2nd Parallel Problem Solving from Nature, pp. 137-144,
1992.

10

Brief Biographical History:
2007- Research Associate at School of Modern Mechanical Engineering,
Waseda University

Main Works:¯ “Adaptive Human-Robot Interaction System using Interactive EC,” Proc.
of IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS2006),
Oct. 2006.

Membership in Academic Societies:¯The Robotics Society of Japan (RSJ)

Journal of Robotics and Mechatronics Vol.20 No.4, 2008

Human-Adaptive Robot Interaction Using IEC

Name:

Name:

Tetsuya Ogata

Shigeki Sugano

Affiliation:

Affiliation:

Associate Professor, Department of Intelligence
Science and Technology, Graduate School of Informatics, Kyoto University

Professor, Department of Modern MechanicalEngineering,School of Creative Science and Engineering, Waseda University

Address:

Address:

Yoshida-honmachi, Sakyo-ku, Kyoto 606-8501, Japan

3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan

Brief Biographical History:

Brief Biographical History:

1997-1999 Research Fellow, The Japanese Society for the Promotion of
Science (JSPS)
1999-2001 Research Associate, Department of Mechanical Engineering,
School of Mechanical Engineering, Waseda University, Japan
2003-2005 Lecturer, Department of Intelligence Science and Technology,
Graduate School of Informatics, Kyoto University, Japan
2005- Associate Professor, Department of Intelligence Science andTechnology, Graduate School of Informatics, Kyoto University, Japan¯“Experience Based Imitation Using RNNPB,” Advanced Robotics, VSP
and Robotics Society of Japan, Vol.21, No.12, pp. 1351-1367, Oct. 2007.

1986- Research Associate, Department of Mechanical Engineering,
Waseda University
1990- Assistant Professor, Department of Mechanical Engineering,
Waseda University
1992- Associate Professor, Department of Mechanical Engineering,
Waseda University
1998- Professor, Department of Mechanical Engineering, Waseda
University
2007- Professor, Department of Modern Mechanical Engineering, Waseda
University
2001- Director of the Waseda WABOT-HOUSE laboratory

Membership in Academic Societies:

Main Works:

Main Works:¯ The IEEE Robotics & Automation Society¯ The Japan Society of Mechanical Engineering (JSME)¯ The Robotics Society of Japan (RSJ)¯ Information Processing Society of Japan (IPSJ)¯ “Human-Robot-Contact-State Identification Based on Tactile
Recognition,” IEEE Transactions on Industrial Electronics, Vol.52, No.6,
2005.
¯ “Robot Design and Environment Design -Waseda Robot-House
Project-,” SICE-ICASE Int. Joint Conf. 2006, Oct. 2006.

Membership in Academic Societies:¯ The IEEE Robotics & Automation Society¯ The Robotics Society Japan (RSJ)¯ The Japan Society of Mechanical Engineers (JSME)¯ The Society of Instrument and Control Engineers (SICE)

Journal of Robotics and Mechatronics Vol.20 No.4, 200811