Interactive Evolutionary Robotics fromDiﬀerent Viewpoints of Observation
Daisuke Katagami1 and Seiji Yamada2
1
Tokyo Institute of Technology, CISS, IGSSE
4259 Nagatsuta, Midori-ku Yokohama 226-8502, Japan, katagami@ntt.dis.titech.ac.jp
2
National Institute of Informatics, Japan, seiji@nii.ac.jp

Abstract
In this paper, we describe inﬂuence of viewpoints ofobservation in an interactive evolutionary robotics
system. We have been proposed a behavior learning
system ICS (Interactive Classiﬁer System) using interactive evolutionary computation. In this system,a mobile robot is able to quickly learn rules by directteaching of a human operator. ICS is a novel evolutionary robotics approach using a classiﬁer system.
We classify teaching methods into internal observation and external one, and investigate inﬂuence ofobservation methods. We have experiments based on
our teaching methods in two kinds of tasks. We foundthat teaching methods from diﬀerent viewpoints ofobservation change teaching eﬃciency because of thediﬀerence between a robot’s recognition and an operator’s one in an environment.

1

Introduction

In previous robot learning studies, optimization ofcontrol parameters has been applied to acquire suitable behaviors in an real environment. Also in mostof such researches, a model of human evaluation has
been used for validation of learned behaviors. However, since it is very diﬃcult to build a human evaluation function and adjust control parameters, a system hardly learns behaviors intended by a humanoperator.
In contrast with modeling human evaluation analytically, we introduce another approach in which asystem learns suitable behaviors using human direct
evaluation without its modeling. Such an interactive
method with Evolutionary Computation (EC) as asearch algorithm is called Interactive EC (IEC)[1] ,and a lot of researches on it have been done thus
far[2][3]. Consequently it can performs subjective
evaluation by human sense. However an operator
suﬀers from cognitive and physical load. In case of arobot reinforcement learning in a real environment,

some researches have been worked[4]. Unfortunately
it makes a lot of time for convergence of learning.
To cope with these problems, we have been proposed
Interactive Evolutionary Robotics (IER)[5][6]. It is
an interactive EC learning method to design a robot
using EC methods like genetic algorithm, genetic
programming and evolutionary strategy. We expect
IER to perform high emergent property of ER and
subjective adaptability of IEC. This method quickly
learns eﬀective rules by simple instructions of a human operator. The objective of IER is to make initial learning more eﬃcient and learn behaviors thata human operator intended through interaction withhim/her.
First of all, we developed a learning system based onclassiﬁer system[7] on IER framework, which is able
to adapt to multiplicity of an environment and a variable dynamic state. we call it as Interactive Classiﬁer
System (ICS). Fig.1 give an environment of teaching
using ICS. The major diﬀerence between ICS anda traditional learning classiﬁer system is to introduce an interactive method. Accordingly, we expectthat the system eﬃciently performs initial learningin an actual environment and can operate concentrative incremental learning. However there is few
framework that an operator observes from robot’sview. Therefore, the system can not make the bestuse of the learning. In this paper, we propose interactive method based on internal observation in orderto solve the problem.
Nordin et al.[8][9] work a robot learning using genetic
programming in a real environment. They argue areal robot learning is possible in a task of an obstacle
avoidance. Unfortunately, it take a long time to learn
the eﬃcient behavior yet. Nicolescu et al.[10] researches learning by interacting with a human like as
our approach. However, it is very diﬀerent from ourapproach for reasons of what it applies experience of
the interaction to evolutionary learning. Mondada
et al.[11] works a robot learning with evolutionaryoperator

Actual Environment

Teacher
Observation

monitorjoystickvibrationteaching

Learner

robot

image
information

Operation

sensorinformation

ICS
ICS : Interactive Classifire System

Figure 2: Teacher view
actionFigure 1: Teaching environment

Teacher
Observation

learning as well as Nordin. However they report thatthe learning takes scores of hours to converge.

2Operation

Learner

Sensor Information

Teaching Method Based on Diﬀerent
Viewpoints of Observation
Figure 3: Learner view

A robot hardly knows how it modiﬁes behaviors ofitself for a task. Furthermore it may be diﬃcult torecognizes processing of a task. It is not an observed
approach through an inside viewpoint of a system,
but an observed one through an outside viewpoint
to be able to recognize achieving of a task.
In case of an operator usually teaches to a robot, an
external observer don’t recognize its internal information. It is general that an operator teaches to arobot by guessing its internal information. Therefore, we had a signiﬁcant diﬀerence between cognition of an operator and one of a robot, which is calleda perceptual aliasing problem. It become a subject of
discussion when a operator teach skills for a task toa robot.
In this paper, we prepare the simple setting basedon the observation to examine how the diﬀerenceinﬂuence to acquired rules by teaching. We call amethod to observe through an outside viewpoint ofa system as teacher view (TV) (Fig.2), and a method
to observe through an inside viewpoint of a system
as learner view (LV) (Fig.3).
We examine the diﬀerence by IER based on teachingwith this TV and LV. To realize this IER, we appliedthese methods to developed ICS, which is a robot
learning system based on interactive EC.

3
3.1

Interactive Classiﬁer System
System overview

ICS applies XCS[12], which is a kind of Learning
Classiﬁer System (LCS), as evolutionary computation and introduces a interactive function. XCS
equipped a function to preserve classiﬁers from overgeneralization to make system’s performance worse.
Moreover, XCS applies restricted mating which is akind of strategy based on genetic algorithms. For
these reason, XCS improves learning performance oftraditional CS. It is constructed as a robot learningmodel that can not only learn through teaching but
also learn autonomously using XCS.
ICS mainly consists of a rule generation component(RGC), a sensor processing component (SPC), a display component (DC) and a reinforcement component (RC) (Fig.4). It was implemented with C language and GTK+ on Linux. It utilizes Video4Linuxfor image processing. The rule generation component makes a new classiﬁer from teaching by the operator. The SPC processes each information of some
sensors and camera, and through it for the RGC.
The DC displays by GUI interface and processes theEnvironment
operator
joystickrobot

monitorsensorinformation"left"

ICS
SPC

DC
01

[P]
p
43
32
14
27
18
24

#011:01
11##:00#0##:11001#:01#0#1:11
1#01:10

e
.01
.13
.05
.24
.02
.17

F
99
9
52
3
92
15

Match Set

Prediction Array

Action Set[M]#011:01#0##:11001#:01#0#1:11

RGC

43
14
27
18

.01 99.05 52.24 3.02 92

nil 42.5 nil 16.6

#011:01
001#:01

43 .01 99
27 .24 3

max
discount

GA
RC+

delay=1

Previous Action Set[A] -1Figure 5: User interface
Figure 4: Overview of Interactive Classiﬁer
System
3. The RGC makes new rules from them and adds
them into a rule list. When nothing is input
from the operator, a mobile robot executes autonomous behaviors.

input from a joystick. Finally the RC performs learning by updating parameters in ICS. Fig.5 shows thedeveloped an interface for the system. The middle ofthe left of the interface indicates some matched classiﬁers in this time. The middle of the right indicatesrobot’s sensor information and state of a joystick.
The experiments are made with a standard miniature
mobile robot Khepera (Fig.6). The mobile robot has
cylindrical shape, a diameter of 6 cm and a height of
5 cm. It possesses two motors and on-board power
supply. The motors can be independently controlled
by a PID controller. It is equipped with eight infrared proximity sensors. The eight infrared sensors
are distributed around the robot in a circular pattern. They emit infrared light, receive the reﬂected
light and measure distances in a short range: 2-5 cm.
The robot is also equipped with a Motorola 68331
micro-controller which can be connected to a computer via serial cable. Moreover, the system utilizes
SONY analog controller DUALSHOCK as a joystick.
Fig.6 shows them respectively.
3.2

ICS procedures

We describe a learning procedure in ICS as follows.
1. At ﬁrst, a human operates robot with a joystickby viewing sensor information displayed on GUI,and the DC processes it.
2. Next, the SPC gets operator’s instruction androbot’s sensor information.

4. Finally, the RC reinforces the classiﬁers by updating their parameters in the actions whichwere previously executed.
In traditional works of robot learning in a real environment, a learning takes pretty much time to converge because of learning by trial and error. In thiswork, we consider that this learning by trial and error is a problem in a real environment. However, it
hardly prepare suitable a priori knowledge in an environment. For this reason, the ICS generate initialindividuals by teaching from human-robot interaction. We can perform eﬃciently initial learning inthis way.

4
4.1

Experiments
Experiments with cognitive observation

We experimented in a real world to investigate difference in two teaching methods: TV and LV as
mentioned earlier. Fig.7 shows an experimental environment. As an experimental task, ICS reduces
the number of steps from any start points to a lightsource which set up as a goal in a ﬁeld surroundedwith white plastic plates. We compared two teaching methods (TV and LV) with a traditional method
in which a robot autonomously learns by simple EC.
To investigate inﬂuence by gap between operator’s

2

3

1

4

0

5

DC motors
IR proximity sensors
7

6

Figure 6: A mobile robot Khepera and a joystick

Figure 7: Experimental Environment

Table 1: Experimental Parameters
Parameters
The number of problems in one experiment
The number of expriments
Maximum size of the population
Probability to do crossover
Probability of mutating one bit

Value
30
1
300
0.8
0.04

recognition and robot’s one, we prepared experiment
1 without an obstacle and experiment 2 with someobstacles.
We consider forty steps as a trial and begin teachingor autonomous exploration for ﬁve trials at random
start points, and test a trial at each of ﬁve startpoints as examination of learned rules by the teachings or the exploration. It performs this procedure
six times, and consequently we have thirty trials as
examination. Table 1 shows experimental parameters.
An encoding of a classiﬁer is as follows. A classiﬁer isthe twenty bit string “#000#100000###100#:01”.
The robot’s condition is the left sixteen bit string, it
represented eight infrared proximity (left eight bit)and light (right eight bit) sensors around the robot
in a circular pattern respectively (Fig.6). The bit is“1” if a input sensor value larger than a threshold, or
else “0”. “#” is a “don’t care” symbol which classiﬁer system employed. The robot’s previous action is
the next two bit string, represented as forward ”11”,left-turn “01”, right-turn “10 ” and back “00”. The
robot’s current action is the two bit string similarly.
A ﬁtness function of ICS deﬁned as follows. Reward F is computed by the sum of eight light sensors

through a sigmoid function.
u=

7

i=0

lightiαlightMAX × 8

F =1
1 + exp(−u)

In the case of TV, the operator can not recognize
a small obstacles as the robot can perceive althoughhe/she looks a whole environment. To represent TV,
an operator teaches a robot using a camera whichlooks a whole environment. Fig.8 shows information
through GUI by a camera which sets up the environment.
In the case of LV, an operator can not look a wholeenvironment although it can recognize directly asmall obstacles and a recognition error which therobot can perceive. To implement LV, ICS uses GUI
which indicates some sensor values. Since an operator hardly recognize an environment by reading only
numerical values of sensor data, we developed GUI
interface which can represent sensor values graphically. Fig.9 shows such sensor information throughGUI interface.
In the experiment 1, we examined the number ofreach times to a light source from ﬁve start points.
Fig.10 shows the number of times to reach to a lightsource (goal of the task) at test trials from ﬁve startpoints. Though LV begins to success the task at earlystage, it doesn’t success later. On the other hand,
TV improves robot’s learning and achieves the task
from all ﬁve start points before LV. TV improves
a robot’s learning in the simple environment which
cognitive diﬀerence is little by teaching is easy because an operator looks a whole environment. However, the diﬀerence is not so large.
In the experiment 2, we introduced an obstacle in the5Teacher View
Learner View
No Teach

Reach Times to Goal

4

3

2

1

0

0

5

10

15

20

25

30

35

Problems

Figure 10: Reach Times to Light Source
Figure 8: External Observation Settings
20

Teacher View
Learner View

Steps to Goal

15

10

5

0

0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22
Problems

Figure 11: Steps to Light Source
Figure 9: Internal Observation Settings

environment of experiment 1 to investigate the eﬀect
by a more diﬃcult task. The obstacle was made of
the transparent plastic board because of perception a
direction of a light source. A robot must reach to thelight source as avoiding the obstacle. We compareda teaching method by TV with LV in the same way
as experiment 1.
We examined the number of steps to a light source.
The number of steps to a light source shows Fig.11.
Since ICS hardly uses information of proximity sensors and acquires eﬀective rules by using only information of light sensors in the environment 1, thediﬀerence of cognition between an operator and arobot is a little. For this reason, the diﬀerence of
two teaching methods was not so large.

However ICS hardly learns from teaching of an operator in case of experiment 2 because there is thediﬀerence between the situation of the robot which
an operator estimated by observation from the outside and the one of the robot in the real world. LV
actually creates eﬀective rules because an operator
performed teaching as he/she checks robot’s internal
status.
After we have experiments TV and LV each twenty
trials, we test a trial at each of ﬁve start points as
examination using each created rules. Table 2 showsthe experimental results.
The robot can not reach to a light source at any startpoints of ﬁve in the examination of TV. On the other
hand, the robot reached to a light source by six steps
and nine steps respectively at two of ﬁve start points
in the examination of LV. We can see that ICS cancreate eﬀective rules independently of start points

Table 2: Experimental Results of Exploit

times
TV
LV

Step toLight Source
1 2 3 4 5
6 9 -Reach to
Goal
total
0
2

[4] E. Uchibe, M. Asada, and K. Hosoda. Behavior
coordination for a mobile robot using modular reinforcement learning. In IEEE/RSJ International
Conference on Intelligent Robots and Systems 1996
(IROS96), pages 1329–1336, 1996.

Table 3: Created Rules by LV Method
Condition
0#0#0#####0#10##1#
0010###0##000#1###
00###0#000#1#0#01####000000101001011

Action1001
1111

Prediction
413.7
364.0
292.0
256.0

Teach
2
1
4
1

since LV improves the robot’s learning by teaching.
Table 3 shows ten rules which have best values ofthe system prediction in created ones after twenty
trials with LV. ICS created eﬀective and commonsense rules like as the robot moves forward when thelight source faces in front of the robot and it moves
right when the light source faces right. In this table, Prediction is a ﬁtness of a classiﬁer and Teach iswhether it is taught by human or its oﬀspring. Its
number is the number of which a taught classiﬁer alternate generations. We can see teaching improves
very well because all the rules whose the system prediction is high created from teaching or its oﬀspring.
Though an operator did not teach a robot to go backin the both experiments, the robot goes back foravoiding an obstacle when the robot collides witha wall and reaches to a light source. Because ICS
created these rules by which a robot works in cooperation with human.

5[3] T. Unemi. A Design of Multi-Field User Interface
for Sumulated Brdding. Asian Fuzzy System Symposium, pages 489–494, 1998.

Conclusion

In this paper, we proposed a novel interactive
method from the viewpoint of observation and investigated its eﬀects in a real world experiments. It
was found that internal observation increase the effects according to arise the diﬀerence between the
robot’s recognition and operator’s one in more complex environment.

References[1] R. Dawkins. The Blind Watchmaker. Longman, Essex, 1986.
[2] Y. Nakanishi. Capturing Preference into a Function
Using Interactions with a Manual Evolutionary Design Aid System. Genetic Programming, pages 133–
140, 1996.

[5] D. Katagami and S. Yamada. Interactive classifiersystem for real robot learning. In IEEE International
Wortkshop on Robot and Human Interaction, pages
258–263, 2000.
[6] D. Katagami and S. Yamada. Real robot learningwith human teaching. In The Fourth Japan-Australia
Joint Workshop on Intelligent and Evolutionary Systems, pages 263–270, 2000.
[7] J. H. Holland and J. S. Reitman. Cognitive systems
based on adaptive algorithms. In Donald A. Waterman and Frederick Hayes-Roth, editors, PatternDirected Inference Systems, pages 313–329, Orlando,
1978. Academic Press.
[8] P. Nordin and W. Banzhaf. Genetic Programming
Controlling a Miniature Robot. In Working Notes
for the AAAI Symposium on Genetic Programming,
pages 61–67, 1995.
[9] P. Nordin and W. Banzhaf. An on-line method to
evolve behavior and to control a miniature robot inreal time with genetic programming. Adaptive Behavior, 5(2):107–140, 1996.
[10] M. Nicolescu and M. J Mataric. Learning and Interacting in Human-Robot Domains. IEEE Transactions
on Systems, Man, Cybernetics, 31(5), 2001.
[11] F. Mondada and D. Floreano. Evolution and mobileautonomous robotics. In In Proceedings of the Evolutionary Engineering, pages 221–249, 1995.
[12] S.W. Wilson. Classifier fitness based on accuracy.
Evolutionary Computation, 3(2):149–175, 1995.