Interactive Evolutionary Computation for Real
Robot from a Viewpoint of Observation
Daisuke Katagami CISS, IGSSE, Tokyo Institute of Technology
4259 Nagatsuta, Midori-ku Yokohama 226-8502, JAPANkatagami@ymd.dis.titech.ac.jp
Seiji Yamada CISS, IGSSE, Tokyo Institute of Technology
4259 Nagatsuta, Midori-ku Yokohama 226-8502, JAPAN
yamada@ymd.dis.titech.ac.jp
Abstract. In this paper, we describe investigation on a viewpoint ofobservation in an interactive evolutionary robotics system. We propose
a behavior learning system ICS (Interactive Classifier System) using interactive evolutionary computation and a mobile robot is able to quickly
learn rules by direct teaching of an operator. Also ICS is a novel evolutionary robotics approach using an adaptive classifier system to environmental changes. We classify teaching methods as internal observationand external one, and investigate the relationship between the observation methods and the results. We have two experiments based on our
teaching methods on a real world.

1

Introduction

In previous robot learning studies, optimization of control parameters has been applied
to acquire suitable behaviors in an real environment. Also in most of such researches,a model of human evaluation has been used for validation of learned behaviors. However, since it is very diﬃcult to build a human evaluation function and adjust control
parameters, a system hardly learns behaviors intended by a human operator.
In contrast with modeling human evaluation analytically, we introduce another approach in which a system learns suitable behaviors using human direct evaluation without its modeling. Such an interactive method with Evolutionary Computation (EC) as
a search algorithm is called Interactive EC (IEC) [1] , and a lot of researches on it havebeen done thus far[2] [3].
Additionally reinforcement learning has been applied to robot learning in a realenvironment[4]. Unfortunately the learning takes pretty much time to converge. Furthermore, when a robot hardly gets ﬁrst reward because of no priori knowledge, thelearning becomes far slower.
To solve these problems, we have been proposed the framework of Interactive Evolutionary Robotics (IER)[5][6]. It is an interactive EC learning method for the purposeof designing a robot using EC methods like genetic algorithm, genetic programming andevolutionary strategy. We can expect IER to perform high emergent property of ER
and subjective adaptability of IEC. This method quickly learns eﬀective rules by simple
instructions of a human operator. The objective of IER is to make initial learning more
eﬃcient and learn behaviors that a human operator intended through interaction withhim/her.

operator

Actual Environment

Teacher
Observation

monitorjoystick

Learnerrobot

vibrationteaching

image
information
sensorinformation

Operation

ICS
ICS : Interactive Classifire System

actionFigure 1: Teaching environment

Figure 2: Teacher viewFirst of all, we developed a learning system based on classifier system[7] on IER
framework, which is able to adapted to multiplicity of an environment and a variable
dynamic state. we call it as Interactive Classifier System (ICS). Fig.1 give an environment of teaching using ICS.
The diﬀerence between ICS and an usual learning classiﬁer system is to introduce
an interactive method. Accordingly, we expect that the system performs eﬃcient initial
learning in an actual environment and can operate concentrative incremental learning.
However there is few framework that an operator observes from robot’s view. Therefore,
the system can not make the best use of the learning. In this paper, we propose
interactive method based on internal observation in order to solve the problem.
2

Teaching Method Based on the Viewpoint of Observation

A robot hardly knows how the robot modiﬁes behaviors of itself for a task and recognizes
that the task is in the process of achieving. Besides, a robot, which observes throughan inside viewpoint of a system, may be unable to recognize that a task is realizing,or rather an operator, which observes through the outside viewpoint of a system, canrecognize. We, then, had a signiﬁcant diﬀerence, what is called a perceptual aliasingproblem, between cognition of an operator and one of a robot. It hence become asubject of discussion when a operator teach skills for a task to a robot.
In this paper, we prepare the simple setting based on the observation to examine
how the diﬀerence inﬂuence to acquired rules by teaching. We call the method whichobserve through an outside viewpoint of a system as teacher view (Fig.2), and themethod which observe through the inside viewpoint of a system as learner view (Fig.3).
We examine the diﬀerence by IER based on teaching with this teacher view andlearner view. To realize this IER, we applied these methods to developed ICS, which isa robot learning system based on interactive EC.
3

Interactive Classifier System

3.1

System overview

ICS applies XCS[8], which is a kind of Learning Classiﬁer System (LCS), as evolutionary computation and equipped a interactive function. XCS equipped a function
which preserve classiﬁers from overgeneralization makes system’s performance worse.
Moreover, XCS applies restricted mating which is a kind of strategy based on genetic

Environment
operator
joystickrobot

monitorsensorinformation"left"ICS

Teacher

SPC

DC
01

Observation

Learner
[P]
p
43
32
14
27
18
24

#011:01
11##:00#0##:11001#:01#0#1:11
1#01:10

e
.01
.13
.05
.24
.02
.17

F
99
9
52
3
92
15

Match Set

Prediction Array

Action Set[M]

Operation

Sensor Information#011:01#0##:11001#:01#0#1:11

RGC

43
14
27
18

.01 99.05 52.24 3.02 92

nil 42.5 nil 16.6

#011:01
001#:01

43 .01 99
27 .24 3

max
discount

GA
RC+

delay=1

Previous Action Set[A] -1Figure 3: Learner view
Figure 4: Overview of Interactive Classifier
Systemargorithms. For these reason, XCS improves learning performance of traditional CS. It
is constructed as a robot learning model that can not only learn through teaching but
also learn autonomously using XCS.
ICS mainly consist of a rule generation component (RGC), a sensor processing
component (SPC), a display component (DC) and a reinforcement component (RC).
It was developed with C language and GTK+ on Linux. It utilizes Video4Linux forimage processing. The rule generation component makes a new classiﬁer from teachingby the operator. The SPC processes each information of some sensors and camera,
and through it for the RGC. The DC displays by GUI interface and processes the input
from a joystick. Finally the RC performs learning by updating parameters in ICS. Fig.5
shows the developed interface of the system.
The experiments are made with a standard miniature mobile robot Khepera (Fig.6).
The mobile robot has a cylinder shape, a diameter of 6 cm and a height of 5 cm. It
possesses two motors and on-board power supply. The motors can be independently
controlled by a PID controller. The eight infrared sensors are distributed around therobot in a circular pattern. They emit infrared light, receive the reﬂected light andmeasure distances in a short range: 2-5 cm. The robot is also equipped with a Motorola
68331 micro-controller which can be connected to a computer via serial cable. Moreover,the system utilizes SONY analog controller DUALSHOCK as a joystick. Fig.6 showsthem respectively.
We describe a learning procedure in ICS as follows.
1. At ﬁrst, a human operates robot with a joystick by viewing sensor information
displayed on GUI, and the DC processes it.
2. Next, the SPC gets operator’s instruction and robot’s sensor information.
3. The RGC makes new rules from them and adds them into a rule list. Whennothing is input from the operator, a mobile robot executes autonomous behaviors
from interaction.
4. Finally, the RC reinforces the classiﬁers by updating their parameters in theactions which were previously executed.

2

3

1

4

0

5

DC motors
IR proximity sensors
7

Figure 5: User interface

6

Figure 6: A mobile robot Khepera and a joystick

In traditional works of robot learning in a real environment, a learning takes prettymuch time to converge because of learning by trial and error. In this work, we consider
that this learning by trial and error is a problem in a real environment. However, it
hardly prepare suitable apriori knowledge in an environment. For this reason, the ICS
generate initial individuals by teaching from human-robot interaction. We can perform
eﬃciently initial learning in this way.
4

Experiments

4.1

Experiment A with Cognitive Observation

We experimented in a real world to investigate diﬀerence in two teaching methods:
teacher view and learner view. Fig.7 shows an experimental environment. As an experimental task ICS reduces the number of steps from any start points to a light source
which set up as a goal in a ﬁeld surrounded with white plastic plates. We compared
two teaching methods with a traditional method in which a robot autonomously learns
by simple EC.
An encoding of a classiﬁer is as follows. A classiﬁer is the twenty bit string “#000#10
0000###100#:01”. The robot’s condition is the left sixteen bit string, it represented
eight infrared proximity (left eight bit) and light (right eight bit) sensors around therobot in a circular pattern respectively (Fig.6). The bit is “1” if a input sensor value
larger than a threshold, or else “0”. “#” is a “don’t care” symbol which classifiersystem employed. The robot’s previous action is the next two bit string, represented
as forward ”11”, left-turn “01”, right-turn “10 ” and back “00”. The robot’s current
action is the two bit string similarly.
A ﬁtness function of ICS deﬁned as follows. Reward F is computed by the sum ofeight light sensors through a sigmoid function.
u=

7

i=0

lightiP aymentRange
lightM AX × 8

F =1
1 + exp(−u)

We consider fourty steps as a trial and begin teaching or autonomous exploration
for ﬁve trials at randomly start points, and test a trial at each of ﬁve start points

Table 1: Experimental Parameters
Parameters
number of problems in one experiment
number of expriments
maximum size of the population
probability to do crossoverprobability of mutating one bit

Figure 7: Experimental Environment

Value
30
1
300
0.8
0.04

Figure 8: External Observation Settings

as examination of created rules by the teachings or the exploration. It performs thisprocedure six times, and consequently, we have thirty trials as examination. Table 1
shows experimental parameters.
In the case of teacher view, the operator can not recognize a small obstacles as therobot can perceive although he looks a whole environment. To represent teacher view,an operator teach to a robot using the camera which looks a whole environment. Fig.8
shows information through GUI by a camera which sets up the environment.
In the case of learner view, an operator can not look a whole environment although
it can recognize directly a small obstacles and a recognition error which the robot canperceive. To implement learner view, ICS uses GUI which represent some sensor values.
Since an operator hardly understands although one looks only numerical values of sensor
data, we developed GUI interface which can represent sensor values by graphs. Fig.9
shows sensor information which represented by graphs through GUI interface.
4.2

Experimental Results of Examination A with Cognitive Observation

In this experiments, we examined the number of steps to a light source, system error,
and ﬁtness. Fig.10 shows the average of the number of steps at test trials from ﬁve startpoints. System error(Fig.11) is the absolute diﬀerence between the system prediction
for the chosen action by a system and the actual external payoﬀ. Fitness (Fig.12) which
is used in genetic algorithms for evaluation.
Teacher view improves a robot’s learning in the simple environment which cognitive
diﬀerence is little by teaching is easy because an operator looks a whole environment.
However, the diﬀerence is not so large. There was no diﬀerence in both observations
regard for system error and ﬁtness. We found out that the learning improves without
they eﬀect on system error or ﬁtness by using two kind of teaching methods.

50

Auto
Learner View
Teacher View
LV BestTV BestStep to the light source

45
40
35
30
25

20
15
10
5
0

0

5

10

15

20

25

30

TrialsFigure 10: Step to Light Source

Figure 9: Internal Observation Settings
0.5

0.5

Auto
Learner View
Teacher View

0.4

System Error

0.4

Fitness

0.3

0.2

0.1

0.1

0

5

10

15

20

25

30

TrialsFigure 11: System Error

4.3

0.3

0.2

0

Auto
Learner View
Teacher View

0

0

5

10

15

20

25

30

TrialsFigure 12: Fitness

Experiment B with Cognitive Observation

We introduced an obstacle in the environment of experiment A to investigate the eﬀect
by a more diﬃcult task. Fig.7 shows the experimental environment. The obstacle
was made of the transparent plastic board because of perception a direction of a lightsource. A robot must reach to the light source as avoiding the obstacle. We compareda teaching method by teacher view with learner view in the same way as experiment A.
We consider twenty steps as a trial and begin teaching for twenty trials at randomly
start points. It is diﬀerent from experiment A because of simplifying an experiment.
We test a single trial at a start point every one trial as examination of created rules.
The experimental parameters is as same as Table 1.
4.4

Experimental Results of Examination B with Cognitive Observation

In experiment B, we examined the number of steps to a light source. The number ofsteps to a light source shows Fig.13.
Seeing from a best value of teaching (TV Best and LV Best), we see that teacher
view improves teaching likewise experiment A because of looking a whole environment.
However, we can see that learner view outperformed teacher view about steps to a lightsource in contrast with experiment A. Since ICS hardly uses information of proximity
sensors and can acquire eﬀective rules by using only information of light sensors in theenvironment, the diﬀerence of cognition between an operator and a robot is a little. For
this reason, the diﬀerence of two teaching methods was not so large.

A Number of Teaching

25

Learner View
Teacher View
LV BestTV Best

20

15

10

5

0

0

5

10

15

20

TrialsFigure 13: Step to Light Source
Table 2: Experimental Results of Exploit

times
teacher view
learner view

Step to Light Source
12345----69---Reach to Goal
total
0
2

Table 3: Created Rules by learner view Method
Condition
0#0#0#####0#10##1#
0010###0##000#1###
00###0#000#1#0#01####000000101001011
0#0#0#####0#10#01#
00#000001001##0##1
00###0#000#1#0##11#000#100000###100##000##001000##0001#000#0001000##0001

Action
10
01
11
11
10
10
11
01
11
11

Prediction
413.7
364.0
292.0
256.0
280.3
244.5
215.0
101.5
100.8
99.4

Teach
2
1
4
1
3
1
1
3
6
10

ICS however hardly learns from teaching of an operator in case of experiment B
because there is the diﬀerence between the situation of the robot which an operator
estimated by observation from the outside and the one of the robot in the real world.
Leaner view actually creates eﬀective rules, because an operator performed teaching asverifying robot’s internal status.
After we have experiments teacher view and learner view each twenty trials, we test
a trial at each of ﬁve start points as examination using each created rules. Table 2 showsthe experimental results. The robot can not reach to a light source at any start points
of ﬁve in the examination of teacher view because teaching does not improve. On theother hand, the robot reached to a light source by six steps and nine steps respectively
at two of ﬁve start points in the examination of learner view. We can see that ICS
can create rules which eﬀective and do not depend on start points since learner view
improves the robot’s learning from teaching.
Table 3 shows ten rules which have best values of the system prediction in createdones after twenty trials with learner view. ICS created eﬀective and common-sense rules
like as the robot moves forward when the light source is in front of the robot, and it
moves right when the light source is right. We can see its teaching improves very wellbecause all the rules, which the system prediction is high, is created from teaching or
its oﬀspring.
Though an operator did not teach a robot to go back in the both experiments, therobot goes back for avoiding an obstacle when the robot colides with a wall and reaches
to a light source. Because ICS created these rules by which a robot works in cooperation
with human.
5

Conclusion

In this paper, we proposed a novel interactive method from the viewpoint of observation.
We classiﬁed the teaching from the viewpoint of observation as external observation andinternal one, and investigated its eﬀects in a real world experiments. It was found thatinternal observation increase the eﬀects according to arise the diﬀerence between the
robot’s recognition and operator’s one in a complex environment.
This study showed the diﬀerence of a teaching eﬀect due to the diﬀerence of observation methods and examined that it is possible to perform a fast learning of a robot byhuman-robot interaction in a real world. Besides, our system learns and automaticallyacquires a complex robot’s program which human hardly describes by the way that an
operator only teachs to a robot simply. Our future direction will be one that acquires
the unconscious information of an operator like reactive behaviors of human and skillsof an operator.
References[1] R. Dawkins. The Blind Watchmaker. Longman, Essex, 1986.
[2] Y. Nakanishi. Capturing Preference into a Function Using Interactions with a Manual Evolutionary Design Aid System. Genetic Programming, pages 133–140, 1996.
[3] T. Unemi. A Design of Multi-Field User Interface for Sumulated Brdding. Asian
Fuzzy System Symposium (AFSS’98), pages 489–494, 1998.
[4] E. Uchibe, M. Asada, and K. Hosoda. Behavior coordination for a mobile robot
using modular reinforcement learning. In IEEE/RSJ International Conference onIntelligent Robots and Systems 1996 (IROS96), pages 1329–1336, 1996.
[5] D. Katagami and S. Yamada. Interactive classiﬁer system for real robot learning. In
IEEE International Wortkshop on Robot and Human Interaction(ROMAN-2000),
pages 258–263, 2000.
[6] D. Katagami and S. Yamada. Real robot learning with human teaching. In The
Fourth Japan-Australia Joint Workshop on Intelligent and Evolutionary Systems,
pages 263–270, 2000.
[7] J. H. Holland and J. S. Reitman. Cognitive systems based on adaptive algorithms. In
Donald A. Waterman and Frederick Hayes-Roth, editors, Pattern-Directed Inference
Systems, pages 313–329, Orlando, 1978. Academic Press.
[8] S.W. Wilson. Classiﬁer ﬁtness based on accuracy. Evolutionary Computation,
3(2):149–175, 1995.