Proceedings 2003 IEEE International Symposium on
Computational Intelligence in Robotics and Automation
July 16-20, 2003, Kobe, Japan

Teacher’s Load and Timing of Teachingbased on Interactive Evolutionary Robotics
Daisuke Katagami
CISS, IGSSE, Tokyo Institute of Technology, JAPANwww.ntt.dis.titech.ac.jp/home/katagami/

Abstract
We have proposed a fast learning method that enables
a mobile robot to acquires autonomous behaviors frominteraction between human and robot. In this research
we develop a behavior learning method ICS (Interactive
Classiﬁer System) using interactive evolutionary computation considering an operator’s teaching cost. As aresult, a mobile robot is able to quickly learn rules bydirectly teaching from an operator. ICS is a novel evolutionary robotics approach using classiﬁer system. In
this paper, we investigate teacher’s physical and mentalload and proposed a teaching method based on timingof instruction using ICS.

1Introduction

Under the situation that it’s diﬃcult to prepare theknowledge for action, the autonomous robot requires theability to accomplish tasks in the environment where
contents human activities. The situation, for example,
may be an dynamic environment or unexpected interaction from human. Therefore, the study for acquisitionof autonomous behavior and adaptation to various environments become necessary.
Recently, reinforcement learning and evolutionary
computation technique were used as the framework oflearning and adaptation. Moreover, the research thatenables a robot get a controller autonomously has attracted attention. When making interaction dynamics
with robot’s embodiment and environment reﬂect inconstruction of a controller, one of the purposes of thesetechniques is eliminating the unsuitable and unnecessary
bias by the designer. Therefore, in the former, it has
been made usual to learn by trial and error to an agent,
without putting in prior knowledge in the framework of
reinforcement learning. However, the execution speed
becomes a problem in applying to a real environment.
Then, some researches of the approach using interaction with the human who exists in environment has
been carried out. Particularly for the robots that do nothave a priori knowledge or commit trial and error in theinitial stage, human instruction is the very eﬀective acquisition technique of autonomous behavior. However,in a certain level of autonomous robot, it is not necessary to follow instruction from human all the time. In
the stage which does not need instruction, robot should
demonstrate its autonomy based on the instruction rules
stored by interaction with human without putting a burden on human. Therefore, we need to the technique ofestablishing a robot’s autonomy from through interaction between human and a robot is required.
Asoh et al. [1] proposed the framework that the map

0-7803-7866-0/03/$17.00 ©2003 IEEE

1096

Seiji Yamada
National Institute of Informatics, JAPANresearch.nii.ac.jp/~seiji/

information of the unknown environment is built by amobile robot, called Jijo-2 which performs a communication by voice conversation with human. However, it
doesn’t get the behavior of the robot by the interaction
through human and a robot. Ishiguro et al. [2] built thestate space of the mobile robot by reinforcement learning. However, it is learning by using as a sample actionthat the introduction human taught. After that, a robot
only builds an internal state and there is no interaction
with human. Horiguchi et al. [4] used the idea of themutual leadership pattern interaction as the design ofthe interaction of the robot with the human and realized the cooperation behavior of the automation processof a mobile robot and human operations by using power
feedback. However, the result of learning didn’t reﬂectedon the behavior acquisition of the robot. Inamura etal. [5] indicate acquirement behavior of a robot using
Bayesian Network based on a dialog with a user. It isdiﬀerent from our technique to get behavior graduallyby the evolutionary computation technique.
Our purpose is realizing a robot’s autonomy by receiving the instruction information as a suitable act fromhuman, and gaining act rules evolutionally with thestate recognition which can solve a task. We call such aframework Interactive Evolutionary Robotics (IER).
In this paper, we propose some method about ateacher’s load and timing of teaching which is key pointin the framework of IER.

2

Interactive Teaching

2.1

Teacher’s Load

Generally, in interactive evolutionary learning, the more
it is taught, the better the performance is. However,human labor is not unlimited. It is clear that it is tradeoﬀ like it is better as instruction cost lowers. Human’slabor has a limit in cooperating with a machine without
tiredness, carrying out comparison evaluation of many
individuals (or rules) for every generation, and inputting
an evaluation value. This has been a serious practicalproblem.
Moreover, as the second problem, the number of individuals and the number of search generations must be
lessened as compared with the usual EC search in order to reduce physical and mental load in case humanevaluates individuals. It makes convergence worse.
In this research, in order to measure a teacher’s load
simply, it divides into mental load and physical load.
We consider the timing of teaching as mental load andthe number of times of teaching as physical load respectively. It is necessary to solve the load problem for utilization of interactive evolutionary learning. There are
four measures in respect to that problem. It is 1)thedesign of the improvement of the input interface to acomputer, 2)the improvement of the presentation interface on a computer, 3)speedup of EC convergence, and4)the fusion method with the usual EC which is not interactive learning.
In this research, we consider the following things as
instruction. First, direct operation of the robot by input
equipment is performed. Next, a rule is automaticallygenerated from the operation and the environment information at that time. In this framework, it is necessary
to perform neither comparison evaluation of many individuals, nor the input of an evaluation value like the conventional interactive evolutionary learning. Thereby, it
is expected physical and mental load is reduced sharply.
These are considered to be the improvement of the input interface for a computer, and the improvement of thepresentation interface from a computer in the frameworkof Interactive Evolutionary Computation.
Moreover, we will consider the case where interactive
learning is applied to real robot environment. When ateacher directs by operating a robot intuitively from input equipment (teaching), a rule is created automatically, and a robot learns autonomously when there are
no directions. We consider that this load problem is reduced by this method in the point that a system learns
autonomously, the point that a rule is automatically created by human’s intuitive instruction, and the point that
additional study can be performed anytime.

2.2

Timing of Teaching

We think that the timing of teaching is greatly concerned with the above-mentioned teacher’s load. However, since it depends on a system side for the timing
which instruction performs, in order to teach in accordance with the timing, human has to wait. Not to mention the experiment in a simulation, a teacher’s loadincreases further in the real environmental learning thatneeds more time for an experiment.
Compared with the above-mentioned general instruction study, we aim at teaching without recognizing thata teacher is teaching in IER. That is, a system willlearn by gaining operation of a teacher as instruction
information automatically only by a teacher operating
a robot and performing a task. This means that it leads
to reduction of an teacher’s load. We investigate theeﬀect by the timing of instruction, in order to realize
such instruction. The timing of the conventional instruction shall be divided into the timing of following
three instruction in this research.
• pre-teaching• passive teaching• anytime teaching

pre-teaching
Exploration in the framework of interactive evolutionary robotics learns from instruction information. Preteaching is the method of performing exploration by instruction at Teaching Mode beforehand, and performing
exploitation at Autonomous Behavior Mode.

passive teaching
We deﬁne passive teaching method as the method of directing teaching at the time of the demand of a system
to a user. Mishima and Asada el al. have improved that1097

the eﬃciency of learning gets worse by passive teaching for a gap (Cross Perceptual Aliasing) of the environmental recognition produced between a teacher anda learner. In study eﬃciency, passive teaching has little futility of teaching and is considered to be a good
method. However, the teacher has to be supervising until a system requires action. Moreover, since it does notknow when the timing comes, it is thought that a mentalload becomes large to the number of instruction.

anytime teaching
The problem how to treat the trade-oﬀ between exploration and exploitation is in one of the important topics
in reinforcement learning. In order to obtain many rewards, you have to choose the optimal action preferentially under the present value function. In this case, it
is said that the present knowledge about action value is
exploit(ed). However, the optimal policy based on thepresent value function is not necessarily really optimal
policy. In order to discover a better policy, it is necessary to raise the accuracy of the present value function.
Therefore, each action needs to be tried several times.
Moreover, in a dynamic environment, in order to acquire
the knowledge which is adapted for new environment, itis necessary to also try the action that was not good
before. It is referred to as exploring environment to try
new action, in order to ﬁnd a better policy. Since wecannot perform exploitation and exploration simultaneously, let both balance be a problem.
On the other hand, it is possible that a teacher gives
instructs to a robot at favorite timing. In this research,
this is called anytime teaching method. Seeing a robot
perform autonomous action, a teacher operates a robot
to favorite timing and makes a task. Thereby, teacher
can instruct to a robot being unconscious of teaching,
without worrying about whether he teaches by seeing
a robot’s action. Thereby, a teacher can teach without worrying about whether being conscious of teaching,whether it teaches, or not when he/she saw a learner’s
all actions. However, it is diﬃcult to include such speciﬁcation in a system side.

3
3.1

Teaching based on Interactive Evolutionary Robotics
Interactive Evolutionary Robotics

Interactive Evolutionary Robotics (IER) is a framework
aiming at performing eﬃcient real environmental robot
study using the evaluation capability of (Interactive
Evolutionary Computation(IEC)). IEC is the methodof including evaluation of human in the evaluation system of a system directly, and searching evolutionally.
Moreover, IER is an approach which designs a robot interactively using the evolutional calculation techniques,
such as a genetic algorithm, genetic programming, andan evolution strategy. The framework ﬁgure of IER isshown in Fig.1.
In general Evolutionary Robotics, it is consideredas the object of a design of a robot’s controller. ER
is designing it through evolution process by selection
pressure under an evaluation function rather than determines the detailed speciﬁcation in top-down. In IER,
evolution is promoted by gaining a rule by interaction
with a designer further. For this reason, while expecting
the high emergent of ER, it is possible to also expect thepliability and sensitivity of IEC in a complicated problem.

RobotTeacher

operator
joystick

Environment
robot

monitorsensorinformation "left"

ICS

evaluation,instruct

GP

0011

[P]

design, action

p
43
32
14
27
18
24

#011:01
11##:00#0##:11001#:01#0#1:11
1#01:10

GA

e
.01
.13
.05
.24
.02
.17

F
99
9
52
3
92
15

Match Set

0110001001000

SPC

DC

01

Prediction Array

Action Set[M]0001100101000#011:01#0##:11001#:01#0#1:111100010011110

43
14
27
18

.01 99.05 52.24 3.02 92

nil 42.5 nil 16.6

Figure 1: Interactive Evolutionary Robotics

43 .01 99
27 .24 3max

Evolutionary Computation
RGC#011:01
001#:01

discount

GA
RC

+

P

delay=1

Previous Action Set[A] -1Figure 2: Overview of Interactive Classiﬁer System
We think that the method in this framework is effective in the learning of an initial stage that must be
performed by trial and error. Moreover, we also expect
the eﬀect of obtaining a solution to the partial solutionthat cannot be solved only by human, through interaction between human and robot. Furthermore, since it is
not dependent on learning algorithm, this technique is
widely applicable to general evolution robotics.

3.2

Interactive Classiﬁer System

By including the interactive function of IEC in (Learning Classiﬁer System (LCS), ICS is the robot study
model that can also perform study by instruction in
addition to autonomous study. XCS[7] which Wilson
proposed is used for LCS that is study algorithm. XCS
adds the parameter that is what improved ZCS[8] and iscalled accuracy . Although the classiﬁer was generalized
by including #(don’t care symbol) in a conditional part,the classiﬁer system or ZCS of Holland were not able toperform it eﬀectively.
This originates in not having a mechanism for theclassiﬁer system itself advancing generalization appropriately, and the phenomenon in which the performanceof a system gets worse by the classiﬁer generalized too
much (overgeneral) has been reported[9]. In XCS, inorder to control the classiﬁer generalized especially too
much, not only the conventional intensity but accuracy
has determined the validity of a classiﬁer. This accuracy is calculated by the error of the reward received
as a result of performing a classiﬁer, and its prediction
value, and it is reported that a rule can be generalized
appropriately by this, without becoming common too
much[7]. The framework ﬁgure of the built system isshown in Fig.2.
ICS consists of a rule generation component (RGC),
a sensor processing component (SPC), a display component (DC) and a reinforcement component (RC). All ofthem are developed on Linux. It is described by the C
language and GTK+. Each module is explained below.
【RGC】 Rule Generation Component creates the ruleby instruction. A teacher operates it using input
equipment, looking at the information displayedon an interface in a robot. A sensor processing part(SPC) receives the operation history of a there,
and the sensor information of the robot at thattime, RGC creates a rule newly from it, and it
adds to a rule list. The creation procedure of a rule1098

was improved so that a rule could be created from
instruction information (the action to which operator operated the robot) on the basis of XCS[7].
1. ICS receives a robot’s sensor information X
and instruction information at from SPC.
2. Some classiﬁers that matched X is moved
from a group [P] to a match set [M]. ICS
turns regularly the Prediction value of classiﬁer which supports each act ai in [M] with a
Fitness value, and creates P (ai ). The value ofP (ai ) is put on Prediction Array, and the actof classiﬁer chosen by P (ai ) is chosen by act
selection methods. Act selection methods are
performed by deterministic selection method
or roulette wheel selection method.
3. If aj = at to compare act aj chosen by act selection methods and act at obtained by teaching, the action part of the rule which has aj
in an action part in [M] will be rewritten toat . A change will not be made if aj = at .
4. The action set [A] which consists of classiﬁers
in [M] which supports selected act aj is created. When act aj or at is sent to an eﬀect
machine, and in case of at , reward rteach is
given immediately. When there is no input ofat , remuneration rimm is returned from environment.
【RC】Reinforcement Component is a reinforcement
learning part in classiﬁer system. It learns by updating the parameter of classiﬁers chosen last time
step. When there is no operation of a teacher, arobot can act autonomously from the rule createdby then.
【DC】Display Component takes charge of the displayof the data processed by SPC. GTK+ is used fordevelopment of an interface. The developed interface is shown in Fig.3.
【SPC】 Sensor Processing Component performs processing of a robot’s various sensors and processingof teaching information. It is sent to DC and RGC
and the processed data is displayed and ICS creates classiﬁers from them.

sensor information
operator

robot

ICS

Teaching Mode

Autonomous Modeteaching

autonomous behavior

Figure 4: Teaching Mode and Autonomous ModeTeaching modeStep 4: If there is no rule belonging to the samecluster, it will add as a rule newly.
Figure 3: User Interface

Step 5: If there is a rule belonging to the samecluster, a strength value will be updated byreward.

At ﬁrst, a human operates robots with a joystickby referring to sensor information displayed on GUI interface, and the DC processes the information. Next,
the SPC gets interaction and sensor information. The
RGC make new rules from them and adds them into arule list. When nothing is input from the operator, amobile robot executes autonomous behaviors from interaction. Finally, the RC reinforces the classiﬁers by
updating their parameters in the actions that were previously executed.
ICS diﬀers from IEC in that operators do not haveto evaluate individuals each time. The operator canalways operate a mobile robot directly, and such direct
operation can take place of the ﬁtness evaluations ofeach individual in IEC. Therefore the operator can doteaching with less load, and can always do concentrative
additional learning for sub-tasks diﬃcult to achieve.

3.3




Autonomous behavior modeStep 1: The robot behaves by conforming tostored rules in Rule List.
Step 2: If the average of the number of the time
steps from GA of just before in a match set
exceeds a threshold, GA will be performed tothe match set.



3.4


pre-teachingStep 1: Prepare the robot’s state space.



1. A teacher directs action to state space.


passive teachingStep 2: It teaches depending on any of the procedure of the timing of three kinds of instruction they are.



Procedure of the timing of teaching

The timing of teaching has three timing described in
Chapter 2.2. Each procedure is shown below. Each is
performed in Step2: in teaching mode.

ICS performs two modes: a teaching mode and an autonomous behavior mode by turns. The procedures of
the two modes are shown in the following.

Step 3: A rule is created by an operator’s directions information and environmental information at the time.



Fig.4 shows the overview of a teaching mode and an
autonomous behavior mode.

Procedure of learningTeaching mode

1. Act A will be performed if there is effective action A to state space.
2. If there are no directions, directions
will be requested to a teacher.



1099





Q Q F
Q O O
O Q Q

O Q F
O Q O
O Q Q

Q Q G
O O Q
Q Q O

O Q G
Q Q O
O O O

O Q F
Q Q Q
Q Q O

Q O F
Q Q O
Q Q Q

Q O G
Q O O
O O O

Q O F
O O O
O Q O

O O F
O Q O
Q O Q

O O G
Q Q O
Q O Q

Q O G
Q O O
O Q O

Q O G
O O Q
Q Q O

Q O F
O Q Q
O O O

O O G
Q Q O
O Q O

O Q F
O Q Q
O O Q

O O G
Q Q O
O Q Q

O O F
O Q O
Q Q Q

Q Q F
O O O
O O Q

*

Table 1: Experimental Parameters
Parameters
number of problems in one experiment
number of experiments
maximum size of the population
number of teachingprobability to do crossoverprobability of mutating one bit

Value
5000
10
800
5, 10, 20, 50
0.8
0.04

* : Animat
F,G : Food
O,Q : Obstacle

Figure 5: Woods2 Environment
anytime teaching
1. To state space, if there are directions
from a teacher, it will perform.

teachings that have about 3 steps by one trial.
A setup of the number of instruction prepared
four setup with 5, 10, 20, and 50 instruction to
one problem. Action for about 3 steps is built by
one trial by instruction. When ten trial is per formed, the classiﬁer of about 30 will be created.
The experiment is asking for 10 trial deed andits average about each setup. The parameter ofan experiment is shown in Table 1.

2. If there are no directions, a robot will
perform exploration autonomously.


4
4.1

4.2

Experimental Results

In this work, we investigated performance (in av-

 erage steps to food) and system error (average

Experiment
Teaching eﬀects and Teacher’s Load

We test a preliminary experiment in order toevaluate the eﬀectiveness our ICS. This is a very
simple domain. We use Woods2 environment
which one of Wood-like environments[7] as anenvironment in the experience. It used as a testbed in several works based on classiﬁer system.
Fig.5 shows Woods2 environment. This environment is markovian multi-step problem. The
left and right edges of Woods2 are connected,
as are the top and bottom. Woods2 has two
kind of ”food” and two kind of ”rocks”. F and
G are the two kind of food, with sensor codes
110 and 111, respectively. O and Q are the twokind of objects, with sensor codes 010 and 011,
respectively. Blanks have sensor code 000. The
system, here regarded as an animat or artiﬁcial animal, is represented by *. To sense its
environment, * is capable of detecting the sensor codes of objects occupying the eight nearest
cells. The encoding of a classiﬁer is as follows.
A classiﬁer, for example, is the 24-bit string
000000000000000010010110. The left-hand threebits are always those due to the object occupying the cell directly north of *, with the remainder corresponding to cells proceeding clockwisearound it. The animat’s available actions consistof the eight one-step moves into adjacent cells,with the move directions similarly coded from 0
for north clockwise to 7 for northwest. If a cell isblank, * simply moves there. If the cell contains
food, * moves to the cell, ”eats” the food, andreceives a reward(rimm = 1000).
We compare our teaching method with Noteaching method by four settings of the number of teaching to investigate relationships withteaching eﬀect and load. The teaching operates
by timing as same as pre-teaching. We try 50

1100

absolute diﬀerence between the system prediction for the chosen action and P which is the sum
total of maximum system prediction and the current reward in Fig.2). Fig.6 shows the steps tothe foods. And, Fig.7 shows system error.
Especially in early stage of learning, ICS outperformed No-teaching system. Our teaching
method is twice as good as Non-Teaching method
in early stage(50 trials) in the steps to foods.
Moreover, there is no eﬀect on system error.
ICS improves the early time learning due to havebeen given the human-robot interaction in advance. Although eﬃciency becomes good about
the load of instruction so that the number of instruction goes up, it turns out that it is diﬃcult
to look for an appropriate point.
In order to investigate how much taught ruleis spread and used eﬀectively for a group, therate in which the rule taught into the group or
its posterity is contained was investigated for every number of problems. Here, the taught rule
expresses the rule made from instruction information, and all the children made by Genetic
Algorithms considering it as parents. The resultis shown in Fig.8. In order to teach by setup of
10 beforehand, the rate is not ﬁlled in the stage
to begin to 1%. However, if 500 problems are
exceeded, 80% of a group will be occupied. The
rule to which about 100% was taught in the stage
beyond 1000 problems occupies. Signs that thetaught rule is used very well and spreads in thegroup are known.

5

Conclusion

We proposed a fast learning method based on
ICS which enables a mobile robot to acquire autonomous behavior from interaction between human and robot. We evaluated the eﬃciency ofthis work by the experiments in the multi-steps
simulation environment.

25

Teach5Teach10
Teach20
Teach50
No-Teach

The Rate of The Taught Classifier

Steps to Food

20

1

15
10
5

0.8

0.6

0.4

0.2

0
0

0
0

50

100

150

200

250

300

350

400

1000

2000
3000
Problems

4000

5000

Problems

Figure 8: The Rate of The Taught Classiﬁers
Figure 6: Step to Food1

telligent Robots and Systems, pp. 1496–1501(1996)

Teach50
No-Teach

System Error

0.8[3] C. Mishima and M. Asada: Active Learningfrom Cross Perceptual Aliasing Caoused by
Direct Teaching; IEEE/RSJ International
Conference on Intelligent Robots and Systems, pp. 1420–1425 (1999)

0.6
0.4
0.2
0
0

50

100

150

200

250

300

350

400

[4] Y. Horiguchi and T. Sawaragi and G.
Akashi: Naturalistic Human-Robot Collaboration Based upon Mixed-Initiative Interactions in Teleoperating Environment;
IEEE International Conference on Systems,
Man, and Cybernetics, pp. 876–881 (2000)

Problems

Figure 7: System Error
ICS has two major characteristics. First, theICS generate initial individuals by teaching fromhuman-robot interaction. We can perform initial learning eﬃciently in this way. Second, auser can add new rules to operate a robot directly at any time during the course of teachingin ICS. Therefore the user can perform teaching
without much load, and can always do concentrative incremental learning for sub-tasks diﬃcult
to achieve.
About the timing of teaching, it is under construction at this time. In the near future, we will
make experiments on the three teaching timing
using real robot such as AIBO to inspect the effect of teaching and user’s load.

References[1] H. Asoh and Y. Motomura and I. Hara andS. Akaho and S. Hayamizu and T. Matsui: Combining probabilistic map and dialog for robust life-long oﬃfce navigation;
IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 807–812(1996)[2] H. Ishiguro and R. Sato and T. Ishida:
Robot Oriented State Space Construction;
IEEE/RSJ International Conference on In-

1101[5] T. Inamura, M. Inaba, and H. Inoue:
User Adaptation of Human-Robot Interaction Model based on Bayesian Network
and Introspection of Interaction Experience;
IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2139–2144(2000)[6] D. Katagami and S. Yamada: Interactive
Classiﬁer System for Real Robot Learning;
IEEE International Wortkshop on Robot andHuman Interaction, pp. 258–263 (2000)[7] S. W. Wilson: Classiﬁer ﬁtness based on accuracy; Evolutionary Computation, Vol. 3,
No. 2, pp. 149–175 (1995)[8] S. W. Wilson: ZCS: a zeroth order classiﬁer
system; Evolutionary Computation, Vol. 2,
pp. 1–18 (1994)[9] D. Cliﬀ and S. Ross: Adding temporary
memory to zcs; Adaptive Behavior, Vol. 3,
No. 2, pp. 101–150 (1994)[10] A. R. Cassandra and L. P. Kaelbling and M.
L. Littman: Acting Optimally in PartiallyObservable Stochastic Domains; 12th National Conference on Artificial Intelligence,Vol. 2, pp. 1023–1028 (1994)[11] S. W. Wilson and D. E. Goldberg: A critical review of classiﬁer systems; The Third
International Conference on Genetic Algorithms, pp. 244–255 (1989)