Active Teaching for an Interactive Learning Robot
D. Katagami

S. YamadaTokyo Institute of Technology
4259Nagatsuta midori-ku
Yokohama, 226-8502, Japankatagami@ntt.dis.titech.ac.jp
Abstract
We have proposed a fast learning method that enables a mobile robot to acquires autonomous behaviors from interaction between human and robot. In
this research we develop a behavior learning method
ICS (Interactive Classifier System) using interactive
evolutionary computation considering an operator’steaching cost. As a result, a mobile robot is able toquickly learn rules by directly teaching from an operator. ICS is a novel evolutionary robotics approach
using classifier system. In this paper, we investigate teacher’s physical and mental load and proposed ateaching method based on timing of instruction using
ICS.

1

Introduction

Some researches of the approach using interaction
with the human who exists in environment has been
carried out. Particularly for the robots that do nothave a priori knowledge or commit trial and error in
the initial stage, human instruction is the very effective acquisition technique of autonomous behavior.
However, in a certain level of autonomous robot, it
is not necessary to follow instruction from human all
the time. In the stage which does not need instruction, robot should demonstrate its autonomy basedon the instruction rules stored by interaction with human without putting a burden on human. Therefore,
we need to the technique of establishing a robot’s autonomy from through interaction between human anda robot is required.
Our purpose is realizing a robot’s autonomy by receiving the instruction information as a suitable act
from human, and gaining act rules evolutionally withthe state recognition which can solve a task. We callsuch a framework Interactive Evolutionary Robotics(IER). In this paper, we propose Active Teaching
method taught by timing regarding a teacher’s cognitive load in IER. We compared it with previous teaching methods by simulation experiment.

National Institute of Informatics
2-1-2 Hitotsubashi Chiyoda-ku
Tokyo, 101-8430, Japanseiji@nii.ac.jp
2

Related Literature

Asoh et al.[1] proposed the framework that a mobile robot built the map information of the unknown
environment, called Jijo-2 which performs a communication by voice conversation with human. However,it doesn’t get the behavior of the robot by the interaction through human and a robot. Ishiguro et al.
[6] built the state space of the mobile robot by reinforcement learning. However, it is learning by using asa sample action that the introduction human taught.
After that, a robot only builds an internal state andthere is no interaction with human. Horiguchi et al.
[4] used the idea of the mutual leadership pattern interaction as the design of the interaction of the robot
with the human and realized the cooperation behavior of the automation process of a mobile robot andhuman operations by using power feedback. However,
the result of learning didn’t reﬂected on the behavior
acquisition of the robot. Inamura et al. [5] indicate
acquirement behavior of a robot using Bayesian Network based on a dialog with a user. It is diﬀerent
from our technique to get behavior gradually by theevolutionary computation technique.

3

Interactive Teaching

3.1

Teacher’s LoadIn this research, in order to measure a teacher’sload simply, it divides into mental load and physicalload. We consider the timing of teaching as mentalload and the number of times of teaching as physical load respectively. Generally, in interactive evolutionary learning, the more it is taught, the better the
performance is. However, human labor is not unlimited. It is clear that it is trade-oﬀ like it is better as
instruction cost lowers. Human’s labor has a limit incooperating with a machine without tiredness, carrying out comparison evaluation of many individuals (orrules) for every generation, and inputting an evaluation value. This has been a serious practical problem.
Moreover, as the second problem, the number of individuals and the number of search generations must be
lessened as compared with the usual EC search in order to reduce physical and mental load in case humanevaluates individuals. It makes convergence worse. As

a result, it is diﬃcult to reduce the number of timesof teaching.
On the other hand, IER consider the following
things as instruction. First, direct operation of therobot by input equipment is performed. Next, a ruleis automatically generated from the operation and theenvironment information at that time. In this framework, it is necessary to perform neither comparison
evaluation of many individuals, nor the input of anevaluation value like the conventional interactive evolutionary learning. Thereby, it is expected physicaland mental load is reduced sharply.
Moreover, we will consider the case where interactive learning is applied to real robot environment.
When a teacher directs by operating a robot intuitively from input equipment (teaching), a rule is created automatically, and a robot learns autonomously
when there are no directions. We consider that thisload problem is reduced by this method in the pointthat a system learns autonomously, the point that arule is automatically created by human’s intuitive instruction, and the point that additional study can be
performed anytime.

evaluation,instructdesign, action
GP

Timing of Teaching

It is very diﬃcult as above-mentioned to reduce thenumber of teaching. Then, in this research, in orderto reduce an informer’s cognitive load, we attention
to the timing of teaching. We think that the timing of teaching is greatly concerned with the abovementioned teacher’s load. Generally, timing of teaching is performed beforehand (Oﬀ-line Teaching), or
has much what is performed at the time of the demandof a system (Passive Teaching). Since these techniques
have left the timing which instruction performs to thesystem side, in order to teach, a man side must standby. Not to mention the experiment in a simulation, ateacher’s load increases further in the real environmental learning that needs more time for an experiment.
Then, we propose the following Active Teaching
methods. We conduct the experiment, which measurescognitive load as compared with the conventional Oﬀline Teaching method and Passive Teaching method.
And we compare them by psychological evaluation.
Each technique is explained below.
3.2.1

Active Teaching

In this teaching method, it is possible that a teacher
gives instructs to a robot at favorite timing. In thisresearch, this is called Active Teaching method. Seeing a robot perform autonomous action, a teacher operates a robot to favorite timing and makes a task.
Thereby, teacher can instruct to a robot being unconscious of teaching, without worrying about whether heteaches by seeing a robot’s action. Thereby, a teacher
can teach without worrying about whether being conscious of teaching, whether it teaches, or not when
he/she saw a learner’s all actions. However, it is diﬃcult to include such speciﬁcation in a system side.

GA
0110001001000
0001100101000
1100010011110

Evolutionary Computation

Figure 1

3.2.2

Interactive Evolutionary Robotics

Oﬀ-line TeachingOﬀ-line teaching is the method of performing exploration by instruction at Teaching Mode beforehand,and performing exploitation at Autonomous Behavior
Mode.
3.2.3

3.2

Robot

Teacher

Passive Teaching

We deﬁne passive teaching method as the methodof directing teaching at the time of the demand of asystem to a user. Mishima and Asada el al. have
improved that the eﬃciency of learning gets worse by
Passive Teaching for a gap (Cross Perceptual Aliasing)of the environmental recognition produced between ateacher and a learner [8]. In study eﬃciency, Passive
Teaching has little futility of teaching and is considered to be a good method. However, the teacher has tobe supervising until a system requires action. Moreover, since it does not know when the timing comes,
it is thought that a mental load becomes large to thenumber of instruction.

4

Teaching based on Interactive Evolutionary Robotics

4.1

Interactive Evolutionary Robotics

Interactive Evolutionary Robotics (IER) is a framework aiming at performing eﬃcient real environmental
robot study using the evaluation capability of Interactive Evolutionary Computation (IEC). IEC is themethod of including evaluation of human in the evaluation system of a system directly, and searching evolutionally. Moreover, IER is an approach which designs
a robot interactively using the evolutional calculation
techniques, such as a genetic algorithm, genetic programming, and an evolution strategy. The framework
ﬁgure of IER is shown in Fig.1.
We think that the method in this framework is effective in the learning of an initial stage that must be
performed by trial and error. Moreover, we also expect
the eﬀect of obtaining a solution to the partial solutionthat cannot be solved only by human, through interaction between human and robot. Furthermore, since it

is not dependent on learning algorithm, this technique
is widely applicable to general evolution robotics.

operator
joystick

Environment
robot

monitorsensorinformation "left"

4.2

The XCS Classiﬁer System

Classifers in XCS have three main parameters: (1)the prediction p, which estimates the payoﬀ that thesystem expects if the classiﬁer is used; (2) the prediction error , which estimates the error of the prediction
p; and (3) the ﬁtness F , which estimates the accuracyof the payoﬀ prediction given by p.
On each time step, the system input is used to build
a matchset [M] containing the classiﬁers in the population whose condition part matches the current sensory
inputs. If the match set does not contain any classiﬁers, a new classiﬁer which matches the current inputs
is created through covering. For each possible action
ai in [M], a systempredictionP (ai ) is computed as theﬁtness weighted average of the predictions of classiﬁers
which advocate action ai is performed. Action selection can be deterministic, i.e. the action with thehighest system prediction is chosen, or probabilistic,i.e. the action is chosen with a certain probability
among the possible action.
Classiﬁers in [M] which advocate the selected action
form the current actionset[A]. The selected actionis then performed in the environment, and a scalar
reward r is returned to the system together with a
new input conﬁguration.
Classiﬁer parameters are updated on each timestep. The updates occur in the action set [A]−i fromthe previous time-step. First, a Q-learning-like payoﬀ P is computed: P = r−1 + γ maxa P (a), wherer−1 is the reward on the previous time-step, P (a) are
the system predictions for the current time-step, andγ is a discoutf actor(0 ≤ γ < 1). Then, each classiﬁer in [A]−1 , is updated as follows. The prediction p
is updated using the W idrow − Hof f deltarule withlearning rate β(0 ≤ β ≤ 1) : p ← p + β(P − p).
The prediction error  is updated with the formula: ←  + β(|P − p| − ). The ﬁtness update is slightly
more complex. Initially, the prediction error is used
to calculate the accuracyκ of each classiﬁer as κ = α(/0 )−n f or  > 0 , else κ = 1. Then, each classiﬁer’sreactiveaccuracyκ is computed: κ = κ/ [A]−1 κ.
Finally the ﬁtnesses are adjusted: F ← F + β(κ − F ).
The genetic algorithm is applied to [A]−1 , though
not usually on every time-step. It selects two classiﬁers
with probability proportional to their ﬁtnesses, copies
them, and with probability χ performs crossover on
the copies; then, with probability µ it mutates eachallele. The resulting oﬀspring are inserted into the
population and two classiﬁers are deleted See (Wilson95) [10].
4.3Interactive Classiﬁer System

We have so far developed learning system Interactive Classiﬁer System[7] using XCS based on IER. ICS
is the robot study model that can also perform study
by instruction in addition to autonomous study. It

ICS
0011

[P]
p
43
32
14
27
18
24

#011:01
11##:00#0##:11001#:01#0#1:11
1#01:10

e
.01
.13
.05
.24
.02
.17

F
99
9
52
3
92
15

Match Set

SPC

DC

01

Prediction Array

Action Set[M]#011:01#0##:11001#:01#0#1:11

43
14
27
18

RGC

.01 99.05 52.24 3.02 92

nil 42.5 nil 16.6

43 .01 99
27 .24 3

max
discount

GA
RC

Figure 2#011:01
001#:01+

P

delay=1

Previous Action Set[A] -1

Overview of Interactive Classifier System

Figure 3

User Interface

included the interactive function of IEC in (Learning Classiﬁer System (LCS). ICS uses the abovementioned XCS which is one of the LCS as study algorithm. The framework ﬁgure of the built system isshown in Fig.2.
ICS consists of a rule generation component (RGC),
a sensor processing component (SPC), a display component (DC) and a reinforcement component (RC).
Each module is explained below.
【RGC】 Rule Generation Component creates therule by instruction. A teacher operates it using
an input equipment, looking at the information
displayed on an interface in a robot. A sensor
processing part (SPC) receives the operation history of a there, and the sensor information of therobot at that time, RGC creates a rule newly fromit, and it adds to a rule list. The creation procedure of a rule was improved so that a rule could
be created from instruction information (the action to which operator operated the robot) on thebasis of XCS[10].

4. If there is no rule belonging to the same cluster,
it will add as a rule newly.
5. If there is a rule belonging to the same cluster, astrength value will be updated by reward.

1. ICS receives a robot’s sensor information X
and instruction information at from SPC.
2. Some classiﬁers that matched X is moved
from a group [P] to a match set [M]. ICS
turns regularly the Prediction value of classiﬁer which supports each act ai in [M] witha Fitness value, and creates P (ai ). The valueof P (ai ) is put on Prediction Array, and theact of classiﬁer chosen by P (ai ) is chosen byact selection methods. Act selection methods are performed by deterministic selection
method or roulette wheel selection method.
3. If aj = at to compare act aj chosen by act
selection methods and act at obtained byteaching, the action part of the rule which
has aj in an action part in [M] will be rewritten to at . A change will not be made if
aj = at .
4. The action set [A] which consists of classiﬁers
in [M] which supports selected act aj is created. When act aj or at is sent to an eﬀect
machine, and in case of at , reward rteach is
given immediately. When there is no input
of at , remuneration rimm is returned fromenvironment.

Autonomous behavior mode

【RC】 Reinforcement Component is a reinforcement learning part in classiﬁer system. It learns
by updating the parameter of classiﬁers chosenlast time step. When there is no operation of ateacher, a robot can act autonomously from therule created by then.
【DC】 Display Component takes charge of the display of the data processed by SPC. The developed
interface is shown in Fig.3.
【SPC】 Sensor Processing Component performs
processing of a robot’s various sensors and processing of teaching information. It is sent to DC
and RGC and the processed data is displayed andICS creates classiﬁers from them.

4.4

Procedure of Learning

ICS performs two modes: a teaching mode and an
autonomous behavior mode by turns. The proceduresof the two modes are shown in the following.
Teaching mode
1. Prepare the robot’s state space.
2. It teaches depending on any of the procedure of
the timing of three kinds of instruction they are.
3. An operator creates a rule by instruction information and environmental information at the time.

1. The robot behaves by conforming to stored rulesin Rule List.
2. If the average of the number of the time steps
from GA of just before in a match set exceeds athreshold, GA will be performed to the match set.
4.5

Procedure of the Timing of Teaching

The timing of teaching has three timing describedin Chapter 3.2. Each procedure is shown below. Eachis performed in Step 2 in teaching mode.
Oﬀ-line Teaching
1. A teacher directs action to state space.
Passive Teaching
1. Act A will be performed if there is eﬀective action
A to state space.
2. If there are no directions, directions will be requested to a teacher.
Active Teaching
1. To state space, if there are directions from ateacher, it will perform.
2. If there are no directions, a robot will perform
exploration autonomously.

5

Experiment

5.1

Experimental Settings

We test a preliminary experiment to evaluate theeﬀectiveness our ICS. This is a very simple domain.
We use Woods2 environment which one of Wood-likeenvironments[10] as an environment in the experience.
It used as a test-bed in several works based on classiﬁer system. Fig.4 shows Woods2 environment. This
environment is markovian multi-step problem. The
left and right edges of Woods2 are connected, as are
the top and bottom. Woods2 has two kind of ”food”
and two kind of ”rocks”. F and G are the two kind offood, with sensor codes 110 and 111, respectively. O
and Q are the two kind of objects, with sensor codes
010 and 011, respectively. Blanks have sensor code
000. The system, here regarded as an animat or artiﬁcial animal, is represented by *. To sense its environment, * is capable of detecting the sensor codes ofobjects occupying the eight nearest cells. The encoding of a classiﬁer is as follows. A classiﬁer, for example,
is the 24-bit string 000000000000000010010110. The

Q Q F
Q O O
O Q Q

O Q F
O Q O
O Q Q

Q Q G
O O Q
Q Q O

O Q G
Q Q O
O O O

O Q F
Q Q Q
Q Q O

Q O F
Q Q O
Q Q Q

Q O G
Q O O
O O O

Q O F
O O O
O Q O

O O F
O Q O
Q O Q

O O G
Q Q O
Q O Q

Q O G
Q O O
O Q O

Q O G
O O Q
Q Q O

Q O F
O Q Q
O O O

O O G
Q Q O
O Q O

O Q F
O Q Q
O O Q

O O G
Q Q O
O Q Q

O O F
O Q O
Q Q Q

Q Q F
O O O
O O Q

*

* : Animat
F,G : Food
O,Q : Obstacle

Figure 4

Woods2 Environment
Figure 5

Step to Food

Figure 6

Population Size

left-hand three bits are always those due to the object
occupying the cell directly north of *, with the remainder corresponding to cells proceeding clockwise around
it. The animat’s available actions consist of the eight
one-step moves into adjacent cells, with the move directions similarly coded from 0 for north clockwise to 7
for northwest. If a cell is blank, * simply moves there.
If the cell contains food, * moves to the cell, ”eats” thefood, and receives a reward(rimm = 1000). ICS used a
population size, N, of 800 classiﬁers. Parameters wereset as follows: α = 0.1, β = 0.2, γ = 0.95, θ = 25,
0 = 0.01, χ = 0.8 and µ = 0.04
5.2

Experiment Description

We conducted the comparison experiment with Active Teaching, Passive Teaching or Oﬀ-line Teaching.
It is one trial, when it arrives at the goal or 50step
movement is carried out. Seven graduate students
were experimented on the subject by considering 50
trial as one experiment.
The primary task in the experiments is to instruct
an agent through numeric keypad. In experiments involving human cognitive load, experiment participants
are sometimes asked to perform a secondary task (ortasks) as they perform a primary task[3]. In our experiment, subjects must solve two digit addition problems
while performing the agent instruction task.
5.3

Experimental Results

Primary Task EﬀectivenessIn this work, we investigated performance in average steps to food (Step to Food) and average of generated population size (Population Size) Fig.5 shows
the steps to the foods. And, Fig.6 shows Population
Size.
Each teaching technique showed performance better than Auto mode in the initial stage. However,since practice is insuﬃcient, the error by instruction
error has arisen. Performance hardly changed about
each teaching method.
In having no instruction (Auto), the number ofrules is stabilized about 600. However, in each teaching method, when man teaches, directions are limited

and the number of rules become about 200. Whenteaching by the Active Teaching method, the numberof rules is increasing to about 400. This is a very interesting result. In order to teach by judgment of an
given occasion in Active, it is thought that the wayof instruction is not ﬁxed and the number of rules
increases. Since the contents of the rules were unveriﬁable this time, it is necessary to analyze in detailfrom now on.
Secondary Task EﬀectivenessThe experiment result in the sub task which solves
the addition problem of 2 ﬁgures is shown below. Fig.7
shows the number of answers. And, Fig.8 shows thenumber of incorrect answers. The number of problems
is hardly solved in Passive Teaching. Since cognitive
load is high, it is shown that time to perform secondary task was hardly able to be taken. Since thetiming which teaches in oﬀ-line teaching was decided,
it could concentrate on each task and many problems
have solved. It can be said that many problems have
solved similarly in active teaching.
When we investigated the number of incorrect answers, it turns out that the problem of two questions
is mistaken among 76 questions on an average in oﬀline. Although speed increases by solving a problem

Figure 7

Figure 8

The number of Answers

The number of Incorrect Answers

continuously, it turns out that a mistake also increases.
Usability
Based on evaluation of usability, seven steps of thequestionnaire survey was conducted about Eﬀectiveness, Eﬃciency, and Satisfaction after the experiment,
respectively. Fig.9 shows the questionnaire of usability.
In all items, evaluation of Active was good andthere was statistically signiﬁcant (p < 0.05) in Eﬃciency.

6

Conclusion

We proposed a Active Teaching method regarding
for teacher’s cognitive load when a teacher instruct a
mobile robot to perform a simulation task. We evaluated the eﬃciency of this method by the primary task
in the multi-steps simulation environment and the secondary task involving human cognitive load. The proposed method had low cognitive load, and was eﬀective in eﬃciency.

References[1] H. Asoh and Y. Motomura and I. Hara and S.
Akaho and S. Hayamizu and T. Matsui: Combining probabilistic map and dialog for robust
life-long oﬃfce navigation; IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 807–812 (1996)

Figure 9

Usability[2] A. R. Cassandra and L. P. Kaelbling and M. L.
Littman: Acting Optimally in Partially Observable Stochastic Domains; 12th National Conference on Artificial Intelligence, Vol. 2, pp. 1023–
1028 (1994)[3] J. Crandall and M. A. Goodrich: Characterizing Eﬃciency of Human Robot Interaction:
A Case Study of Shared-Control Teleoperation;
IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1290–1295 (2002)[4] Y. Horiguchi and T. Sawaragi and G. Akashi:
Naturalistic Human-Robot Collaboration Basedupon Mixed-Initiative Interactions in Teleoperating Environment; IEEE International Conference on Systems, Man, and Cybernetics,
pp. 876–881 (2000)[5] T. Inamura, M. Inaba, and H. Inoue: User
Adaptation of Human-Robot Interaction Model
based on Bayesian Network and Introspectionof Interaction Experience; IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2139–2144 (2000)[6] H. Ishiguro and R. Sato and T. Ishida: Robot
Oriented State Space Construction; IEEE/RSJ
International Conference on Intelligent Robots
and Systems, pp. 1496–1501 (1996)[7] D. Katagami and S. Yamada: Interactive Classiﬁer System for Real Robot Learning; IEEE
International Workshop on Robot and Human
Interaction, pp. 258–263 (2000)[8] C. Mishima and M. Asada: Active Learning from Cross Perceptual Aliasing Caoused
by Direct Teaching; IEEE/RSJ International
Conference on Intelligent Robots and Systems,
pp. 1420–1425 (1999)[9] S. W. Wilson and D. E. Goldberg: A critical review of classiﬁer systems; The Third International Conference on Genetic Algorithms,
pp. 244–255 (1989)[10] S. W. Wilson: Classiﬁer ﬁtness based on accuracy; Evolutionary Computation, Vol. 3, No. 2,
pp. 149–175 (1995)