Real Robot Learning with Human Teaching
D. Katagami

S. Yamada

CISS, IGSSE, Tokyo Institute of Technology
4259 Nagatsuta, Midori-ku Yokohama 226-8502, JAPAN{katagami,yamada}@ymd.dis.titech.ac.jp
Abstract
In this paper, we describe a fast learning method for a mobile robot which acquires autonomous behaviors
from interaction between a human and a robot. We develop a behavior learning method ICS (Interactive
Classifier System) using evolutionary computation and a mobile robot is able to quickly learn rules so that ahuman operator can directly teach a physical robot. Also the ICS is a novel evolutionary robotics approach
using an adaptive classifier system to environmental changes. The ICS has two major characteristics forevolutionary robotics. For one thing, it can speedup learning by means of generating initial individuals fromhuman-robot interaction. For another, it is a kind of incremental learning methods which adds new acquired
rules to priori knowledge by teaching from human-robot interaction at any time.

1

Introduction

In previous robot learning studies, optimization of parameters has been applied to acquire suitable behaviors in a real environment. Also in most of such studies, a model of human evaluation has been used forvalidation of learned behaviors. However, since it is very diﬃcult to build human evaluation function andadjust parameters, a system hardly learns behavior intended by a human operator.
In contrast with modeling human evaluation analytically, we introduce another approach in which a system
learns suitable behavior using human direct evaluation without its modeling. Such an interactive methodwith Evolutionary Computation (EC) as a search algorithm is called Interactive EC [1][2], and a lot of studies
on it have been done thus far[3][4][6].
The most signiﬁcant issue of Interactive EC is how it reduces human teaching load. The human operator
needs to evaluate a lot of individuals at every generation, and this evaluation makes him/her so tired.
Specially in the interactive EC applied to robotics, the execution of behaviors by a robot signiﬁcantly costs
and a human operator can not endure such a boring task.
Additionally reinforcement learning has been applied to robot learning in a real environment[5]. Unfortunately the learning takes pretty much time to converge. Furthermore, when a robot hardly gets the ﬁrst
reward because of no priori knowledge, the learning convergence becomes far slower. Since most of the timethat are necessary for one time of action moreover is spent in processing time of sense system and action
system of a robot, the reduction of learning trials is necessary to speedup the learning.
In our study, we build an interactive EC framework which quickly learns rules with operation signal ofa robot by a human operator as teacher signal. Our objective is to make initial learning more eﬃcient andlearn the behaviors that a human operator intended through interaction with him/her. To the purpose, we
utilize a classiﬁer system as a learner because it is able to learn suitable behaviors by the small number oftrials, and also extend the classiﬁer system to be adaptive to a dynamic environment.
In this system, a human operator instructs a mobile robot while watching the information that a robot canacquire as sensor information and camera information of a robot shown on the screen top. In other words,
the operator acquire information from a viewpoint of a robot instead of a viewpoint of a designer. We callsuch a framework Interactive Classifier System (ICS), and Fig.2 shows its overview.
The characteristic points of ICS which diﬀerent from previous studies of LCS are to add frameworkusing human-robot interaction. Accordingly, we suppose that ICS can perfrom initial learning in a realenvironment and always can do conentrative incremental learning.

operator

Actual Environment

monitorrobot

joystickvibrationteaching

image
information
sensorinformation

ICS
ICS : Interactive Classifire System

actionFigure 1 Overview of Interactive Classiﬁer System

Operator performs teaching with joystick by direct operating a physical robot. the ICS inform operator
about robot’s state by a robot send a vibration signal of joystick to the ICS according to inside state.
Our ICS is signiﬁcantly diﬀerent from previous passive interactive techniques in which a human selects
good individuals from given ones. The ICS is an active interactive technique to learn suitable rules because
a human is allowed to actively operate it. This technique reduces the psychological load of an operator
and does learning without getting tired. Because a user must evaluate each aquwired rule at every steps inprevious wroks with EC. This is a problem because of the psychological load and a physical one. However
the user always add new rules by which operate a mobile robot directly at any time in ICS. For this reason,
this technique reduces the psychological load.
We investigate the eﬀectiveness of ICS by a task of wall-following and soccer game in which real robots
are used.

2

Classifier system(Learning) Classiﬁer System (CS, LCS) is a kind of method of Genetic Algoritm-Based Machine Learning(GBML). LCS consists in Performance Function based on Production System, Reinforcement Learning basedon Credit Assignment and Rule Generation Function based on GA.
In this study, we used XCS [7][8] as a classiﬁer system for ICS. XCS is a classiﬁer system in which eachclassiﬁer maintains a prediction of expected payoﬀ, but the classiﬁer’s ﬁtness is given by a measure of theprediction’s accuracy. These aspects of XCS result in its population tending to form a complete and accurate
mapping X × A → P form inputs and actoins to payoﬀ predictions.

3Interactive Classifier System

ICS consists of a rule generation component (RGC), a sensor processing component (SPC), a display
component (DC) and a reinforcement component (RC).
The rule generation component make a new classiﬁer from teaching by an operator. The SPC processes
each information of infrared sensors, optical sensors and CCD camera, and through it for the RGC. The
DC displays by GUI interface and processes the input from a joystick. Finally the RC performs learning by
updating parameters in ICS.
At ﬁrst, a human operates robots with a joystick by viewing sensor information displayed on GUI interface,
and the DC processes the information. Next, the SPC gets interaction and sensor information. The RGCmake a new rules from them and adds them into a rule list. When nothing is input from the operotor, amobile robot executes autonomous behaviors from interaction. Finally, the RC reinforces the classiﬁers by
updating their parameters in the actions which were previously executed.

Environment
operator
joystickrobot

monitorsensorinformation"left"

ICS
SPC

DC
01

[P]
p
43
32
14
27
18
24

#011:01
11##:00#0##:11001#:01#0#1:11
1#01:10

e
.01
.13
.05
.24
.02
.17

F
99
9
52
3
92
15

Match Set

Prediction Array

Action Set[M]#011:01#0##:11001#:01#0#1:11

RGC

43
14
27
18

.01 99.05 52.24 3.02 92

nil 42.5 nil 16.6

#011:01
001#:01

43 .01 99
27 .24 3

max
discount

GA
RC+

delay=1

Previous Action Set[A] -1Figure 2 Schematic illustration of ICS

ICS diﬀers from IEC in that operators have the load for evaluation of individuals at every time. However
the opeartor always can operate a mobile robot directly, and such direct operation of a robot is considered
for a human operator far less than the ﬁtness evaluation in IEC. Therefore the operator can do teaching
without the load, and always can do conentrative additional learning for sub-tasks diﬃcult to achievement.
Fig.1 shows overview of ICS.
3.1

Rule Generation Component (RGC)

In previous methods, the initial individuals are generated randomly and learn at an early stage throughtrial and error. Our method speedup the learning at an early stage by automatically generating the initialindividuals from interacton between a robot and a human operator. Whenever this component generates
a new classiﬁer, the population is scanned to see if the new classiﬁer has the same conditon and action as
any existing classiﬁer. If there is a classiﬁer with identical condition and action, the classiﬁer is incremented
by one. If there is no existing classiﬁer with identical condition and action, the new classiﬁer is added tothe population. Consequently, our method can subjectively evaluate as same as IEC which integrate humanevaluation into our system.
Fig.2 shows the schematic illustration of ICS.
3.2

Display Component (DC)As mentioned before, our system consists of the RGC which generates a new classiﬁer from human-robot
interaction, the SPC which processes infrared sensors and images from a CCD camera and the DC which has
GUI interface and processes input from a joystick. This system is developed with C langage and GTK+ on
Linux and using Video4Linux for image processing. Fig.3 shows developed GUI interface on DC.

Figure 3 User Interface

3.3

Sensor Processing Component (SPC)

We use the joystick for a SONY PlayStation via parallel port as a joystick on SPC for teaching by operator.
The speciﬁcation of the joystick is as followings.
• It has fourteen buttons and the system can use all buttons in degital mode.
• Moreover, the system can use two buttons in analog mode.
• It has two vibration modes.
Because of this two kinds of vibration, the system can communicate stimulus to an operator directly. To
take an example use vibration functions, an operator can feel directly robot’s sensations because the systemcommunicate stimulus by vibration when the mobile robot approach obstacles. Hence, we think the systemhave an eﬀect on acquisition of a reﬂective behavior and an instinctive one by teaching.
3.4

Renforcement Component (RC)the RC reinforces the classiﬁers by updating their parameters in the actions which were previously executed.

4

Some charactaristics of ICS

4.1

Initial individuals from human-robot interaction

In previous works of robot learning in a real environment, the learning takes pretty much time to converge
because of learning by trial and error. In this work, we assumed that this learning by trial and error is a
probrem in a real environment. However, it hardly prepare suitable priori knowledge in a environment inadvance. For this reason, the ICS generate initial individuals by teaching from human-robot interaction. We
can perform eﬃciently initial learning in this way.
4.2

Adding rules at any time

In previous wroks with EC, a user must evaluate each aquwired rule at every steps. This is a problem
because of a mental load and a physical one. In contrast with EC, the user always add new rules by which
operate a mobile robot directly at any time in ICS. Therefore the user can do teaching without the load,
and always can do conentrative incremental learning for sub-tasks diﬃcult to achievement.
4.3

Teaching method

ICS performs two modes: a teaching mode and an autonomous behavior mode by turns. The proceduresof the two modes are shown in the following.

sensor information
monitorrobot

operator

ICS

Teaching Mode :Autonomous Mode :teaching

autonomous behavior

Figure 4 Teaching Mode and Autonomous Mode

Teaching mode
Step1

Opeator have teaching to mobile robot by performing tasks practically.

Step2

ICS generate a new rule from human-robot interaction and sensor information.

Step3

If there is no existing classiﬁer with identical condition and action, the new classiﬁer is added to the
population.

Step4

If there is existing classiﬁer with identical condition and action, the existing classiﬁer is incremented
by one.

This procedure is as same as classiﬁer generation method of XCS called for Macroclassifiers[8].
Autonomous behaviour mode
Step1

The robot behaves by conforming to stored rules in Rule List.

Step2

If there is no rule to match input in Rule List, perform a new rule generated from any other rules.

Step3

The genetic algorithm acts on the match set after ﬁxed time steps.

In the autonomous behavior mode, the system ﬁnds a similar stored rule and executes it. If the system
don’t ﬁnd a similar rule, it generates a new rule integrated with any human-robot interaction. Fig.4 shows
the overview of a teaching mode and an autonomous behavior mode.
4.4

Mining and recycling of rules

ICS extract basic rules in each environment from acqueired rules by ICS. We use datamining tools to
realize. In contrast with the learning by trial and error, ICS can perfrom initial learning in a real environment
by recycling acquwired rules as initial rules acquwired rules in new niche envrinment.

5

Experiment

5.1

Environment settings

We test a preliminary experience in order to evaluate the eﬀectiveness our ICS. This is a very simple
domain. We use Woods2 environment which one of Wood-like environments as an environment in theexperience. It used as a test-bed in several works based on classiﬁer system. Fig.5 shows Woods2 environment.
This environment is markovian multi-step problem. The left and right edges of Woods2 are connected, as are

0

1

2

3

1

Q

Q

F

2

O

O

O

3

O

O

Q

4

29

0

Table 1 Experimental Parameters*
Parameters
number of problems in one experiment* : Animat
number of expriments
F,G : Food
O,Q : Obstacle maximum size of the population
number of teachingprobability to do crossoverprobability of mutating one bit

4

Value
5000
10
800
50
0.8
0.04

14

Figure 5 Woods2 Environment
25

1

Teach
No-Teach

0.8
System Error

Step to Food

20

Teach
No-Teach

15
10
5

0.6
0.4
0.2

0

0
0

50

100

150

200

250

Problems

Figure 6 Step to Food

300

350

400

0

50

100

150

200

250

300

350

400

Problems

Figure 7 System Error

the top and bottom. Woods2 has two kind of “food” and two kind of “rocks”.F and G are the two kind offood, with sensor codes 110 and 111, respectively. O and Q are the two kind of objects, with sensor codes 010
and 011, respectively. Blanks, denoted by “.”, have sensor code 000. The system, here regarded as an animat
or artiﬁcial animal, is represented by *. To sense its environment, * is capable of detecting the sensor codes ofobjects occupying the eight nearest cells. The animat’s available actions consist of the eight one-step moves
into adjacent cells, with the move directions similarly coded from 0 for north clockwise to 7 for north-west.
If a cell is blank, * simply moves there. If the cell contains food, * moves to the cell, “eats” the food, andreceives a reward.
We compair our teaching method with No-teaching method. The teaching operates by timing as same as
Covering method by Wilson. The timing is the case of no applicable classiﬁer in the environment or the caseof no classiﬁer have accuracy below the threshold. We try 50 teachings that have about 3 steps by one trial.
5.2

Experimental Results

In this work, we investigated the steps to the foods and system error. Table 1 shows experimental parameters.
Fig.6 shows the steps to the foods. And, Fig.7 shows system error.
Specially in early stage of learning, ICS outperformed No-teaching system. Our teaching method is twice
as good as Non-Teaching method in early stage(50 trials) in the steps to foods. However, there is no eﬀectFigure 8 Teaching based on sensor information

Figure 9 Teaching based on observation

on system error. ICS improves the early time learning due to have been given the human-robot interactionin advance.

6

Experiment in real world

6.1

Teaching method

Teaching based on sensor information An operator performs teaching to the mobile robot, not from aview point of overlooking the envronment, but teach as seeing displayed sensor information (in other words,
from robot’s view point) on GUI interface and images from a CCD camera. As a consequence, operator
performs teaching within the bounds of information the robot is able to acquire. Thus ICS generates
suitable robot program easily.
Fig.8 shows the appearance of teaching based on sensor information.
Teaching based on observation In contrast to teaching based on sensor information, an operator observes
the mobile robot directly from a view point of overlooking the envronment and considers next operation forteaching. Thus, the system can perform teaching save labor for tasks in need of a detailed teaching.
Fig.9 shows the appearance of teaching based on observation by human.
6.2

Tasks of experiments

We will make experiments with two kinds of tasks. As exmanle tasks, a wall following task with a single
learning robot and a simpliﬁed soccer game with multiple learning robots.
A wall following task In a wall following task, the purpose is to acquire a wall following behavior. Several
human operators perform teaching to a robot respectively. If each acquired robot program is same each other,
we consider it is a solution of a wall following task. We will make experiments with above two teaching andcompare with previous work.
A soccer game with multi robots In a simpliﬁed soccer game with multiple learning robots, ICS
extracts the diﬀerence of acquired rules because of cooperative and competitive behaviors.
Fig.10 shows overview of experimental environment.
we analyze rules acquired in experiment and consider it.

Figure 10 Experimental environment for soccer tasks

7

Conclusion

We proposed a fast learning method based on ICS for mobile robots which acquire autonomous behaviors
from experience of interaction between a human and a robot. We evaluated the eﬃciency of this work by theexperiments in the multi-steps environment.
ICS has two major characteristics. For one thing, the ICS generate initial individuals by teaching fromhuman-robot interaction. We can perform eﬃciently initial learning in this way. For another, the user always
add new rules by which operate a mobile robot directly at any time in ICS. Therefore the user can do teaching
without the load, and always can do conentrative incremental learning for sub-tasks diﬃcult to achievement.
Consequently, ICS can speedup robot learning in a real environment. Moreover, since ICS has simple
procedures and can be widely applied to something which have input ﬁxed devices from human, it has a wide
range of technological application. Such as the autonomous driving of a car and the automatic pilot of aplane. We expect that it evolve this interactive approach in future.

References[1]

R. Dawkins. The Blind Watchmaker. Longman, Essex, 1986.

[2]

R. Dawkins. The Evolution of Evolvability. In Langton, C. G., editor, Artiﬁcial Life, pages 201–220.
Addison-Wesley, 1989.

[3]

Y. Nakanishi. Capturing Preference into a Function Using Interactions with a Manual Evolutionary
Design Aid System. Genetic Programming, pages 133–140, 1996.

[4]

M. Ohsaki, H. Takagi, and T. Ingu. Methods to Reduce the Human Burden of Interactive Evolutionary
Computation. Asian Fuzzy System Symposium (AFSS’98), pages 495–500, 1998.

[5]

E. Uchibe, M. Asada, and K. Hosoda. Behavior coordination for a mobile robot using modular reinforcement learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems 1996
(IROS96), pages 1329–1336, 1996.

[6]

T. Unemi. A Design of Multi-Field User Interface for Sumulated Brdding. Asian Fuzzy System Symposium (AFSS’98), pages 489–494, 1998.

[7]

S.W. Wilson. ZCS: a zeroth order classiﬁer system. Evolutionary Computation, 2:1–18, 1994.

[8]

S.W. Wilson. Classiﬁer ﬁtness based on accuracy. Evolutionary Computation, 3(2):149–175, 1995.