STNS-R: A LEARNING METHOD FOR SEAMLESS
TRANSPLANTATIONFROM A VIRTUAL AGENT TO A PHYSICAL ROBOTATSUSHI UENO*, HIROAKI SOEDA"', HIDEAKI TAKEDA** * and MASATSUGU KIDODE** Graduate School of Information Science, Nara Institute of Science and Technology, Ikoma, Nara, Japan 630-0101** Intelligent Systems Research Division, National Institute of Informatics, Tokyo, Japan 101-8430

Abstract
In this paper, we are concerned with the problem
of how a physical robot can get an appropriate internal representation to its task and environment. Learning from experience is effective for the problem, but
it is very time-consuming to learn a representationfrom the beginning in a real environment. On theother hand, the representation learned only in a simulated environment has the risk of not serving the purpose in a real environment because of the uncertainty
in sensors, actuators, and the environment. In order tohave the best of both worlds, it is effective to transplant the learned state representation of a virtual
agent to a physical robot. For this purpose, we improved our developed incremental learning architecture for use in the real environment and developed a
new architecture, called STNS-R. In this architecture,
inappropriate negative instances caused by uncertainties are found on the basis of the distribution of instances and removed in order to correct the distorted
shapes of the states. The effectiveness of STNS-R isshown in the experimental results.

Keywords
Categorical learning, uncertainty in the real environment, reactive agent, reinforcement learning.

1 Introduction
Machine learning is a promising and necessary
method to realize an autonomous agent because it
does not need a priori knowledge. Since most oflearning methods require many trials to learn thegiven environment, they are usually performed in asimulated environment. It means that they are notapplicable to robots in a real world. We need aseamless learning method that is applicable either ina simulated or in a real environment, i.e., the samealgorithm is applicable and acquired knowledge isvalid in both environments.

' Currently, he Is with H.Q.S CORPORATION.
0-7803-7087-2/01/$10.000 2001 IEEE

371

A physical robot must face various situations andcope with a huge amount of information. Learningfrom experience is effective for such a robot in
acquiring an appropriate state representation for thecurrent task. It is, however, very time-consuming tolearn a representation from the beginning in a realenvironment. On the other hand, the state representation learned only in a simulated environment has therisk of not serving the purpose in a real environment
because of the uncertainty in sensors, actuators, andthe environment. In order to have the best of bothworlds, it is effective to transplant the learned state
representation of a virtual agent to a physical robot.
Incremental learning methods are suitable forthis type of learning task because they can adapt
seamlessly to changes of the environment. In OUT
previous work [ 1, 21, we proposed Situation Transition Network System (SnVS), an architecture that
performs categorical learning and behavioral learning
incrementally in parallel with task execution. In categorical learning, it makes a state representation andmodifies it according to the results of behaviors. Simulation results showed that it could learn efficiently
and adapt to unexpected changes of the environment
such as sensor trouble or actuator trouble.
For a preparatory experiment, we tested our
method with a physical robot that started to learn
with a STNS learned enough in a simulated environment. In the experiment, it could maintain the state
representation and behaviors, but its performance got
worse. Then, we improved STNS, and developed a
new architecture by which a state representation
learned in a simulated environment can be transplanted seamlessly and adapted to a physical robot.
This paper proposes the improved STNS for the realworld, called SWS-R.
The remainder of this paper is structured as follows: In the next section, we review our previous
work, including explanations for the research area wetackled. Next, we show the seamless learning method
to learn an appropriate state representation either in asimulated or in a real environment, and finally, weshow the experimental results and give a discussion
with concluding remarks.

2

The Domain and Our Previous Work

2.1 Categorical Learning in Reactive Agents

Physical robots are usually designed with fixedcategories for recognizing their environments. However, human-designed categories are not always suitable for them. Learning the categories from experiences is effective in adapting to the environment.
We think that categories should be created on thecriterion how well the robot can act. Several methods
have been proposed which segment the state space
enclosing input vectors from which the robot reaches
the same reward by the same sequence of behaviors
in the same state (segmerztation-based-on-behaviors)[1.-6]. This policy is suitable for reactive agents thatassign one feasible behavior to each state because input vectors at which the robot should take the same
behavior put together into the same state by this policy. Thus, this policy makes highly abstracted state
representation specialized to the current task.
There are four previous works on this policy except for ours. Asada et al. [ 3 ] proposed a method that
divides the state space by hyperellipsoids that enclose input vectors from which the robot achieves thegoal or already acquired state by a variable sequenceof one kind action primitive. Albus et al. [4] proposeda method that divides the state space using recursively a standard clustering algorithm. Yairi et al. [ 5 ]proposed a method that segments the state space
using Bayesian classifier. Ishiguro et al. [6] proposeda method that divides states by hyper-planes in which
the robot receives different rewards (or delayed rewards) when executing the same action.
In OUT previous work [l, 21, we developed thistype of categorical learning to a new architecture,
STNS, which performs segmentation-based-on-behaviors by incremental learning: the system extracts
states and maintain them while executing the task on
the basis of experiences of taskexecuting behaviors.
This type of incremental learning contributes to autonomy, adaptability and efficiency of the learning
system, and it is suitable for agents that continue task
execution without breaking down against changes ofthe environment. In the next section, we explain t h sarchitecture in outline.

Fig. 1: The Structure of STNS
plan on the STN, and at last activates a behavior
module according to the plan.
Categorical learning and behavioral learning are
performed on the data that consists of the input, thecorresponding situation, the selected behavior, andthe received reward. This data puts into a history
database. It keeps the data for a fixed period anddeletes it after that. Therefore, STNS can adapt flexibly to the changes of environment and the internal
representation without being misguided by outdated
data.
In categorical learning, the system constructs astate representation and maintains it by segmentationbased-on-behaviors policy. Reinforcement learningof a behavior policy is performed on this state representation: the system constructs ag MDP (Markov
Decision Problem) model of the environment which
consists of the transition probabilities between situations and the expectations of immediate rewards accompanying transitions by the maximum likelihood
estimation. It decides an appropriate behavior by thep o k y iteration algorithm [7] and planning on themodel in order to maximize discounted sum of rewards received over a period of time. In each behavior step, the system adjusts the shapes of situations in
categorical learning and modifies the MDP model inbehavioral learning reflecting the last data. If big
modification is needed, the system extracts or eliminates situations in categorical learning and modifies
the MDP model reflecting the new state representation in behavioral learning on the data in the history
database. In this way, these two learning processes
are performed in parallel while executing the task.
In the next section, we explain the state representation used in STNS. For further details of the wholearchitecture, please see [ l ] or [ 2 ] .

~2.2 STNS

2.3 State Representation in STNSAs shown in Fig. 1, STNS consists of a situation
classifier, a situation transition network (STN), andseveral behavior modules. In each behavior step, thesystem identifies the current situation (or state) where
the xuirent input should be included, then makes a

STNS segments the state space into some situations. In STNS, a situation is defined as a set of input
vectors from which the system can meet with the specific result by the specific behavior. The specific behavior is called the condition behavior of the situa-372

tion. The specific results are divided into two types,
i.e., R-situation and T-situation. In a situation basedon immediate rewards called R-Situation, the result is
to receive a specific big reward (called the goal reward). In a situation based on situation transitions
called T-Situation, the result is to transit to a specificsituation (called the parent situation). If every chainof T-Situations is anchored to an R-Situation, every
situation is guaranteed to lead to a goal reward by thesame sequence of behaviors.
In incremental learning, the system should be
able to decide rough shapes of situations from a limited amount of data, and to decide finer shapes asdata increase. For this purpose, we contrived thebitten h-vper-ellipsoidrepresentation. In this representation, each situation is shaped by the positive instances and the negative instances that are decided
based on the definition of the situation. As shown in
Fig. 2, this representation is a mixture of macroscopic
recognition and microscopic recognition. In macroscopic recognition, the boundary of a situation is a
contour of Mahalm-obis' distance from the populationof the positive instances. This boundary forms ahyperellipsoid. This recognition is quick and roughthat can be formed even from very few data. Micro-

3.2 Results

First, we made a virtual agent with STNS and let
it learn enough in a simulated envlronment corresponding to the real environment. Then, we transplanted the well-learned STNS to a physical robot
and let it continue learmng in the real environment.
This procedure was planned 111 consideration of practical use where learmng in a real environment is veryt i m e c o n s u n g and should be transferred to a simulated environment as much as possible. Fig. 5a shows
the state space of the transplanted STNS after 2000
behavior steps learmng 111 the simulated environment.
Learning converged thoroughly until then. Relearning in the real environment was performed until 1000
behavior steps. We repeated thls experiment fiveFig. 4: Our Physical Robottimes. Table 1 shows the average numbers of behaviors that are needed to reach the goal. All numbers of
behaviors to the goal was averaged in each experiRecognilion on (he bitlen hyper-ellipsoid represenlalion

Fig. 2: The Bitten Hyper-Ellipsoid Representation

PemptoaI Jnput (2dimcmlanal):
The real-valuzd (% y) coordmdes of Ute center of the god on lhe coardinafe sy61em fixed lo the robot
The angm IS setllrdal the centeer of Ute robot, and the x-axis points lo the front of rhe robot

scopic recognition is realized by the Nearest Neighbor methods and grows finer as data increase. By
mixing these two types of recognition, a fine andflexible recognition is realized.

/7k

3 Preparatory Experimentrobot

3.1 Task and Environment
Fig. 3: The Navigation Problem

The task for a mobile robot is to reach a goal
area as shown in Fig. 3. Fig. 4 shows a picture of thephysical robot with an omnidirectional vision sensor
with a hyperboloidal mirror. It can find a goal target
from everywhere in the plane and calculate its coordinates on the coordinate system fixed to the robot.

ment. The average of these numbers is 2.53 and thestandard deviation is 0.05.
As a comparative experiment, the virtual agent
continued learning until additional 1000 behavior

373

Table1: Results of the Original STNS

steps. We repeated this experiment ten times. The

2.57
5

2.57

total average of the numbers of behaviors to the goalis 2.04. Compared with this number, the result of
STNS is 23.4 % worse.
Fig. 5b shows an example of state spaces after
1000 behavior steps relearning in the real environment. The small circle (the radius is 50cm) at the center of each space denotes states in which the rover arrives at the goal. The big circle (the radius is about
283cm) surrounding each space denotes the longest
distance between the rover and the goal. Comparedwith the state space before relearning (Fig. 5a) or the
optimum state space (Fig. 5c), its shape is rather distorted.

a. Before relearning

4 The Improved STNS for the RealWorld
4.1 Problems with Our Previous Work

The preparatory experiment showed that learningin the real environment was worse than in the simulated one in performance. We suspect that incorrect
distortion of state shapes causes the low performanceof STNS in the real environment. A s shown in Fig. 6,
causes of the low performance are connected witheach other and form a vicious circle. Distortion ofstate shapes is related to all causal relationships denoted by solid arrows in this figure. By reducing this
distortion, the performance will be improved to aconsiderable extent.
This incorrect distortion of state shapes is causedby experience in the real environment that has various types of uncertainty as shown by broken arrows
in Fig. 6. In STNS, each state is represented by usingb. After relearningc. Optimum state spacecondition behaviors
C- forward move CJ clockwise rotation+ backward movecounterclockwise rotation

Fig. 5: State Spaces before and after Relearning andthe Optimum State Space

Fig. 6: Vicious Circle Causative of Low Performance374

positive instances and negative instances collected
while executing the task as shown in Fig. 2. It works
well in a simulated environment where every instanceis represented exactly, but it may have problems in areal environment where instances have uncertainty intheir representation.

the former cannot discriminate in the neighborhoodof boundaries of states.
This discrimination process is performed when a
new negative instance or a new positive one is added
to a state. When a new negative one is added, the system checks it with all positive instances in the state.
If it is judged inappropriate, it is removed kom thestate. When a new positive one is added, the system
checks all negative instances one at a time with all
positive ones including the newest. All negative instances judged inappropriate are removed from thestate.
The advantage of the method is that it is based
only on the distribution of instances and needs neither a priori knowledge on uncertainty of sensors, actuators, and the environment nor redundant sensory
information. It is good at maintaining a state that has
a convex shape. On the other hand, it cannot maintaina ring-shaped state or a state divided into several
areas. Therefore, it is unsuitable for problems inwhich the optimum state representation has suchstates. In the problem to which it applied, the ideal
loci, that is the loci without uncertainties, of the condition behaviors in the state space should fulfill bothof the following two conditions:
1. Loci are continuous.
2. If two loci pass two sufficiently neighboring
points, they are neighboring in their lower
courses.
These conditions are broken in many problems inwhich behavior modules make complicated behaviors.
Fig. 8 shows three examples of such cases. In thesimple problem tackled in this paper (shown in Fig.
3), these conditions are fulfilled.
The proposed method is limited to a problem
with a two-dimensional state space. We are tackling
the problem of extending the idea of discriminatingwith the maximum visual angle to a multidimensional
state space.

4.2 The Method
The proposed method so-called STNS-R is an
improved version of STNS to adapt to uncertainty insensors, actuators, and the environment in the realworld. In order to correct the distorted shapes of thestates, STNS-R removes inappropriate negative instances caused by the uncertainty tiom the state representation. It judges appropriateness of instancesfrom their distribution using the following heuristics:
the more positive instances exist around a negative
instance, the more suspicious the validity of the instance is. Removal of inappropriate positive instances
is not performed because positive ones may be much
more than negative ones and it is difficult to judge
their appropriateness from fewer negative ones
around them. Negative instances are apt to be shortbecause collecting many negative instances is in conflict with achieving high performance in task execution.
Fig. 7 shows how to discriminate between inappropriate negative instances and appropriate ones in
STNS-R. It calculates all visual angles between adjacent positive instances in view of a negative instance.
If the maximum angle is less than a threshold, thesurrounded negative instance is judged inappropriate.
The threshold is fixed empirically to 90 degrees inthis task. We chose not the minimum distance between a positive instance and a negative instance but
the maximum visual angle between adjacent positive
instances as an attribute for discrimination because

"I

0

0-

-)(

90 degrees

OverdegreesJ00a. When the condition 1 is broken

0a. An inappropriateb. An appropriate

negative instancenegative instance

0 a positive instance
X a negative instance1

the maximum visualangle

b. When the condition 2 i s broken

Fig. 7: An Inappropriate Negative Instance and an
Appropriate One

Fig. 8: Unsuitable Cases for Our Method375

representation. Transplanting a state space from a virtual agent to a physical robot can accelerate learning
very much.
STNS-R is effective also in learning from the beginning in a real environment because it can selectappropriate data among a corrupt set of data becauseof uncertainty in the real environment.
As mentioned in Section 4.2, there is limitationin the problem area to which the proposed method fordiscriminating inappropriate instances can be applied.
We are tackling the problem of improving the definition of state in order to simplify state shapes, as well
as extending the state representation to be more expressive.

5 Experimental Results-'

In order to test the improved STNS, STNS-R, we
conducted the same experiment as the original STNS:
we transplanted a state representation learned enoughin a simulated environment (Fig. 5a) to the physicalrobot and let it continue learning until 1000 behavior
steps. We repeated this experiment five times. With
STNS-R, it could adapt seamlessly to the real environment and maintain the state representation, behaviors, and high performance all through the experiment. Table 2 shows the average numbers of behaviors that are needed to reach the goal in the same
form as Table 1. The average of these numbers is
2.28 and the standard deviation is 0.02. Comparedwith the average of the virtual agent in an ideal environment without uncertainties, 2.04 as mentioned
above, the deterioration of performance were limited
to 11.8 %. This results means that STNS-R prevented
47.9 % of unnecessary deterioration in performanceas compared with the original STNS.

References
1. U. Ueno, H. Takeda and T. Nishida, Learning of
the Way of Abstraction in Real Robots, Proc. of theIEEE International Conference on Systems, Man andCybernetics (SMC'99) (CD-ROM), vol. 1, pp. 1174611751, 1999.
2. U. Ueno and H. Takeda, Cooperation of Categorical and Behavioral Learning in a Practical Solution to
the Abstraction Problem, New Generation Computing, vol. 19, no. 3, pp. 257-282,2001
3. M. Asada, S. Noda and K. Hosoda, Action-Based
Sensor Space Categorization for Robot Learning,
Proc. of the 1996 IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS 96), pp.

Table2: Results of STNS-R
Experiment #
1
2
-

3
4
5

Ave. # of Behaviors to the Goal
2.25
2.30
2.28
2.27
2.30

1502-1509, 1996.

6

4. J. Albus, A. Lacaze and A. Meystel, Multiresolutional Intelligent Controller for Baby Robot, Proc. ofthe loth International Symposium on Intelligent Control, 1995.
5. T. Yairi, S. Nakasuka and K. Hori, Sensor Fusion
for State Abstraction Using Bayesian Classifier, Proc.
of the IEEE International Conference on Intelligent
Engineering Systems (INES'98), pp. 99-104, 1998.
6. H. Ishiguro, R. Sat0 and T. Ishida, Robot Oriented
State Space Construction, Proc. of the 1996
IEEERSJ International Conference on Intelligent
Robots and Systems (IROS 96), pp. 1496-1501, 1996.
7. S. Russell and P. Norvig, Artificial Intelligence: A
Modem Approach, Prentice Hall, 1995.

Conclusions

We have proposed a new architecture, called
STNS-R, by which a state representation learned in asimulated environment can be transplanted seamlessly and adapted to a physical robot. In this architecture, inappropriate negative instances caused byuncertainties in the real environment are found on thebasis of the distribution of instances and removed inorder to correct the distorted shapes of the states. The
effectiveness of STNS-R was shown in the experimental results. The advantage of STNS-R is that we
can switch the learning environment from a simulated
one to a real one at anytime because of the incrementality of learning. Even if the transplanted state representation should not be mature, it would be adaptedby learning in the real environment.
STNS-R is the only learning method, as far as
we know, which can transplant seamlessly a state
representation learned in a simulated environment toa physical robot. Any other learning method needs alarge number of data of random behaviors in a realenvironment in order to construct an appropriate state

3 76