An Acquisition of the Relation between
Vision and Action using Self-Organizing
Map and Reinforcement Learning
Kazunori Terada, Hideaki Takeda and Toyoaki Nishida
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara 630-01, Japan
E-mail : kazuno-t@is.aist-nara.ac.jp

Abstract

an object is perceived through only tactile sensors.
On the other hand, using a visual sensor or a sonar
An agent must acquire internal representation ap- sensor, an agent can estimate what happen in the
propriate for its task, environment, sensors. As a future result from performing selected action. Howlearning algorithm, reinforcement learning is often ever, this is correct only under following assumputilized to acquire the relation between sensory in- tion. Visual input represent 3 dimensional space.
put and action. Learning agents in the real world In other words, an agent can reconstruct 3 dimenusing visual sensors is often confronted with critical sional space from visual input. A newborn agent
problem how to build necessary and sucient state should not know even this assumption. Therefore,
space for the agent to execute the task. In this paper, an agent should acquire relation visual input and
we propose acquisition of relation between vision physical world by interacting with its environment.
and action using Visual State-Action Map (VSAM).
Many works have addressed on acquiring a beVSAM is the application of Self-Organizing Map havior or constructing internal representation through(SOM). Input image data is mapped on the node interaction between an agent and its environment.
of learned VSAM. Then VSAM outputs the appro- In the eld of computer vision, purposive vision
priate action for the state. We applied VSAM to have proposed [1] [2]. In simple terms, this apreal robot. The experimental result shows that a real proach suggests that vision has a purpose, and Perrobot avoids the wall while moving around the envi- ception Action system should be organized by beronment.
havior. Pfeifer [8] proposed the concept of sensorymotor coordination. It means that all interaction
with the environment is to be conceived as a sensorymotor coordination.
a learning algorithm, reinforcement learnA sensor based agent performing a task in an envi- ing As
isoften
utilized. Reinforcement provides agentsronment must acquire internal representation about with the capabilityof learning to act optimally by
its environment. In order to adapt uncertain and experiencing the consequencesof actions, without
unforgiving environments, an agent should acquire any domain knowledge. The interaction
betweenthe internal representation by interacting with its an agent and the environment is only focusedenvironment. Evaluating current sensory input us- Learning agents in the real world using visualupon.
sening internal representation, an agent can output an sors is often confronted with critical problem how
appropriate action for the situation which the agent to build necessary and sucient state space for thefaces.
to execute the task. This problem includes
An agent gets information about its environ- agent
two
problems.
ment through various sensors, e.g., visual sensor,
tactile sensor and sonar sensor. Tactile sensor dif- 1. How to reduce the size of the state space into
fers with the other sensors about the point that itproper size with which reinforcement learnoutputs a signal when the body of the agent physiing can be performed. An abundance of vically touches an object. A crucial state for agent's
sual sensors has resulted in a state space sotask such as colliding with a obstacle or graspinglarge that it overwhelms the agent's limited

1 Introduction

resources for computation, storage and learning experience.
Input image data
2. How to select the appropriate axes of state
space. For a given tasks, an agent must focus
its attention upon only necessary features inenormous visual input data. Axes of the state
Self-organization map
space are often selected depending on humanintuition.
Reinforcement learningThis two problems is often handled as combined
problem. Nakamura and Asada [7] have proposed aMotor command
method using a motion sketch which closely relates
robot actions to sensor patterns for reducing thesize of the state space. This approach uses features
Figure 1: Architecture
selected by humans such as optical ows. MaCallum [6] has proposed an approach using U-Tree, areinforcement learning algorithm that uses selective
to construct necessary and suattention and short term memory. This approach cientFurthermore,
state
space
the agent, we used following
uses features previously selected human and some assumption. If theforstate
consists of n pixels
features appropriate for tasks are automatically se- of binary image data, thespace
numberof whole state
lected. Ishiguro and Sato [3] have proposed an ap- space becomes 2n . However, the state
space conproach using Empirically Obtained Perceivers which tains unsuitable states which does not reallyclassi es sensor patterns by statistically analyzing We consider one image pattern as one state. exist.
the actions, sensor patterns, and rewards given as
input of this system is the vector of the imresults of task executions. However, they performed age The
data
and the output is an action. The input
their experiments in simulated world.
vectorisand mapped to a unit of selfIn recent researches of learning agent perform- organizingclustered
map
(SOM)
is used as internal
ing in the real world, raw image data was not used. state space. To perform owhichline
learning,
units
Instead of using raw image data, many researchers of SOM has a set of possible action and each
next
state
have used processed image. This paper proposes resulted from performing a selected action. Thenan approach to construct a state space using Visual relation between the states and actions is computed
State-Action Map (VSAM) which is the application using reinforcement learning. We call this architecof Self-Organizing Map (SOM)[4]. VSAM is inter- ture Visual State-Action Map (VSAM).
nal representation of the agent. Input image data
Figure 1 shows the architecture of VSAM. The
is mapped on the node of VSAM. Then VSAM out- processof constructing VSAM is decomposed into
puts the appropriate action for the state. O -line three phases.
reinforcement learning is performed on VSAM andappropriate relation between vision and action is 1. Clustering of image data previously collectedacquired.
using self-organizing map.
A similar works to our research approach havebeen already reported. Krose [5] proposed a sen- 2. Exploration of the environment. During exploration, the image data was mapped on thesor based navigation scheme which makes use of awinning node of SOM. Then, the winning node
global representation of environment by means of aand corresponding action was stored, and esself-organizing map. They performed their work in
timation of the state transition probability
simulation and used 16 range sensors. We, rather
pij (a) is made. pij (a) is de ned as the probfocus upon real world agent and acquiring relation
ability that, given an initial state i, the agent
between vision and action.
is in the state j after action a.
3. Learning. Since transition of each state wascalculated, reinforcement learning in SOM was
The purpose of this system is to output approperformed.
priate action for the state. For this purpose thesystem have two features. First, we use the nodes
of SOM as state space, and each node of SOM is thecluster of image data. Second, to decide the appropriate outputs for each state, we use reinforcement
learning.

2 Architecture3 Clustering of images
3.1 Self-Organizing Maps

The SOM de nes a mapping from the input data
space Rn onto several dimensional array of nodes.
With every node i, a parametric reference vector Figure 2: Two line gures that are orthogonal
mi = [i1; : : : ; 1n ]T 2 RT n is associated.
An input
vector X = [x1; : : : ; xn ] 2 Rn is connected to allnodes in parallel via variable scalar weights ij and Consider Figure 2; the two binary patterns are combe compared with all the mi. The smallest of Eu- pletely orthogonal, although looking very similar toclidean distances kX 0 mi k can be made to de ne us, and would be interpreted as totally dissimilar by
SOM. To avoid this problem, we use blurring which
the winner node, signi ed by the subscript c.
is one of the simplest preprocessing method. If thevalue of a cell (i; j ) is 1, the value of neighborhood
cells follows the equations described below.
kX 0 mc k = min
fk
X
0
m
kg
(1)
i
i
8
=0
>
>
< 10:66 dd =The output yi of the node i is
C (d) = 0:33 d = 12

>
>
:0
1 i=c
d>2
yi = 0 otherwise
Where d is the distance from cell (i; j ).
Learning of SOM is performed by renewing theweight of winner node mi and a neighborhood setof array points around winner node.
Reinforcement learning is an attractive frameworkformi(t + 1) = mi(t) + hci(t)[X (t) 0 mi(t)] (2) the unsupervised learning of action policies forautonomous agents. The tasks are given by a huWhere t = 0; 1; 2; : : : is an integer, the discrete- man as rewards embodied in the external world.
time coordinate. The function hci (t) is called neighQ-learning [9] is based on estimating the value ofborhood function.
Q(s; a), which is the expected future discounted reward for taking an action a in the input state s andcontinuing with the optimal policy. The discounted(N + 1 0 krc 0 rik) reward
is the sum of all future reward weighted byhci (t) = h(krc 0 ri k; t) = (t) 1 c
Nc
how
close
they are. The value of Q(s; a) is renewed.
(3)22
Where rc 2 R and ri 2 R are the location
vector of nodes c and i, respectively, in the array.
Q(st ; at ) Q(st ; at )+The value of (t) is identi ed with a learning-rate(rt + max
Q(st+1 ; b) 0 Q(st ; at )) (6)b2A
factor (0 < (t) < 1). Nc is a neighborhood setof array points around node c. Both (t) and theWhere t is the present time, is a discount facradius of Nc are usually decreasing monotonically tor (0 < < 1), and rt is the reward received atin time (during the learning process).
time t.
The following is a Q-learning algorithm we used(t) = 0 (1 0 Tt )(4) here.
1. s the current statet
Nc = (d0 0 1)(1 0 ) + 1(5)T
2. a Select an action randomly
Where 0 is constant value. d0 is the initial
3. s0 Next state, r Immediate reward in s0
value of Nc . T is whole learning steps.
4. Renew Q(s; a)4 Reinforcement learning

3.2 Preprocessing of Optic Patterns

SOM algorithms compare input pattern vector 4.1 Acquisition of Visual State-Action
with weight vectors using Euclidean di erences, whereby Map
it is not meant to tolerate even insigni cant amounts Many applications of reinforcement learning wereof transformation in elementary operations such as implemented by simulation, since enormous computranslation, rotation, or scale changes of patterns. tational time is required to converge the learning.

To accomplish o -line learning with real data we the robot can move 6 directions (Figure 4(b)). The
propose Visual State-Action Map (VSAM) which is robot has a CCD camera which outputs a picture
the application of SOM. Each node of VSAM has 6 of 160 2 120 pixels at each time step and two tactile
sub-nodes as selectable action and each node has a sensors in front of the hand.
link to the next state from performed selected acThe environment used this experiment is 90cm2
tion(Figure 3). VSAM also has a slot of reward 180cm rectangular space surrounded by 40cm height
which is received when the robot touches the wall. wall. The boundary line between the oor and the
The following is the algorithm to acquire the data wall is marked black. The width of boundary line
set of VSAM.
is 5cm.
a1
a2
a3
a4
a5
a6

(1,1)

a1
a2
a3

a1
a2
a3

a1
a2
a3

a4
a5
a6

a4
a5
a6

a4
a5
a6

a1
a2
a3

a1
a2
a3

a1
a2
a3

a4
a5
a6

a4
a5
a6

a4
a5
a6

(1,2)

(1,3)

5.2 Generation of SOM

Input vector of SOM used in this experiment is 12 2
9 cells binary picture which resolution is decreased
from 160 2 120 pixels. If 20% of the pixels in a cellis black, the value of input vector is 1, else 0. With
201 piece of image data set collected by random
movements in the environment, SOM as a internal
state space has been constructed (Figure 5). Wholelearning times was 200 steps. 70 nodes of SOM ismapped from input vector.

Figure 3: VSAM
1. ai select an action (i = 1; 2; : : : ; 6)
2. Ut de ne winner node
3. r reward
4. Memorize Ut and r to ai of Ut01

5 Experimental Results
5.1 Robot and Environment

a1
CameraTactile sensor

a6

a2

a5

a3
a4

(a) Robot(b) Actions

Figure 4: Robot

Figure 5: SOM
The nodes in circles drawn in Figure 5 have similar patterns from a view of human intuition. The
average of Euclidean distances between neighborhood nodes is 1:4.

5.3 Construction of VSAM

The transition of the state of VSAM is collected byrandom movements.
The robot used in this experiment is shown inResult from executing 600 actions, 317 sets of
Figure 4(a). The robot is driven by two motors to relation between state and action in 420 possible
which we can send a motor command 01; 0; +1 in- sets were required. Table 1 is the examples of acdependently. Selecting a combination of each com- quired action sets. For example, if the current statemand;is (1; 9) and a selected action is a4, the next stateis (3; 3). Reward is also required, when the tactile
a1 (1; 1); a2 (1; 0); a3 (01; 0)
sensor in front of the hand of robot touches the wall.
a4 (01; 01); a5(0; 01); a6 (0; 1)

Table 1: Example of VSAM
1,9
2,9
3,9
4,9
5,9
6,9

a1

a2

a3

a4

3,3 1,7 0,4 3,32,2 0,6 6,9 3,3
3,6 0,5 6,8 3,40,5 7,9 2,53,9 8,9 8,7 2,6
2,9

a5

a6

0,6
2,5 5,53,5 5,67,65,6 7,5
3,9 ,6,8

reward
0
0
0
0
0
-1

Image Data

UHF Tuner

Motor Command
Sensor Data
Wireless Modem

Figure 7: Con guration of system
The value of reward is 01. The blank columns of
the table means that the relation was not required. UHF receiver and sent to SGI workstation throughvideo port.
We applied the VSAM to the real mobile robot.
5.4 Learning
Figure 8 shows the result of this experiment
Using the required transition of the state of VSAM,o -line Q-learning was performed. Figure 6 shows
12
3
the examples of learned Q value which was suciently converged. The shaded arrow means the Q
value, black arrow means high Q value and whitearrow means low Q value. We can see that the Q
value of the action toward the wall is low. Under a5
46
policy by which a robot selects an action of which Q
value is maximum, it seems like that a robot avoid
the wall.
Figure 8: The action avoiding the wall. 1,4,6: The
agent turns the corner.; 2,3,5: The agent goes along
the wall.
high
In Figure 8, we can see that the robot avoids thewall. When it comes to the corner, it changes the
direction, and run along the wall.
low
The advantage of having visual sensors is to estimate what happen in the future result from performing selected action. As a consequence of perFigure 6: Learned Q value: When agent is going forming reinforcement learning, value of action havealong the wall, Q value of the action toward wall is calculated. When agent is going along the wall, Q
low and Q value of the action leaving from the wall value of the action toward wall is low and Q valueof the action leaving from the wall is high. Thisis high.
means that agent previously know the negative reward which is given agent when it collide with wall.

6 Experimental result for a real7 Discussion and Conclusion
mobile robot
In this paper, we proposed an approach to the prob-

Experiment with real mobile robot ware per- lem which concerns the construction of the state
formed with R3 robot (IS Robotics, Inc.). Figure 7 space. Clustering of input image vector using selfshows a con guration of system used in this exper- organizing map, we have constructed a state space
iment. The motor commands is sent to R3 robot which does not depend upon human intuition. To
via wireless modem. The image taken by a CCD perform o -line learning with real data, we havecamera mounted on the robot is transmitted to a used Visual State-Action Map as a model of an environment.

navigation. In Proceedings of the IEEE/RSJ/GI
Using VSAM, reinforcement learning which dealInternational Conference on Intelligent Robots
with high dimensional state space in the real worldand Systems, pages 9{14, 1994.
was performed. However, in order to apply the proposed method to more complex environment andtasks, the state space constructed on VSAM must [6] A.K. McCallum. Learning to use selective attention and short-term memory in sequential tasks.
be clustered based on certain viewpoint of feature.
In Proc. of The Fourth International Conference
To de ne the axes of state space explicitly, anon Simulation of Adaptive Behavior (SAB96),
agent must focus its attention on the changes ap1996.
peared in the image data. In our work, the agent
did not explicitly select the axes of state space after [7] T. Nakamura and M. Asada. Motion sketch:learning, nevertheless, from the view of input genAquisition of visual motion guided behaviors.
eralization, an agent must decide axes which is the
In Proc. of Int. Joint Conference on Ariti cialfeature of action of object.
Intelligence, pages 126{132, 1995.
To recognize the objects in the real world, anagent must know various visual feature. Since vi- [8] Rolf Pfeifer and Christian Scheier. Sensorymotor coordination: the metaphor and besual feature strongly concerns physical feature, acyond. In R. Pfeifer and R.A. Brooks, ediquiring the relation between the tactile sensor intors, Robotics and Autonomous Systems. Speputs and visual input is very important. In ourcial Issue on Practice and future of autonomous
work, agent has a few tactile sensors. An agent
agents., 1996.
must have many tactile sensors to recognize various object of which feature is unique. Finally, we
discuss the problem about dynamic environment. [9] C.J.C.H. Watkins and P. Dayan. Technical note:
Q-learning. Machin Learning, 8:39{46, 1992.
Applying the method of VSAM to dynamic environment, state space will multiply. Because one nodeof VSAM represent one state, and trivial change ofimage according to change of environment will generate new state. Therefore an agent must consider
an object as independent object (not a pattern),
and observe critical objects for agent's action.

Acknowledgments

The author thanks Takayuki Nakamura and Atsushi
Ueno for discussions about learning algorithm .

References[1] Yiannis Aloimonos. What i have learned. In
CVGIP: Image Understanding, pages 60:74{85,1994.
[2] R. James Firby, Roger E. Kahn, Peter N .
Prokopowicz, and Michael J. Swain. An architecture for vision and action. In International Joint Conference on Arti cial Intelligence, IJCAI-95, pages 72{79, 1995.
[3] R. Sato H. Ishiguro and T. Ishida. Robot oriented state space construction. In Proc. ofIEEE/RSJ International Conference on Intelligent Robots and Systems 1996 (IROS96), pages
1496{1501, 1996.
[4] T. Kohonen. Self-Organizing Maps. Springer,
1997.
[5] B.J.A. Krose and M. Eecen. A self-organizing
representation of sensor space for mobile robot