The 2010 IEEE/RSJ International Conference onIntelligent Robots and Systems
October 18-22, 2010, Taipei, Taiwan

Position Tracking System of Everyday Objects
in an Everyday Environment
Kouji Murakami, Tsutomu Hasegawa, Kousuke Shigematsu, Fumichika Sueyasu,
Yasunobu Nohara, Byong Won Ahn, Ryo Kurazume
Abstract— We propose an object tracking system for a service
robot working in an everyday environment. The system iscomposed of an intelligent cabinet, a floor sensing system anda data management system. The position of an object can
be classified into three areas: 1) in/on furniture, 2) on thefloor, 3) held by a human or a robot. Being equipped witha RFID reader and loadcells, the intelligent cabinet measures
the position of an object in/on itself. The floor sensing system
which uses a laser range finder, measures the position of an
object on the floor and the position of a human walking ina room. The data management system integrates the position
data of the intelligent cabinets and the floor sensing system,
and it performs position measurement of an object carried bya human. The data management system provides robots withposition information to support robot activities.

Town management system (TMS) collects data from embedded sensors and integrates them to establish a real time
database of the environment. TMS supports the robotic activities by providing information of the environment. This paper
describes a position tracking system of everyday objects inan everyday environment. The system has been developed
as one component in the informationally structured environment, the RobotTown.
GPS

Rob o t to wn with Distributed
Senso rs and RFID tags

Radio
LAN
Distributed
Vision System

I. I NTRODUCTION
Daily life support work in a care house for elderly persons
is a promissing application of a service robot. Most work
includes a go-and-fetch task of objects. If the robot canperform the go-and-fetch task of objects specified by anelderly person at the care house, the burden on health care
workers is reduced. However, an everyday environment is a
dynamically changing 3D real space because there are humanbeings walking and working around, and the placementof objects will change frequently. Although it may be asimple and well-ordered environment for human beings, it
is too complex to recognize changes for a conventional
self-contained autonomous robot equipped with as many
sensors as possible within its limited body. We will not beable to expect a capable robot to execute various tasks inthe everyday environment in the near future. An alternative
approach to the realization of a service robot working in
the everyday environment is to structure the environment in
an informative way [1][2][3][4][5]. Instead of solely relying
on the sensors on the robots themselves, we distribute laser
range finders, pressure sensors and other sensors embeddedin the environment and connected to a computer network to
support autonomous robotic activities.
We are currently developing technologies to implement
such a structured environment platform: Robot Town [6][7](Fig. 1). In this platform, a data management system called
This work is supported by NEDO (New Energy and Industrial Technology
Development Organization, Japan) project ’Intelligent RT Software Project’.
K. Murakami, T. Hasegawa, K. Shigematsu, F. Sueyasu, Y. Nohara, and R.
Kurazume are with Graduate School of Information Science and ElectricalEngineering, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka, Japan (e-mail: mkouji@ait.kyushu-u.ac.jp) .
B.W. Ahn is with Division of Marine Engineering, Mokpo Maritime
University, 571 Jukgyo-dong, Mokpo-si, Jeollanam-do, Republic of Korea.

978-1-4244-6676-4/10/$25.00 ©2010 IEEE

Buildings with RFID tags,
Sensors, Indoor map etc.

Intelligent
Wheel
car/Robot

RFID tag

Fig. 1.

Embedded
SensorConcept of RobotTown.

Mori et al. [8] proposed a position management system inwhich the positions of objects with RFID tags are measured
by using a RFID reader attached to a cabinet; however,the system cannot track an object when the object is outof the cabinet. Deyle et al. [9] reported a position search
scheme by moving a directional antenna and receiving thesignal from a RFID tag attached to an object; however, theposition resolution of the scheme is low. Nishida et al. [10]used an ultrasonic sensor for position measurement. The
positions of objects with ultrasonic tags are measured byusing ultrasonic receivers placed on the ceiling. However it
is difficult to attach the ultrasonic tags to many objects due tothe cost and the size of the tag. Pressure sensors embeddedon the floor has been introduced to track human activities[11]. However, this method cannot detect the positions oflightweight objects. A vision sensor is often used for position
tracking. But the vision system will often fail to track
objects due to occlusion, lighting conditions, and complex
backgrounds. Moreover constant surveillance by using vision
sensors causes an invasion of privacy.
Many papers on the position tracking system have been
written however these have reported on a human tracking
sytem rather than an object tracking system. An object issmaller than a human, and the object is taken in and out of asmall space such as a cabinet. Therefore previous work can
not be directly applied to the object tracking sytem.

3712

We propose an object tracking system which is based on
the structurization of an everyday environment. The featuresof the object tracking system are as follows.
1) position tracking of objects which are smaller than ahuman2) position tracking of objects in an everyday environment3) position tracking of objects carried by human
II. P OSITION TRACKING OF EVERYDAY OBJECTSThe placement of an object can be classified into threeareas as follows.
1) in/on a cabinet,
2) on the floor,3) held by a human or a robot
We develop an object tracking system which is applicableto these three cases. In the case of 1), information structurization of a cabinet is implemented by installing a RFID
reader, a loadcell etc. to the cabinet. In the case of 2), we
consider information structurization by using an laser range
finder(LRF) which has wide sensing area. There are not
only objects but also persons on the floor. So they haveto be distinguished. We use supplementarily a pyroelectric
sensor for this distinction. In the case of 3), it is difficult
to directly track objects carried by a human. We assume
that the movement of an object is caused by human beings.
So position tracking of an object carried by a human canbe replaced by human tracking while the human carries
the object. In order to implement this replacement, thesystem has to decide whether a human has an object ornot. This decision is achieved by data integration between
the information of an object taken out of a cabinet and theposition of the human. When the object is taken out of thecabinet, the object is related to the human who is closest to
the cabinet. While the human carries the object, the positionof the object is replaced by the position of the human.
Based on the considerations mentioned above, we arranged
the requirements for the object tracking system as follows.
1) position measurement of objects in a cabinet
2) position measurement of objects and humans on thefloor3) recognition of an object type(object or human) on thefloor4) integration and management of position data of objects
and humans
A function of providing a robot with the information of
objects via a network is also needed. So we add the following
requirement.
5) provision of the information of objects to a robot
The developed system satisfies all the requirements. There
are some reports on object tracking. Each of them satisfies
some requirements mentioned above. However no measurement system which satisfies all the requirements has beenreported in previous work.

III. S YSTEM S TRUCTURE
The object tracking system is composed of three components as follows.
A) an intelligent cabinetB) a floor sensing systemC) a data management system
The intelligent cabinet recognizes objects stored in it. It
detects objects taken in and out, and measures the positions
and attributes of objects. The floor sensing system measures
the positions of objects on the floor and the positions ofhumans walking in a room. The data management system
has a database and a network interface. The data management
system integrates the position data of the intelligent cabinet
and the floor sensing system, and performs object tracking.
The data management system provides a robot with theinformation of existing objects in the environment. The data
flow of the object tracking system is shown in Fig. 2. Eachcomponent is explained in the following subsections.

Fig. 2.

Data flow of object tracking system.

A. Intelligent Cabinet
The intelligent cabinet recognizes the positions and attributes of objects stored in it. Objects are often placed
closely to each other in the cabinet, therefore mutual occlusion of objects will often occur. A vision camera and a LRF
are not suitable to the position measurement in the cabinet.
In contrast, a loadcell and a RFID reader are not affected by
occlusion. The loadcell measures the weight of objects ona shelf of the cabinet. The RFID reader recognizes a RFID
tag with a radio wave. Therefore we use the loadcell and the
RFID reader to develop the intelligent cabinet.
1) Design of Intelligent Cabinet: Four loadcells are installed to the bottom of an acrylic plate. The plate issupported by the four loadcells. The plate with loadcells areplaced on a shelf of the cabinet. An object taken in andout of the cabinet is detected as the change of loads. The
position of the object is also calculated from the outputs of
the loadcells. The arrangement of four loadcells is shown in
Fig. 3. Let the position of i th loadcell be pi . The weight ofthe object is the sum of the outputs of the four locadcells(eq.
(1)).

3713

f = f1 + f2 + f3 + f4

(1)

The position p of the object on the plate is calculated as
the center of gravity of the outputs of the loadcells (eq. (2)).
p=

f1 p1 + f2 p2 + f3 p3 + f4 p4
f(2)

Fig. 4.

Fig. 3.

Appearance of intelligent cabinet.

Position measurement by using loadcells.

We use RFID to recognize attributes of objects. A RFID
tag has a unique ID. By attaching RFID tags to the objects
in an environment, the tag ID is used as the key to manage
the positions and attributes of the objects in the data management system. The attributes such as name, weight, size
are registered to the data management system in advance.
The tag ID is used as the key for this registration.
The position of an object is related to its tag ID as follows.
When an object is taken into the cabinet, its tag ID is newly
recognized by the RFID reader. After the object is put on ashelf of the cabinet, the position of the object is calculated
from the outputs of the loadcells. Finally its position isrelated to the tag ID which was previously recognized.
2) Intelligent Cabinet Implementation: The intelligent
cabinet is shown in Fig.4. A rectangular RFID antenna isplaced on a shelf of the cabinet. An acrylic plate withloadcells is placed over the RFID antenna. We used four
loadcells(Measurement Specialties Inc., FC22), one RFID
reader(TAKAYA Corp., TR3-LD003D-4), and one RFID
antenna(TAKAYA Corp., TR3-SA101M). A passive RFID
tag used is for the high frequency(HF) band at 13.56 MHz(Texas Instruments Incorporated: RI-I01-112A, ISO15693).
We conducted an experiment to measure the positions andattributes of objects in the intelligent cabinet. Three bottles
with RFID tags were taken in and out of the cabinet (Fig. 5).
The experimental result is shown in Figs. 6 to 7. Occlusionof bottles occurs in Fig. 6. Two bottles are piled and stuckin Fig. 7. In each figure, the right image shows the measured
position and attributes. A green square indicates the positionof a bottle. The bottle name as attribute is also shown near
the square. The name was derived from the data management
system by using the ID of the tag attached to the bottle.
The measurement of the positions and attributes of objects
is successfully demonstrated.

Fig. 5.

Everyday objects with passive RFID tags.

1) Position Measurement of Objects using LRF: It isdifficult to install many sensors densely on the floor whichis spacious. A LRF has a wide sensing area, and it is robust
against the lighting condition which is changed frequently inan everyday environment. Therefore we used a LRF for the
position measurement on the floor.
The human/object tracking which uses a LRF, is performedin two steps: exclusion of stationary area in the range data,
and then detection of blobs in the rest of the range data.
These steps are implemented by using a simple backgroundsubtraction. Objects on the floor are identified in the range
data out of stationary area by following procedures(Fig. 8).

B. Floor Sensing System
The floor sensing system measures both the positions ofhumans walking in a room and the positions of objects on
the floor. Moreover the system has to distinguish between
objects and humans on the floor. We developed the floor
sensing system by using a LRF and a pyroelectric sensor.

3714

Fig. 6.

Fig. 7.

Position measurement of objects.

Position measurement of piled objects.

(a)Fig. 9.

Walking model (red : right leg, blue : left leg).

(b)Fig. 8.

Estimation of object position.

1) Points in the range data located within a certain threshold distance value are merged to be a profile of an
object.
2) Width of a profile w is measured as the width of theobject (Fig. 8(a)).
3) The center of the object profile is computed to be apoint at distance w/2 on the extended line from theorigin of LRF to the midpoint of the profile (Fig. 8(b)).
Tracking of a pedestrian is performed from a series ofmeasured leg positions. Extracted legs are paired and tracked
using the Kalman filter. The position of the human body isdefined as the midpoint of the paired legs. The pair of legs
has the following features.
1) Distance of the legs is shorter than a certain thresholdvalue.
2) Traveling direction of the legs is almost the same.
The two legs satisfying these conditions in definite time
period are paired. Consistent tracking of a pair of legs overa sequence of frames of range data is performed using the
Kalman filter: the position of the paired legs in the next frame
is estimated from the previous frames. Then the profilesof the paired legs are extracted in the range data of thenext frame as the ones that are the closest to its estimatedpositions. Repeating the process, the legs are tracked. The
Kalman filter is extended to use acceleration of the legs as
the additional external input [12]. The acceleration of theleg is determined by using walking model(Fig. 9). In themodel, the movement of the two legs is defined in fourphases. The present phase is determined using past sequenceof the velocity and acceleration of the leg. When the present
phase is determined, the acceleration of the leg is determinedaccordingly.
We conducted an experiment to track two pedestrians. A
LRF (Hokuyo Electronic Co. Ltd., UTM-30LX) was placedon the floor in an indoor environment of 6m × 6m. The system tracked the pedestrians by using the LRF. The tracking
result is shown in Fig. 10. Two pedestrians are successfully
tracked while they move along a different trajectory.

Fig. 10.

Tracking result of two pedestrians.

2) distinction between pedestrian and object: The human/object tracking system can distinguish between humans
and everyday objects if humans are walking. However if ahuman stands still, the system fails in this distinction. So we
use a pyroelectric sensor to distinguish between a human andan object. This sensor detects changes of infrared radiation inits view area. The changes of infrared radiation occurs whensomething comes into the view area, because its temperature
is different from the temperature of the background. The
magnitude of the sensor output depends on the magnitudeof the changes of infrared radiation. The temperatures ofa human and an object are different. Therefore we candistinguish them by applying the threshold test to the sensor
output.
The pyroelectric sensor detects changes of infrared radiation at the current frame and infrared radiation in previous
frames. If a human stands still, infrared radiation does notchange. Then the sensor can not detect the human as changes
of infrared radiation while the human does not move inthe view area. We use a chopper to resolve this issue. This
chopper masks a pyroelectric sensor periodically. When thesensor is unmasked, the sensor detects changes of infrared
radiation even if the human does not move. The pyroelectric
sensor unit with a chopper is shown in Fig. 11.
We conducted an experiment to distinguish between ahuman and an object. The pyroelectric sensor unit (PanasonicCorp, NaPiOn) [13] was installed to the ceiling. Fig. 12shows the output of the sensor while each of them is in thesensing area. The vertical axis shows the sensor output. The
horizontal axis shows elapsed time. The sensor is unmasked
twice by a rotation of a chopper. In the case of the bottle,
the output value is almost constant because its temperature3715

is almost the same as the background. Thereofore when
the position measurement system detects something new inthe sensing area of the pyroelectric sensor unit, we candistinguish it by applying the threshold test.

(a) masked(b) unmasked

Fig. 11. Pyroelectric sensor unit with a chopper. A pyroelectric sensor ismasked by a chopper in (a). The sensor is unmasked in (b).

D. Object Tracking System
When an object is taken out of an intelligent cabinet,
the object is related to a person who is closest to thecabinet. The positions of pedestrians are tracked by the floor
sensing system. It is difficult to directly track the small object
carried by the person. However the object tracking system
recognizes who has the tracked object by using the relation
information between the object and the person. Therefore
object tracking can be replaced by human tracking while theperson carries it. TMS manages the position of the object
held by the person as the position of the person. When theobject is placed in the intelligent cabinet or is placed on
the floor, the relation between the object and the person
is released. The floor sensing system identifies the person
who is closest to the object when the object is put on thefloor. TMS has information of the object which the person
held. Then the attributes of the object placed on the floor are
recognized by using previously tracked information, thougha RFID reader is not installed to the floor.
IV. E XPERIMENT
We conducted an experiment of tracking objects by theobject tracking system which is composed of the intelligent
cabinet, the floor sensing system and TMS.

(a) object
Fig. 12.

(b) human

Measurement result of a pyroelectric sensor unit with a chopper.

C. Town Management System TMS
The position information measured by the intelligent cabinet and the floor sensing system is managed and robot
interaction with the intelligent environment is ensured byTown Management System (TMS) as a data management
system. TMS has the following functions:
1) Communication with robots and sensors embedded in
an environment2) Data storage, revision, and retrieval
3) Provision of the information related to the ID of a
RFID tag attached to an object
4) Provision of the ID of a RFID tag related to specificdata such as an object position and name
Network interface of TMS is implemented as Web service
based on the Simple Object Access Protocol (SOAP). Thenthe adding of new sensors and the separating of installed
sensors are easy. Database service of TMS is implemented
by using MySQL. A tag ID is used for the key to update
the position and attribute information of objects. Function
programs we developed to enable robots and measurement
systems to interact with TMS are provided in an application
program interface (API) library. An example of API function
is locating objects in the environment. Once the link to the
API library is coded in a robot’s control program, the robot
obtains environment information from TMS by executing the
API function at arbitrary timing.

A. Experimental setup
Three pieces of furniture were informationally structured.
Loadcells and a RFID reader are installed to a box, thisintelligent box measures the positions and tag IDs of objects
stored in it. Loadcells are installed to a table, this intelligent
table measures the positions of objects on it. A RFID reader
is installed to a garbage can, this intelligent garbage canrecognizes the tag IDs in it. We call them intelligent cabinets
for the sake of convenience. The three intelligent cabinets
and the floor sensing system were placed in an indoor
environment. The arrangement of them is shown in Fig. 13. A
laser range finder(Hokuyo Electronic Co. Ltd., UTM-30LX)
was placed on the floor for the floor sensing system. Itsscanning plane is parallel to the floor, and the height of thescanning plane is 60mm from the floor. Pyroelectric sensor
units with a chopper were placed on the ceiling. The position
information measured by the intelligent cabinets and the floor
sensing system is integrated in TMS.
Intellig ent tab leIntellig ent box

pyroelectric
sensor units
Intellig ent
garbage ca n
bed

LR F

Fig. 13.

Experimental setup.

Passive RFID tags are attached to all objects in theenvironment. A tag ID is used as the key to relate the attribute3716

information with the position information of an object in
TMS. The attribute information is registered to TMS in
advance. The object tracking system is demonstrated after
these preparations. We assume that the object tracking system
will be used in a care house. So the preparations are madeby care workers.
B. Object Tracking Result
In the experiment, the subject goes around in the environment. The subject handled two bottles. One of them wascarried from the intelligent box onto the floor. The other was
carried from the intelligent box onto the intelligent table. The
positions of the bottles and the subject were tracked. The
sequence of the experiment is as follows:
1) The subject moves to the front of the intelligent box.
2) The subject takes bottle A out of the intelligent box.
3) The subject carries bottle A.
4) The subject leaves bottle A on the floor.
5) The subject moves to the front of the intelligent box.
6) The subject takes bottle B out of the intelligent box.
7) The subject carries bottle B to the front of the intelligent table.
8) The subject puts bottle B on the intelligent table.
The tracking result is shown in Figs. 14 through 16. The
yellow circle indicates the position of the subject. The purple
circle and the blue circle indicate the positions of bottle A
and bottle B respectively. bottle A and bottle B are shownas small circles while the subject carries them.
The system successfully tracks bottle A from the intelligent box onto the floor in Fig. 15. When bottle A is takenout of the intelligent box, it is related to the subject who hastaken it. Object tracking can be replaced by human tracking
while the subject carries it. When bottle A is put on thefloor, its position is newly measured by the floor sensing
system. The floor sensing system cannot recognize what
the measured object is. However the floor sensing system
identifies the subject who is closest to the object when theobject is put on the floor. TMS has information of the object
which the subject holds. Then the object is recognized asbottle A by using previously tracked information in TMS.
The system successfully tracks also bottle B from theintelligent box onto the intelligent table in a similar way
in Fig. 16.

Fig. 14.

Experimental result.

Fig. 15.

Result of object tracking.

V. C ONCLUSION
We developed the object tracking system. The system
is composed of three components. The intelligent cabinet
recognizes objects stored in it. The floor sensing systemmeasures the position of an object on the floor and theposition of a human walking in a room. TMS integrates the
position data of the intelligent cabinet and the floor sensing
system, and provides a robot with the position information.
The object tracking system is successfully demonstrated.
If a person held several objects, the system cannot recognize which object was put on when the person put an object
on the floor. In order to deal with this case, we will apply aprobability model to the object tracking system in the future
work.
This work is supported by NEDO (New Energy and Indus-3717

[8] R. Fukui, H. Morishita, T. Mori and T. Sato, “Development of a Homeuse Automated Container Storage/Retrieval System”, in Proceedingsof IEEE International Conference on Intelligent Robots and Systems(IROS2008), pp.2875-2882, 2008.
[9] T. Deyle, H. Nguyen, M. Reynolds and C. Kemp, “RF Vision: RFID
Receive Signal Strength Indicator (RSSI) Images for Sensor Fusion
and Mobile Manipulation”, in Proceedings of IEEE International
Conference on Intelligent Robots and Systems , pp.5553-5560, 2009.
[10] Y. Nishida, H. Aizawa, T. Hori, N.H. Hoffman, T. Kanade and M.
Kakikura, “3D Ultrasonic Tagging System for Observing Human Activity”, in Proceedings of IEEE International Conference on Intelligent
Robots and Systems, pp.785-791, 2003.
[11] T. Mori, T. Matsumoto, M. Shimosaka, H. Noguchi and T. Sato,“Multiple Persons Tracking with Data Fusion of Multiple Cameras andFloor Sensors Using Particle Filters”, Workshop on Multi-camera andMulti-modal Sensor Fusion Algorithms and Applications (M2SFA22008), 2008.
[12] K.Nakamura, H.Zhao, and et al., “Tracking pedestrians using multiple
single-row laser range scanners and its reliability evaluation”, Systems
and Computers in Japan, vol.37, no.7, pp.1-11, 2006.
[13] http://panasonic-denko.co.jp/ac/e/index.jsp

Fig. 16.

Result of object tracking.

trial Technology Development Organization, Japan) project’Intelligent RT Software Project’.
R EFERENCES[1] Y. Nishida, T. Hori, T. Suehiro, and S. Hirai “Sensorized environment
for self-communication based on observation of daily human behavior,” Proc. of IEEE/RSJ Int. Conf. on Intelligent Robots and Systems,
pp.1364-1372, 2000.
[2] J.H. Lee and H. Hashimoto, “Intelligent Space - concept and contents,”
Advanced Robotics, Vol.16, No.3, pp.265-280, 2002.
[3] Y. Nakauchi, K. Noguchi, P. Somwong, T. Matsubara, and A. Namatame, “Vivid Room: Human Intention Detection and Activity Support Environment for Ubiquitous Autonomy,” Proc. of IEEE/RSJ Int.
Conf. on Intelligent Robots and Systems, pp.773-778, 2003.
[4] K. Kogure, N. Hagita, Y. Sumi, N. Kuwahara, and H. Ishiguro,“Toward Ubiquitous Intelligent Robotics,” Proc. of IEEE/RSJ Int.
Conf. on Intelligent Robots and Systems, pp.1826-1831, 2003.
[5] C.D. Nugent, D.D. Finlay, P. Fiorini, Y. Tsumaki, and E. Prassler,“Home Automation as a Means of Independent Living,” IEEE Trans.
Automation Science and Engineering, Vol.5, No.1, pp.1-9, 2008.
[6] K. Murakami, T. Hasegawa, R. Kurazume, and Y. Kimuro, “A Structured Environment with Sensor Networks for Intelligent Robots,” Proc.
of IEEE Int. Conf. on Sensors, pp.705-708, 2008.
[7] K. Murakami, T. Hasegawa, R. Kurazume, and Y. Kimuro, “Supporting Robotic Activities in Informationally Structured Environment
with Distributed Sensors and RFID Tags”, Journal of Robotics andMechatronics, Vol.21, No.4, pp.453-459, Aug. 20093718