A Wearable Augmented Reality System
Using Positioning Infrastructures and a Pedometer
Ryuhei TENMOKU Masayuki KANBARA Naokazu YOKOYA
Vision and Media Computing Laboratory
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, 630-0101 Japan{ryuhei-t, kanbara, yokoya}@is.aist-nara.ac.jp
Abstract
This paper describes a wearable augmented reality system using positioning infrastructures and a pedometer. To
realize augmented reality systems, the position and orientation of user’s viewpoint should be obtained in real time. The
proposed system measures the orientation of user’s viewpoint by an inertial sensor and the user’s position using positioning infrastructures in environments and a pedometer.
The system specifies the user’s position using the position
ID received from RFID tags or IrDA markers which are thecomponents of positioning infrastructures. When the user
goes away from them, the user’s position is alternatively estimated by using a pedometer. We have developed a navigation system using the proposed techniques and have proventhe feasibility of the system with experiments.

1. Introduction
Since computers have made remarkable progress in resent years, a wearable computer can be realized[1, 2]. At
the same time, Augmented Reality (AR) which merges thereal and virtual worlds has received a great deal of attention as a new method for displaying information[3, 4, 5].
If we construct an annotation overlay system for real sceneusing an AR technique with a wearable computer, it canbe applied for many applications[6, 7, 8, 9, 10]. Figure 1
shows an example of the annotation overlay system which
imposes annotations on the real scene using AR techniqueswith wearable computers.
To realize AR systems, the position and orientation ofuser’s viewpoint are needed for acquiring the relationshipbetween the real and virtual coordinate systems. Usually,
Global Positioning System (GPS) is used to measure theuser’s position outdoors[11]. In AR systems which use
GPS, a differential GPS[12] or a real time kinematic GPS

Annotations
Figure 1. An example of a wearable AR application.

is used to measure the user’s position accurately. However, since both of them need radio communication to correct GPS data, the hardware configuration of the system
will be complicated. Moreover, GPS cannot be used indoors. Therefore, if we construct a wearable AR system
which can be used both indoors and outdoors, we must combine other positioning methods which can be used indoors
such as an electro-magnetic sensor[13] or image markers.
Thomas et.al. constructed a wearable AR system which canbe used both indoors and outdoors combining a differential
GPS and image markers[14]. However, there remain someproblems such that the hardware configuration of the system
is complicated and the system needs to switch the sensors
between indoors and outdoors.
The purpose of the present work is to construct a wearable annotation overlay system which can be used bothindoors and outdoors seamlessly. In this paper, we propose a wearable annotation overlay system using positioning infrastructures and a pedometer. Positioning infrastructures are used for specifying the user’s position at appointed
points, and the pedometer is used for estimating the user’s
position by measuring the user’s movement. Moreover, anInertial Sensor
Display Device
INTERTRAX2(INTERSENSE)

User’s Position
CameraDragonfly(PointGreyResearch)

Clip On Display(Micro Optical)

Annotation
Overlay Image

Real Scene Image
RFID Tag Reader

V720 (OMRON)

IrDA Receiver
Pedometer

Computer
Original

3D Motion Sensor(NEC TOKIN)

User’s Position

Annotation
Database
INSPIRON 8100
(DELL)

Figure 2. Hardware configuration of the proposed system.
inertial sensor which is attached to the headset detects theorientation of the user’s viewpoint. The system generates
annotation overlay images from information such as the position, the orientation, and the database of annotations thatare held in a wearable computer, and shows annotation overlay images to the user in real time through a display device
attached to the headset.
This paper is structured as follows. Section 2 describes
the proposed wearable AR system using positioning infrastructures and the pedometer. In Section 3, experimental results with a prototype system are described. Finally, Section
4 describes summary and future work.

2. Wearable Annotation Overlay System
In this chapter, our proposed wearable annotation overlay system is described in detail. In Section 2.1, the hardware configuration of the proposed system is described.
Section 2.2 and 2.3 describe how the system measures theuser’s position and orientation respectively. Section 2.4 describes the method of generating annotation overlay images.

2.1 Hardware Configuration of Prototype System
Figure.2 shows a hardware configuration and data flowof the proposed wearable augmented reality system. The
user equips five sensors, a notebook PC and a display device. Five sensors can obtain the position and orientation of
the user’s viewpoint and the real scene image. These data
are sent to a notebook PC. The notebook PC generates annotation overlay images from these data and a database ofannotations stored in the PC. The notebook PC sends annotation overlay images to a display device which is attachedto the user’s headset. The user can see it through the display
device. Components of the proposed system are describedin more detail in the following.
Sensors The user equips following five sensors. Electric
power of them is supplied from the notebook PC ora 9V battery. The data is transmitted to the computer
through USB, IEEE1394, or serial connection.
Inertial sensor (Intersense: InterTrax2 ) An inertial
sensor is attached to the user’s headset and mea-sures the orientation of the user’s viewpoint. The
inertial sensor can obtain data at 256Hz.
Camera (Point Grey Research: Dragonfly) A camera
is attached to the user’s headset and captures thereal scene image from the user’s viewpoint. It
can capture RGB 24bit color image of 640 × 480
pixels. The camera can capture the image at 30
frames per second.

IrDA marker

RFID tag

IrDA receiver (Original) An IrDA receiver is attached to the user’s bag as shown in Figure 2. The
IrDA receiver can receive infrared ray including
position IDs.
RFID tag reader (OMRON: V720): An RFID tag
reader is attached to the user’s wrist as shown in
Figure 2. The RFID tag reader can read the position IDs recorded in RFID tags.
Pedometer (NEC TOKIN: 3D motion sensor) A pedometer can measure pitch, roll, and yaw. It
can also measure accelerations in the horizontal directions. The pedometer can obtain data at
125Hz.

Specify the user’s position using RFID tag
Specify the user’s position using IrDA marker
Estimate the user’s position using a pedometer

Figure 3. Method for measuring the user’s position.

Computer (DELL: Inspiron8100, PentiumIII 1.2GHz,
512Mbytes memory) A computer is carried in theuser’s shoulder bag. It has a database of annotation
information for generating annotation overlay images.
Display device (MicroOptical: Clip On Display) A display
device is a video see-through display device. It is attached to headset as shown in Figure 2. It can present
a 640 × 480 color image to the user. Power is supplied from a control box which is carried in the user’s
shoulder bag.

2.2 Measurement of User’s Position
The proposed system measures the user’s position indoors and outdoors using positioning infrastructures and apedometer as illustrated in Figure 3. The system identifies
the user’s accurate position using positioning infrastructures
when the user is near the components of positioning infrastructures. If the user goes away from them, the user’s position is estimated by using a pedometer. An accumulative error of the pedometer is corrected when the system receives
the position ID again.
2.2.1

Identification of User’s Position Using Positioning Infrastructures

In this paper, RFID tags and IrDA markers are used as thecomponents of positioning infrastructures. The componentsof positioning infrastructures have their own position IDs,(a) RFID tags.

(b) RFID tag reader.

Figure 4. RFID tags and a tag reader.

and the user equips the sensors which can receive position
IDs. The position ID which is recorded in a RFID tag shownin Figure 4(a) can be read by bringing a tag reader shown in
Figure 4(b) to the RFID tag. It is also possible to transmit
the position ID by receiving infrared ray from IrDA markers shown in Figure 5(a) with an IrDA receiver in Figure5(b). In our approach, RFID tags and IrDA markers are set
up to the appointed points as components of positioning infrastructures, and the system can specify the user’s position
receiving position IDs with the RFID tag reader or the IrDA
receiver that are put on the user.In the following, characteristics of these two devices are explained in detail.
IrDA device : Figure 6 shows an example of arranging
IrDA markers. The user equips an IrDA receiver which
can identify IrDA markers on the shoulder bag asshown in Figure 2. IrDA markers always send theirunique position IDs. When the user comes into the in-(a) IrDA marker.

(b) IrDA receiver.

Figure 5. IrDA receiver and marker.
Figure 7. An example of arranging RFID tags.

Figure 6. An example of arranging IrDA markers.

frared ray range of the IrDA marker, the user’s position
is identified by receiving the position ID with the IrDA
receiver. The infrared ray range of an IrDA marker isa cone whose diameter is about 3 meters and height isabout 5 meters since the infrared ray has the directivity.
Therefore, when IrDA markers are set up to the ceilingin indoor environments as shown in Figure 6, the infrared ray range is a circle whose diameter is about
1.5 meter at floor level. Consequently, the system canspecify the user’s position automatically when passing
under an IrDA marker.
RFID tag : RFID tags are set up to the appointed points
as shown in Figure 7 and the RFID tag reader is set
up to the user’s wrist as shown in Figure.2. When theuser brings the RFID tag reader close to the RFID tag,the system can receive the position ID and identifies
the user’s position. It is assumed that RFID tags areused in outdoor environments since they do not needelectric power.
2.2.2

Estimation of User’s Position Using a Pedometer

The pedometer consists of an electronic compass and acceleration sensors. The former can detect which direction theuser walks toward. The latter can count how many steps theuser walks[16]. These two data and the user’s pace make itpossible to estimate the user’s position in the neighborhoodof components of positioning infrastructures.
When the user walks, the periodic pattern appears in
the user’s waist acceleration in the back-and-forth direction. Figure 8 shows the typical relationship between user’s
walking actions and the user’s waist acceleration in the
back-and-forth direction. In Figure 8, a solid line shows
the user’s waist acceleration in the back-and-forth direction.
Solid straight lines parallel to the vertical axis mean the time
when the user walks one step which is measured manually.
Considering these data, it is found that the user’s waist acceleration in the back-and-forth direction changes rapidly
within the fixed time while the user is walking one step. Our
system finds the user’s walking action by detecting these
rapid changes of the user’s waist acceleration. In Figure
8, dotted straight lines mean the time when the user walks
one step detected from the user’s waist acceleration. Comparing them with solid straight lines parallel to the verticalaxis, their perfect correspondence can be found. Therefore,
the user’s walking action can be detected with sufficient accuracy. The system regards the user’s waist direction measured with the compass when the user’s walking action isdetected as the user’s walking direction.
In the proposed method above, thresholds for detecting
rapid changes of the user’s waist acceleration are decided
from the user’s waist acceleration of the past. Accordingly,
the system can detect the user’s walking action accurately.
The accumulative error of the pedometer includes two kindsof errors mainly. One is the difference between the user’s
actual pace and the user’s average pace estimated in advance, and the other is the difference between the actual
walking direction and the user’s waist direction. Both ofthem are accumulated according to the user’s walking distance. In our system, the accumulative error of the pedometer is canceled whenever the system receives the position ID
from the component of positioning infrastructures.
2.2.3

Evaluation of Measureing the User’s Position

We have carried out an experiment for evaluating the measurement of the user’s position using positioning infrastruc-Acceleration [G]
1

③

②

④
0

①
10m
0

2000

User’s waist
acceleration in
the back-andforth direction

User’s walking
action manuallydetected

4000
Time [msec](a) Experimental environment

User’s walking
automaticallyaction detectedfrom theacceleration

Figure 8. Relationship between the number ofuser’s steps and the user’s waist accelerationin the back-and-forth direction.

tures and a pedometer. In the experimental environmentshown in Figure 9 (a), the user walked along arrows in
1 , 
2 , 
3 ,
the situation that IrDA markers were set up at and 
4 . In Figure 9 (b), red crosses show the user’s position at each step, blue circles show that the accumulative
error of the pedometer was reset by using positioning infrastructures. This experiment shows that the accumulative error of pedometer is at most 20 or 30 percent of theuser’s walking distance from components of positioning infrastructures. Especially, when the user walks where the2 and 3 ), theterrestrial magnetism is distorted (between accumulative error of pedometer becomes larger. However,
the accumulative error of pedometer can be bounded if components of positioning infrastructures are set up densely.

2.3 Measurement of User’s Orientation
The direction of the user’s viewpoint is measured with
the inertial sensor attached to the user’s headset. However,
it is difficult to measure the user’s direction correctly with
only the inertial sensor since it has some kinds of errors
generally. The errors of the inertial sensor include the drift
error which is caused by the change of temperatures and theaccumulative error which is caused by the high-speed rotation or being used for a long time. In our system, the drift
error is estimated using input images from the camera, and10m(b) Estimated user’s position
Figure 9. Experiment for evaluating the measurement of the user’s position.

the drift error is reduced[15]. As another issue of the inertial sensor, the data acquired from the inertial sensor are notsynchronized with input images from the camera. The asynchronousness of two data is reduced by delaying the data of
the inertial sensor. In our prototype system, the accumulative error of the inertial sensor is reset by the user manually.
In the future, it is necessary to reduce the accumulative error of the inertial sensor combining an electronic compass
and a gravity sensor.

2.4 Generation of Annotation Overlay Images
Figure 10 shows an example of annotation overlay images generated by the prototype system. Annotation overlay
images are real scene images from the user’s viewpoint withfour kinds of information shown in Table1: pointing arrows,
guiding annotations, indicators, and detailed information.
The pointing arrow and the guiding annotation are placedIndicators

Pointing Arrow

G
(i)

F

E

(g)
D

(h) 3
C
(f)
(e)

Component of
positioning
infrastructures(d)2

1 RFID tag
2 3 IrDA marker

A(c)
Guiding Annotation

Guiding annotation(a)

Detailed Information(b)
Figure 10. An example of annotation overlay
image.

Table 1. Kinds of annotations.
Pointing arrows
Guiding annotations
Indicators
Detailed annotation

Linking guiding annotations
to the objects in the real scene
Showing the names ofreal objects
Showing the state of the system
to the user
Showing the detailed informationof the real object

in the appointed points in the real scene. The closer theuser walks to a guiding annotation, the larger it is shown
to the user. Therefore, the user can intuitively link guiding
annotations to real objects. Detailed annotations show theuser detailed information about the real object. A detailed
annotation appears on the lower right side of an image while
the user notices the corresponding guiding annotations. The
system recognizes that the user notices a guiding annotation
when it is on the center of an annotation overlay image.
As mentioned above, the system needs the database ofannotations to generate annotation overlay images that arepresented to the user. The database of annotations consistsof the following data:• Image files of guiding annotations and detailed annotations,• Directions of pointing arrows,• Positions of annotations and IrDA markers.

B

1

Indoorenvironment

Figure 11. An experimental environment.

The system selects appropriate annotations from thisdatabase according to the user’s estimated position and direction, and generates the annotation overlay images in realtime.

3 Experiments with the Prototype System
We have carried out some experiments using the proposed wearable augmented reality system. We have developed a navigation system which works in our campus where
IrDA markers and RFID tags are placed at appointed points
in both indoor and outdoor environments.
Figure 11 illustrates a part of the experimental environ1 means the position wherement. In Figure 11, the point 
the RFID tag is set up, 
2 and 3 mean the positions where
the IrDA markers are set up. The user walked along the
1 to 
3 in Figure 11. Seven guiding
dotted arrows from annotations are arranged at points A,...,G in Figure 11, andoverlaid on the real scene image. Figure 12(a),...,(i) show
annotation overlay images when the user was at the point(a),...,(i) in Figure 11, and the user’s orientation was along
the solid arrows at the point (a),...,(i) in Figure 11, respectively. Figure 12(a), (b), (d), (g), and (h) show the annotation overlay images when the system identified the user’s
position using positioning infrastructures, and (c), (e), (f),and (i) in Figure 12 are ones at the user’s position estimatedwith the pedometer when the user walks some steps fromcomponents of positioning infrastructures.
Through this experiment, even if the user walks some

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 12. Examples of annotation overlay images.

meters from components of positioning infrastructures inboth outdoor and indoor environments, the system can estimate the user’s position using the pedometer and the user
can obtain the images on which annotations are correctly
overlaid. The accumulative error of the pedometer is one or
two meters at most when the user walks within ten meters.
It does not affect overlaying annotations seriously. However, the accumulative error of the pedometer increases withthe user’s walking distance estimated with the pedometer.
Thus components of positioning infrastructures should bearranged so well that the accumulative error of the pedometer does not amount so much. This experiment also showsthat the system can be used indoors and outdoors seamlessly
since the system measures the user’s position indoors andoutdoors seamlessly. The annotation overlay image can bepresented to the user at 25fps. The weight of the user’s
headset is about 200g and the user’s shoulder bag containing

the notebook PC and batteries is about 5kg. The system canrun with the battery more than three hours continuously. As
shown in Figure 12(g), when there exist occlusions among
objects in the scene, the system may overlay wrong annotations. Therefore, the user cannot intuitively link annotations
to the correct real objects.

4 SummaryThis paper has described a wearable augmented reality
system which can be used in indoor and outdoor environments seamlessly using positioning infrastructures and a pedometer. RFID tags and IrDA markers are used as components of positioning infrastructures, and the system specifies
the user’s position at appointed points by receiving position
IDs with RFID tag reader and an IrDA receiver. When theuser goes away from components of positioning infrastruc-tures, the user’s position is alternatively estimated by using
a pedometer constructed of an electronic compass and acceleration sensors. The direction of the user’s viewpoint ismeasured with the inertial sensor. We also have shown thefeasibility of the proposed system demonstrating with experiments in both indoor and outdoor environments. The
proposed system can be extended to a navigation system forfacilities or tourist attractions which cover both indoor andoutdoor environments.
The issues for improving and extending the proposedsystem are listed below:• Network sharing of the database of annotations
The database of annotations held in a server computer can be obtained automatically using the wireless network. Therefore, it is not necessary to store
the database of annotations in the user’s computer inadvance.
• Occlusions between real objects and annotations
Using a simple 3D model of the real scene, the system
could represent annotations with correct occlusion.
• Arrangement of the positioning infrastructures
IrDA markers are set up to the ceiling in indoor environments with a 9v battery severally, since IrDA markers need the electric power. In the future, we will develop IrDA marker and fluorescent light lamp units inorder to arrange and maintain them easily.
• Showing positions of components of positioning infrastructures
It is necessary to let the user know positions of components of positioning infrastructures, if the proposedsystem is applied to a wide area. We will develop aninterface to show the annotations at positions of components of positioning infrastructures.
Acknowledgments
This research is partially supported by Core Research for Evolutional Science and Technology (CREST) Program “Advanced Media Technology for Everyday Living” of Japan Science and Technology Corporation (JST) and also by the Grant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports,
Science and Technology (MEXT).

References[1] S. Mann: “Wearable Computing: A First Step Toward Personal Imaging, ” IEEE Computer, Vol. 30, No. 2, pp. 25-32,
2002.
[2] T. Starner, J. Weaver, and A. Pentland: “A Wearable Computer Based American Sign Language Recognizer, ” Proc.
1st Int. Symp. on Wearable Computers, pp. 199-202, 1997.
[3] R. Azuma: “A Survey of Augmented Reality, ” Presence,Vol. 6, No. 4, pp. 355-385, 1997.

[4] M. Kanbara, T. Okuma, H. Takemura and N. Yokoya: “A
Stereoscopic Video See-through Augmented Reality System
Based on Real-time Vision-based Registration, ” Proc. IEEE
Int. Conf. on Virtual Reality 2000, pp. 255-262, 2000.
[5] S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner,
T. Holler, and S. Sestito: “Information Filtering for Mobile
Augmented Reality, ” Proc. IEEE/ACM 1st Int. Symp. on
Augmented Reality, pp. 3-11, 2000.
[6] S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner,
T. Höller, and S. Sestito: “Personal Guidance System for the
Visually Impaired, ” Proc. Int. Conf. on Assistive Technologies, pp. 85-90, 1994.
[7] M. Kourogi, T. Kurata, and K. Sakaue: “A Panorama-based
Method of Personal Positioning and Orientation and Its Realtime Applications for Wearable Computers, ” Proc. 5th Int.
Symp. on Wearable Computers, pp. 107-114, 2001.
[8] K. Satoh, K. Hara, M. Anabuki, H.Yamamoto, andH.Tamura: “TOWNWEAR: An Outdoor Wearable MR System with High-precision Registration, ” Proc. 2nd Int. Symp.
on Mixed Reality, pp. 210-211, 2001.
[9] R. Tenmoku, M. Kanbara, and N. Yokoya: “A wearable augmented reality system using an IrDA device and a passometer, ” Proc. SPIE, Vol. 5006, pp. 478-486, 2003.
[10] P. Daehne, and J. Karigiannis: “ARCHEOGUIDE: System
Architectureof a Mobile Outdoor Augmented Reality System, ” Proc. 1st Int. Symp. on Mixed and Augmented Reality, pp. 263-264, 2002.
[11] H. Petrie, V. Johnson, T. Strothotte, A. Raab, S. Fritz, and
R. Michel: “MoBIC: Designing a Travel Aid for Blind andElderly People, ” Jour. of Navigation, Vol. 49, No. 1, pp.
44-52, 1996.
[12] S. Feiner, B. MacIntyre, T. Höller, and A. Webster: “A Touring Machine: Prototyping 3D Mobile Augmented Reality
Systems for Exploring the Urban Environment, ” Proc. 1st
Int. Symp. on Wearable Computers, pp. 74-81, 1997.
[13] A. State, G. Horita, D. Chen, W. Garrett, and M. Livingston:“Superior Augmented Reality Registration by Integrating
Landmark Tracking and Magnetic Tracking, ” Proc. SIGGRAPH’96, pp. 429-438, 1996.
[14] B. Thomas, B. Close, J. Donoghue, J. Squires, P. Bondi, M.
Morris, and W. Piekarski: “ARQuake: An Outdoor/Indoor
Augmented Reality First Person Application, ” Proc. 4th Int.
Symp. on Wearable Computers, pp. 139-146, 2000.
[15] R. Tenmoku, M. Kanbara, N. Yokoya, and H. Takemura:“Annotation Overlay with a Wearable Computer Using Augmented Reality, ” Proc. 1st CREST Workshop on Advanced
Computing and Communicating Techniques for Wearable
Information Playing, pp. 27-32, 2002.
[16] S. Lee and K. Mase: “Incremental Motion-based Location
Recognition, ” Proc. 5th Int. Symp. on Wearable Computers,
pp. 123-130, 2001.