Nara Palace Site Navigator:
A Wearable Tour Guide System Based on Augmented Reality
Masayuki Kanbara†, Ryuhei Tenmoku†, Takefumi Ogawa‡,
Takashi Machida‡, Masanao Koeda†, Yoshio Matsumoto†, Kiyoshi Kiyokawa‡,
Haruo Takemura‡, Tsukasa Ogasawara†, and Naokazu Yokoya††Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, 630-0192, JAPAN{kanbara, ryuhei-t, masana-k, yoshio, ogasawar, yokoya}@is.naist.jp‡Cybermedia Center, Osaka University{ogawa, machida, kiyo, takemura}@ime.cmc.osaka-u.ac.jp

Abstract
This paper describes a wearable tour guide system using augmented reality techniques which can merge the realand virtual worlds. The proposed system is supported to
be used in indoor and outdoor sightseeing sites which havesome sightseeing spots. In the proposed system, the someuser localization methods, which can be used both indoors
and outdoors, are used to guide the user along the tour routeand providing the user with multimedia contents stored in a
networked shared database. Especially, the proposed system presents annotations and virtual objects to user using
augmented reality techniques so as to be able to understandthe augmented environment intuitively. In addition, we introduce a system by which virtual objects can be merged
into a bird’s eye view image acquired by an unmanned helicopter.

1. Introduction
Since computers have made remarkable progress in resent years, a wearable computer can be realized[1, 2]. At thesame time, Augmented Reality (AR) which merges the realand virtual worlds has received a great deal of attention asa new method for displaying location-based information[3,4, 5]. If we construct an annotation overlay system for realscene using an AR technique with a wearable computer, it
can be applied for many applications[6, 7, 8, 9]. Figure 1
shows an example of annotation overlay system which imposes annotations on the real scene using AR techniqueswith wearable computers. If the AR system can be realized with a wearable computer, user can get location-based
information anyplace at any time. Therefore, the system is

Figure 1. A wearable AR application.
applicable to human navigation or tour guide systems which
provide wearable computer users with location-based information such as sightseeing information or route guide.
To realize AR systems with wearable computers, wehave to resolve some problems. The important two problems of them are user localization and management oflocation-based information provided user. The former
means the registration problem which requires to estimate
the relationship between the real and virtual worlds, to overlay virtual objects such as annotation and virtual buildings
at the accurate position of the real scene. The later includes how to provide users with appropriate informationand how to effectively visualize the information. Both ofuser localization and information management problems are
more difficult and complex than the conventional AR, because wearable computer users move anyplace and try to
get location-based information at any time.
Usually, to resolve the user localization problem, the position and orientation of user’s viewpoint should be measured in real time. A number of methods have been proposed for localization in indoor and outdoor environments.
For examples, Global Positioning System (GPS) is used

Figure 2. The aerial photograph of Narapalace site.

to measure the user’s position outdoors. In AR systems
which use GPS, a differential GPS or a real time kinematicGPS[10, 11] is often used to measure the user’s position
accurately. For indoor environments, several approaches
which use fiducial markers or IR markers as positioning infrastructures are proposed. On the other hand, the problem of information management has not been investigatedin detail yet. In general, a user’s computer needs to holduser’s location-based information to overlay annotations on
the real scene image. Up to this time, since a database ofannotation information is usually held in a wearable computer in advance, it is difficult for the database of annotation
information to be easily updated or added by information
providers.
This paper introduces new approaches which overcome
both of the user localization and information management
problems. We selected Nara palace site as a platform of experiment environment. Nara Palace Site Navigator which isa wearable tour guide system using augmented reality techniques is constructed. Nara palace site is the remains ofJapanese ancient capital palace which existed 1300 years
ago. Nara palace site shown Figure 2 has 2 square kilometers and locates in the west of Nara city. It has museums
which exhibit hangover and an outdoor archaeological sitewhich has scattered sightseeing spots such as ancient building excavation sites. We propose user localization methods
for this experiment environment which includes indoor andoutdoor scenes. In addition, we attempt to generate an augmented scene which is captured from bird’s eye view. Finally, we describe a networked shared database of locationbased information and view management method for effective visualization management.

2. Framework of Nara Palace Site Navigator
Some navigation systems using a wearable computer ormobile device have already been proposed until now[12,13, 14]. ARCHEOGUIDE system is a mobile system for
guiding tourists[15, 16]. This system is intended to presentarchaeological information to the user at cultural-heritage
sites. The archaeological information in this system includes virtually reconstructed buildings which are overlaid
on ruins of buildings using augmented reality techniques.
This system demonstrates that augmented reality provides
an effective method for guiding tourists. We try to establish a wearable AR system which can provide the user withaugmented scene in several experiment environments.
Figure 3 shows an overview of Nara palace site navigator. The user can acquire generated augmented scene as
guide information under three kinds of environments including indoor and outdoor scenes and bird’s eye view. In
outdoor environments, to estimate user’s position widely,
the position is measured by combining the user’s absolute
position acquired by GPS and infrared marker and deadreckoning data. In indoor environments, to measure theuser’s position accurately, the infrared marker and fiducial
marker set up on the ceiling of building are used as positioning infrastructure. On the other hand, the augmented scenefrom bird’s eye view is generated by measuring the helicopter’s position and orientation using GPS and gyroscopeattached to the helicopter. We also propose a management
method for location-based information. The proposed approaches are based on using a networked shared database,which can effectively provide the user with appropriate information and intuitive visualization.
This paper is structured as follows. Sections 3 and 4
describe the localization method in outdoor and indoor environments, respectively. Section 5 describes the method
and experiments of bird’s eye view augmentation. In Section 6, the approaches of networked shared database andview management are described in detail. Finally, Section
7 summarizes the present work and also describes future
work.

3

Outdoor user localization

To acquire the user’s position in wide outdoor environments, the position is measured by combining specificationof the user’s absolute position and dead reckoning data as
shown in Figure 4[17]. In the proposed method, the user’s
position is measured by combining the user’s absolute position which is measured by the GPS and the user’s relative
movement which is measured by the pedometer.

3.1

Specification of user’s absolute position

The user’s absolute position is measured using a handheld GPS in outdoor environments. When the GPS is usedsolely for specifying the position in outdoor environments,
the error fluctuates between 5 meters and 50 meters which

Outdoor AR
Outdoor localization
using GPS and dead reckoning

Management of location-based information

Indoor AR

Bird’s eye view AR• Networked shared database• View management

Indoor localization using
infrared marker and fiducial marker

AR system using unmanned helicopter

Figure 3. Overview of Nara Palace Site Navigator.

Specifying absolute position

Dead reckoningGPSsatellite(a) DRM-III.

(b) MDP-A3U9.

Figure 5. Dead reckoning sensors.

Figure 4. The proposed positioning method.

is caused by various reasons: the number of captured GPS
satellites, the weather, and so on. In environments whichhave no high-rise buildings, the error of the standard GPS is
within 20 meters. The GPS can specify the user’s position
at 1Hz.

rection. However, the sensor just like ”MDP-A3U9 (NECTOKIN)” (Figure 5 (b)) which is composed by acceleration
sensors and a digital compass also can estimate the user’s
relative movement. The ”MDP-A3U9” can measure the acceleration of the user’s waist to detect the user’s walking
action using the acceleration sensor and measure the user’s
walking direction using the digital compass.

3.3
3.2

Estimation of user’s relative movement

We can estimate the user’s relative movement using a pedometer represented by the dead reckoning module, ”DRMIII (Point Research Corporation)” as shown in Figure 5 (a)both indoors and outdoors. The ”DRM-III” estimates theuser’s relative movement by integrating these three data: theuser’s average pace which is input beforehand, the detection of the user’s walking action, and the user’s walking di-

Combining the absolute position and deadreckoning

In our method, even when the accuracy of the GPS drops
temporarily, the user’s current position is measured by using past GPS data and dead reckoning data. Thus the positioning accuracy is defined as the sum of the GPS accuracy which is included the standard GPS data, NMEA data
and 5% of the distance traveled. The system calculates thevalue for GPS data at every moment and adopts the position

Infrared
markerStereo-camera
Orientation
sensorIrID

IrDA signal
decoder(a)(b)

Triangulation

HMD
PC

Get data fromsensors

Overlay navigation
information

Hand-heldmouse
Location detection

(c)

(d)

Figure 6. Examples of generated images.

data whose value of positioning accuracy is minimum as theuser’s position.
The user’s position can be measured at user’s step rateby using the dead reckoning method with specifying theuser’s absolute position. Therefore, the user’s position canbe measured with the same timing as the user’s movement.
However, the accumulative error occurs according to theuser’s walking distance when we use the proposed deadreckoning method. The accumulative error includes two
kinds of errors mainly. One is the difference between theuser’s actual pace and the user’s average pace estimated inadvance, and the other is the difference between the actual
walking direction and the user’s waist direction. Both ofthem are accumulated according to the user’s walking distance. In our method, the accumulative error of dead reckoning is canceled whenever the system receives the GPS
data.

3.4

Figure 7. Infrared marker based indoor tracking technique.

4.1

We have developed an indoor position and orientation
tracking system, which combines infrared markers with ahead-mounted stereo camera and an orientation sensor as
shown in Figure 7. In this system, most markers can be casually placed without the need for calibration, as unknown
markers in the workspace are automatically mapped. The
proposed method is robust with respect to lighting conditions, also working in complete darkness, and is less obtrusive in terms of aesthetics compared with conventional
marker-based approaches. An extended Kalman filter is employed to reduce triangulation and orientation errors by integrating the signals acquired by multiple sensors. The location accuracy of the system was evaluated through a seriesof experiments that showed that the error for a stationary
subject is less than about 10 cm, while that for a moving
subject is within 10% of the movement distance.

Examples of outdoor augmented scene
4.2

We have developed a prototype wearable augmented reality system which presents location-based information in
outdoor environments. The user’s position is measured byusing the proposed method. Figure 6 shows examples ofgenerated images which are presented to the user. Figures
6(a) and (b) show a position-based annotation overlay image. Figure 6(c) and (d) show images on which a virtual
building is overlaid.

4

Infrared marker based hybrid tracking

Indoor user localization

We have developed two kinds of indoor position and orientation tracking systems[18]. We explain the proposedsystems in detail in the following sections.

Fiducial marker based technique

We have developed another indoor user positioning technique, which is inexpensive, easy to setup, and available in awide area. This technique uses many fiducial markers casually stuck on the ceiling with a camera looking upward. In
the technique, an arbitrary small number of fiducial markers
have to be calibrated, while the rest of them can be roughlyplaced. Positions of uncalibrated markers are propagatively
estimated based on adjacent markers. User’s global position is then detected based on the estimated marker positions. Experimental results have shown that jittering is a
few millimeter and accumulative error is not critical. Thisapproach is applicable to a large scale indoor environments
since a marker can be roughly placed in a short time.

(a) Helicopter(b) Operater(c) Omnidirectional image(d) Perspective image

Figure 10. Measured GPS data and gyroscope
data.

Figure 8. Immersive teleoperating system.

Figure 11. Annotation and virtual Nara palace
overlaid on perspective images.

Figure 9. Bird’s eye view AR system overview.

5

Augmented reality of bird’s eye view scene

We have proposed an immersive teleoperating
system[19] for unmanned helicopters using an omnidirectional camera as shown in Figure 8. We applyed thedeveloped system to an annotation-based guide system.
User can see bird’s eye view augmented scene at Narapalace site. Figure 9 shows the overview of this system.
On the helicopter, an omnidirectional camera, a GPS,a gyroscope, and a PC are mounted. Position/attitude
data and an omnidirectional image are transmitted to theoperator during the flight via a wireless LAN. A perspective
image is generated from a received omnidirectional image,and displayed on the Head Mounted Display(HMD) which
the operator wears. Annotations to real objects and virtual
objects are overlaid on the perspective image. The operator
has a database of annotations which consists of the names
and the position information of neighboring objects. The
displayed image changes depending on the head direction
which is measured by the gyroscope attached to the HMD.
The operator holds a controller with both hands andcontrols the helicopter wearing a backpack which contains
a laptop PC.

We have carried out an experiment using this system.
Figure 10 indicates GPS and gyroscope data. Figure 11
shows examples of the annotation overlaid images. Using
the current position and attitude of the helicopter, some annotations and virtual Nara palace site are overlaid on theperspective image. The attitude of the operator’s head isdisplayed at upper side and left side of the image. Additionally, the attitude of the helicopter is indicated by a virtual helicopter at left below of the image. The position ismeasured correctly and annotations are overlaid on actual
buildings generally.

6

Management of location-based information

In this session, we describe management methods for thelocation-based information in wearable AR system. In thissection, as a first step, we handle only annotation information such as a name of building. To effectively manage
the information, we propose some techniques: networked
shared database for wearable AR, development toolkit forthe shared database, and visualization method.

6.1

Shared database of annotation information

Figure 12 shows an overview of shared database systemof annotation information[20]. In this study, the databaseis shared via a wireless network. The database of annotation information is stored in the server and is shared bymultiple users of wearable AR systems and information
providers. Consequently, users of wearable AR systems
can obtain annotation information at anytime via a wireless network and can see the newest annotation overlay images without holding the database of annotation information in advance. On the other hand, information providers
can provide efficiently the newest annotation information
for users of wearable AR systems by updating and adding
the database with a web browser. In Section 6.1.1, the construction of the database of annotation information is described. Section 6.1.2 describes how to update the databasewith a web browser. Section 6.1.3 describes how the user
obtains annotation information.
6.1.1 Construction of annotation database
The database contains some kinds of location-based contents. Each annotation is composed of a pair of contents(name and detail) and their positions. Components of
the annotation information are described in detail in the following.
Position: Three-dimensional position of an annotation in
the real world. Three parameters (latitude, longitude,height) are stored in the database.
Name: The name of the object which is overlaid in the realscene as annotation information.
Detail: Detailed information about the object to be annotated. When user’s eyes are fixed on the object, thedetail about the object is shown in the lower part of theuser’s view.
6.1.2 Updating the shared database
The annotation information can be corrected, added anddeleted by information providers with a web browser. An
interface for information providers to update the database isa web browser as shown in Figure 13. Information providers
can easily update the database by accessing a prepared web
page and by transmitting the data of annotation information.
The annotation updating procedure is described below.
1. Specification of position
Information providers can zoom in and out to maps
(Figure 13: C, D) using buttons (Figure 13: A). Besides, the providers can move in the map by clicking
any point on the map. In this way, the providers canFigure 12. Overview of a networked annotation databese shared by users.

specify the position of a new annotation to be added.
It should be noted that position parameters such as
latitude, longitude and height are automatically determined based on the specified position on the map.
2. Input of name
Information providers input the object name to web
page. The name is sent to the server and the pictureof annotation is automatically generated in the server.
3. Input of details
Using the same method as in the input of the name,information providers send details of objects. The
providers also can send a picture, a sound file and amovie file as details.
The providers can efficiently send the newest annotation
information using a web browser. Note that users of wearable AR systems can also update the database. In this case,the server shows the user a map of his neighborhood according to the user’s position acquired by positioning sensors. Since the user is able to update the database, the user
can immediately correct the position error of annotations byconfirming the overlaid image.
6.1.3 Getting annotation information
In this work, the database server is prepared assuming thatthe user’s wearable computer can access the database via awireless network. Annotations to be presented to the user
are determined based on the user’s current position. First,
the user’s position is measured by some positioning sensors
such as GPS which are equipped by the user. The user’s

Client 1’s view

Client 2’s view

Figure 15. Clients’ view in the experiment.

Figure 13. Input form of annotation information.
Client 1

ServerPosition

Annotation
Annotation

MIDDLEMIDDLEMIDDLE

NEAR

NEAR

NEAR

(a) Depth of field(b) Relative size(c) Aerial perspective(frame)

FAR

FAR

FAR

MIDDLEMIDDLEMIDDLE

NEAR

NEAR

NEAR
(f) Motion parallax

User’s view

Figure 16. Metaphors of monocular depth
cues for depth visualization.

HMD

Annotation

User
Program

Position

Figure 14. A data flow diagram of a networked
wearable AR system architecture.

wearable computer then obtains proper annotation information based on the measured user’s position. The server automatically decides which annotation should be provided.
Consequently, the user’s wearable system can obtain thenewest annotation information at anytime.

6.2

FAR

(d) Aerial perspective (e) Texture gradient(signboard)

AnnotationComm.
library

DB Manager

FAR

Rendering library
Annotation

Addition,
Deletion, etc.

User’s view

Client 2

Annotation

Annotation
DB

HMD

Renderinglibrary

DB library
SearchUser
ProgramAnnotation

PositionComm. library

Annotation

Annotation
Position

PositionComm. library

Position

User
programFAR

Development toolkit for networked wearable
AR systems

We have also developed a development toolkit for a networked wearable AR system. Our toolkit is designed withconsideration of annotation data volume, ease of annotation
updates and hardware architecture diversity. In particular,
the toolkit hierarchically manages annotation data based ona geographic distribution of clients and annotation. The system transmits annotations to clients using dedicated priority management architecture. Specifically, user preference,location, viewing direction, and levels of details etc. aretaken into consideration for fast and effective data transfer.
An experiment shows that annotation information was properly transmitted from the server to two clients that use theproposed toolkit and different position detection techniques,
based on their position information (see Figures 14 and 15).

6.3

Annotation visualization techniques

We have proposed a number of depth visualization techniques for virtual object annotations. It is often difficult toperceive the depth of virtual objects in an augmented reality (AR) environment due to poor depth cues and visualcongestion. Existing techniques are intended primarily fornormal objects having 3D geometry. Our techniques dynamically modify visual attributes of each annotation, suchas color or line-style of the frame, based on its distance tothe user or occlusion relationship between the annotationand the real environment to facilitate perception of annotations’ location and spatial relationship (see Figure 15).
Preliminary experiments indicate that three representative
depth visualization methods are effective for reducing response time, depth recognition error, and head moving distance. These approaches should be particularly effective forwearable AR systems that use a monocular HMD.

7

SummaryThis paper has described wearable tour guide systems
and related augmented reality for Nara palace site whichis the remains of Japanese ancient capital palace. In particular, we proposed the techniques of indoor and outdoor
user localization, bird’s eye view augmentation and management methods of location-based information for wearable computer users. The user can acquire the guide information in three environments: outdoor, indoor and bird’s
eye view scenes. We also have proposed the management
method of overlaid information. The proposed methods are
the networked shared database for effectively providing theuser with the annotation information, and the visualization
method of overlaid information. For future work, we plan
to improve the Nara palace site navigator as a guide tour
application and to integrate the proposed localization methods.
Acknowledgments
This research is partly supported by Core Research for Evolutional Science and Technology (CREST) Program ”Advanced Media Technology for Everyday Living” of Japan
Science and Technology Agency (JST).

References[1] S. Mann: “Wearable computing: A first step toward personal
imaging,” IEEE Computer, 30, 2, 1999.
[2] R. D. Vaul, M. Sung, J. Gips and A. S. Pentland: “Mithril
2003: Applications and architecture,” Proc. of Int. Symp. on
Wearable Computers 2003, pp. 4–11, 2003.
[3] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Juiler
and B. MacIntyre: “Recent advances in augmented reality,”IEEE Computer Graphics and Applications, 21, 6, pp. 34–
47, 2001.
[4] S. Feiner, B. MacIntyre and D. Seligmann: “Knowledgebased augmented reality,” Communications of the ACM, 36,7, pp. 52–62, 1993.
[5] M. Kanbara, N. Yokoya and H. Takemura: “Registration forstereo vision-based augmented reality based on extendible
tracking of markers and natural features,” Proc. of IAPR Int.
Conf. on Pattern Recognition, II, pp. 1045–1048, 2002.
[6] T. Kurata, T. Kato, M. Kourogi, J. Keechul and K. Endo: “Afunctionally-distributed hand tracking method for wearable
visual interfaces and its applications,” Proc. of IAPR Workshop on Machine Vision Applications, pp. 84–89, 2002.
[7] M. Kourogi, T. Kurata and K. Sakaue: “A panorama-based
method of personal positioning and orientation and its realtime applications for wearable computers,” Proc. of Int.
Symp. on Wearable Computers, pp. 107–114, 2001.
[8] W. Piekarski and B. Thomas: “Bread crumbs a techniquefor modelling large outdoor ground features,” Proc. of Int.

Symp. on Mixed and Augmented Reality, pp. 269–270,
2002.
[9] S. Feiner, B. MacIntyre, T. Höller, and A. Webster: “A
touring machine: Prototyping 3d mobile augmented reality
systems for exploring the urban environment,” Proc. of Int.
Symp. on Wearable Computers, pp. 74–81, 1997.
[10] T. Hollerer, S. Feiner and J. Pavlik: “Situated documentaries:
Embedding multimedia presentations in the real world,”
Proc. of Int. Symp. on Wearable Computers ’99, pp. 79–86,
1999.
[11] T. Hollerer, S. Feiner, T. Terauchi, G. Rashid and D. Hallaway: “Exploring mars: Developing indoor and outdoor userinterfaces to a mobile augmented reality system,” Computers
and Graphics, 23, 6, pp. 779–785, 1999.
[12] R. Oppermann and M. Specht: “Adaptive mobile museum
guide for information and learning on demand,” Proc. of HCIInternational ’99, pp. 642–646, 1999.
[13] J. Baus, A. Kruger and W. Wahlster: “A resorce-adaptive
mobile navigation system,” Proc. of Int. Conf. on Intelligent
User Interfaces 2002, pp. 15–22, 2002.
[14] R. Malaka and A. Zipf: “Deep map - challenging it researchin the framework of a tourist information system,” Proc.
of Int. Congress on Tourism and Comm. (ENTER2000),
pp. 15–27, 2000.
[15] D. Stricker, J. Karigiannis, I. T. Christou, T. Gleue andN. Ioannidis: “Augmented reality for visitors of cultural heritage sites,” Proc. of Int. Conf. on Cultural and Scientific Aspects of Experimental Media Spaces, pp. 89–93, 2001.
[16] T. Gleue and P. Dähne: “Design and implementation of amobile device for outdoor augmented reality in the archeoguide project,” Proc. of Archaeology, and Cultural Heritage Int.
Symp., 2001.
[17] R. Tenmoku, M. Kanbara and N. Yokoya: “A wearable augmented reality system using positioning infrastructures anda pedometer,” Proc. of Int. Symp. on Wearable Computers,
pp. 110–117, 2003.
[18] M. Maeda, T. Ogawa, T. Machida, K. Kiyokawa and
H. Takemura: “Indoor localization and navigation using
ir markers for augmented reality,” Proc. of Int. Conf. on
Human-Computer Interaction, pp. 283–284, 2003.
[19] M. Koeda, Y. Matsumoto and T. Ogasawara: “Developmentof an immersive teleoperating system for unmanned helicopter,” Proc. of Int. Workshop on Robot and Human Commnunication, pp. 47–52, 2002.
[20] K. Makita, M. Kanbara and N. Yokoya: “Shared database ofannotation information for wearable augmented reality system,” Proc. of SPIE Electronic Imaging, 5291, pp. 464–471,
2004.