Cultural Heritage Tourism
with Augmented Reality

Takafumi Taketomi, Koji Makita,Tomokazu Sato, and Naokazu Yokoya
Nara Institute of Science and Technology, Japan

Cultural Heritages in Nara

Todai-ji

Kawaradera

Yakushi-ji

Heijyo-kyo

There are several now-defunct cultural heritages.

Cultural Heritage Reconstruction Methods•Physical reconstruction at the original place•Tourists can intuitively imagine the atmosphere of the time.
•Huge money need to reconstruct buildings.

•Reconstruction by CG in virtual space•Buildings are reconstructed with low financial cost.
•Tourists cannot intuitively imagine the atmosphere of the time.

•Reconstruction by augmented reality (AR) at the original place•Tourists can intuitively imagine the atmosphere of the time.
Buildings
g are reconstructed with low financial cost.
•Buildings

Requirement for reconstruction by AR:
Geometric registration of real and virtual worlds
in an outdoor environment

Geometric Registration Methods
in Augmented Reality
Vision-based approach•The method can achieve pixel-level
geometric registration.
•Robust geometric registration isdifficult in a wide area.

Correspondence between
landmarks and feature points
Landmark database

Input image[Taketomi et al. 2008]

Sensor-based approach•It is difficult to achieve pixel-level
geometric registration.
•Robust geometric registration is achievedin a wide area.
[Azuma et al. 1999]

Our Goal
Virtual cultural heritage reconstruction on the real site
in an outdoor environment
Approach
We have developed and tested two types of AR system
for realization of virtual cultural heritage tourism.
•Vision-based System
High quality AR images are generated to watch key buildings
in a limited area.

•Sensor-based System

Roughly registered AR images are generated to watch theappearance of the buildings in a wide area.

Vision-based SystemHardware Configuration
Geometric registration is achieved by extrinsic
camera parameter estimation.
PC for extrinsic camera
parameter estimation
3D Magnetic Sensor

Extrinsic camera parameters
HMD
Captured Image

Camera for lightsource estimation

PC for renderingg
AR imageg
CG models are rendered by using estimated light source [Kakuta et al. 2007].

Flow of Vision-basedGeometric Registration
Database construction in the offline process(A-1) Acquisition of depth map and surface texture(A-2) Acquisition of landmark information

Camera parameter estimation in the online process(B-1) Initialization using magnetic sensor(B-2) Search for corresponding pairsof landmarks and image features(B-3) Camera parameter estimation using correspondences

We have firstly tested previously developed the landmark-based
geometric registration method [Taketomi et al. 2008] for AR sightseeing.

Original Database Construction Method[Taketomi et al. 2008]
Distorted!!
•3-D position of landmarkmark•Image templates of different
fferent viewpointsOmnidirectional image
Generation of landmark information

Landmark DatabaseSparse 3-D informationby the SFM process
3-D measurement of the environment

Registration to the database

Problem: Appropriate compensation for image templates
only from sparse 3-D information is difficult.

Extrinsic Camera Parameter Estimation
Extrinsic camera parameters are estimated using
corresponded landmarks.
Selection of landmarks
Landmark Database

Matching

Input image

Detection of features

Extrinsic camera
parameters

AR imageCorresponded Landmark Usingan Original Landmark Database
Difficult to find corresponding
landmarks existing close to the user

Circles show positions of the corresponded landmarks in the input image.

Database Construction Using
Omnidirectional Laser Range Sensor

Withoutdistortion
ndmark•3-D position of landmarke•Image templates of different viewpoints

Omnidirectional image
Generation of landmark information

Landmark Database
Omnidirectional dense depth map
3-D measurement of the environment

Registration to the database

Image templates are appropriately compensated for
different viewpoints using a dense depth map.

Corresponded Landmarks Usinga New Landmark Database

Landmarks in the groundwere found.

Circles show positions of the corresponded landmarks in the input image.

Preliminary Test for Vision-based Geometric
Registration in The Target Environment

Sensor-based SystemHardware Configuration
AR images are displayed on the ultra-mobile PC’s display.

VAIO type U•CPU: 1.2GHz,•Camera is at the back sideof the device.
•Weight: 500g
•Angular accuracy: 1c•Angular resolution: 0.01c•Weight: 25g

InertialCube2

Appearance of the user

•Positioning accuracy: 40cm
•Sampling rate: 4Hz•Weight: 500g
P4-GPSPosition and posture of the system are acquired by GPSand inertial sensor respectively.

Reflection of lighting condition
for the CG Model
7 patterns of lighting condition are prepared and used to produce
appropriate shadows on the CG model.

AM 10:00

AM 11:00

PM 00:00

Cloudy day

PM 01:00

PM 02:00

PM 03:00

Fine dayLighting condition is manually switched depending on time and weather.

Preliminary Test for Sensor-based Geometric
Registration in The Target Environment

Public Experiment in Asuka Village
To demonstrate the effectiveness of our developed system,
we had carried out a public experiment at the Kawaradera.
•Date: November 21 ~ 23, 2009
•Time: 10:00 ~ 15:00 for each day

Kawaradera temple ruins
CG models of Kawaradera* CG models were provided by Ikeuchi Laboratory,The University of Tokyo.

Appearance of the Public Experiment

Problems• Generated AR images were sometimes felt unnatural due toincorrectly occluded people by virtual object.
• Vision-based system could not work well due to fast head motion
and hiding landmarks by people.

Huge man seems to be standingat the back of the building.

Conclusion
We have reconstructed the defunct temple withaugmented reality by using two types of system.
•Vision-based System
High quality AR images are generated in a limited area.
•Sensor-based System
Roughly registered AR images are generated in a wide area.

Future work
•Occlusion problem between virtual object and tourist
must be solved to achieve realistic AR sightseeing.
•Fusion of vision and sensor based registration is necessary
to achieve robust and accurate geometric registration.
Acknowledgements
This research was partially supported by the Ministry of Education, Culture, Sports,
Science and Technology, Grant-in-Aid for Scientific Research (A), 19200016.