Fly-through Heijo Palace Site
Historical Tourism System Using Augmented Telepresence
Fumio Okura

Masayuki Kanbara

Naokazu Yokoya

Nara Institute of Science andTechnology(NAIST)
8916-5, Takayama, Ikoma
Nara, Japan

Nara Institute of Science andTechnology(NAIST)
8916-5, Takayama, Ikoma
Nara, Japan

Nara Institute of Science andTechnology(NAIST)
8916-5, Takayama, Ikoma
Nara, Japan

fumio-o@is.naist.jp

kanbara@is.naist.jp

yokoya@is.naist.jp

ABSTRACT
We have developed an augmented telepresence system whichenables virtual tourism beyond time and space. Augmentedtelepresence provides a user with both the view of a remotelocation and related information using augmented realitytechniques. This study deals with the geometric and photometric registration problems to generate movie-quality augmented omnidirectional videos automatically. The user can
look around the scene from the sky above Heijo palace Site
which is an ancient capital in Nara, Japan in the technicaldemonstration.

Categories and Subject Descriptors
H.5.1 [Information Interfaces and Presentation]: Multimedia
Information Systems—Artiﬁcial, augmented, and virtual realities

Keywords
Augmented telepresence, aerial omnidirectional image, unmanned airship1.

INTRODUCTION

The developed system for demonstration superimposes
computer graphics (CG) of an old palace built 1,300 years
ago on recorded video captured from an unmanned airshipusing an augmented telepresence technique. The year 2010
was the 1,300th anniversary of Nara Heijo-kyo, an ancient
capital in Japan. Today, almost all of the ground in Heijokyo is covered by a ﬁeld of grass excepting some buildings
and bases that were physically restored for the commemorative events of the anniversary. The proposed system generates augmented videos by superimposing CG models ofbuildings that existed 1,300 years ago on the “plain grassland” images using an augmented telepresence technique.
Omnidirectional aerial videos are used to provide users witha wider view than on the ground.
In this study, augmented telepresence (AT) refers to a
type of telepresence that provides the user with both theview of a remote site and related information using augmented reality (AR) techniques. Note that this study deals
with oﬄine AT; that is, AT using recorded video, by which
the user can experience the augmented scene oﬄine. Ofﬂine AT should consider not only the problems of geometCopyright is held by the author/owner(s).
MM’12, October 29–November 2, 2012, Nara, Japan.
ACM 978-1-4503-1089-5/12/10.

ric and photometric registration like usual AR systems, butalso the immersiveness of oﬄine telepresence. Related AT
systems using recorded images cannot resolve geometric orphotometric registration problems, and/or they do not provide for immersive telepresence. Ghadirian et al. [1] visualized vegetation information of various ages in a GIS
database by superimposing CG models of plants generatedusing the database onto a single panoramic image. In addition, Grosch [2] realized an oﬄine AT system for an interior
design using a single panoramic image and generated 3D
shapes of a real scene. Although Grosch’s proposal [2] contains realtime lighting using a light source designated by theuser, the work did not use multiple omnidirectional images
to enable the user to change their viewpoints.
In this study, we have developed an oﬄine AT systemusing an unmanned airship and an omnidirectional multicamera system (OMS) [4]. This study resolves the geometricand photometric registration problems in order to generate
high-quality augmented videos using omnidirectional videos.
The omnidirectional videos captured by the OMS attachedto the airship increase the immersiveness of the telepresence by providing the user with the sense of being able tolook around the location. To render augmented scenes, we
use image-based-lighting (IBL) and global illumination (GI)techniques with an omnidirectional environmental map.

2. AUGMENTED OMNIDIRECTIONAL
VIDEO GENERATION
To overcome the geometric registration problem, the proposed system estimates the camera position and orientation. The proposed system employs oﬄine process which isa high-accuracy camera position and orientation estimation
method [6] with a video and position data measured by GPS.
Each frame of the omnidirectional video is aligned using theposition and orientation of the OMS (e.g. Figure 1(a)).
The omnidirectional video has missing area including sections that is out of ﬁeld of view of OMS and backgroundscenery occluded by the airship as shown in Figure 1(a). To
use the omnidirectional video as an environmental map forIBL, the missing area must be ﬁlled-in and complete environmental maps must be generated. To ﬁll-in missing areas
and generate a complete environmental map, other frames
in the video are searched for an area similar to the missingarea, and the pixel intensities belonging to this area is copied
to the pixels of the missing area. The all-sky-model [3] isused to ﬁll-in the areas having no corresponding areas, where
the background scene is occluded in all the frames. The allsky-model is a statistical model representing luminance of

(a) An omnidirectional image aligned by the orientation of omnidirectional camera. A missing area appears at the top of the image.

(b) An omnidirectional image with virtual buildings
using the illumination environment of real world.
Missing areas are ﬁlled using sky model.

Figure 1: Omnidirectional images generated by the proposed augmented telepresence system.

Figure 2: Appearance of virtual tourism system.
various skies with some parameters. In this study, the parameters are estimated from intensities of background scene
except missing areas.
Augmented video is rendered using a commercial GI rendering software 3ds Max (Autodesk, Inc.) and Mentalray(Mental Images GmbH.), as shown in Figure 1(b). The virtual objects are registered geometrically using the estimatedlocation and position of the OMS. An omnidirectional augmented video is generated through IBL and GI rendering
using a complete environmental map.

3.

CONFIGURATION OF
DEMONSTRATION SYSTEM

Figure 2 shows appearance of the system for the demonstration using HMD and motion tracker. The augmented
omnidirectional video is converted to view-dependent perspective video (see Figure 3) in real-time [5] using view direction of the user captured by an electromagnetic motion
tracker FASTRAK (Polhemus, Inc.) attached to the HMD.
The view-dependent perspective video is presented to theuser via a HMD.
The user can freely look around the augmented scene moving viewpoint on the captured path of the omnidirectional
image sequence. The system also switches sequences withand without virtual objects. A similar system was demonstrated at the Commemorative Events for the 1,300th Anniversary of Nara Heijo-kyo Capital, and over a thousandpeople experienced the system.

4.

ACKNOWLEDGMENTS

This research was partially supported by Grant-in-Aidfor Scientiﬁc Research (A), No. 23240024, from the Ministry of Education, Culture, Sports, Science and Technology

Figure 3: A view-dependent perspective image generated
from an augmented omnidirectional image.
(MEXT) and by the “Ambient Intelligence” project fundedby MEXT.

5. REFERENCES[1] P. Ghadirian and I. D. Bishop. Integration ofaugmented reality and GIS: A new approach to realistic
landscape visualisation. Landscape and Urban
Planning, 86:226–232, 2008.
[2] T. Grosch. PanoAR: Interactive augmentation ofomnidirectional images with consistent lighting. In
Proc. Computer Vision/Computer Graphics
Collaboration Techniques and Applications, pages
25–34, 2005.
[3] N. Igawa, Y. Koga, T. Matsuzawa, and H. Nakamura.
Models of sky radiance distribution and sky luminancedistribution. Solar Energy, 77:137–157, 2004.
[4] F. Okura, M. Kanbara, and N. Yokoya. Augmented
telepresence using autopilot airship andomni-directional camera. In Proc. Ninth IEEE Int’l
Sympo. on Mixed and Augmented Reality , pages
259–260, 2010.
[5] Y. Onoe, K. Yamazawa, H. Takemura, and N. Yokoya.
Telepresence by real-time view-dependent imagegeneration from omnidirectional video streams.
Computer Vision and Image Understanding,71(2):154–165, 1998.
[6] Y. Yokochi, S. Ikeda, T. Sato, and N. Yokoya. Extrinsic
camera parameter estimation based-on feature tracking
and GPS data. In Proc. Seventh Asian Conf. on
Computer Vision, volume 1, pages 369–378, 2006.