Fly-through Heijo Palace Site:
Augmented Telepresence Using Aerial Omnidirectional Videos
Fumio Okura, Masayuki Kanbara, Naokazu Yokoya
Nara Institute of Science and Technology (NAIST)∗(a)

(b)

(c)

Figure 1: (a) An omnidirectional image aligned by the posture of omnidirectional camera. An invisible area appears at the top of the image.
(b) An augmented omnidirectional image with virtual constructs superimposed using camera position and posture under the illumination
environment of the real world. Invisible areas are inpainted from the input image. (c) A view-dependent perspective image converted from theomnidirectional augmented image in real-time.

1Introduction

This study developed a virtual tourism system “beyond time andspace” using augmented telepresence. Telepresence is a techniquethat provides a view of a remote site using immersive displays. In
this paper, augmented telepresence (AT) refers to a kind of telepresence that provides a user with both a view of a remote site andrelated information using augmented reality (AR) techniques.
The 1300th anniversary of Nara Heijo-kyo capital, an ancient capital in Japan, was last year. The proposed system superimposes
CG of the Heijo palace which was built 1300 years ago “beyondtime” on recorded videos captured from an unmanned airship at
the remote site “beyond space.” Related AT works using recorded
videos (e.g. [Ghadirian and Bishop 2008]) cannot resolve geometric and photometric registration problems, and/or they do not provide for immersive telepresence. This study handles geometric andphotometric registration problems for automatic movie-quality registration. In addition, the proposed system uses omnidirectional
videos captured by an omnidirectional multi camera system (OMS)
equipped on an airship to increase the immersiveness of telepresence by providing looking-around behavior for the user. To renderaugmented scenes, we use image-based-lighting (IBL) and global
illumination (GI) techniques with an omnidirectional environmental map. Invisible areas of the environmental map including areas
which the OMS cannot capture and areas which the airship occludes
a background scenery, are inpainted automatically.

2Approach

To overcome the geometric registration problem, the proposed system estimates the camera position and posture. The proposed system uses non-realtime processing with a high-accuracy camera position and posture estimation method [Yokochi et al. 2006] with avideo and position data measured by GPS. Each frame of the omnidirectional video is aligned using the position and posture of theOMS (e.g. Figure 1(a)).
Environmental maps for IBL are generated from an omnidirectional∗ e-mail: {fumio-o, kanbara, yokoya}@is.naist.jp

video to realize photometric registration. Environmental maps had
to be generated without invisible areas. Such areas also decrease
the immersiveness of telepresence. To inpaint invisible areas andgenerate a complete environmental map, other frames in the video
are searched for an area corresponding to the invisible area using
SSD, and the pixel intensities belonging to this area is copied to thepixels of the invisible area. The All-sky-model [Igawa et al. 2004]is used to inpaint areas having no corresponding areas, where thebackground scene is occluded in all the frames. The All-sky-model
is a statistical model of various skies, with the intensities of pixels
calculated from some parameters estimated from the whole sky.
Augmented omnidirectional video is rendered using the complete
environmental maps and the estimated position and posture of theOMS (c.f. Figure 1(b)). Commercial software is used for the rendering process. The augmented omnidirectional video is converted
to view-dependent perspective images (c.f. Figure 1(c)) in realtime [Onoe et al. 1998], and is presented to the user via a HMD.
The user can freely look around the augmented scene. The proposed system was demonstrated at the Commemorative Events forthe 1300th Anniversary of Nara Heijo-kyo Capital, and over a thousand people experienced the system.

References
G HADIRIAN , P., AND B ISHOP, I. D. 2008. Integration of augmented reality and GIS: A new approach to realistic landscapevisualization. Landscape and Urban Planning 86, 226–232.
I GAWA , N., KOGA , Y., M ATSUZAWA , T., AND NAKAMURA , H.
2004. Models of sky radiance distribution and sky luminancedistribution. Solar Energy 77, 137–157.
O NOE , Y., YAMAZAWA , K., TAKEMURA , H., AND YOKOYA , N.
1998. Telepresence by real-time view-dependent image generation from omnidirectional video streams. Computer Vision andImage Understanding 71, 2, 154–165.
YOKOCHI , Y., I KEDA , S., S ATO , T., AND YOKOYA , N. 2006.
Extrinsic camera parameter estimation based-on feature tracking
and GPS data. In Proc. Seventh Asian Conf. on Computer Vision(ACCV2006), vol. 1, 369–378.