Fly-through Heijo Palace Site:

Augmented Telepresence Using Aerial Omnidirectional Videos
Fumio Okura, Masayuki Kanbara, Naokazu Yokoya

Nara Institute of Science and Technology (NAIST) Email: fumio-o@is.naist.jp
Goal and approach

Capturing omnidirectional video

Realizing offline augmented telepresence (AT) system
in wide outdoor environment using aerial views

Augmented Reality(AR): beyond time

Registration problems
between real and virtual worlds• Geometric registration• Photometric registration

Telepresence: beyond space

Augmented Telepresence(AT): beyond time and spaceare accurately resolved offline.

Space-time AT with Heijo-kyo,an old Japanese capital

CG of ancient Heijo-kyo capital(Courtesy of Toppan Printing Co., Ltd.)

Other applications:Map, Landscape simulation, etc.

Unmanned airship• Length: 12m• Payload: 15kg
• Max. speed: 50km/h
The largest battery-powered
unmanned airship in the world.

Equipments• Omnidirectional Multi-camera System (OMS)
Ladybug3 (Point Grey Research, Inc.)• Differential GPS
P4-GPS (Hitachi Zosen Corp.)• Fiber-optic gyroscope
TISS-5-40 (Tokyo Keiki)

Geometric registration

Photometric registration(1) Camera position and posture estimation using structure-frommotion (SfM) and GPS measurement [Yokochi, et al.]Environmental maps for
Image-based-lighting (IBL) include“missing area.”

Structure-from-motion

GPS measurement

Estimation process uses
only video.
× Accumulative errors appear.
× Scale factor is unknown.

× Posture cannot be measured.
No accumulative errors appear.
Scale factor is acquired as
GPS coordinate system.

Estimating camera position and posture
without accumulative errors with scale factor.

Capturedomnidirectional image(1) Completion of ground area
1. An area similar to the missing area issearched from previous frames using
Sky area
SSD criterion.
Groundarea
2. Intensities of corresponding pixels
are copied to the missing area.

Geometric registration:
Aligned omnidirectional
image(2) Completion of sky areaN(2) Alignment of the omnidirectional images
1. Omnidirectional images are mapped on
a sphere.
T
2. The sphere is rotated by 𝐑−1=𝐑𝑖𝑖.

1. Aligned sky images are
unified in the wholesequence.

NorthHorizon

Photometric registration:
Rendered augmentedimage

𝐑 i : Estimated posture of OMS(3) Posture refinement of OMS
1. Optical-flows near the horizon of two consecutive frameson aligned video are calculated.
2. 3DOF rotation of two consecutive frames 𝐑 𝐞𝐫𝐫 is estimated byminimizing an energy function defined as the sum of squares oflengths of the optical-flows mapped on the sphere.
𝐓
3. 𝐑−𝟏=𝐑𝐞𝐫𝐫
𝐞𝐫𝐫 is multiplied with posture of OMS.
4. The processes are applied to the whole sequence.

2. Parameters inAll Sky Model [Igawa, et al.]are estimated from theunified image.
3. Intensities in the missing
area are copied from theestimated model.

Parameter estimationin All Sky Modelcompare
CloudySi = 0.0

・・・・・・・・・・・・・・・・・・・

Sunny
Si = 1.0 Si = 2.0

・・・・・・

Si = 0.6Min.
difference(3) Rendering augmented images

View-dependent
perspective images

Y. Yokochi, S. Ikeda, T. Sato and N. Yokoya: “Extrinsic camera parameter estimation based-on feature
tracking and GPS data,” Proc. Asian Conf. on Computer Vision (ACCV2006), Vol. I, pp. 369–378, 2006.

Vision and Media Computing Laboratory, Graduate School of Information ScienceNara Institute of Science and Technology (NAIST)

Augmented images are rendered using commercial global illumination(GI) rendering engine, 3ds Max (Autodesk, Inc.) with IBL.
N. Igawa, Y. Koga, T. Matsuzawa and H. Nakamura: “Models of sky radiance distribution and sky luminancedistribution,” Solar Energy, Vol. 77, pp. 137-157, 2004.