NAIST-IS-DD1161016

Doctoral Dissertation

Imaging and Rendering Framework for
Photorealistic Mixed-Reality World Exploration

Fumio Okura

March 24, 2014

Department of Information Science
Graduate School of Information Science
Nara Institute of Science and Technology

A Doctoral Dissertation
submitted to Graduate School of Information Science,
Nara Institute of Science and Technology
in partial fulfillment of the requirements for the degree ofDoctor of ENGINEERING
Fumio Okura
Thesis Committee:
Professor Naokazu Yokoya
Professor Hirokazu Kato
Associate Professor Masayuki Kanbara
Associate Professor Tomokazu Sato

(Supervisor)(Co-supervisor)(Co-supervisor)(Co-supervisor)

Imaging and Rendering Framework for
Photorealistic Mixed-Reality World
Exploration∗
Fumio Okura

Abstract
The exploration of the real world in a virtual simulation has become oneof the most important applications of augmented virtuality (AV) in accordancewith the popularization of virtual globe applications (e.g., Google Earth), where
AV is one aspect of mixed reality (MR) research fields according to the realityvirtuality continuum. A large-scale physical environment can be virtualized using
modern three-dimensional (3D) reconstruction techniques, such as vision-based
and multi-sensor integration approaches. Image-based rendering provides promising improvements to the appearance of automatically reconstructed 3D models,
which usually include errors in 3D shape.
In certain applications that allow an interactive virtual exploration of the realworld, virtual objects can be superimposed for the visualization of disaster areas
or non-existent buildings. In such applications, the virtual objects have to berendered frame by frame. Real-time rendering approaches have been studied forphotometric registration between real and virtual objects in augmented reality(AR), and have become one of the most important research fields in the areaof MR. The goal of these researches is to achieve the same quality as oﬄine
rendering, but this has not been achieved yet.
This study proposes a framework for acquiring real-world scenes, and rendering both the real and virtual worlds for photorealistic MR-world exploration.
∗

Doctoral Dissertation, Department of Information Science, Graduate School of Information
Science, Nara Institute of Science and Technology, NAIST-IS-DD1161016, March 24, 2014.

i

Spherical and/or high-dynamic range (HDR) imaging techniques have been studied for photorealistic virtual-object superimposition using image-based lighting.
Most studies have dealt with imaging on the ground, whereas some diﬃculties
remain in spherical aerial imaging for realizing a bird’s-eye view, which is used
for most virtual globe applications. This thesis first describes two approaches
for acquiring full spherical aerial images using an unmanned airship and omnidirectional cameras, where the images can be used for the photorealistic superimposition of virtual objects. In addition to MR-world exploration, the acquired
images/videos can be employed for aerial-immersive panoramas, i.e., an extension of Google Street View type applications for a bird’s-eye view. To achieve
the photorealistic superimposition of virtual objects onto a virtualized real-world,
we propose a rendering approach that combines the oﬄine rendering of virtual
objects with image-based rendering. The proposed rendering approach takes advantage of the higher quality of oﬄine rendering without the high computational
cost of online computer graphics (CG) rendering, i.e., it incurs only the cost of
online computations for light-weight image-based rendering.
Using the proposed imaging and rendering frameworks either fully or partly,
various MR applications can be developed. We also demonstrate some examples
of such MR applications based on the proposed imaging and rendering frameworks.

Keywords:
mixed reality, high-dynamic range imaging, omnidirectional imaging, photorealistic rendering, free-viewpoint image generationii写実的な複合現実空間生成のための画像取得およびレンダリングフレームワーク ∗
大倉 史生

内容梗概

Google Earth 等の Virtual Globe アプリケーションの普及とともに、インタラ
クティブに視点を移動しながら現実空間を閲覧することは、複合現実感（Mixed
Reality: MR）の一分野である Augmented Virtuality (AV) における最も重要な応用の一つとなっている。近年、これらの応用として、事前撮影された画像群を用いた多視点ステレオ等による三次元復元により実空間を仮想化し、効率的に広範
囲の情景を提示することが可能となってきた。さらに、自動復元された三次元形状の誤差に起因する見えへの悪影響を軽減するため、イメージベースの画像生成アプローチを取り入れた自由視点画像生成についての研究が盛んに行われている。
インタラクティブに実空間の閲覧を可能とするアプリケーションにおいて、災害や建造物等の景観シミュレーションのために、仮想物体を合成する応用が考えられる。そのためには、提示画像のフレームごとに、リアルタイムのレンダリングにより仮想物体の見えを計算する必要がある。特に拡張現実感（Augmented
Reality: AR）において、CG 分野で盛んに研究されてきたリアルタイムレンダリ
ング手法を AR に適用することにより光学的整合性を向上することが試みられてきたが、現在も、これらの手法はオフラインで時間をかけたレンダリング品質には追いついていない。
本論文は、実空間と仮想物体が写実的に融合された複合現実空間を構築する
ための、画像取得および実・仮想環境のレンダリングのためのフレームワーク
について述べる。従来、仮想物体合成のための実空間の撮影手法として、全方位カメラ等を用い、全天球かつ高ダイナミックレンジの画像を取得する手法が実現されてきた。しかし、多くの手法は地上撮影を対象としており、多くの Virtual∗

奈良先端科学技術大学院大学 情報科学研究科 情報科学専攻 博士論文, NAIST-IS-DD1161016,
2014 年 3 月 24 日.

iii

Globe アプリケーションで用いられるような空からの視点での全天球撮影は困難であった。そこで、本論文ではまず、複合現実空間生成に適した空撮手法として、
全方位カメラと無人飛行船を用いた不可視領域のない全天球映像の生成方式を提案する。撮影された全天球映像は複合現実空間の生成だけでなく、Google Street
View に代表される実環境の見回しアプリケーション等を空からの視点に拡張することができる。本論文ではさらに、全天球撮影に基づき仮想化された現実空間に仮想物体を写実的に合成するための、オフラインレンダリングと自由視点画像生成を融合した仮想物体合成アプローチを提案する。本研究で提案する仮想物体
合成は、オンライン処理における仮想物体のレンダリングのための計算量を必要とせず、簡略化された自由視点画像生成の処理コストのみで複合現実空間を提示可能とする。
画像取得およびレンダリングフレームワークを部分的に用いることによって、様々なアプリケーションを構築可能である。本論文では、フレームワークの提案だけでなく、それらを用いた複合現実感アプリケーションの構築例についてもあわせて述べる。

キーワード
複合現実感, 高ダイナミックレンジ画像, 全方位画像, 写実的レンダリング, 自由
視点画像生成iv

Contents
Chapter 1

Introduction

1

Chapter 2
Related Work2.1. Imaging Technique for MR-World Generation . . . . . . . . . . .
2.1.1 Field of View . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 Dynamic Range . . . . . . . . . . . . . . . . . . . . . . . .
2.1.3 Capturing Location . . . . . . . . . . . . . . . . . . . . . .
2.2. Rendering Technique for Real and Virtual Objects . . . . . . . . .
2.2.1 Real-World Virtualization . . . . . . . . . . . . . . . . . .
2.2.2 Superimposition of Virtual Objects . . . . . . . . . . . . .
2.3. Contribution of This Study . . . . . . . . . . . . . . . . . . . . .

6
6
7
10
13
20
21
27
29

Chapter 3
Full Spherical Aerial High Dynamic Range Imaging 35
3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2. Spherical Aerial Image Completion Using Sky Model . . . . . . . 36
3.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.2 Aerial Imaging Using an OMS Mounted on an Aerial Vehicle 37
3.2.3 Completion of Missing Areas on Ground Scenery . . . . . 38
3.2.4 Completion Based Only on the Intensity of Spherical Images 39
3.2.5 Completion of Remaining Missing Area Using All Sky Model 41
3.2.6 Experiment with Spherical Image Completion . . . . . . . 43
3.3. Full Spherical Aerial HDR Imaging Using Two OMSs . . . . . . . 46
3.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3.2 Multi-Exposure Aerial Image Capture from Two OMSs . . 48
3.3.3 HDR Image Generation from Multi-Exposure Images . . . 52v

3.3.43.3.5

Composition of HDR Images Captured by Two Cameras .
Experiment with Spherical Image Generation Using Two
Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4. Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1 Comparison of Two Approaches . . . . . . . . . . . . . . .
3.4.2 Auto-Exposure Control . . . . . . . . . . . . . . . . . . . .
3.5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

56
58
63
63
64
66

Chapter 4
Photorealistic Rendering for MR-World Exploration 67
4.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2. Free-Viewpoint Image Generation Framework for Photorealistic
Superimposition of Virtual Objects in Real-World Virtualization . 68
4.3. Fly-Through Application Using Full Spherical Aerial Images, 2D
Grid Structure, and VDTM . . . . . . . . . . . . . . . . . . . . . 70
4.3.1 Oﬄine Process: Photorealistic Superimposition of VirtualObjects on Structured Viewpoints . . . . . . . . . . . . . . 72
4.3.2 Online Process: Free-Viewpoint Image Generation with Reduced Computational Cost . . . . . . . . . . . . . . . . . . 75
4.4. Experiment of Application Using 2D Structure . . . . . . . . . . . 76
4.5. Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.5.1 Quality of Real-World Images on Structured Viewpoints . 80
4.5.2 Quality of Virtual-Object Superimposition of Oﬄine Process 80
4.5.3 Density of Structured Viewpoints . . . . . . . . . . . . . . 82
4.5.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Chapter 5

Photorealistic MR Applications Based on Proposed
Framework91
5.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.2. HDR Immersive Panorama . . . . . . . . . . . . . . . . . . . . . . 92
5.2.1 Immersive Panorama System with HMD and Head Tracker 93
5.2.2 Display Methods for Spherical Aerial HDR Images . . . . . 95
5.2.3 Subjective Evaluation Using Immersive System . . . . . . 100
5.2.4 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . 101vi

5.3. Augmented Immersive Panorama . . . . . . . . . . . . . . . . . .
5.3.1 Augmented Immersive Panorama System Using Full Spherical Aerial Image Sequence . . . . . . . . . . . . . . . . . .
5.3.2 Public Experiment . . . . . . . . . . . . . . . . . . . . . .
5.4. Real-Time AR . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1 Oﬄine Process: Rendering of Virtual Objects at Structured
Viewpoints . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.2 Online Process: Free-Viewpoint Image Generation . . . . .
5.4.3 Mobile Device Experiment . . . . . . . . . . . . . . . . . .
5.4.4 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5. MR-World Interactive Exploration . . . . . . . . . . . . . . . . .
5.5.1 Fly-Through Application using 3D Structure . . . . . . . .
5.5.2 Walk-Through Application Using Spherical Images Captured on Ground, 1D Structure, and Morphing . . . . . . .
5.6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 6

Conclusions

103
103
105
106
106
108
108
109
112
112
113
120
121

Acknowledgements

124

References

126

List of Publications

145

Appendix A Spherical HDR Imaging on the Ground Using OMS 151

vii

List of Figures
1.1
1.2

Reality-virtuality continuum [MK94] and augmented virtuality techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Basic pipeline of MR-world generation. . . . . . . . . . . . . . . .

2.1
2.2
2.3

Fisheye camera and acquired image in [XT97]. . . . . . . . . . . .
HDR radiance reconstruction from multi-exposure images [DM97].
Spherical image completion for missing areas in the ground portion[KMSY10]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Pedestrian removal in spherical images [AHK+ 10]. . . . . . . . . .
2.5 An image from a multi-exposure spherical image combined with agenerated HDR spherical image. . . . . . . . . . . . . . . . . . . .
2.6 Aircraft altitude. . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.7 Spherical aerial image captured from an OMS mounted on bottomof airship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8 Google Street View [Vin07, ADF+ 10] . . . . . . . . . . . . . . . .
2.9 QuickTime VR [Che95] . . . . . . . . . . . . . . . . . . . . . . . .
2.10 Immersive displays. . . . . . . . . . . . . . . . . . . . . . . . . . .
2.11 Geometry-image continuum [KSA00] and adequate scenes for freeviewpoint image generation methods. . . . . . . . . . . . . . . . .
2.12 The proposed flow in imaging and rendering frameworks. . . . . .
2.13 Applications partially using proposed imaging and rendering frameworks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1
3.2
3.3

Unmanned airship and mounted sensors . . . . . . . . . . . . . .
Completion of missing areas on ground scenery. . . . . . . . . . .
Completion of an environmental map. . . . . . . . . . . . . . . . .
viii

2
3
8
12
15
16
17
18
19
22
22
23
24
30
33
37
38
40

3.4
3.5

Estimation of Si in the All Sky Model [IKMN04]. . . . . . . . . . 43
Spherical image sequence with missing areas, which were captured
by OMS mounted on unmanned airship. . . . . . . . . . . . . . . 44
3.6 Full spherical image sequence completed using the sky model. . . 45
3.7 Full spherical HDR image generation using two OMSs. . . . . . . 47
3.8 Overview of full spherical aerial HDR imaging. . . . . . . . . . . . 48
3.9 Unmanned aircraft and two OMSs. . . . . . . . . . . . . . . . . . 49
3.10 Illustrative example of HDR histogram for n = 4. . . . . . . . . . 51
3.11 HDR images with and without multi-exposure image alignment. . 52
3.12 Process for aligning multi-exposure images. . . . . . . . . . . . . . 54
3.13 Masks used for alignment of images from the two cameras. . . . . 56
3.14 Example of chromatic change due to ND filters. . . . . . . . . . . 57
3.15 Captured multi-exposure images and corresponding exposure times. 60
3.16 Full spherical images with cropped intensity using exposures of themulti-exposure images. . . . . . . . . . . . . . . . . . . . . . . . . 61
3.17 Full spherical image tone-mapped in accordance with the method
by [RSSF02]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.18 Full spherical tone-mapped images generated from video frames. . 62
3.19 Completion of missing area based on spherical image completion
using sky model. . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.20 Visualization of negative eﬀect of large quantization steps. . . . . 65
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8

Structured viewpoints designated during oﬄine process. . . . . . .
Online free-viewpoint image generation using only neighboring structured viewpoints (in case of 2D structured viewpoints.) . . . . . .
Pre-generation of real-scene images at grid points. . . . . . . . . .
Penalty definitions of our oﬄine VDTM. . . . . . . . . . . . . . .
Example of augmented spherical scene at a grid point. . . . . . .
Captured points of spherical images, structured viewpoints, andonline-generated viewpoints. . . . . . . . . . . . . . . . . . . . . .
3D models of the virtual palaces in the Heijo-kyo capital. (Courtesy of Toppan Printing Co., Ltd.) . . . . . . . . . . . . . . . . .
Combined and simplified 3D models of the real world and the virtual world, used for the online process. . . . . . . . . . . . . . . .
ix

69
70
71
74
75
77
78
78

4.9 Free-viewpoint images from various viewpoints. . . . . . . . . . .
4.10 Spherical real-world images generated at grid points. . . . . . . .
4.11 A sample frame in sequences used for an evaluation of the oﬄine
superimposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.12 Interface used for evaluation using video sequence with a changeof grid size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.13 Augmented views from the same viewpoint with a change in gridsize (horizontal view direction). . . . . . . . . . . . . . . . . . . .
4.14 Augmented views from the same viewpoint with a change in gridsize (downward view direction). . . . . . . . . . . . . . . . . . . .
4.15 Naturalness scores of augmented video sequences with a change ingrid size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.16 Negative eﬀects using a larger grid sizes. . . . . . . . . . . . . . .

79
81
83
85
86
87
88
89

5.1 Immersive system used in the experiments. . . . . . . . . . . . . . 93
5.2 Illustrative tone curves for four display methods. . . . . . . . . . . 94
5.3 Image generated through LDR-like representation. . . . . . . . . . 96
5.4 Result using Reinhard’s tone-mapping method [RSSF02]. . . . . . 97
5.5 Seed pixels for GrabCut [RKB04]. . . . . . . . . . . . . . . . . . . 98
5.6 Regions determined using GrabCut [RKB04]. . . . . . . . . . . . 98
5.7 Result of region-wise tone-mapping. . . . . . . . . . . . . . . . . . 99
5.8 Examples of view-direction dependent tone-mapping. . . . . . . . 100
5.9 Results of subjective experiment involving the four display methods.102
5.10 Augmented spherical image sequence for augmented immersive
panorama application. . . . . . . . . . . . . . . . . . . . . . . . . 104
5.11 Appearance of augmented immersive panorama systems. . . . . . 105
5.12 Modified rendering framework for real-time AR. . . . . . . . . . . 107
5.13 Panoramic appearance and depth map of virtual objects at a structured viewpoint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.14 Top view of structured viewpoints where virtual objects are rendered oﬄine, capturing path of real-world spherical images for AR
application, and online-generated viewpoints. . . . . . . . . . . . . 109
5.15 Real-time AR based on proposed rendering framework. . . . . . . 110

x5.16 Comparison of scenes generated through online free-viewpoint image generation and oﬄine rendering. . . . . . . . . . . . . . . . .
5.17 Free-viewpoint images from various altitudes. . . . . . . . . . . .
5.18 Artifacts on occluding boundary between virtual objects and virtualized real objects. . . . . . . . . . . . . . . . . . . . . . . . . .
5.19 Reconstructed real-world 3D models and camera pose of the OMS
used. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.20 Top view of structured viewpoints where MR-world is rendered,capturing path of real-world spherical images, and online-generated
viewpoints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.21 Panoramic appearance of MR-world at a structured viewpoint for
virtual walk-through. . . . . . . . . . . . . . . . . . . . . . . . . .
5.22 Virtual walk-through using 1D structure. . . . . . . . . . . . . . .

111
114
115
116

117
118
119

A.1 Multi-exposure images whose exposure values are determined using
the method described in Section 3.3.2. . . . . . . . . . . . . . . . 152
A.2 A spherical HDR image captured in outdoor environment. . . . . 152

xi

List of Tables
2.1
2.2
2.3
2.4
2.5
2.6

Large FOV imaging approaches. . . . . . . . . . . . . . . . . . . .
Comparison of diﬀerent aircrafts. . . . . . . . . . . . . . . . . . .
Capturing location of large FOV imaging. . . . . . . . . . . . . .
Goals of virtual object superimposition for MR-world exploration.
Intended environment of proposed framework. . . . . . . . . . . .
Classification of MR applications. . . . . . . . . . . . . . . . . . .

10
19
20
30
31
34

3.1
3.2

Amount of misalignment owing to changes in position and orientation of the camera. . . . . . . . . . . . . . . . . . . . . . . . . .
Quantization steps of the HDR image. . . . . . . . . . . . . . . .

53
64

4.1

Amount of images required to prepare a grid structure. . . . . . .

84

5.1
5.2

System configuration for augmented immersive panorama. . . . . 106
Frame rates (fps) of the proposed rendering in our AR applicationand simple rendering of the textured virtual model. . . . . . . . . 112

xii

Glossary
1D

one dimensional

2D

two dimensional

3D

three dimensional

ANOVA analysis of varianceARaugmented reality

AVaugmented virtuality

CCD charge coupled device
CGcomputer graphics

CMOS complementary metal oxide semiconductor
DoF degrees of freedom
FOV field of viewGI

global illumination

GLSL OpenGL Shading Language
GPS global positioning system
GPU graphics processing unit
HDR high dynamic range

xiii

HMD head mounted display
IBL

image-based lighting

IBR

image-based rendering

LDR low dynamic range
LI

local illumination

MBR model-based renderingMR

mixed reality

MTB median threshold bitmap
MVS multi-view stereo
ND

neutral density

OMS omnidirectional mutli-camera systemPC

personal computer

RANSAC random sample consensus
SfM

structure from motion

SNR signal to noise ratio
SSD

sum of squared diﬀerences

UAV unmanned aerial vehicle
VDTM view-dependent texture mappingVRvirtual realityxivChapter 1
Introduction
For thousands of years, human-beings have virtualized the real world, and explored the virtualized real world. The map is a typical example of virtualization.
One of the oldest maps remains on a rock at Valcamonica, Italy, which was drawn
in approximately B.C. 1500 [Bra97]. From the invention of the map, people havevirtualized all over the world, and explored maps for learning about the world.
In 1827, Joseph Nicéphore Niépce developed the photographic technology, which
is a novel methodology for virtualizing the real world. This technology enabled
to explore the world with realistic representations. Information science, mixed
reality (MR) in particular, has largely extended the role of the real world virtualization. Telepresence [Min80,Ste92] is a primary approach of information sciencebased on virtualization, which is a technique presenting users highly immersive
real-world scenes of remote site; that is, a goal of this technique is to oﬀer people
a virtual transporter. As a result of the popularization of virtual globe applications [But06] (e.g., Google Earth), we can currently explore the virtualizations
not only for learning the world, but also for immersing into the world. Interactive exploration of the real world in a virtual simulation (e.g., telepresence andvirtual globes) is now one of the most important uses of augmented virtuality(AV), which is an aspect of MR research fields according to the reality-virtuality
continuum [MTUK94], as shown in Figure 1.1. In such applications, a large-scale
physical environment can be virtualized using modern three dimensional (3D)reconstruction techniques, such as vision-based [ASS+ 09, ASSS10, WACS11] andmulti-sensor integration approaches [BMOI08, PNF+ 08]. To improve the appear1

0L[HG 5HDOLW\ 05
5HDO
(QYLURQPHQW$XJPHQWHG
5HDOLW\ $5$XJPHQWHG
9LUWXDOLW\ $9

9LUWXDO
5HDOLW\ 95([DPSOHVRIDXJPHQWHGYLUWXDOLW\WHFKQLTXHV )UHHYLHZSRLQWLPDJHJHQHUDWLRQ 'UHFRQVWUXFWLRQ ([SORUDWLRQRIYLUWXDOL]HGUHDOZRUOG 7HOHSUHVHQFH ,PPHUVLYHSDQRUDPD  05ZRUOGJHQHUDWLRQDQGH[SORUDWLRQ

Figure 1.1. Reality-virtuality continuum [MK94] and augmented virtuality techniques.

ance of automatically constructed 3D models, free-viewpoint (sometimes referred
to as novel- or arbitrary-viewpoint) image generation using a hybrid of imageand model-based rendering [DTM96,ZC04] provides promising solutions [SSS06].
Human-beings also have created countless new worlds. History of creation isolder than that of virtualization. A recent study [PHGD+ 12] figured out thatsome old rock paintings, in which imaginary or abstract scenes were occasionally
depicted, were drawn over 40.8 thousand years ago. In MR fields, the history ofcreation is also older than that of virtualization. Virtual reality (VR) is an MR
technique that creates virtual worlds in the cyberspace, and to present it for users
with high immersive values (right edge of reality-virtuality continuum shown in
Figure 1.1). The Ultimate Display [Sut65,Sut68], whose concept was proposed in1965 by Ivan Sutherland, is the roots of VR as well as of the other MR studies.
It is the first concept of immersive technology using a head mounted display(HMD), and yet predicted an ultimate goal of VR. In the last few decades, virtual
objects are occasionally superimposed onto the real world in real-time. Suchtechnique is referred to as augmented reality (AR) [Azu97, ABB+ 01], and has
recently attracted lots of attention from many creators as well as from industry.
In certain applications that allow an interactive exploration of the virtualized real world, virtual objects can also be superimposed for the visualizationof disaster areas or non-existent buildings [Cho09, Gro05, GB08]. In this thesis,

2Real-world

Imaging

Rendering

Imaging ofreal-world

Virtualization ofreal-world
MR-world

Virtualobjects

Superimpositionof virtual objects

Figure 1.2. Basic pipeline of MR-world generation.

such type of MR applications is referred to as MR-world exploration. Figure 1.2
illustrates the basic pipeline of generating an MR world, which consists of imaging, real-world virtualization, and virtual-object superimposition. We refer to thegeneration process as MR-world generation.
In MR-world exploration applications, virtual objects are rendered on a frameby-frame basis, along with virtual-object superimposition in AR applications. A
goal of MR applications using virtual-object superimposition is to realize an MR
world where virtual objects are superimposed into real scenes as if they actuallyexist. Photometric registration [KY02] between real and virtual objects is one of
the most important factors in realizing photorealistic virtual-object superimposition. It is basically achieved in two steps: estimating the real-world illumination,and rendering virtual objects using the estimated illumination. There are numerous AR/MR studies that have focused on one or both of these steps.
In traditional approaches to real-world illumination estimation, scenes of realenvironments are acquired directly using mirrored balls [Deb98] or cameras withfisheye lenses [GM00, SSI02, ALCS03]. Some recent studies have been able tosuccessfully estimate the illumination from AR background images without theuse of additional cameras or objects [JND12, GRTS12]. However, most studies
have dealt with the illumination acquisition from the ground, and there havebeen some diﬃculties in acquiring real scenes from the sky that can be used3

for realizing fly-through MR-world exploration applications with a photorealistic
superimposition of virtual objects. Such fly-through views are used for mostvirtual globe applications.
This study proposes the use of aerial imaging techniques that overcome the difficulties in realizing fly-through MR-world exploration. For photometric and immersive presentations of real and virtual objects, real-world images are captured
through full spherical high dynamic range (HDR) imaging using omnidirectional
cameras mounted on an unmanned airship. In addition to MR-world exploration,
the acquired images/videos can be employed for aerial immersive panoramas, i.e.,an extension of Google Street View type applications for a bird’s-eye view.
The process of generating the appearance of virtual objects using an estimatedillumination is achieved by employing real-time rendering techniques [HDH03,
KY02, GM00, GCHH03, KTM+ 10, GDS10, KK12, LB12], such as those used incomputer graphics (CG) research. The quality of real-time rendering has been
improved with advancements in computer technology. However, real-time rendering has not achieved the same quality as oﬄine rendering, which is generally
time-consuming. Although real-world illumination conditions and virtual-object
reflectance properties can be estimated/determined accurately, part of their information is wasted because real-time rendering cannot represent all physical phenomena. Notably, computational costs become greater when high-quality rendering techniques are used, or when virtual objects consisting of numerous polygons
are applied. In these types of situations, it is diﬃcult to perform state-of-the-art
real-time rendering techniques on mobile devices. For this reason, cinema-quality
mobile AR has not been achieved yet. In film-making, on the other hand, CG effects are rendered using high-quality and time-consuming illumination methods,
and a large number of manual operations. We contend that the eﬀorts involved insuch oﬄine and high-quality processes can be employed to improve virtual-object
expression in MR applications.
To achieve the photorealistic superimposition of virtual objects utilizing oﬄine
eﬀort, we propose a new rendering approach that combines the oﬄine renderingof virtual objects and free-viewpoint image generation. The proposed rendering
approach generates the appearance of virtual objects using free-viewpoint image
generation from oﬄine-rendered images with multiple viewpoints to take advan-

4

tage of the higher quality of oﬄine rendering and without the computational costof online CG rendering; i.e., it incurs only the cost of the online computations forlight-weight free-viewpoint image generation. In addition to aerial spherical images acquired by the proposed imaging techniques, this approach accommodates
the input of general real-world scenes with only minor modifications.
The remainder of this thesis is organized as follows. Chapter 2 describes otherworks related to imaging and rendering techniques for MR-world generation andpresentation, followed by descriptions about contribution of this study. Next,
the aerial spherical imaging approaches are detailed in Chapter 3, and the proposed rendering approach is introduced in Chapter 4. The proposed imaging andrendering approaches can be employed for various MR applications using partsof the framework with minor modifications. Chapter 5 describes examples of
MR applications based on the proposed framework, along with discussions of themethodology and limitations in developing each application. Finally, Chapter 6
summarizes this thesis.

5Chapter 2
Related Work2.1. Imaging Technique for MR-World Generation
We first discuss the requirements for imaging techniques used to achieve ouraim, that is, photorealistic MR-world generation. Since the original invention ofphotography by Joseph Nicéphore Niépce in 1827, numerous imaging techniques
have been developed. Digital monocular cameras are currently widespread andare attached to most mobile devices. However, ordinary monocular imaging is not
suﬃcient for MR-world generation, the requirements of which can be summarized
into the following two categories.
• Eﬀective acquisition of large environments
In most MR-world exploration applications, a real scene is virtualized using
images captured from numerous locations and directions. To acquire an
entire scene of the target environment, eﬀective imaging techniques are
required. Ineﬃcient methods, such as the use of conventional handheld
monocular cameras, can easily be used to generate the missing parts of areal-world virtualization drastically lacking user immersion. In addition,such missing parts may cause a lack of illumination of the real world, which
negatively aﬀects the appearance of the virtual objects.

6• Accurate acquisition of the scene luminance
The rendering of virtual objects is an estimation process of their surface
color. To achieve photorealistic rendering, the surface color should be estimated accurately using the surrounding illumination and reflectance property of the surface. This implies that an accurate acquisition of the realworld illumination should be achieved for photorealistic MR. Unfortunately,
ordinary cameras are unable to acquire accurate luminance information ofa light source owing to a saturation of the pixel intensity.
According to the above requirements for photorealistic MR-world generation,
we will review the existing imaging techniques in terms of the field of view,
dynamic range, and capturing location.

2.1.1 Field of View
Large field-of-view (FOV) imaging techniques facilitate the eﬀective acquisitionof a real scene.
Fisheye camera
Traditional imaging systems basically employ a CCD/CMOS array and a lens, theprojection model of which can be approximated by a planer perspective projection
similar to a pinhole camera. To realize large FOV imaging with such traditional
cameras, a fisheye lens, which has a very short focal length, is employed [Miy64].
Fisheye lenses can be employed without any special equipment beyond an ordinary camera and the lens itself, and acquire about 180-degree images1 , as shownin Figure 2.1. Xiong and Turkowski [XT97] proposed a registration method for
multiple fisheye cameras to acquire a FOV of larger than 180 degrees. However,in principle, they cannot acquire 360-degree, omnidirectional scenes using a single
shot.

1

Fisheye lenses wider than 180-degree can be developed in principle. In a particular case,Nikon has released a 220-degree fisheye lens (e.g., Fisheye Nikkor 6mm f/5.6 released in 1969).

7

Figure 2.1. Fisheye camera and acquired image in [XT97].

Omnidirectional camera using mirrors
Omnidirectional imaging, in which the entire direction of a scene is acquired, is a
popular and ultimately large FOV imaging technique [Yag99]. Omnidirectional
imaging techniques employing a traditional camera and mirrored surfaces havetraditionally been studied. Image-based lighting (IBL), a CG rendering method
based on real-world illumination, originally employed two mirrored spheres andtwo times of capturing [DTM96]. In the field of robotics, particularly for thenavigation of mobile robots, imaging systems using a mirror and a video camera
have been developed to capture real-time omnidirectional videos [Nay88, YK90,
YYY93, Nal96, Nay97]. This type of omnidirectional camera is referred to as
catadioptric camera. Such studies have developed mirrored surfaces and realized
omnidirectional projection models with a common principal point. However, it
is diﬃcult to acquire a large vertical FOV using catadioptric cameras.

8Omnidirectional multi-camera system
In accordance with the downsizing of traditional cameras, omnidirectional multicamera systems (OMSs) have been developed. OMSs employ multiple cameras
synchronized to acquire the entire direction, and combine multiple images intoa single omnidirectional image. With a well-considered configuration, an OMS
can capture a nearly full sphere with a single shot. In this thesis, such an omnidirectional image covering nearly an entire sphere is referred to as a sphericalimage. Although most OMSs do not achieve common principal points among
multiple cameras, they easily achieve a higher resolution and wider vertical FOV
than catadioptric cameras. OMSs have become popular as imaging systems
used for Google Street View and certain commercial products used in research,such as Ladybug by Point Grey Research, Inc. (http://www.ptgrey.com/). The
growth of OMSs has brought omnidirectional/spherical imaging techniques tothe consumer market in recent years. Notably, Ricoh Theta, released by Ricoh(http://theta360.com/) in 2013, is expected to standardize the role of omnidirectional imaging for the general public. This product is equipped with two CMOS
sensors with an over 180-degree fisheye lens, and combines two images into asingle spherical image. Omnidirectional imaging techniques are widespread, andare expected to become even more popular and important.

The best large FOV imaging method for MR-world generation
Table 2.1 summarizes major approaches for large FOV imaging. With the recent
growth of OMS, high-resolution spherical imaging using OMS has become notso costly compared to former approaches. Because most OMSs do not achievecommon principal points among multiple cameras, they cause misalignments on
panoramic spherical images. Although a dynamic panoramic stitching method
can be applied similar to a real-world exploration application developed by Uyttendaele et al. [UCK+ 04] to reduce the visual artifacts due to the misalignment,
it is diﬃcult to compensate the diﬀerence of the principal points.
In terms of MR-world generation, which is expected to be employed in large
environments, the misalignment problem can be negligible small. The distance
between the principal points of neighboring cameras in Ladybug2, which is a fa9

mous OMS developed by Point Grey Research, is approximately 2cm. In the casethe distance to the real scene is 15m, misalignment of the scene on the sphericalimage is 0.07◦ , which is less than one pixel of the maximum resolution of thecamera. It intends that the misalignment is ignorable if the target environment
consists of long-distant view, which is a general situation in outdoor shooting.
Although it required to pay careful attention to the misalignment issues, imaging
using OMS is a promising approach to fulfill a requirement of imaging techniquesfor MR-world generation; that is, eﬀective scene acquisition of large environments.

2.1.2 Dynamic Range
To employ real-world images as an illumination environment for rendering virtual
objects, the images should be unsaturated [DM97]. Unfortunately, the dynamic
range of many outdoor scenes during the daytime is too high for a proper captureusing standard 8-bit (low dynamic range, or LDR) cameras, as sunlight can reachapproximately 217 -times brighter levels than darker areas of the sky or cloud
cover [STJ+ 04]. HDR imaging techniques, which acquire higher than 8-bit realworld images, have been studied to overcome this problem.

Table 2.1. Large FOV imaging approaches.
Fisheye camera
One-shot360◦ imaging
Resolution
Cost
Commonprincipal point
WeightOmnidirectional
camera w/ mirrors√

Omnidirectional
multi-camera system√

×√(narrow vertical FOV)×√(near full sphere)√√√< 1 kg< 1 kg×

10

× (formerly)√(recently)×> 1 kgHDR camera
Some HDR cameras equipped with special CCD/CMOS sensors with a highquality A/D converter are available for film-making, and achieve 10- or 12bit/channel video recording. ViewPLUS, Inc. (http://www.viewplus.co.jp/) has
released an 18-bit HDR video camera, called Xviii, for research use. Spherical
HDR cameras have also been developed for use in virtual reality. SpheronVR(http://www.spheron.com/) has released a 13,000 pixel × 5,300 pixel full spherical panorama camera, the bit-depth of which reaches a dynamic range of 108 : 1;
however, the acquisition of a single shot takes several minutes owing to the useof a line-scan sensor.
A spherical HDR camera that acquires a spherical image in a single shot,
can be a suitable imaging technique for MR-world generation, in terms of its
eﬀectiveness and accuracy. However, it is so far diﬃcult to find such a special
HDR camera.
Multi-exposure imaging
An HDR imaging method using LDR images taken by multiple exposures (multiexposure images), shown in Figure 2.2, is widely used to capture scenes with ahigh dynamic range using commodity camera hardware. Exposure of the digital
camera, the amount of the light incoming to a sensor, is determined by changing
exposure time, aperture size, and sensor gain. Multi-exposure images are captured by changing exposure so as to capture both bright and dark portions. An
HDR image is generated by appropriately combine these images. This approach
was originally proposed by Debevec et al. in 1997 [DM97], and multi-exposure
imaging is currently being implemented in most commercial still-cameras as anauto-bracketing function. It should be noted that the response curve (sometimes
referred to as the response function) of such cameras is basically non-linear, thatis, the pixel intensities do not vary linearly in accordance with the physical luminance. To successfully generate an HDR image from multi-exposure images,
the response curve should be calibrated and linearized. Two representative calibration techniques were developed by Debevec et al. [DM97] and Mitsunaga et
al. [MN99]. Debevec et al. [DM97] approximated the camera response using alogarithmic function, whereas the other method [MN99] employs a polynomial
11

(a) Multi-exposure images.

(b) Recovered HDR radiance.

Figure 2.2. HDR radiance reconstruction from multi-exposure images [DM97].

12

function. The former method [DM97] is currently being implemented as a software library for HDR image composition [DT01]. For multi-exposure imaging,
exposure time should be changed rather than other parameters to avoid change of
depth of field and increase of noises. Because changeable exposure time is often
restricted by the hardware, the aperture size is also changed in some practicalexamples of multi-exposure imaging [DM97].
When images are captured during motion, however, a misalignment will naturally occur. The alignment of multi-exposure images is one of the most important
issues in HDR imaging research. A method proposed by Kang et al. [KUWS03]
tracks the scene movement using the Lucas and Kanade tracker [ST94], andthen registers the images using a warping technique. To track the movement, amore sophisticated method employs a median threshold bitmap (MTB) [War03],
which is robust under large exposure changes. This MTB-based method has been
improved to realize a more robust alignment [Gro06,JLW08]. However, large misalignments remain diﬃcult to fix using alignment methods. For this reason, it isbetter to use fewer multi-exposure images to generate HDR images, and to ensurethat the camera velocity and frame rate remain constant.
The best HDR imaging method for MR-world generation
As discussed in Section 2.1.1, using large FOV cameras, OMSs in particular, isa suitable solution for eﬀective scene acquisition of a large environment. It iscurrently diﬃcult to find special OMSs equipped with HDR sensors. Therefore,
multi-exposure imaging is a practical solution for HDR spherical imaging. Note
that, the misalignment between the multi-exposure images should be taken into
account when the camera is not fixed during multi-exposure imaging, which is
expected to be a common situation in MR-world generation.

2.1.3 Capturing Location
The large FOV and/or HDR imaging techniques described above are employed
under various situations. In this section, we categorize these imaging systems interms of the capturing locations: from the ground and the sky.

13

Imaging from the ground
Most imaging techniques are intended for use on the ground. Omnidirectional
imaging is often employed in immersive panorama-based real-world exploration
applications as detailed in Section 2.2.1. Certain immersive panorama applications acquire spherical images/videos from a vehicle-mounted OMS [ADF+ 10]. In
addition to a vehicle-mounted capture, spherical images can be captured throughvarious settings using a tripod- or human-mounted OMS [UCK+ 04].
Unless an OMS realizing almost full spherical imaging is employed, missing
areas often remain in a single-capture image, as shown in Figure 2.3. Such missing
areas are caused by limitations in the FOV of the OMS and/or occlusions of the
FOV by persons or vehicles mounting the camera. Kawai et al. have dealt withthe completion of such occlusions, particularly for spherical videos captured on
the ground [KMSY10]. In the method in [KMSY10], missing areas in the ground
portion of spherical video sequences were filled in by estimating the intensities ofthe missing areas based on the intensities of the same locations taken from otherviewpoints. The result of the method in [KMSY10] is shown in Figure 2.3.
When developing immersive panorama applications in crowded areas (e.g., anunderground city), a large number of pedestrians occlude the background scenery.
In the application for an underground city visualization developed by Arai etal. [AHK+ 10] (see Figure 2.4), time-series images captured at fixed locations
were unified such that only the intensity of the background scenery is used basedon the temporal median intensity. Because some pedestrians may remain in a
temporal-median based approach, an image inpainting method [KSY09] was also
employed to completely remove them.
HDR spherical images can be captured through multi-exposure imaging fromthe ground. Such images have been traditionally captured for IBL using mirrored spheres [DTM96] mounted on a tripod. Naturally, we can employ an OMS
for HDR spherical image capturing in a straightforward manner. Appendix A
describes a simple approach used to acquire a spherical HDR image from theground. Figure 2.5 shows an example of an HDR spherical image captured in
an underground city, whose radiance is transformed into an 8-bit image using atone-mapping method [RSSF02]. In addition to IBL, these images help improvethe visibility of the scenery in immersive panorama applications.
14

(a) Spherical image with a missing area.

(b) Complete spherical image.

Figure 2.3. Spherical image completion for missing areas in the ground portion[KMSY10].

15

(a) Spherical image with pedestrians.

(b) Spherical image with pedestrians removed.

Figure 2.4. Pedestrian removal in spherical images [AHK+ 10].

16

(a) A multi-exposure image.

(b) HDR spherical image whose radiance is compressed
through the method in [RSSF02]

Figure 2.5. An image from a multi-exposure spherical image combined with agenerated HDR spherical image.

17

Height

Satellite

100km
10km
Manned aerial vehicle

1km
100m

Unmanned aerial vehicle

10m
Crane

1m

Ground vehicle

Figure 2.6. Aircraft altitude.

Aerial imaging
Aerial imaging using monocular cameras is traditionally used for various applications (e.g., satellite maps and 3D reconstruction). Most map applications employ a bird’s-eye viewpoint using aerial images, rather than a viewpoint fromthe ground, because a bird’s-eye view is advantageous in grasping large-area
scenes at a glance. Aerial images are usually captured from a satellite or airplane at an altitude of over 1 km, as illustrated in Figure 2.6. Unmanned aerial
vehicles (UAV) have recently attracted attention for low-altitude aerial imaging [HJD+ 04, GMG+ 08], which can acquire higher resolution images than highaltitude aerial imaging. In particular, lighter-than-air UAVs (e.g., unmanned
airships), which do not require fuel to float, are often used for low-altitude aerial
imaging [PARJ+ 06,FYS+ 08,HJSL04]. The benefits of lighter-than-air UAVs, i.e.,
fuel-eﬃciency and portability, are shown in Table 2.2.
Although aerial-imaging techniques have been widely used, there are few studies dealing with omnidirectional/spherical imaging from the sky. Omnidirectional
cameras are occasionally mounted on UAVs for visual servo (i.e., vehicle control)tasks [HS03, MCMO10]. In addition to vehicle control, omnidirectional aerial
imaging can be used for immersive panoramas, eﬀective 3D reconstruction, and18

Figure 2.7. Spherical aerial image captured from an OMS mounted on bottom of
airship.
virtual object superimposition. Figure 2.7 shows a spherical image captured fromthe sky by an OMS mounted beneath an unmanned airship. A problem tends tooccur in spherical aerial images acquired using an OMS, i.e., the occurrence of
missing areas. While this problem is also typical in images taken from the ground,
it is a diﬃculty inherent to aerial imaging. Such missing areas are caused by limitations in the FOV of an omnidirectional camera and/or occlusions of the FOVby the hardware used. The primary occlusion shown in Figure 2.7 is from theaircraft on which the camera is mounted. Such areas subtract from the immerTable 2.2. Comparison of diﬀerent aircrafts.
Airplane
Helicopter
Unmanned airship
AltitudeHigh (over 1 km)
Speed
High
Fuel-eﬃciency
×
Weight> 1000 kg

MiddleMiddle×> 100 kg19

Low (lower than 150 m)
Low√(lighter than air)Approx. 10 kg(used in our study)

sive value of the image and disqualify it from use in rendering system using IBL.
While some studies have dealt with missing areas in spherical images [KMSY10],
the solutions achieved are not intended for use in aerial imaging.
Imaging issues for MR-world generation
There are large variations of the target environment of MR-world generation; it
includes both ground and aerial scenarios. Imaging from the ground is employed
for street-view applications; while most virtual globe applications employ aerial
imaging. Table 2.3 simply summarizes the problem in aerial imaging in termsof the capturing location. Although the intensity of the sky is important for
immersion and virtual object rendering in MR-world exploration applications,spherical aerial imaging using OMSs causes missing areas in the sky portion in
the spherical images. This fact indicates that we should overcome the problems
in aerial imaging to the eﬀective and accurate MR-world generation.

2.2. Rendering Technique for Real and VirtualObjects
Numerous rendering approaches have been proposed. This section classifies therendering approaches for both real and virtual objects according to the flow of
the photorealistic MR-world generation, illustrated in Figure 1.2. In a rendering
pipeline used in photorealistic MR-world generation, real-world scenes, acquired
from the imaging technique described in Section 2.1, are first utilized for virtualization. Virtual objects are then superimposed into the real-world virtualization
using CG-rendering techniques.

GroundAerialTable 2.3. Capturing location of large FOV imaging.
Spherical imaging
Full sphericalw/ missing area
imaging√√(completion of ground portion [KMSY10])√
×20

2.2.1 Real-World Virtualization
Real-world scenes acquired at numerous locations by numerous imaging techniques have been employed for real-world virtualization. The aim of the virtualization process is to develop a tool for exploring the real world. Web-map
applications including Google Street View and Google Earth, which employ virtualization techniques, have recently become popular.
Virtualization approaches can be divided into the following two categories interms of a changeable viewpoint:
1. Immersive panoramas
An immersive panorama is a real-world rendered image allowing the user
to change their viewing direction rather than their viewpoint. Immersive
panoramas are employed in Google Street View.
2. Free-viewpoint image generation
Using free-viewpoint image generation, the user can freely configure boththeir viewpoint and direction. Virtual globe applications (e.g., Google
Earth) employ this technique.
Immersive panoramas
An immersive panorama is an image representation using spherical images, which
are acquired as described in Section 2.1. It enables users to change their viewing direction from a fixed location where the image was captured. Immersive
panoramas are currently very popular with Google Street View [Vin07,ADF+ 10],as shown in Figure 2.8.
The primary system used for creating an immersive panorama is QuickTime
VR [Che95], which employs spherical panoramic images composed from monocular camera images captured from various directions at a single location. A
panoramic image is converted into a planar-perspective image, as shown in Figure 2.9, whose viewing direction can be configured freely using the Apple QuickTime framework.
Immersive panorama applications can become more immersive using an HMD
or large screen, as shown in Figure 2.10. Onoe et al. [OYTY98] proposed a realtime perspective-conversion system using spherical images captured from omnidi21

Figure 2.8. Google Street View [Vin07, ADF+ 10]

Figure 2.9. QuickTime VR [Che95]

22

(a) Head-mounted display.

(b) Immersive screen with a motion
platform [HKY09].

Figure 2.10. Immersive displays.

rectional cameras, and developed an immersive system using an HMD. An HMD,
which is equipped with an orientation sensor, displays a view-direction-dependent
perspective image converted from a spherical image. Unlike QuickTime VR, thissystem can present video sequences acquired during motion by an omnidirectional
camera. A system developed by Ikeda et al. [ISKY04] employs a locomotion interface and large screens to realize the sense of walking along a captured path ofa spherical video sequence. The employment of motion platforms [HKY09], asshown in Figure 2.10(b), can also contribute to increase the immersive value forspherical videos captured in a shaky environment, such as on a roller coaster.
Free-viewpoint image generation
Free-viewpoint image generation (sometimes referred to as arbitrary- or novelviewpoint image generation) is a technique for generating freely configurable views
from multiple images captured at multiple locations, and was developed in thefields of CG and computer vision. Free-viewpoint image generation approaches
can be categorized according to the geometry-image continuum [KSA00] (see Figure 2.11), which consists of model-based rendering (MBR), image-based rendering(IBR), and a hybrid of the two.
23

Geometry-image continuumAppearance-based(IBR)

Physically-based(MBR)
Singlegeometry,
singletextureFixedgeometry,
viewdependent
texture(VDTM)

Viewdependent
geometry,
singletextureViewdependent
geometry,
viewdependent
textureSpriteswith
depth

Conventional graphics pipeline

Layered Lumigraph Lightdepth
field
imagesrendering

Warping

Interpolation

Adequate target scenes
Small

Error of 3D shapes

Large

Far

Distance to target object

Near

Lambertian

Reflectance property

View-dependent

Figure 2.11. Geometry-image continuum [KSA00] and adequate scenes for freeviewpoint image generation methods.

An MBR approach is a traditional computer graphics/vision pipeline thatfirst reconstructs 3D shapes of real environments, and then maps images of thereal environment onto the 3D shapes as textures. Some web-maps, including
Google Earth, present textured 3D models of buildings. As the most basic MBR
approach, handcrafted 3D models can be utilized; for example, a system developed by Shimamura et al. [STYY00] realizes small changes in viewpoint by
mapping an omnidirectional image onto a hand-made 3D model. With the recent growth of computer vision techniques, the 3D shape of the real world canbe automatically reconstructed by using multiple images and/or laser scanners.
3D shapes can be acquired from the environments ranging from a small desktop
space [IKH+ 11] to a large outdoor environment [MAW+ 07]. Laser scanners are24

occasionally employed for the automatic 3D modeling of a large outdoor environment [BMOI08, NCS+ 09]. In addition, there are numerous approaches using
vision-based 3D reconstruction methods that employ multiple images captured
at multiple locations. In vision-based approaches, the position and orientation ofa camera capturing multiple images are first estimated through structure-frommotion with bundle adjustment techniques [TMHF99]. Bundle adjustment has
recently been speeded up dramatically, and can accommodate large outdoor environments [ASS+ 09,LA09,ASSS10,WACS11]. A multi-view stereo method is thenused to reconstruct a 3D model with a dense surface from multiple images basedon their estimated position and orientation [PNF+ 08, SY10, JP11]. These largescale reconstruction approaches have been employed for recent web-maps. The
quality of the free-viewpoint images generated through MBR is directly aﬀectedby the accuracy of the 3D shapes, that is, unnatural distortions or missing areas
in such views are easily exposed.
On the other hand, IBR generates free-viewpoint images without the use of
explicit 3D shapes. The ultimate form of IBR is to acquire and represent all
possible light rays from all positions and directions, where the rays are describedas having a plenoptic function [MB95]. To interpolate light rays from a limited
number of image acquisitions, view interpolation techniques have been studied,such as view morphing [MD96], light-field rendering [LH96, NTKH97, DLD12],and lumigraph rendering [GGSC96, BBM+ 01]. Light-field-based representations
have also been recently employed for 3D scene reconstruction for a complex
environment [KZP+ 13]. IBR is currently being employed in numerous applications [SCK07]. A multi-perspective panorama [ZFPW03, KCSC10], which ispatched together from parts of multiple images, was developed to interpolate
the viewpoint from immersive panorama applications, such as Microsoft Street
Slide [KCSC10]. Hori et al. [HKY10] proposed a free-viewpoint stereo image generation system utilizing a multi-perspective panorama approach. Although IBR
reduces missing areas in the resulting images, this approach requires images captured at a large number of locations and directions. A recent study reported thatthe amount of perceived distortion increases along with the distance between theuser’s viewpoint and position where the image was captured [VCL+ 11, VRC+ 13].
In recent years, the main approach for free-viewpoint image generation has

25been the use of hybrid techniques that combine MBR and IBR with the goal of resolving their respective problems. The primary approach of hybrid free-viewpoint
generation is view-dependent texture mapping (VDTM), which was originallyproposed by Debevec et al. [DTM96]. This technique selects and blends multiple
textures acquired from multiple cameras, and then maps the blended textures
onto 3D models. There have been various improvements in VDTM, with focus
on the quality of image generation, the calculation cost, and/or specific applications [DYB98, TKIS00, AS02, PDG05]. Although the textures acquired at theproper positions are selected in VDTM approaches, large errors in the 3D shapes
still cause large distortions in the resultant free-viewpoint images. State-of-theart hybrid rendering methods appropriately transform 3D shapes depending on
the position of the virtual viewpoint, while also selecting and blending the textures [IHA02,SY10]. Note that the 3D shapes are often transformed based on theappearance of object shape in two dimensional (2D) images (e.g., object silhouette [CSD11] and region segments [CDSHD13]) rather than photo consistency,
which is usually used in 3D reconstruction for MBR. These new hybrid-rendering
approaches are referred to as view-dependent geometry and texture, and havebeen employed in some recent applications [OUSY13]. Although 3D shapes may
include large errors, these approaches improve the quality of free-viewpoint images, (e.g., when there are missing regions in the 3D shapes).
Here, we describe a guideline for selecting a free-viewpoint generation method
in terms of the target environment. Suitable approaches should be selected depending on the errors of 3D shapes, the distance to the scenes, and the reflectanceproperty of the target environment as shown in the bottom of Figure 2.11. In thecase that the target environment can be approximated as Lambertian, and yet
the error of the 3D shape is small, physically-based rendering approaches, such
as VDTM, generate reasonable results. On the other hands, if the 3D shape ispoorly reconstructed or the distance to target objects from the viewpoint is small,appearance-based approaches should generate a result preferable to physicallybased one. For particularly complex environments such as scenes from near thetree or objects that have complex reflectance, some hybrid-rendering methods
utilizing complex processes basically improve the quality of the resultant images
at the cost of heavy computation [SKG+ 12, KLS+ 13].

26

Real-world virtualization approach for MR-world exploration
We basically deal with MR-world exploration based on both immersive panorama
and free-viewpoint image generation. When the OMSs are employed for theimaging, the diﬀerent between two approaches is whether a free-viewpoint image
generation technique is employed. If not, immersive panorama applications aredeveloped. Free-viewpoint image generation provides viewpoint changes, but
degrades the quality of the resultant images; thus, immersive panorama is still
valuable for applications that should not display debased scenes.

2.2.2 Superimposition of Virtual Objects
Virtual objects are sometimes superimposed on a real-world virtualization [Cho09,
GB08, Gro05] in certain applications such as landscape simulations. Google
Earth has provided a framework for superimposing CG models using Google
SketchUp [Cho09] (acquired by Trimble Navigation Ltd. in 2012), which has beenused by numerous visualization researchers [Wol08, XBF+ 09]. To realize changes
in viewpoint as in SketchUp, virtual objects must be rendered frame by frame.
It is similar process in real-time AR, which superimposes virtual objects onto areal-scene acquired in real-time. The issues inherent to the virtual-object superimposition of a real-world virtualization, specifically those that provide changes
in viewpoint, are also common in AR applications, i.e., both require real-timerendering techniques. In real-time AR, the virtual objects are superimposed
through the following steps: 1) illumination estimation of the real world, and 2)
the rendering of virtual objects.
Illumination estimation of the real world
Illumination estimation from monocular images has been studied in the CG research field [NHIN86,SSI03], and some recent AR studies have achieved real-time
estimation [JND12, GRTS12]. On the other hand, MR-world generation throughthe virtualization of the real world often employs imaging from large FOV cameras, as discussed in Section 2.1.1. For such situations, we can directly utilize
the luminance acquired by a camera for the illumination environment. Somesystems for virtual object superimposition employ mirrored balls [Deb98, KY02],27

fisheye cameras [KY02, SSI02, ALCS03], or omnidirectional cameras to acquire areal-world illumination.
Rendering of virtual objects
Numerous rendering methods have been proposed in the fields of CG and AR [Cro77,
WPF90,HLHS03], and can be divided into two categories: local illumination (LI)and global illumination (GI) rendering. LI rendering does not simulate multiple
reflections (interreflection) on the surface of CG objects, whereas GI rendering
realizes photorealistic rendering by calculating the interreflection between multiple object surfaces, at the expense of the calculation cost.
[Local illumination (LI) rendering]
Owing to its lightweight computation, LI rendering has been widely used forreal-time applications including real-time AR. To realize shadow and shade presentations, a shadow map technique is often employed. For AR application, thistechnique has been expanded to render multiple virtual objects [GM00], soft shadows [GCHH03], and dynamic objects [KY02]. In contrast, Haller et al. [HDH03]
utilize shadow volumes to achieve faster rendering.
[Global illumination (GI) rendering]
The photorealistic appearance of a virtual object can be generated through GI
rendering, such as photon mapping [Jen96]. There have been numerous GI rendering methods for oﬄine (i.e., non-real-time) virtual-object superimposition ontoreal-world scenes, such as in movie special eﬀects. Fournier et al. [FGR93] utilized Progressive Radiosity [CCWG88] for virtual-object superimposition. IBL,
an illumination acquisition and rendering framework using a large-FOV imaging
technique proposed by Debevec [Deb98], has often been employed in film-making.
Although the original IBL employs ray-tracing [Gla89] using an illumination environment acquired by mirrored balls, some recent rendering software using IBL
employs other GI techniques. These oﬄine GI-rendering techniques are available
as libraries [War94, PH10] and commercial software. In the past, however, it wasdiﬃcult to employ such techniques for AR owing to their heavy computational
requirements.
28

Some AR studies before early 2000s realize the near real-time GI rendering of virtual objects using an image synthesis from the appearance preliminarily
rendered in various illumination environments [SHK+ 05,NSD94], where the viewpoints of the user and virtual objects must be static. With new advancements incomputer technology, recent studies on photometric AR have realized real-time GI
rendering in dynamic environments, including the use of a pre-computed radiancetransfer [GDS10], diﬀerential rendering [KTM+ 10], real-time ray-tracing [KK12],
and reflective shadow maps [LB12]. These approaches employ state-of-the-art
real-time rendering methods, which have been studied in the CG field, for AR
virtual object representation. However, real-time GI-rendering methods have notachieved the same quality as oﬄine rendering, and their heavy computational requirements are too costly for mobile AR applications. Therefore, cinema-quality
mobile AR has yet to be achieved. Even in film-making, for example, CG models
are rendered using high-quality illumination, time-consuming methods, and many
manual operations.
Virtual object superimposition for MR-world exploration
Unlike real-time AR applications, it is expected that MR-world exploration does
not required to deal with environment with illumination change, because it preliminarily virtualize the real-scene and the illumination is static as shown in
Table 2.4. In addition, the real-world illumination can be directly acquired by
OMSs. It is possible that a virtual object superimposition method suitable for
MR-world exploration applications can be developed; in which the method realizes highly photorealistic image synthesis, although it does not accommodate theillumination change.

2.3. Contribution of This Study
Figure 2.12 illustrates the concrete flow of the proposed framework. As discussed
in Section 2.1, novel imaging techniques are required for photorealistic MR-world
generation using aerial images. That is, although spherical HDR images can beacquired from the ground, spherical aerial HDR imaging should be developed.
Based on the discussions in Section 2.2, we propose a rendering framework that29

Imaging

Real
world

Rendering

Aerial spherical
HDR imaging

MR
world

Rendering for
MR-world exploration

Other sphericalimaging methods

Virtualobjects: Processes proposed in this study: Previously proposed processes

Figure 2.12. The proposed flow in imaging and rendering frameworks.

enables the user to configure their viewpoint freely in a static MR-world, where
both real and virtual objects are presented photorealistically. The proposed rendering method can be employed for spherical images captured from both theground and the sky, as summarized in Table 2.5.
This study proposes two diﬀerent imaging methods: 1) spherical aerial HDR
image generation from a spherical image with missing areas captured from an
OMS, and 2) a new imaging technique using two OMSs mounted on the top

Table 2.4. Goals of virtual object superimposition for MR-world exploration.
required, ×: not required.
Real-time (usual) AR√
Photorealistic representation√
Illumination estimationof the real world√
Accommodating real-time
illumination change

30

√
:

MR-world exploration√
××

and bottom of a UAV. The former approach can be employed for images alreadycaptured in an ordinary way using an OMS at the expense of the accuracy of thescene luminance. The latter technique requires a special hardware configurationto acquire the scene luminance with high accuracy. It should be noted that theimaging process is done oﬄine. Details of the proposed imaging approaches aredescribed in Chapter 3.
The proposed rendering method is divided into oﬄine and online processes
to take advantage of high-quality oﬄine rendering for the real-time representation of the MR-world. In the oﬄine process, real scenes are virtualized usinga hybrid free-viewpoint image generation approach, which utilizes automaticallyreconstructed large-scale 3D models. In addition, the virtual objects are rendered through a novel real-time rendering method that can employ any oﬄine
rendering method, that is, scenes that can be rendered photorealistically to the
maximum extent. Therefore, strict illumination settings, time-consuming rendering, and any manual adjustment can be included. Oﬄine-rendered images
are transformed through real-time free-viewpoint image generation, which preserves the high-quality textures. Details of the proposed rendering framework
are described in Chapter 4.
In addition to the MR-world exploration using the proposed imaging andrendering framework, we also show that our framework can be employed in various applications with minor modifications of its implementation, as well as some
performance evaluations. Figure 2.13 illustrates the flow of following three applications using part of the proposed framework:
1. HDR immersive panorama (see Section 5.2): Full spherical HDR images canbe employed in immersive panorama applications such as Google Street
View, by applying perspective projection of the sphere-mapped spherical
Table 2.5. Intended environment of proposed framework.
Ground Sky√
Previous full spherical imaging
×√
Proposed full spherical imaging
×√√
Proposed rendering for MR-world exploration

31

images.
2. Augmented immersive panorama (see Section 5.3): By directly superimposing virtual objects into full spherical aerial images without employing
free-viewpoint image generation, the augmented spherical images can beused for augmented immersive panorama applications. This application
does not provide a change in viewpoint other than at the location where
the images were initially captured.
3. Real-time AR (see Section 5.4): By using real-world scenes captured in realtime, instead of a real-world virtualization, the proposed rendering framework can be employed for (video see-through) real-time AR applications.
4. MR-world interactive exploration (see Section 5.5): When the proposed
framework is fully employed, the produced application realizes the capability of interactive MR-world exploration. Section 5.5 introduces two applications based on the implementation described in Section 4.3.
Table 2.6 summarizes the features of MR applications. As shown in the table,
our study covers broad application areas. Our applications with applicationspecific discussions and evaluations, except for MR-world exploration detailed in
Chapter 4, are described in Chapter 5.
The contributions of this study are summarized as follows:• Proposing a couple of the world’s first full spherical aerial HDR imaging
methods with practical examples.
• Proposing a photorealistic and light-weight rendering method for MR-world
exploration based on free-viewpoint image generation and oﬄine superimposition of virtual objects.
• Developing MR applications based on the proposed framework, in which
the applications can be used for various uses.
The remainder of this thesis describes the proposed imaging and rendering frameworks, as well as the developed applications.

32

Real
world

Imaging

Rendering

Spherical imaging

Perspective
projection

Viewdirection
dependent
images(a) Immersive panorama.
Real
world

Virtualization ofreal-world

Spherical imaging

Freeviewpoint
images(b) Free-viewpoint image generation.

Real
world

Aerial spherical
HDR imaging

Perspective
projection

Other HDR
spherical imaging

Viewdirection
dependent
images(c) HDR immersive panorama.

Real
world

Aerial spherical
HDR imaging

Offline renderingof virtual objects

Other sphericalimaging methods

Viewdirection
dependent
AR images

Virtualobjects(d) Augmented immersive panorama.

Real
world

Real-time
imaging

Rendering for
MR-world exploration

AR
images

Virtualobjects(e) Real-time AR.

Figure 2.13. Applications partially using proposed imaging and rendering frameworks.
33

Table 2.6. Classification of MR applications.
application, ×: not available.

√: available, †: depending on

Viewpoint HDRchangePrevious
applications
Applications
using
proposedframework×√

Immersive panorama
Free-viewpoint
image generation
HDR
immersive panorama
Augmented
immersive panorama
Real-time AR
MR-world exploration

×
×
√
√

34

×
×
√
†
†
†

Virtualobject

Real
scene

×
×

Oﬄine
Oﬄine

×

Oﬄine

√
√
√

Oﬄine
Real-timeOﬄine

Chapter 3
Full Spherical Aerial HighDynamic Range Imaging
3.1. Overview
In this chapter, we propose two imaging techniques for acquiring full sphericalaerial HDR images. To realize photorealistic MR-world generation, large-scale
real-world scenes should be eﬀectively and accurately captured. As discussed
in Section 2.1, some studies have dealt with spherical HDR imaging from theground; however, there has been no consideration regarding aerial imaging. The
proposed aerial-imaging system resolves a problem concerning missing areas that
commonly occur when spherical images are acquired using OMSs.
Missing areas, as shown in Figure 2.7, appear in all single-capture sphericalimages. Such areas are caused by limitations in the FOV of the OMS and/or
occlusions from the vehicles mounting the OMS. In Figure 2.7, the primary occlusion is the aircraft upon which the camera is mounted. Such areas decrease theimmersive value of the image and disqualify it from use in photorealistic renderingsystems. Although there is a study dealing with missing areas on spherical images
acquired from the ground [KMSY10], it is diﬃcult to employ it for aerial imaging
because background scenes of missing areas, most of which occludes the sky, cannot be captured by changing the viewpoint. These missing areas can be captured
in theory, such as by performing inverted flight; however, these approaches are35

not practical in most cases.
This study proposes two approaches overcoming this problem:
1. Spherical aerial image completion using a sky model
Input: A spherical images captured in an ordinary manner using an OMS
mounted at the bottom the aerial vehicle.
Output: Full spherical HDR images.
Summary: Missing areas in the spherical images are completed based on
the All Sky Model [IKMN04], a statistical model for the luminance andradiance of the sky. Although generated luminance may not reliable, thisapproach can be used for existing resources captured by ordinary OMS
aerial imaging. Note that this approach is utilized by inputting multiple
images captured at multiple locations to estimate the captured orientation
using vision-based camera pose estimation methods.
2. Full spherical aerial HDR imaging using two OMSs
Input: Multi-exposure spherical images captured using two OMSs fromthe top and the bottom of an aerial vehicle.
Output: Full spherical HDR images.
Summary: The multi-exposure images are combined with geometric andphotometric alignment. This approach requires a special hardware configuration; however, it can acquire the HDR scene luminance with highaccuracy.

3.2. Spherical Aerial Image Completion Using Sky
Model
3.2.1 Overview
In spherical images captured from the sky, there may be areas that an OMS cannot capture, or parts of the background scenery occluded by the vehicle. Missing
areas must be filled in for these images to be usable as a part of an environmental
light map upon which virtual objects are rendered. While an additional camera
can be used for such purposes, as described in Section 3.3, it is usually diﬃcult
36

Fyber-optic gyroscope,
TISS-5-40 (Tokyo Keiki)
Omnidirectional multi camera system,
Ladybug 3 (Point Grey Research)
Differential GPS,
P4-GPS (Hitachi Zosen Corporation),
is equipped on the top of the airship
Figure 3.1. Unmanned airship and mounted sensors

to capture all directions in a single pass using aerial imaging. As in typical aerial
imaging where the camera is mounted on the bottom of the capture vehicle, in
our situation, missing areas occluded by the capture vehicle appear frequently
in the sky. We overcome this problem using a sky model, which is a statisticalluminance model of the sky for filling in missing areas. This is an easy way toacquire full spherical images from ordinary aerial imaging using an OMS. In thissection, we describe a configuration for spherical aerial imaging using an OMS
and an unmanned airship, followed by a completion method.

3.2.2 Aerial Imaging Using an OMS Mounted on an Aerial
VehicleIn our experimental settings, spherical images are captured using an OMS (Point
Grey Research, Ladybug3) mounted on an unmanned airship, as shown in Figure 3.1. The airship was also equipped with a diﬀerential-GPS (Hitachi Zosen
Corporation, P4-GPS), and a fiber-optic gyroscope (Tokyo Keiki, TISS-5-40). All
sensors on the airship are connected to a laptop PC for storing the captured data.
Figure 2.7 in Section 2.1.3 shows a panoramic image generated from perspective
images acquired by six monocular cameras in an OMS using their intrinsic andextrinsic parameters [ISY03], with the missing area shown in the upper part.
Six degrees-of-freedom (DoF) camera poses (3 DoF positions and 3 DoF orientations) are estimated from the multiple spherical images captured from the OMS
by a structure-from-motion (SfM) approach. Using the camera pose information,
the spherical images are aligned: the i-th image in the input sequence is spheri37

(a) Masked portion shows a missing area.
A part of the missing area overhangs on
ground scenery.

(b) Missing area on ground scenery is completed.

Figure 3.2. Completion of missing areas on ground scenery.

cally mapped, and the sphere is then rotated by R−1= RT
i using the estimatediorientation Ri of the i-th image.
We assume that both the OMS and the vehicle are in the same position relative
to each other and that the missing areas are manually identified. The positionof a missing area does not change in the entire image set when the pose of thecamera is relatively fixed to the capture vehicle. According to these assumptions,the completion process for an environmental map is as follows:
1. Completion of missing areas on ground scenery.
2. Completion based only on the intensity of the spherical images.
3. Completion of the remaining missing area using the All Sky Model [IKMN04].

3.2.3 Completion of Missing Areas on Ground Scenery
Although most parts of a missing area appear in the sky, the airship may also
occlude a portion of the ground scenery, as shown in Figure 3.2(a). We supposethat there are no ground objects at a high elevation angle, and in this study, an
area at a high elevation angle of over 5 degrees is defined as the sky area. The
38

other area is defined as the ground area. The intensities of the previous frame
are used to filling-in a missing area in the ground section. We suppose that achange in the position of the OMS is suﬃciently low between two consequent
frames, and a missing area that appears is far away from the OMS, particularly
for the ground area. A section similar to the missing area is searched from theprevious frame using the Sum of Squared Diﬀerences (SSD), and the intensities of
the corresponding pixels are copied to the pixels of the missing area. The groundarea of the spherical video is filled-in as shown in Figure 3.2(b) by applying thisprocess to the whole sequence.
Because this process assumes the input images as a sorted video sequence,the process should be skipped for unsorted multiple images. In this case, missing
areas may remain on the resultant images.

3.2.4 Completion Based Only on the Intensity of Spherical
ImagesTo estimate the intensity of the pixels in a missing area in the sky, we suppose thatthe illumination of the sky and clouds does not change during capturing the image
sequence, and that the sky and clouds are infinite. Under these assumptions, theintensity of the sky is generalized, and one sky condition can be estimated in
the entire image set because the intensity in the same direction does not change
significantly while all images are captured. The aligned spherical images are first
converted into sky images in equisolid angle projection, which have a uniform
solid angle per pixel, as shown in Figure 3.3(a). The intensity vuni of a pixel in a
direction in the unified sky image is estimated from the intensity vi of the pixelin the same direction in the i-th image of the input image set (see Figure 3.3(b)).
{∑∑∑1i αi vi (i αi ̸= 0)αiivuni =(3.1)∑
undef ined
( i αi = 0),{
1 (vi is not in the missing area)
αi =0 (vi is in the missing area).
Because this process assumes that the illumination of the sky and clouds does
not change, short sequences of only dozens or hundreds of frames are better whenprocessing images of a drastically changing environment.
39

(a) A sky image in equisolid angle projection, where the missing area is masked.

(b) The completion results of the sky area
based only on the intensity of the sphericalvideo(c) A sky image generated using All Sky
Model [IKMN04](d) A completed sky image.

Figure 3.3. Completion of an environmental map.

40

3.2.5 Completion of Remaining Missing Area Using All
Sky Model
The same part of a missing area can be occluded in the entire image set (i.e.,∑
i αi = 0); therefore, the intensity vuni cannot be determined in Equation (3.1).
In the All Sky Model [IKMN04], the luminance (and radiance) distribution of
the sky is statistically modeled, which can be used to complete the remaining
missing area. The All Sky Model is known to be a good approximation of the sky
distribution in all-weather scenarios using a sky index, which is a variable used
for indicating the sky conditions. When generating a complete sky using the All
Sky Model, the generated sky image is represented as an HDR image because thecalculated intensities of the pixels can exceed the 8-bit value.
In this model, the sky luminance, La(γs , γ), is defined as the product of thezenith luminance, Lz(γs ), and the relative sky luminance distribution, L(γs , γ, Si).
The relative sky luminance distribution varies depending on the weather. Thus,
the All Sky Model [IKMN04] can be described as follows:
La(γs , γ, Si) = Lz(γs )L(γs , γ, Si),

(3.2)

La(γs , γ, Si) : luminance of a sky element,Lz(γs ) : zenith luminance,
L(γs , γ, Si) : relative sky luminance distribution,
γs : direction (2 DoF polar coordinates) of the sun,γ : direction (2 DoF polar coordinates) of the sky element, andSi : sky index.
Here, the sky index, Si, takes a value of 0.0 ≤ Si ≤ 2.0 depending on the weather.
A larger Si indicates fine weather, and a lower value indicates a diﬀused (cloudy)
sky. Si is calculated using the amount of global solar radiation in the environment.
Solar altitude γs is calculated automatically using the acquired time and date of
the images, as well as the location in latitude-longitude coordinates. Normally,
the luminance of the sky, La, and the zenith luminance, Lz, are represented using
a physical unit, which is an unknown value in our case. In this study, we used the41intensity of the pixels with linear gamma, which are proportional to the physicalvalues.
The zenith luminance, Lz, which is required to calculate the intensity of thesky, is usually unknown because the zenith is occluded during an entire sequence.
Equation (3.2) is modified to calculate Lz from La and L as follows:Lz =La(γ, Si).
L(γ, Si)(3.3)

Here, we treat Si as a variable, because the global solar radiation, which is
required to calculate Si, cannot be acquired due to existence of missing area.
γs is assumed to be a constant in our situation, which is calculated from thecapturing time and date. The intensity of the pixels that are not in the missing
areas can be used as La, but the Si must still be estimated to calculate L. Once
the Si is estimated, Lz can be calculated using the acquired La. The Lz is ideally
not varying, but when calculated using the real intensity it might include errors.
We therefore estimate the Si with the minimum variance of Lz calculated fromthe intensity of each pixel in the sky images, which are generated by the method
described in Section 3.2.4. Here, V denotes a set of pixels fulfill two conditions:
not belonging to missing area, and not saturated. If γk denotes the direction of
pixel k in a sky image, the optimal sky index Siopt is estimated based on theminimum variance of Lz as follows:∑()¯ acq (γk , Si) 2 ,Siopt = arg min
Lzacq (γk , Si) − Lz
(3.4)
Si

k∈V

Laacq (γk )Lzacq (γk , Si) =,L(γk , Si)
where Lzacq denotes the calculated Lz from the acquired real intensity Laacq , and¯ acq is the average of Lzacq among all γk . Using Equation (3.4) is equivalentLz
to calculating the Si with the minimum error between the generated virtual
skies, using every Si and the real intensity, as illustrated in Figure 3.4. In our
implementation, optimal Si are searched from Si = 0.0 through Si = 2.0 by tiny
increments (i.e., 0.1).
The intensity of the remaining missing area is calculated from Equation (3.2),
using Siopt as the Si. The intensity may vary at the boundary between areas
filled in using the method described in Section 3.2.4 and areas filled in using
42

Sky diffusion

Higher
Si = 0.0

Lower

Si = 1.0…

Si = 2.0…
Si = 0.6 (Minimum variance)

Figure 3.4. Estimation of Si in the All Sky Model [IKMN04].

the method shown in Section 3.2.5; therefore, their boundary is alpha-blended
into the estimated sky image. Figure 3.3(c) shows a sky image generated by themodel, and Figure 3.3(d) shows an example of a complete sky image.

3.2.6 Experiment with Spherical Image Completion
To confirm that the spherical images are completed without unnatural artifacts,
we conducted an experiment using a spherical image sequence captured by an
OMS mounted on an unmanned airship. The spherical images were completed
based on the method described in Section 3.2.2 using a sky model, where sky
index Si, which indicates weather condition, for this experiment was estimatedto be 0.6 (slightly overcast weather). The captured image sequence consistedof 1,900 frames, and the camera pose was successfully estimated for the whole
sequence by a SfM approach using GPS measurements [YISY06]. Figure 3.5 shows
frames sampled from the first 1,400 frames of the input image sequence. As in
the completed images shown in Figure 3.6, full spherical images were successfully
generated using the completion method without notable artifacts.

43(a) First frame.

(b) 200th frame.

(c) 400th frame.

(d) 600th frame.

(e) 800th frame.

(f) 1,000th frame.

(g) 1,200th frame.

(h) 1,400th frame.

Figure 3.5. Spherical image sequence with missing areas, which were captured by
OMS mounted on unmanned airship.

44

(a) First frame.

(b) 200th frame.

(c) 400th frame.

(d) 600th frame.

(e) 800th frame.

(f) 1,000th frame.

(g) 1,200th frame.

(h) 1,400th frame.

Figure 3.6. Full spherical image sequence completed using the sky model.

453.3. Full Spherical Aerial HDR Imaging Using
Two OMSs
3.3.1 Overview
If a wealth of equipment was available for imaging, a new imaging techniquecould be employed, and the completion method described in Section 3.2 would
not be necessary. This technique generates full spherical HDR images from multiexposure images captured from two OMSs mounted on the top and bottom ofan unmanned aircraft. That is, missing areas in a spherical image captured
from the top (see Figure 3.7(a)) and bottom (see Figure 3.7(b)) of the vehicle
are mutually completed, as shown in Figure 3.7(c), because the whole directionof the scene is acquired from at least one camera. Compared with the image
completion technique, this approach is expected to acquire HDR scene luminancewith higher accuracy.
As shown in Figure 3.8, this procedure is divided into the following threestages:1. Multi-exposure aerial image capture from two OMSs.
Multi-exposure images are captured using two OMSs mounted on the top
and bottom of an unmanned aircraft. Since the camera on top of the vehicle
mainly captures the sun and sky, it has neutral density (ND) filters attached.
To capture high- and low-luminance images eﬀectively, the exposure times
are controlled automatically.
2. HDR image generation from multi-exposure images.
HDR spherical images are generated from the multi-exposure images captured from the top and the bottom of the aircraft. Misalignments are corrected by estimating changes in the camera orientation over the sequenceof captured images.
3. Composition of HDR images captured by the two cameras.
Full spherical HDR images are synthesized from the HDR images generated
during the second stage. Because the precise geometric relationship between
images from the top and the bottom of the aircraft is diﬃcult to fix (owing
46

(a) Image captured from top of the aircraft.

(b) Image captured from bottom of the aircraft.

(c) Full spherical image.

Figure 3.7. Full spherical HDR image generation using two OMSs.

47

BottomTop1) Multi-exposure
aerial image capture2) HDR image generationfrom multi-exposure images3) Composition of HDR images
captured by two cameras
Figure 3.8. Overview of full spherical aerial HDR imaging.

to the slight deformability of the unmanned aircraft), we align the images by
estimating the relative rotation from one camera to the other for each frame.
We also correct chromatic changes that may occur in images captured using
ND filtering.

3.3.2 Multi-Exposure Aerial Image Capture from Two OMSs
Configuration of the aerial imaging system using two OMSs
We used a 12-m remote-controlled aircraft, which is the same as the one used in
Section 3.2.2. As shown in Figure 3.9, two OMSs (Point Grey Research Ladybug2)
were mounted on the top and bottom of the aircraft. The cameras were connected
to a laptop PC for time-stamped storage. ND filters (Fujifilm Corp., ND 2.0)
blocking all but 1% of the visible light were attached to the camera on top of thevehicle. Figures 3.7(a) and 3.7(b) show panoramic images with the limb darkening
removed [ISY03]. Note that the amount of limb darkening diﬀers between images
captured with and without ND filters.

48

Figure 3.9. Unmanned aircraft and two OMSs. OMSs were mounted on both the
top and bottom the aircraft. ND filters were attached to the top camera, and a
GPS antenna was also mounted on the top of the aircraft.

Automatic exposure control
The exposure times used to capture multi-exposure images are automaticallydetermined from the intensity of the previously captured images. Many stillcameras provide an auto-bracketing function for capturing multi-exposure images.
They determine the proper exposure value using standard auto-exposure controls,
and use fixed multiples of this value for the remaining exposure values [KUWS03].
Grossberg et al. [GN03] proposed a method for determining the unfixed exposuresets from the dynamic range of a scene. To determine a more appropriate exposure
set, we use an HDR histogram of the previous multi-exposure image sequence toreduce the side eﬀects of large quantization steps on human vision, such as pseudoedges resulting from the use of too few multi-exposure images. Note that ourapproach optimizes the set of exposure times, whereas other modern studies havesought to minimize the signal-to-noise ratio (SNR) [HDF10]. Such minimization
can also be integrated into our approach as a means for improving the appearanceof HDR images, particularly those captured in dark environments.
New exposure times, snew1 , ..., snewn , are determined from multi-exposure im49

ages captured using old exposures sold1 , ..., soldn . In practice, for storing the exposure times, we set n = 4 as the number of exposures in our experiment to matchthe number of registers available in the Ladybug2 cameras.
The shortest exposure time, snew1 , is the value at which the scene captured
using sold1 can be done so without saturation, as follows:
0.5sold1(L1 = M AX IN )
M AX IN + θshsnew1 =
(3.5)
sold1 (L1 ≤ θsh )

2L1



sold1
(otherwise),
where L1 denotes the maximum intensity in the image captured using sold1 ,and M AX IN denotes the maximum representable intensity of the LDR image (M AX IN = 255 for 8-bit images). Further, θsh is a threshold that limits
snew1 to θsh < L1 < M AX IN when the response curve is regarded as linear.
When only a small number of multi-exposure images are acquired, side eﬀects
of the large quantization steps, such as pseudo-edges, can appear in the generated
HDR images. Other exposure times, snew2 , ..., snewn , are therefore determined toreduce such eﬀects. To estimate the exposure times that have minimum quantization steps, an HDR histogram is calculated from the captured images, as
illustrated in Figure 3.10. Hi is the maximum intensity captured using the i-th
shortest (i ≥ 2) exposure time snewi in the HDR histogram. Using Hi , Mi iscomputed through
Mi =

Hi
∑

jk ,

(3.6)

k=H

where jk is the number of pixels whose intensity is k, and H is defined as{0(i = n)H=(3.7)
Hi+1 (otherwise).
The quantization step corresponding to snewi in the HDR histogram can be calculated as follows:
∆i =

Hi
.
M AX IN
50(3.8)

Number of pixels

0Intensity
Figure 3.10. Illustrative example of HDR histogram for n = 4, where ∆i denotes
quantization steps and Hi is the maximum intensity that can be captured using
exposure time, snewi . Further, M3 is the number of pixels in the filled area of thehistogram, with the other Mi determined in a similar fashion.

Finally, snewi (i ≥ 2) is determined by varying snewi to minimize the energy function, Es , where Es is the sum of the products of the quantization step ∆i and thenumber of pixels corresponding to the exposure time snewi :∑Es =(Mi ∆i ).
(3.9)
i

Es is a non-linear function with multiple local minima. We apply a simple coarseto-fine search technique to find a reasonable solution.
The above process for determining the exposure values is repeated every several seconds to allow for changes in the real-world lighting environment. Notethat snew1 may not converge within a single cycle, and that several cycles areoften needed.
51

(a) Without alignment.

(b) With alignment.

Figure 3.11. HDR images with and without multi-exposure image alignment.

3.3.3 HDR Image Generation from Multi-Exposure Images
Alignment of multi-exposure images
Misalignments occur among multi-exposure images owing to changes in the position and orientation of the camera while capturing images at diﬀerent exposure times. This causes blurring in the resulting HDR images, as illustrated in
Figure 3.11(a). Our approach to resolving misalignments in aerial imaging is
explained as follows.
For an aircraft moving at 5m/s and rotating at 30◦ /s, the misalignment angles
among multi-exposure images are calculated as shown in Table 3.1. When a52

camera captures a scene from a great distance, which is a common situation inaerial imaging, the amount of misalignment is predominantly aﬀected by changes
in the camera orientation. We correct such misalignments by estimating thechanges in camera orientation from the captured image sequence, as illustrated in
Figure 3.11(b). Note that this correction method does not take a position change
into consideration, and therefore ignores misalignments from fast-moving objects
captured at short distances. Fortunately, local alignment methods, proposed forgeneral multi-exposure imaging, can be applied when significant camera motion
occurs.
Figure 3.12 shows the diﬀerent steps of our multi-exposure image alignment
process. In the first step, two images with the same exposure are selected, and thecamera rotation between them is estimated, as shown in Figure 3.12(a). Amongthe multi-exposure images, those having the fewest saturated pixels (with anintensity of 255) and underexposed pixels (with an intensity of less than 16 in
our implementation) are selected. The corresponding points (pm , qm ) between
the selected images are determined using the KLT tracker [ST94] and projected
onto a unit sphere. The parameters for rotation Rci are estimated by nonlinearlyminimizing the energy function El , which is defined as the sum of the squared
Euclidean distances between the projected corresponding points |pm , qm |:∑El =|pm , qm |2 .
(3.10)m

Table 3.1. Amount of misalignment owing to changes in position and orientationof the camera: one cycle represents the time taken to capture a single multiexposure image set (0.25 s).

Change ofposition
Orientation

Distance
to object
50 m
100 m
500 m

Misalignment
per a cycle
1.43◦
0.726◦
0.143◦

n.a.

7.5◦

53

Transition of exposure time(a) Estimating rotation between images that have the same exposure.

‫܀‬,‫܀‬,‫܀‬,‫܀‬,

(b) Calculating rotation between neighboring images using spherical linear interpolation.

Reference image(c) Calculating rotation to reference image from other images.

Figure 3.12. Process for aligning multi-exposure images.

54Note that RANSAC [FB81] is used to reduce errors from mismatches.
Next, as shown in Figure 3.12(b), the rotation parameters between each two
adjacent images Rf (i,j) are calculated by interpolating Rci using a spherical linear
interpolation.
The rotations of an arbitrary reference image from neighboring images arethen calculated from Rf (i,j) , as shown in Figure 3.12(c). Multi-exposure images
are aligned with the reference image through a transformation using the obtained
rotation parameters. To align multi-exposure images for an entire video sequence,
Rf (i,j) must first be calculated for the entire sequence, after which each frame inthe sequence can be treated as a reference image.
HDR image generation
To compose the intensity of the multi-exposure LDR images, we use the HDR
imaging method proposed by Debevec et al. [DM97]. In applying this method, it
is necessary to consider the light attenuation owing to the ND filters attached tothe camera on top of the aircraft. If the response curve is linear, or linearized inadvance, the intensities in the HDR image, Ih , can be calculated using the LDR
intensity Il , the nominal light transmittance η of the ND filter, and the exposure
time t[s] of the LDR image:{ Iκ tl (without ND filters),
Ih =(3.11)Ilκ ηt(with ND filters).
Although the scale factor κ is meaningless when only the relative pixel intensities are needed, it can be determined and used to calculate the absolute radiance values [W/sr/m2 ]. The HDR intensities calculated from each set of multiexposure images are composed in accordance with the method proposed describedin [DM97].

55

Figure 3.13. Masks used for alignment of images from the two cameras. Top left:
Black pixels indicate the missing area from the top of the aircraft. Bottom left:
Black pixels indicate the missing area from the bottom of the aircraft. Right:
White pixels indicate an overlapping area.

3.3.4 Composition of HDR Images Captured by Two Cameras
Alignment between the two cameras
The HDR images captured from the top and bottom of the aircraft are aligned byestimating the relative rotation between the cameras. To estimate the parametersof this rotation, missing areas in the images are manually masked in advance,
as shown on the left side of Figure 3.13. Note that only one mask image isrequired for an entire sequence captured by a given camera, as the missing area
for spherical images does not significantly change. The area marked by white
pixels on the right side of Figure 3.13 indicates the overlapping area where thescene has been captured by both cameras without occlusions. This area is defined
as a conjunction of the negation of missing areas in the paired images.
The corresponding points are detected by applying the KLT tracker [ST94] to56

(a) Without ND filter (image captured
from bottom of airship). Average intensity: (R, G, B) = (111.5, 108.3, 119.7).

(b) With ND filter (image captured
from top of airship). Average intensity:(R, G, B) = (95.1, 103.5, 111.9).

Figure 3.14. Example of chromatic change due to ND filters. A pair of HDR
images converted to the same exposure is shown in close-up, at region where thetwo images overlap.

feature points in the overlapping area of the image captured from the bottom of
the aircraft. The rotation parameters between the cameras from top to bottom
are determined using the method described in Section 3.3.3
Correction of chromatic change due to ND filters
Although ND filters are designed to transmit all wavelengths of light equally, they
do cause a certain amount of chromatic changes [STJ+ 04]. The ND filters usedin our experiment transmit less red light, as shown in Figure 3.14. Such changes
are corrected by estimating the linear intensity transformation parameters fromthe intensities in the overlapping areas of the images from the two cameras. The
RGB values (Rtop (x), Gtop (x), Btop (x)) for pixel x in the image captured from57

atop the aircraft are converted using the following linear transformations:′
Rtop(x) = βr Rtop (x),(3.12)

G′top (x)= βg Gtop (x),(3.13)′
Btop(x)= βb Btop (x),(3.14)

where βr is estimated byβr =Rbot (x)Σx∈A R
top (x)NA,(3.15)

where (Rbot (x), Gbot (x), Bbot (x)) denote the RGB values of the corresponding pixel
x in the image captured from the bottom of the aircraft, A denotes the overlappingarea, and NA indicates the number of pixels belonging to A. In addition, βg andβb are estimated in a same fashion.
Combination of corrected HDR images
Full spherical HDR images are generated by combining paired HDR images thathave been realigned and chromatically corrected, as described above. The intensities in the overlapping area of the full spherical image are determined by alphablending the two images. Based on the intensities, Itop and Ibot , of a pixel in theoverlapping area of the two images, the intensity, If ull , of the corresponding pixelof the full spherical image can be calculated using
If ull = αIbot + (1 − α)Itop ,(3.16)

where α varies linearly between zero at the upper boundary of the overlapping
area and unity at the lower boundary.

3.3.5 Experiment with Spherical Image Generation Using
Two Cameras
Generation of still image
To confirm that the HDR image generated using our proposed method reflects
the real environment with reasonable fidelity, we conducted an experiment in
58

which we generated a full spherical HDR image from still images captured using
an unmanned aircraft. The aircraft was flown at 3 m/s at an altitude of 130 m
while capturing the multi-exposure images. M AX IN = 255 and θsh = 192 wereused for this experimental environment.
The captured multi-exposure images and corresponding exposure times areshown in Figure 3.15. Figures 3.7(a) and 3.7(b) show HDR images from the top
and bottom of the aircraft, respectively. The full spherical HDR image composedfrom the images in Figures 3.7(a) and 3.7(b) is shown in Figure 3.7(c). Note thatthe two realigned and chromatically corrected images were successfully combined,leaving no obvious artifacts. Figure 3.16 shows the full spherical images whose
intensity was cropped from the HDR image using the exposures equal to thosein Figures 3.15(e) and 3.15(h). From this, we can confirm that the full spherical
HDR image was satisfactorily combined from the captured multi-exposure images.
The full spherical HDR image can also be visualized using typical tone-mapping
methods such as the one in [RSSF02], as shown in Figure 3.17. Such tone-mapped
images are suitable for immersive panoramas, which provide users with a senseof looking around a location, and the ability to see textures of various radiances.
Generation of video sequence
To generate full spherical HDR videos for fly-through applications, we applied ourapproach to a video sequence consisting of 500 frames. The aircraft was flown
at a speed of 5 to 8 m/s and an altitude of 130 m while capturing the multiexposure images. The frame rate of the generated HDR video was 16 fps, whichis the same as that of the multi-exposure source video. The processing time
was approximately 10 s per frame on a desktop PC with a Core i7-2600 CPU(3.40 GHz, 4 Cores). The position and orientation of the camera on the bottomof the aircraft were estimated using a SfM technique for OMS without GPS
measurements [SIY04]. Figure 3.18 shows samples of the full spherical framesthat are aligned using camera pose information by same manner as described in
Section 3.2.2. Note that the position and orientation of the video frames weresuccessfully estimated. This information can be used for a geometric registration
between real and virtual objects in augmented immersive panoramas.

59

(a) Top: 0.1 ms (with ND filters).

(b) Top: 4.1 ms (with ND filters).

(c) Top: 14.4 ms (with ND filters).

(d) Top: 43.3 ms (with ND filters).

(e) Bottom: 0.1 ms.

(f) Bottom: 0.4 ms.

(g) Bottom: 1.1 ms.

(h) Bottom: 2.4 ms.

Figure 3.15. Captured multi-exposure images and corresponding exposure times.

60

(a) Exposure for the image in Figure 3.15(e).

(b) Exposure for the image in Figure 3.15(h).

Figure 3.16. Full spherical images with cropped intensity using exposures of themulti-exposure images.

Figure 3.17. Full spherical image tone-mapped in accordance with the method
by [RSSF02].

61

(a) First frame.

(b) 100th frame.

(c) 200th frame.

(d) 300th frame.

(e) 400th frame.

(f) 500th frame.

Figure 3.18. Full spherical tone-mapped images generated from video frames.
The images were aligned using estimated relative camera orientation.

62

Figure 3.19. Completion of missing area based on spherical image completion
using sky model.

3.4. Discussions
3.4.1 Comparison of Two Approaches
We compared the two approaches for acquiring full spherical aerial HDR images.
Figure 3.19 shows the same frame as Figure 3.18(b) but with the missing area
completed using the sky model. Note that, from a subjective viewpoint, theestimated smooth textures in Figure 3.19 may seem unnatural and decrease theimmersive value of the image.
To investigate the eﬀects when using full spherical images for a photorealistic
virtual object superimposition using IBL, we compared the estimated intensity
shown in Figure 3.19 with the acquired intensity. The average intensity of thesky estimated using the sky model was quite diﬀerent from the intensity capturedby the camera, (1 : 0.218), as illustrated in Figures 3.18(b) and 3.19. When theimage in Figure 3.19 is used for IBL, virtual objects with Lambertian surfaces become 0.218-times brighter than using the image in Figure 3.18(b). Furthermore,in Figure 3.18(b), the highest intensity (which aﬀects the appearance of cast shadows) was ten-times larger than the intensity in Figure 3.19. These results indicate
63that the approach described in Section 3.3, which employs an additional skywardcamera for capturing the actual intensity of a missing area, produces better IBL
rendering than current approaches using model-based sky area completion.

3.4.2 Auto-Exposure Control
We compared the auto-exposure control method described in Section 3.3.2 withordinary controls for HDR imaging. Most control methods, including that usedby [KUWS03], first determine the base exposure automatically, and then calculate
the remaining exposure values at fixed stops from the base exposure. To avoid
saturating the scene, we used the shortest exposure time, sc1 [ms], as the base
exposure for this experiment. The other exposure times, sc2 , ...sc4 , were thencalculated for every two stops, meaning that sci was set to 22 = 4 times longer
than sci−1 (i.e., fixed multiples). Note that advancing two stops for multi-exposure
imaging is the default commonly used in commercial auto-bracketing cameras.
Table 3.2 shows the calculated maximum intensity and quantization steps foreach approach. The maximum intensity of each multi-exposure image is a value

Table 3.2. Quantization steps of the HDR image using the approach describedin Section 3.3.2 and the conventional approach based on fixed multiples of theshortest exposure time.
ExposureAcquirable
Quantizationtime[ms] max. intensity
step
Proposed

Fixed
multiples

0.1
4.1
14.4
43.3

1000
24.4
6.94
2.31

3.92
0.10
0.03
0.009

0.1
0.4
1.6
6.4

1000
250
62.5
15.6

3.92
0.98
0.25
0.06

64

(a) Close-up of a generated HDR image.

(b) Gradient of the image in Figure 3.20(a)using Sobel filter.

(c) Image re-quantized
multiple exposures.

(d) Gradient of the image in Figure 3.20(c)by Sobel filter.

using

fixed-

Figure 3.20. Visualization of negative eﬀect of large quantization steps.

relative to the maximum intensity in the scene, which is defined as 1000. The
quantization steps of the acquired multi-exposure images (8-bit) were converted
into the HDR intensity by dividing the maximum intensity by 255 in the same
way as ∆ in Section 3.3.2. A smaller value is better here, as it help prevent visual
artifacts such as pseudo-edges. A comparison of the longest exposure time indicates that the quantization step of the conventional method was 6.7-times larger
than that of our proposed approach. This diﬀerence can particularly aﬀect areasof low intensity, such as most ground features. Figure 3.20 shows close-ups of agenerated HDR image of the same frame shown in Figure 3.17. Figure 3.20(c)shows an artificially re-quantized image generated from this image using the exposure times of a conventional approach, sci (1 ≤ i ≤ 4). Figures 3.20(b) and65

3.20(d) show the gradient images of the frames in Figures 3.20(a) and 3.20(c),respectively. Pseudo-edges appear in the image in Figure 3.20(d), mainly in low
intensity areas. This indicates that our proposed approach can prevent visual
artifacts produced when using a conventional approach.

3.5. Summary
This chapter described two approaches for acquiring full spherical aerial images
suitable for photorealistic MR-world generation. The first approach, which was
described in Section 3.2, only requires ordinary spherical aerial imaging equipment
consisting of an aerial vehicle and an OMS mounted on the bottom of the vehicle.
Spherical images captured from the equipment include missing areas, and are
therefore completed using the All Sky Model [IKMN04], which is a statisticalmodel of the sky luminance and radiance. The second approach requires special
hardware configurations, that is, two OMSs mounted on the top and bottomof an aerial vehicle, as detailed in Section 3.3. The two OMSs cover the whole
direction at a particular viewpoint; therefore, full spherical images are generated
by completing mutual missing areas appearing in the spherical images captured
by each OMS. In addition, to generate HDR images, the two OMSs each capture
multi-exposure images.
Although both approaches generate HDR results, the experiments demonstrated that the approach using two OMSs generated a notably higher intensity
compared with the spherical image completion using a sky model. This result
indicates that the additional skyward camera largely improves the accurate acquisition of real-world illumination, which can be used for virtual-object superimposition using IBL. When we employ the results of model-based sky completion
for IBL, it may be necessary to manually edit the illumination information.

66

Chapter 4
Photorealistic Rendering for
MR-World Exploration
4.1. Overview
In some applications that allow an interactive virtual exploration of a virtualized
real-world, virtual objects can be superimposed for a visualization of disasterareas or non-existent buildings [XBF+ 09, OKY10]. As discussed in Chapter 1,
the virtual objects are rendered frame by frame in applications that allow users
to change their viewpoint. Unlike the previous real-time rendering used in AR,
this study proposes a framework that combines the oﬄine rendering of virtual
objects and free-viewpoint image generation to take advantage of the high qualityof oﬄine rendering without the computational cost of online CG rendering, i.e., it
incurs only the cost of online computations for free-viewpoint image generation.
In addition, the generation of structured viewpoints (e.g., at every grid point)
reduces the computational costs required in an online process. The advantagesof our framework are as follows:• High-quality superimposition of virtual objects by combining pre-renderingand free-viewpoint image generation.
• Reducing the computational cost in online free-viewpoint image generation
using pre-generated structured viewpoints.

67

This chapter describes a practical implementation that provides a free-viewpoint
using a 2D grid structure based on our framework, which superimposes virtual
buildings onto real scenes virtualized. This application inputs full spherical aerial
images acquired by the method described in Chapter 3. The virtual objects are
rendered based on IBL using spherical images.

4.2. Free-Viewpoint Image Generation Frameworkfor Photorealistic Superimposition of Virtual Objects in Real-World Virtualization
The proposed rendering framework inputs multiple spherical images capturing
the real world, including but not limited to spherical aerial images acquired fromthe imaging techniques proposed in Chapter 3. The rendering framework canbasically be employed for general situations using spherical images.
Our framework is divided into two parts: an oﬄine process and an online
process. The oﬄine process inputs both real-world images captured at various
positions, and 3D models of the virtual objects, and generates augmented scenes
at numerous viewpoints. Depending on the target environment, the viewpoints
are given as a multi-dimensional structure, such as on a line, grid, or cube, as
illustrated in Figure 4.1. Real-world views from these structured viewpoints are
first generated using a free-viewpoint image generation method using sphericalimages and reconstructed 3D shapes of the real-world objects. If the viewpoints
are densely designated, the quality of online-generated images is expected toincrease; however, large amount of data storage is required to store pre-generated
images. The oﬄine process can include manual operations, and therefore, in
principle, cinema-quality superimposition can be achieved with time and eﬀort.
The online process generates a perspective image from the user’s location
through real-time free-viewpoint image generation using oﬄine-rendered textures.
This process does not depend on the method used to estimate the position and orientation of the user’s viewpoint, and does not designate a specific free-viewpoint
image generation method. A simple guideline for selecting proper free-viewpoint
image generation methods is described in Section 2.2.1. The structured view68

1D(line)
2D(grid)

3D(cube)

Figure 4.1. Structured viewpoints designated during oﬄine process.

points, which are generated in the oﬄine process, are basically expected to reduce the computational costs of the online process. In image-based or hybrid
free-viewpoint image generation, the texture selection and blending processes refer to m multi-viewpoint images, with respect to each of n polygons of the 3D
shape in physically-based methods, or n pixels of the image in appearance-based
methods. This process can include up to O(m ∗ n) calculations. The online process in the proposed framework realizes O(1) calculations for texture selection andblending, which refers only to neighboring structured viewpoints, as illustrated in
Figure 4.2. In principle, the dimension of user’s location changeable in online canbe diﬀer from that of the structured viewpoints; i.e., 3D viewpoint change canbe realized using one dimensional (1D) or 2D structured viewpoints. However,it should receive careful attention that image generation from a location where
faraway from any structured viewpoints clearly leads degradation of the image
quality.
As described above, the rendering framework consists of simple processes andis able to accommodate various methods. In the remainder of this chapter, theproposed framework is detailed using an example of our specific implementation.

69

Pre-renderedaugmented views
Viewpoint generated in
online process

Figure 4.2. Online free-viewpoint image generation using only neighboring structured viewpoints (in case of 2D structured viewpoints.)

4.3. Fly-Through Application Using Full Spherical Aerial Images, 2D Grid Structure, andVDTM
This section describes an example of practical implementation based on the proposed rendering framework, using full spherical aerial images generated as described in Section 3. Note that, proposed framework itself does not restrict the
input to the aerial images. The application is summarized as follows:• Intended use: A virtual tourism application superimposing 3D modelsof historical buildings. The user’s viewpoint is configured in the sky; it isintended to be used like virtual globe applications.
• Input: Full spherical aerial images generated as described in Section 3.
• Dimension of structured viewpoint: 2D grid.
• Free-viewpoint image generation method: VDTM [DTM96]. Because
this application employs aerial views, it is expected that the 3D shapes of
the real-world are far from the viewpoints. As discussed in Section 2.2.1,
physically-based hybrid rendering approaches should be appropriate for ourapplication. VDTM is one of the common physically-based hybrid rendering
approaches.
70Captured points of images
Structured viewpoints (Grid)

Figure 4.3. Pre-generation of real-scene images at grid points: unstructured
captured viewpoints are re-sampled into a grid viewpoint structure using oﬄine
free-viewpoint image generation.

In our application, real-world views are first generated at every grid point, as
illustrated in Figure 4.3, using a VDTM approach [DTM96]. Virtual objects arethen photorealistically rendered for every structured viewpoint using real-world
illumination without any missing areas of the illumination environments in thespherical images, which are generated using the completion method described in
Section 3.2.
Augmented free-viewpoint images, whose viewpoint can be freely configured
on a plane, are presented to users during the online process. The online freeviewpoint images are generated from structured augmented viewpoints, which
are generated during the oﬄine process. The computational cost of online freeviewpoint image generation is reduced using a simplified version of VDTM, where
the textures of four neighboring structured viewpoints are blended into simplified
3D models using a bilinear weighting function (as illustrated in Figure 4.2). The
online process requires only four (i.e., O(1)) weight calculations to blend textures,
while the oﬄine process needs O(m ∗ n) calculations with m polygons and ncameras.

71

4.3.1 Oﬄine Process: Photorealistic Superimposition of
Virtual Objects on Structured Viewpoints
During the oﬄine process, the augmented images are generated at the structured
viewpoints (at every grid point in our implementation) from the real-world images
captured at hundreds of positions and from 3D models of the virtual objects. The
flow of the oﬄine process is as follows:
1. Generation of real-scene images for structured viewpoints
2. Rendering of virtual objects for structured viewpoints
Camera pose information and the 3D shapes of real-world objects are used togenerate the views from the grid points. The views are constructed from the
input images using VDTM, a hybrid free-viewpoint image generation of modeland image-based rendering [DTM96]. The virtual objects are superimposed with
the IBL on the completed environmental maps of the structured views, which are
generated for every grid point.
Generation of real-scene images for structured viewpoints
Six DoF camera pose and dense real-world 3D shapes are first reconstructed. We
employed a modern vision-based SfM approach using captured spherical images
and GPS measurements. VisualSFM [Wu13], which is an SfM application using
multi-core bundle adjustment [WACS11], estimates the relative camera positions;however, it does not provide an absolute scale, which is required for geometricregistration between real and virtual objects without adjusting their coordinates
manually. For our situation, camera position information using an absolute scale
was acquired from the GPS installed on our aerial vehicle, and a non-linear minimization of the SSD between the GPS measurement and the camera position
estimated using SfM was performed. Some approaches combining SfM and GPS
measurements using constrained/extended bundle adjustments [KTSY10, Lhu11]
may also be suitable to our situation with slight improvements. CMPMVS [JP11],
a state-of-the-art multi-view stereo (MVS) software, was used for further reconstruct dense 3D polygonal shapes of real-world objects. The spherical images
were preliminarily converted into six perspective images (cube maps) to prepare72the input data for the SfM and MVS software, which uses perspective images as
the input. The estimated camera pose is also used for the completion of missing
areas in spherical images as described in Section 3.2. Note that the input images
for this application are not prepared as a video sequence; a part of the completion
process assuming video input, as described in Section 3.2.3, is not performed.
The reconstructed 3D models of large outdoor environments consist of millionsof polygons, which causes a heavy amount of computations for free-viewpoint
image generation. Our framework therefore generates densely packed structured
viewpoints of the real world during the oﬄine process to reduce the amount ofcomputations of the online process. By generating dense arrangement viewpoints,
the 3D models are expected to be greatly simplified during the online process
because of the small disparity between neighboring pre-generated viewpoints. In
our application, spherical views of the physical environment are pre-generated atevery grid point on the designated plane, allowing changes in the user’s viewpoint,
as illustrated in Figure 4.3.
In the VDTM [DTM96], the images captured from multiple positions areprojected and blended appropriately into each reconstructed polygon. A large
hemispherical geometry, assumed to be infinite, is used as the background geometry for portions of the scene that are not reconstructed, such as the sky. The
color, Ipj , of the surface on the j-th polygon is determined by alpha-blending the
texture color, Ici , projected from i-th camera, depending on the viewpoint to begenerated. Ipj is calculated using the weight for blending ωof f (i), which is thereciprocal of the penalty function, ωang (i), as follows:Ipj =ωof f (i) =∑ ωof f (i) · Ici∑,i ωof f
i

(4.1)

1
.
ωang (i) + ϵ

Here, ωang (i) denotes the distance between the i-th camera and the desired ray,which is defined as a vector from the viewpoint to the center of the polygon (seeFigure 4.4). In addition, ϵ is a tiny value used to avoid dividing by zero. Note thatour definition of the penalty ωang (i) is commonly used in selecting appropriaterays, such as in an image-based stereo-view generation approach based on lightray selection [HKY10]. The visibility of the 3D shape from the i-th camera is73Virtual
viewpoint
Polygoni-th camera

Figure 4.4. Penalty definitions of our oﬄine VDTM.
calculated using the camera’s depth maps as in a study on VDTM using the
per-pixel visibility [PDG05].
The value of ωof f (i) varies depending on the positions of the polygons, i.e.,
this process requires m-times penalty calculations for the i-th camera when there
are m polygons. As the worst case, the calculation of the weight ωof f (i) requiresO(m∗n) with n cameras. Even if the calculation process is reduced using only thek-th best cameras, as in [BBM+ 01], or using a graphics processing unit (GPU) for
parallelization, the rendering process with the binding and switching of thousandsof textures generates a large amount of overhead for graphic pipelines.
Rendering of virtual objects onto structured viewpoints
Virtual objects are rendered onto each structured real-world spherical view using
a commercial GI-rendering engine, such as 3ds Max (Autodesk, Inc.) or Mentalray (Mental Images GmbH.). Camera pose information, estimated through SfM,
is used for geometric registration between the virtual objects and the real environment. The augmented scenes are generated through IBL and GI rendering, as
in commercial movies, using complete environmental maps and dense 3D shapesof the real world to create ambient and occlusion eﬀects. It should be notedthat the illumination environment for virtual objects is manually edited from theoriginal full spherical images by setting up additional lights. The allowance of
manual editing is an important advantage of the proposed rendering framework.
Figure 4.5 shows an example of an augmented spherical image at a grid point.
The real-world scene was generated using VDTM, as described in Section 4.3.1,
and the CG models of the buildings were rendered oﬄine.

74

Figure 4.5. Example of augmented spherical scene at a grid point.

4.3.2 Online Process: Free-Viewpoint Image Generation
with Reduced Computational Cost
The online process generates planer perspective images from freely configured
viewpoints in real-time, using augmented spherical images at the structured viewpoints and simplified 3D models of both real and virtual objects.
We enabled the free-viewpoint image generation on a 2D plane using the pregenerated structured viewpoints at the grid points. VDTM is improved so as
to reduce the amount of online computations. During the online process, theviews at four neighboring grid points are projected onto 3D surfaces and blended
with bilinear weights, which were calculated based on the positions of the grid
points and the viewpoint to be generated, as illustrated in Figure 4.2. Thus,
Equation (4.1) is modified to use the bilinear weight, ωbilinear (k), as follows:
Iprj =

4∑

ωbilinear (k) · Icrk ,

(4.2)

k=1

where Iprj denotes the surface color on the j-th polygon and Icrk is the color of
the pixel in the k-th (1 ≤ k ≤ 4) neighboring view projected onto the surface.
This process does not require O(m ∗ n) weight calculations because ωbilinear (k)75does not depend on the positions of the polygons, but only on the position of the
viewpoint. It requires only four calculations for ωbilinear (k) while generating animage.
In our application, 3D models of both real and virtual objects are combined
and simplified down to 1.4% of the original numbers of polygons using a quadricbased mesh decimation method [GH97]. Although the simplification of the 3Dmodels does not reduce the cost of calculating ωbilinear (k), the rendering cost forthe 3D models in a graphics pipeline is linearly reduced, depending on the numberof polygons. The online process can be easily implemented on a GPU, such as
using OpenGL Shading Language (GLSL), with a per-pixel visibility test using
the depth maps at the structured viewpoints [PDG05].

4.4. Experiment of Application Using 2D Structure
To confirm that the application based on the proposed framework generates theappropriate scenes in a practical environment, augmented free-viewpoint scenes
were generated on a 400m×400m plane at an altitude of approximately 50 m fromthe ground. The application setting was based on the palace site of Heijo-kyo, anancient capital city of Japan, where the original palaces no longer exist. The grid
points, which were used for the positions of the structured viewpoints, were designated for every 20m × 20m area. The positions of the captured spherical images
and the configured grid are shown in Figure 4.6. The input spherical images were
captured at various altitudes, and the altitude of the grid plane was designatedas the average altitude of the captured points. The superimposed virtual buildings of Heijo Palace, presented in Figure 4.7, were approximately 1 km in lengthfrom east to west and 1.3 km from north to south, and consisted of 4,255,650
polygons. The real-world environment was reconstructed using 3,290,880 polygons from 174 spherical images. During the online process, 3D models of both
real and virtual worlds were combined and simplified into a model of 104,054
polygons (approximately 1.4% of the original 3D model), as shown in Figure 4.8.
Examples of augmented images, which were generated during the online process, are shown in Figure 4.9. The viewpoint of each image is shown in Figure 4.6.
76

200

N
W

(d)

E
S

(c)

100

0

(b)

-100

-200
-200


(a)
-100
Captured points

0100
Generated viewpoints

200 [m]Figure 4.6. Captured points of spherical images, structured viewpoints, andonline-generated viewpoints. Structured viewpoints were designated at every
20m × 20m grid point area.

The online process with a per-pixel visibility test process was implemented using
GLSL on two devices:
1. A desktop PC with an Intel Core i7-3930K (3.20GHz, 6 cores), 56.0GB of
RAM, and an NVIDIA GeForce GTX 690 (2 GB of texture memory).
2. A tablet device equipped with an Intel Core i7 3667U (2 GHz, 2 cores),
8 GB of RAM, and an Intel HD Graphics 4000 CPU-integrated graphics
processor.
77

Figure 4.7. 3D models of the virtual palaces in the Heijo-kyo capital. (Courtesyof Toppan Printing Co., Ltd.)

Figure 4.8. Combined and simplified 3D models of the real world and the virtualworld, used for the online process.

78

(a)

(b)

(c)

(d)

Figure 4.9. Free-viewpoint images from various viewpoints, which are shown in
Figure 4.6, with and without the photorealistic superimposition of virtual objects.

79

The online process was carried out on these devices faster than 600 fps and 60
fps, respectively. A notable performance improvement was demonstrated compared with the oﬄine VDTM, which performed at less than 1 fps with a CUDA
implementation on the same desktop PC used during the online process.

4.5. Discussions
4.5.1 Quality of Real-World Images on Structured Viewpoints
We compared the real-world scenes generated from a grid point where the input
spherical images were densely captured, and from a grid point generated far fromany captured points, because the density of the input spherical images may affect the quality of pre-generated real-world scenes. Figure 4.10 shows examplesof spherical real-world images that were pre-generated at the grid points. Figure 4.10(a) shows a generated viewpoint where the input images were densely
captured ((0, 0) in Figure 4.6); the viewpoint was successfully generated without artifacts owing to free-viewpoint image generation. Blurring appears in theimages in Figure 4.10(b), which is far from any captured points ((-200, -200)
in Figure 4.6), because of the errors in the reconstructed 3D surfaces and the
low-resolution textures that were projected from the distant cameras. This indicates that planning the capturing positions of the images is also important toimprove the appearance of the real-world virtualization in our framework. For 3D
reconstruction using automated vehicles, approaches to determine the next-best
view and plan eﬃcient capturing paths have been studied [Pit99,BL06]. Such approaches can be adapted for data acquisition for free-viewpoint image generation
with further investigation on the quality of the generated images.

4.5.2 Quality of Virtual-Object Superimposition of Oﬄine
Process
We conducted a small subjective evaluation of the quality of the oﬄine superimposition of virtual objects. We compared an augmented video sequence generated

80

(a) Viewpoint within densely captured points at position (0, 0) in Figure 4.6.

(b) Viewpoint far from captured points at (-200, -200) in Figure 4.6.

Figure 4.10. Spherical real-world images generated at grid points.

81

by IBL using an environmental map that was completed by our application anda conventional sequence that was created using a light parallel to the angle ofthe sun as calculated based on the date and time of the captured sequence. Note
that, to eliminate the eﬀects of free-viewpoint image generation and evaluate thesuperimposition quality by itself, the free-viewpoint image generation technique
was not used to generate the sequences used in this evaluation. Figure 4.11 showsa frame of the sequence used in the evaluation. After watching both sequences,ten examinees in their twenties or thirties, evaluated the level of naturalness (1:highly unnatural, 5: highly natural) of the synthesis of real and virtual objects.
As a result, the sequence generated using IBL and a completed environmental
map received an average score of 4.4, while the conventional sequence scored anaverage of 2.1., which are significantly diﬀerent (p < 0.01, t-test). The process
for oﬄine rendering based on IBL was eﬀective against the use of an ordinary
illumination environment, in which the strength and distribution of the skylight
were not carefully considered.

4.5.3 Density of Structured Viewpoints
Although the use of a larger interval of structured viewpoints can reduce thestorage usage during the online process, it can degrade the appearance of theresultant images. Table 4.1 shows the relation between grid sizes and amountof images. Because the proposed rendering requires huge data amount in small
grid size, it is important to investigate the trade-oﬀ between the data amountand image quality, and find useful trends to determine an appropriate grid size.
We performed a subjective experiment to investigate the eﬀects of viewpoint
density by changing the grid size in the application described in Section 4.3.
Such subjective evaluation should be preferable than objective evaluations, e.g.,comparing SNR between images generated using diﬀerent grid sizes, because we
intend to investigate the eﬀects for human vision rather than the objective image
quality.
In this experiment, 14 examinees in their twenties or thirties scored the naturalness of video sequences using the interface shown in Figure 4.12. The augmented videos shown to the examinees were generated by our application using
six diﬀerent grid sizes: 10, 20, 40, 60, 80, and 100 m. In addition, we prepared a82

(a) Using full spherical image as illumination: Scored an average of 4.4.

(b) Using only a parallel light: Scored anaverage of 2.1.

Figure 4.11. A sample frame in sequences used for an evaluation of the oﬄine
superimposition. Ten examinees evaluated the naturalness of the synthesis of realand virtual objects in five scales (1: highly unnatural, 5: highly natural).

reference video without using the structured viewpoints; i.e., virtual objects weredirectly rendered onto every frame in the video sequences, in which real-world
scenes were generated by VDTM frame-by-frame in oﬄine. It is equivalent togenerate infinite number of grid points using 0 m grid size. We prepared two
types of view direction for each grid size: downward and horizontal direction.
Sampled frames from each video are shown in Figures 4.13 and 4.14. The examinees scored each sequence on a scale of 0 to 100 using a trackbar interface,
regarding the score of reference video as 50.

83

Figure 4.15 shows average scores of the naturalness by the experiment. To
remove outliers, 20% of scores (the highest 10 % and the lowest 10 %, respectively) were ignored to calculate the average and the standard deviation, as wellas performing a multiple comparison test. The naturalness decreased with larger
grid size. We performed a Williams multiple comparison test (p < 0.05), in
which the control group was set as the score of the reference video, assuming thescore distribution is monotonically decreasing. Scores of which grid size is larger
than 60 m, were significantly unnatural comparing with those of reference video,
which was generated with grid size 0 m. This trend was same between horizontal and downward view direction. Results by grid size 10 or 20 m surprisingly
demonstrates better quality than the reference video; it is thought that the flicker
appeared on the reference video, because the blending weight among real-world
textures determined by VDTM frequently changed. 20 m grid size, which wasused for our experiment described in Section 4.4, showed the performance similar
to using smaller (i.e., 10 m) grid as well as using the reference video, under our
experimental environment. Although this comparison result cannot be directly
used for determining the best grid size, which is a trade-oﬀ between the quality and the number of pre-generated images, it intends that the larger grid size(i.e., larger than 60 m in this experiment) causes significant negative eﬀects touser experiences. Figure 4.16 indicates the negative eﬀects using a larger grid,
Table 4.1. Amount of images required to prepare a 400m × 400m grid structure.
The structured views are assumed to be stored as 8-bit 4-channel (RGB anddepth) cube-maps of 1024 × 1024 pixels.
Grid size
10m
20m
40m
80m
100m
200m

# of grid points Data amount (not compressed)
1681
40.3 GB441
10.6 GB121
2.9 GB36
864 MB25600 MB
9216 MB84

Figure 4.12. Interface used for evaluation using video sequence with a change ofgrid size. Examinees evaluated the naturalness of the sequence in a scale of 0 to
100.

i.e., the low resolution of the entire scene, and the skewing/blurring of the virtual building posts. When a larger grid was used, the resolution of the resultant
images was lower owing to the textures projected from the distant viewpoints.
The skewing and blurring posts of the buildings were due to the simplified 3Dmodels, in which some of the vertices were removed, and the eﬀects became more
pronounced in the larger grid. In addition, a significantly large grid occasionally generated clearly incorrect textures because some surfaces of the complex 3D
shapes were not visible from all four neighboring grid points.
The trend of the relation between the quality and grid sizes basically varies
depending on the environment: the complexity of the model and the distance to85

(a) Reference video (0 m grid size).

(b) Grid size: 10 m.

(c) Grid size: 20 m.

(d) Grid size: 40 m.

(e) Grid size: 60 m.

(f) Grid size: 80 m.

(g) Grid size: 100 m.

Figure 4.13. Augmented views from the same viewpoint with a change in gridsize (horizontal view direction).

86

(a) Reference video (0 m grid size).

(b) Grid size: 10 m.

(c) Grid size: 20 m.

(d) Grid size: 40 m.

(e) Grid size: 60 m.

(f) Grid size: 80 m.

(g) Grid size: 100 m.

Figure 4.14. Augmented views from the same viewpoint with a change in gridsize (downward view direction).

87

100

Standard Deviation w/o
20% outlier

Naturalness score (0-100)

90
80
70
60

Score of reference video (50)

50
40
30
20

Significantly low scorescompared to reference video

10
0

10

20

40

60

Grid size [m]Horizontal view

80

100

Downward view

Figure 4.15. Naturalness scores of augmented video sequences with a change ingrid size.

the objects. Surfaces on the 3D shapes that are not visible from all neighboring
grid points, can be calculated preliminarily; and further, they occupy a large
area on the free-viewpoint image when the distance to the surface is small. As a
trade-oﬀ between a reduction in the number of images and the appearance of thegenerated images, a promising solution is to change the density of the structured
viewpoints, depending on the complexity of the model and the distance to theobjects, using quad-trees or similar structures.

4.5.4 Limitations
The proposed framework has some limitations. First, the number of pre-generated
images can be large because of the oﬄine rendering of virtual objects for multiviewpoint images. This issue is common to free-viewpoint image generation,
particularly for IBR. To address this, multi-viewpoint image compression ap88

Grid size: 20 m

Grid size: 20 m

Grid size: 60 m

Grid size: 100 m

Figure 4.16. Negative eﬀects using a larger grid sizes: Closeups from Figures 4.13
and 4.14.

proaches [MRG03, SKC03, LLW04] for IBR can be employed in the proposed
framework. Second, although dynamic objects can be superimposed by thisframework if the objects are initially rendered, such use is impractical. The
third limitation is that this framework do not accommodate real-time illumination change in the illumination environment in a straight-forward manner. To
overcome this, certain AR photometric registration methods can be employed,such as image synthesis from the appearances rendered under various illumination environments [SHK+ 05].
The proposed rendering framework can be potentially employed for MR-world
exploration applications using a static environment, such as Google SketchUp. Although there are limitations for use under a dynamic environment, the proposed
rendering framework can be a promising approach for a variety of applications,89

such as a landscape simulation. The historical application introduced in Section 4.3 is a typical example.

4.6. Summary
This chapter described a novel framework for the photorealistic superimpositionof static virtual objects into the setting of a real-world virtualization. Oﬄine
rendering and a free-viewpoint image generation were combined to preserve thequality of oﬄine rendering in a free-viewpoint MR environment, which provides
the user with freely configurable viewpoints. The computational cost of the online
process is highly reduced through the pre-generation of structured viewpoints andthe pre-rendering of virtual objects at these viewpoints.
We introduced a practical implementation for fly-through application basedon the proposed framework, which inputs spherical images captured through themethod described in Section 3.2. The application provides a free configurationof the user’s viewpoints on a 2D plane through a VDTM-based free-viewpoint
generation technique using pre-generated viewpoints at the grid points. In theexperiments conducted at a historical site, our implementation demonstrated ahigh performance. These experiments also illustrated the eﬀects of rendering using IBL in an oﬄine process, which improves the appearance of virtual objects,
as well as some problems that should be examined further, such as the determination of the most eﬀective grid sizes. Increasing the dimension easily leads toan enormous number of images. This issue should be considered when developing
large applications. In Section 5.5, two applications based on this implementation
are described.

90

Chapter 5
Photorealistic MR Applications
Based on Proposed Framework
5.1. Overview
The proposed framework can accommodate various specific imaging and rendering methods. Using the proposed imaging and rendering frameworks either
fully or partly, many kinds of MR applications can be developed as discussed in
Section 2.3. This chapter describes following four applications with applicationspecific discussions and evaluations:
1. HDR immersive panorama
Full spherical aerial HDR images can be employed in immersive panorama
applications by applying perspective projection of sphere-mapped sphericalimages. However, we should appropriately compress the dynamic range
when displaying HDR images for ordinary display devices. We developed apractical immersive panorama application and investigated diﬀerent display
methods using dynamic range compression.
2. Augmented immersive panorama
By directly superimposing virtual objects into full spherical aerial images
without the use of a real-world virtualization, the augmented spherical images can be used for augmented immersive panorama applications. This application does not provide a change in viewpoint other than at the location91

where the images were initially captured. A historical entertainment application has been developed based on an augmented immersive panorama foruse in a public demonstration.
3. Real-time AR
With some modifications, the proposed rendering framework can be employed for (video see-through) real-time AR applications of real-world scenes
captured in real-time, instead of a real-world virtualization. A simple realtime AR application was developed based on 1D structured viewpoints andan image-based free-viewpoint image generation method.
4. MR-world interactive exploration
When the proposed framework is fully employed, the produced application
realizes the capability of interactive MR-world exploration. An exampleof the implementation described in Section 4.3, is easily extended to othersituations with a small modification. We developed two applications: anextension of the implementation to use 3D cubic structure, and a simple
application for the virtual walk-through using 1D structure based on an
ordinary spherical imaging using an OMS during walking on the ground.

5.2. HDR Immersive Panorama
The full spherical aerial HDR images generated in Section 3.3 can be used for
immersive panorama applications, which allow users to freely look around thescene from the location where the images were captured. This section describes
an immersive panorama application based on an HMD and a head tracker using the spherical HDR images. To show HDR images on ordinary displays, theradiance has to be compressed through a process referred to as tone-mapping.
Numerous tone-mapping methods have been developed [RHD+ 10] to search forsuitable approaches that successfully compress the radiance of an image whilesaving its specific textures and without generating unnatural artifacts. In mosttone-mapping methods, HDR images are converted using parameters determinedfrom the whole image, such as the maximum and logarithmic mean intensities.
However, immersive panorama applications generally crop and convert a spheri92

Figure 5.1. Immersive system used in the experiments.

cal image into a perspective image such that the entire image is not displayed all
at once. There may be more suitable tone-mapping methods than those used forusual perspective images. It is therefore important to investigate tone-mapping
approaches suitable for such applications to improve the fidelity of scenes withlarge dynamic ranges.

5.2.1 Immersive Panorama System with HMD and Head
Tracker
We conducted a subjective evaluation of four display methods using an immersive
system comprising an HMD and a head tracking device [OYTY98]. Using thissystem, a spherical image is converted into a perspective image in real-time basedon the user’s viewing direction, as detected by an electro-magnetic sensor (Fastrak, Polhemus) mounted on the HMD (HMZ-T1, Sony Corp.). Figure 5.1 shows
the system used in our experiments. From the location where the spherical image

93

LDR intensity
255

0

LDR intensity
255

0

HDR intensity(a) LDR-like representation.

/'5LQWHQVLW\JURXQGHDR intensity(b) Reinhard’s tone-mapping [RSSF02].

LDR intensity
255

VN\

Depending onview direction

0+'5LQWHQVLW\(c) Region-wise tone-mapping.

(d) View
mapping.

HDR intensity
direction

dependent

tone-

Figure 5.2. Illustrative tone curves for four display methods.
was captured, users can freely change their viewing direction by changing theirhead pose; i.e., the viewing direction corresponds to the user’s head direction in
real-time. We used spherical aerial HDR images using taken from OMSs using
the method described in Section 3.3 for an immersive panorama system.
The four display methods outlined below were evaluated. Figure 5.2 illustrates
an example of the tone curves for each of these methods.

94

1. LDR-like representation
This is a luminance transformation method that uses a linear tone curve,
such as that shown in Figure 5.2(a), and generates an LDR image with thesame appearance as that generated through traditional LDR imaging.
2. Reinhard’s tone-mapping [RSSF02]
This uses the maximum and log-mean intensities to tone-map the HDR
image. It is one of the most popular tone-mapping methods.
3. Region-wise tone-mapping
Spherical aerial images are divided into bright sky and dark ground regions. Using the method of [KOKY12], these regions are then tone-mapped
independently using diﬀerent parameters.
4. View-direction dependent tone-mapping
An input HDR image is tone-mapped based on the intensity of the field ofview of a perspective image converted using an immersive system based on
the user’s viewing direction. The intensity of a position in the scene varies
according to this viewing direction.

5.2.2 Display Methods for Spherical Aerial HDR Images
Let us now look at the implementation of each display method and the results ofa subjective review.
LDR-like representation
A linear intensity transformation is used to generate an image that looks the sameas any image generated through traditional LDR imaging. A tone curve can be
defined as{
1.0 (Ih > 1.0)
Il =
(5.1)
2v Ih (otherwise),
where Ih is the intensity in an HDR image, Il denotes the intensity of an LDR
image normalized to the range [0, 1], and v is the exposure value of the generated
LDR image. We determined v to reduce overexposed and underexposed pixels
95

Figure 5.3. Image generated through LDR-like representation.

in the generated LDR image. For this experiment, underexposure is defined as
1
Il < 16, and overexposure as Il = 1.0. Note that, as shown in Figure 5.3, many
overexposed and underexposed pixels remain in the resulting image, which wasconverted from an HDR image captured as described in Section 3.3.
Reinhard’s tone-mapping [RSSF02]
The tone-mapping method proposed by Reinhard et al. [RSSF02] is composed oftwo steps. In the first step, the intensity Ih of an HDR image is transformed into
LDR intensity Il by a function involving the log-mean I¯h :αIs = ¯ Ih ,
(5.2)
Ih
(
)
Is 1 + I 2Is
max
Il =
.
(5.3)
1 + Is
Note that Is is a scaled intensity using scale factor α, and Imax denotes the maximum intensity scaled using Equation (5.2). The second step, so-called “dodgingand-burning” is applied to areas of low and high intensity to improve their visibility (detailed in [RSSF02]). Parameter y is automatically estimated by [Rei02].
Figure 5.4 shows the result of tone-mapping the same frame used in Figure 5.3.
96

Figure 5.4. Result using Reinhard’s tone-mapping method [RSSF02].

Region-wise tone-mapping
Dividing images into sky and ground regions, we applied the approach proposedby Kitaura et al. [KOKY12] to tone-map these regions using separate parameters.
To fully automate this process, we improved it for spherical aerial images. Specifically, we used a priori knowledge of aerial images to support automatic GrabCut
segmentation [RKB04] in place of a user input of the seed pixels. We assumed
that the camera pose was estimated by SfM in the same fashion as described in
Section 3.3.5, and that the captured HDR image was aligned to fit the horizonin the scene to the horizontal scan line of the image. Under these assumptions,the seed pixels for the dark (ground) region can be taken from the lower-half of
the image, as shown in Figure 5.5. The distribution of seed pixels for the bright(sky) region depends on the landscape of the scene. We used the upper-third of
the image as seed pixels for the bright region, resulting in a successful separationof the sky from the ground, as shown in Figure 5.6.
Region-wise tone-mapped images, such as that shown in Figure 5.7, are thenproduced by applying Reinhard’s tone-mapping [RSSF02] to each region independently.

97

Seeds ofsky region

Seeds ofground region

Figure 5.5. Seed pixels for GrabCut [RKB04]: y denotes the vertical componentof the image coordinates; and ymax is the height of the image.

Figure 5.6. Regions determined using GrabCut [RKB04].

98

Figure 5.7. Result of region-wise tone-mapping.

View-direction dependent tone-mapping
For immersive panorama applications in particular, it is worth investigating
tone-mapping methods whose tone curves vary according to the user’s viewing
direction. We applied a new approach similar to the tone-mapping for HDR
video [KUWS03]. This approach consists of two processes:
1. Oﬄine process: Pre-calculation of log-mean and maximum intensities.
2. Online process: Tone-mapping using these pre-calculated parameters.
The log-mean and maximum intensities were calculated from planer perspective images generated for each viewing direction (in this study, we calculated
the parameters for every one degree of latitude and longitude). Equations (5.2)
and (5.3) were then applied for tone-mapping in real-time using the calculatedparameters and α = 0.18, which is the default value used in Reinhard’s tonemapping [RSSF02]. Note that although this experiment used an oﬄine process,it is possible to implement a fully real-time process by leveraging the GPU computation, as in other real-time tone-mapping methods [GWWH03]. The results,
shown in Figure 5.8, reveal how the intensity at a given position in a scene varies
according to the user’s viewing direction.
99

(R,G,B) = (70,71,94)(R,G,B) = (27,27,36)

Figure 5.8. Examples of view-direction dependent tone-mapping: The intensityof the same position in the scene varies according to the viewing direction.

5.2.3 Subjective Evaluation Using Immersive System
To investigate the features of the four display methods described above for spherical aerial HDR images, we conducted a subjective evaluation using an immersive
system. Examinees used an HMD to view scenes generated using these four methods, and responded to each of the following questions with a rating of one (worst)
to five (best):
Q1: Were the scene and textures easily recognizable? (Visibility)
Q2: Did the tone, brightness, and appearance of the textures appear to be natural? (Naturalness)
Q3: Did you get the feeling that you were actually flying through the sky? (Immersion)
Q4: Were you satisfied with the display method as a whole? (Total)
Note that these questions were originally given in Japanese. The examinees were
ten people in their twenties or thirties; some of them were familiar with the VR
and image processing techniques. The examinees were also allowed to comment
freely after answering the questionnaire. The order in which the four display
methods were presented was randomized, and the examinees were allowed toswitch the display mode among the representations as many times as they wished.
100

In this experiment, we stated following two hypotheses:
1. LDR-like tone-mapping, which does not represent high dynamic range intensity, is evaluated worth than other methods. The representable dynamic
range directly aﬀects the visibility of the scene (Q1); while the degree ofsatisfaction (Q4) should demonstrate the similar trends. It intends that the
combination of HDR imaging and appropriate representation are eﬀective
for immersive panorama applications.
2. Region-wise and view-direction dependent tone-mapping are scored higher
than to tone-map a whole spherical image by Reinhard’s tone-mapping,because of its advantages in the visibility.
The results of the evaluation are shown in Figure 5.9. This figure also indicates the pairs that had a significant diﬀerence (p < 0.05 and p < 0.01), as calculated using the Tukey’s test. In total, Reinhard’s method and the view-direction
dependent method had higher ratings than the LDR-like method. Region-wisetone-mapping was significantly lower than the other three, especially with regard
to naturalness.

5.2.4 Discussions
Eﬀects of HDR imaging for immersive panorama
The results demonstrate that the first hypothesis is correct with some exceptions.
For Q4 in Section 5.2.3, with the exception of region-wise tone-mapping, nonlinear
tone-mapping methods had significantly higher ratings than LDR-like representation methods (as shown in Figure 5.9). We can therefore confirm that HDR
imaging and appropriate display methods are superior for immersive panoramicapplications, as well as for traditional uses of HDR images such as conventional
photography and IBL. This superiority is most noticeable in the reduction ofoverexposed and underexposed pixels. Furthermore, since immersion is one of
the chief goals of virtual reality [Moe97], it is noteworthy that HDR imaging does
not decrease the immersion despite the use of a nonlinear intensity transformationin the tone-mapping, which can be regarded as unnatural.

101











00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

9VE \/'5 NH
5H QKDUG

1D XUD QHVV,PPHUV RQS 
5HJ RQ
9 HZ G UHF RQ

7R D
6 DQGDUG GHY D RQ

Figure 5.9. Results of subjective experiment involving the four display methods.

The best HDR image display method for immersive panorama
This experiment partially refuted our second hypothesis. Although region-wisetone-mapping had a higher visibility rating (along with other HDR tone-mapping
methods), significantly lower ratings were obtained for the other questions. Free
comments from the examinees suggested that the most critical factor in determining the naturalness was the darkness of the sky relative to the intensity of
the ground.
Contrary to the expectation, there are no significant diﬀerences between Reinhard’s tone-mapping method and the view-direction dependent tone-mapping.
Free commentary from the examinees indicated that the view-direction dependent tone-mapping produced unnatural intensity variations as the viewing direction changed in real-time. This trend would change under diﬀerent applications
102and target environments; e.g., if users stay watching groundward scenes with little change of the luminance, the artifacts by view-direction dependent one does
not occur. It indicates that the parameters related to human factors, such as
acceptable rates of intensity variation, should be investigated to guide further
development. If it can overcome this perceived deficiency, or can find suitable applications, view-direction dependent tone-mapping may become a more populardisplay method.

5.3. Augmented Immersive Panorama
By directly rendering virtual objects into spherical images, an augmented immersive panorama can be developed. Such rendering is considered for applications
using the proposed imaging framework, as well as the oﬄine rendering process inthe proposed rendering framework without an explicit real-world virtualization.
To display the augmented spherical images, we can basically employ the same
hardware configuration as in an immersive panorama without the superimposition of virtual objects, which provides a change in the user’s viewing direction.

5.3.1 Augmented Immersive Panorama System Using Full
Spherical Aerial Image Sequence
We developed a virtual historical tourism application based on the augmented
immersive panorama technique using the full spherical aerial image sequence described in Section 3.2.6. Unlike the free-viewpoint rendering described in Chapter 4, this application does not provide a free configuration of the user’s viewpoint,
but enables the users to move along the captured path of the spherical image sequence. The virtual objects of the old palaces, which are from the same 3D modelas that used in Section 4.3, were rendered using IBL without generating structured viewpoints. The resultant augmented spherical image sequence consistsof 1,900 frames, whose real-world scenes are completed by a method describedin Section 3.2, and Figure 5.10 shows the frames sampled from this sequence.
Note that the selected frames shown in Figure 5.10 correspond to those shown in
Section 3.2.6.

103

(a) First frame.

(b) 200th frame.

(c) 400th frame.

(d) 600th frame.

(e) 800th frame.

(f) 1,000th frame.

(g) 1,200th frame.

(h) 1,400th frame.

Figure 5.10. Augmented spherical image sequence for augmented immersive
panorama application.

104

(a) TV monitor system.

(b) HMD system.

Figure 5.11. Appearance of augmented immersive panorama systems.

5.3.2 Public Experiment
In 2010, an experimental public demonstration using the augmented immersive
panorama application was conducted during the commemorative event for the
1,300th anniversary of Nara Heijo-kyo capital for a two-week period, and over
one-thousand people experienced the application. We developed two display systems using an HMD and a TV monitor, which are shown in Figure 5.11. The configuration of the HMD system is similar to that used in the immersive panorama
system described in Section 5.2, which changes the viewing direction of the augmented scene using the user’s head pose measured using a gyroscope mounted
on the HMD. In the system using a TV monitor, the viewing direction could bechanged using a joystick. The specific configuration of the demonstration system
is shown in Table 5.1. Note that in the presented system, the user can switch
sequences with and without virtual objects to achieve a better user experience.
We obtained a large amount of feedback from the users during the demonstration, and most of them were positive comments. We also heard many suggestions
regarding the demand for annotations of the virtual buildings and animating ofvirtual objects. From these user suggestions, in addition to realizing a photorealistic superimposition, a high-quality user interface is also required for thesevirtual historical tourism systems.

105

5.4. Real-Time AR
We developed a simple real-time AR application using mobile device based on the
proposed rendering framework. In our implementation, a user explores AR scenes
from viewpoints on initially designated linear paths. Based on the flow illustrated
in Figure 5.12, the appearances of virtual objects and depth maps at structured
viewpoints were generated oﬄine. During the online process, we employed amorphing technique [MD96], which is an appearance-based free-viewpoint image
generation method.

5.4.1 Oﬄine Process: Rendering of Virtual Objects at
Structured Viewpoints
During the oﬄine process, structured viewpoints are designated on a line every
couple of meters. At each viewpoint, the appearances of the virtual objects
and depth maps are rendered, as shown in Figure 5.13. A spherical image isacquired from a location near the structured viewpoints; the image is employed as
the illumination for the virtual objects with manual adjustments (e.g., manuallydesignating additional light sources).

Table 5.1. System configuration for augmented immersive panorama.
HardwareModel
Specifications
HMD
systemPC
HMD

TV
monitorsystemPC
TV
Monitor

CPU: Intel Core2 Duo T7300
Graphic: Intel HD Graphics
Resolution: 640×480
Vertical FOV: 32◦
3 DoF gyroscope
FaithCPU: Intel Core i7 X920
MTX2 i7720 Graphic: GeForce GTX 285M
Sharp AQUOS Display size: 65 inches
LC-65RX1W Resolution: 1920×1080
DELL
Vostro 1400
iWear
VR920

106

Virtual objects

Offlineprocess

Onlineprocess

Real-world

Renderingof virtual objects on
structured viewpoints

Free-viewpoint
image generation

Augmented scenes

Figure 5.12. Modified rendering framework for real-time AR.

Figure 5.13. Panoramic appearance and depth map of virtual objects at a structured viewpoint.

1075.4.2 Online Process: Free-Viewpoint Image Generation
The online process superimposes a virtual object into a real scene by morphing [MD96] the two structured viewpoints nearest to the position of the camera
capturing the scene. Pre-generated depth maps are used for the 3D shape of thescene, and the appearance of the virtual object is morphed with respect to eachpixel of the depth map. This process does not depend on the number of polygons,
but rather depends on the number of pixels in the depth map. The morphed textures are blended using a simple weight, i.e., the reciprocal ratio of the distances
to the two neighboring viewpoints. The structured viewpoints enable an easy
selection of the neighboring views and O(1) calculation of the blending weight.
This process is similar to other appearance-based free-viewpoint image generation
methods simplified for the use of only two views, such as unstructured lumigraphrendering [BBM+ 01].

5.4.3 Mobile Device Experiment
To investigate the quality of generated scenes and the computational costs of theonline process, a basic AR application for a historic site was developed. The
application superimposed into a real scene a virtual seven-stair tower, which nolonger exists, of the famous Todaiji Temple in Japan. This model was createdwith reference to the miniature model of Todaiji at the time of its foundation,and is exhibited in the Hall of the Great Buddha, with architectural validation by
Yumiko Fukuda, Hiroshima Institute of Technology, and patina expression technology validation by Takeaki Nakajima, Hiroshima City University. The structured viewpoints and captured path of the real scene are shown in Figure 5.14.
Eleven structured viewpoints were designated at every 1m, and the virtual object
was rendered at every viewpoint using oﬄine GI-rendering software. The virtual
object appearance and depth maps were generated as cube maps, for which a total of 66 MB of storage was required. In this experiment, to eliminate the eﬀects
of errors in the geometric registration of virtual objects during the online process, the real scenes were preliminarily captured using an OMS, and the captured
position and orientation were preliminarily estimated using SfM [Wu13]. In thisexperiment, a perspective-image sequence generated from a spherical video was

108

-1

0

1

2

3

4

5

6

7

8

9

10

11
1
0

(a)

(b)
Captured sequence for AR
Generated viewpoints

Structured viewpoints-1[m]

Figure 5.14. Top view of structured viewpoints where virtual objects are renderedoﬄine, capturing path of real-world spherical images for AR application, andonline-generated viewpoints.

used as the input for the online process. Figure 5.15 shows examples of augmented
scenes, as well as the real images used as the background scene. The online image
generation process was performed at 30 fps with a GLSL implementation on atablet device equipped with an Intel Core i7 3667U (2 GHz, 2 cores), 8 GB of
RAM, and an Intel HD Graphics 4000 CPU-integrated graphics processor.

5.4.4 Discussions
Image quality from online process
To confirm the quality of the images generated using the online process, we compared the AR scenes with an oﬄine-rendered scene. Figure 5.16(a) shows an
online-augmented scene, and Figure 5.16(b) shows the scene generated throughoﬄine rendering using the same viewpoint and illumination used in the scene
shown in Figure 5.16(a). The minor diﬀerence between the two images, as shownin Figure 5.16(c), indicates that our online process successfully generates a similar
appearance as the oﬄine-rendered images. Because oﬄine rendering includes special boundary processes, such as anti-aliasing, errors mainly occur in the boundaries between the real and virtual scenes. These processes can be employed forthe proposed framework, along with some AR studies involving the registrationof image quality between real and virtual objects [FBS06].

109

(a)

(b)

Figure 5.15. Real-time AR based on proposed rendering framework. Scenes withand without superimposition.

Computational cost
We compared the performance of the proposed framework-based AR application
with that of an ordinary rendering method. Our AR application superimposes a
virtual model of 1,131,736 polygons, while our implementation does not depend
at all on the number of polygons. Table 5.2 describes the performance of the
proposed application and the rendering of a textured virtual model, where thelatter method does not employ any special lighting eﬀects. The devices used
for the performance measurement were the same as those used in the experimentdescribed in Section 4.4. Our application performance was 10- to 100-times fasterthan the simple rendering of the textured virtual model.
110

(a) Generated by our online process.

(b) Produced through oﬄine rendering.

(c) Diﬀerence between scenes in (a) and (b).

Figure 5.16. Comparison of scenes generated through online free-viewpoint image
generation and oﬄine rendering. The online process generates a similar appearance as the oﬄine-rendered image. Errors mainly occur in the boundaries between
the real and virtual scenes.

111

5.5. MR-World Interactive Exploration
5.5.1 Fly-Through Application using 3D Structure
The application described in Section 4.3 can be extended for using 3D structure
by straightforward approach, generating multiple grids in multiple altitude. It
should be noted that, this approach requires a larger amount of images than using
2D structure.
Implementation
We developed a fly-through application using 3D structure in the same target
environment as described in Section 4.3. Augmented free-viewpoint scenes weregenerated on three grids in multiple altitude, each of these covers 400m × 400m.
In addition to the grid used in the 2D application, whose altitude was set inapproximately 50 m from the ground, we also generated grids in 10 and 30 m
altitude by the same manner as the oﬄine process of 2D application. Fromthe experiment in Section 4.5.3, there are not a significant degradation of thefree-viewpoint images in 40 m grid, the grid points were designated for every
40m × 40m area, which is larger than that in the implementation in Section 4.3.
The horizontal and vertical intervals of structured viewpoints are diﬀerent (40m and 20 m). It is because the target environment includes some two-storiedbuildings; it clearly occurs the missing textures for a large vertical interval ofthe viewpoints. Note that diﬀerences of interval do not increase the complexity

Table 5.2. Frame rates (fps) of the proposed rendering in our AR application andsimple rendering of the textured virtual model.
Proposed

Textured Model

480

5.1

30

2.0

Desktop PC w/
NVIDIA GeForce GTX 690
Tablet PC w/
Intel HD Graphics

112

of the rendering process. In the application using 3D structure, online VDTM
blends the textures at eight neighboring structured viewpoints, although the 2D
application uses four textures. The eight neighboring structured viewpoints areplaced at each vertex of the cube including the user’s viewpoint, and they realize
O(1) calculations for the blending weight.
Results and discussions
Examples of augmented images, which were generated during the online process,are shown in Figure 5.17. This application was implemented in the same desktop PC used in the experiment described in Section 4.4, and achieved similar
performance as the 2D application (faster than 600 fps).
Augmented images in low altitude include a lot of visual artifacts. It is thought
to be the cause of two problems: 1) The user’s viewpoint is relatively close in low
altitude. The errors of the shape due to 3D reconstruction and polygon reduction
largely aﬀect the appearance of the image, and 2) some parts of complex 3D
shapes were not visible from all neighboring structured viewpoints. Artifacts dueto the problem 1) are shown in the appearance of occlusion boundary as shownin Figure 5.18. The errors of shape cause blur on the occlusion boundary fromscenes viewed from high altitude. The artifacts become larger in low altitude;i.e., clearly incorrect textures were generated. These negative visual eﬀects canbe reduced by designating narrow grid; however, it leads to use of an enormous
number of images.

5.5.2 Walk-Through Application Using Spherical Images
Captured on Ground, 1D Structure, and MorphingThis section describes a walk-through application based on the proposed rendering approach. The application is summarized as follows:• Intended use: An MR walk-through application superimposing 3D models
of buildings.
• Input: Spherical LDR images captured from the ground during moving.

113

(a) Horizontal location as in Figure 4.9(a).

(b) Horizontal location as in Figure 4.9(b).

(c) Horizontal location as in Figure 4.9(c).

(d) Horizontal location as in Figure 4.9(d).

Figure 5.17. Free-viewpoint images from various altitudes. The altitude is approximately 40 m (left figure), 20 m (right figure) from the ground.

114

(a) A complex shape viewed from high altitude (approximately 50 m).

(b) A complex shape viewed from low altitude (approximately 15 m).

Figure 5.18. Artifacts on occluding boundary between virtual objects and virtualized real objects.

• Dimension of structured viewpoint: One dimensional structure (on aline.)• Free-viewpoint image generation: Morphing technique [MD96] using
depth maps from structured viewpoints. Because the target scene is expected to be close to the user’s viewpoint, as discussed in Section 2.2.1,
appearance-based hybrid rendering approaches should be appropriate forour application. Morphing is a typical appearance-based rendering approach.
In our application, real-world views are first generated at equally-spaced points
on a line using a morphing technique. Virtual objects are then photorealisticallyrendered for every structured viewpoint. It should be noted that, the virtual115Figure 5.19. Reconstructed real-world 3D models and camera pose of the OMS
used.

models and the real environment are not semantically related; there are no any
historical meanings of the contents.
Implementation
Camera pose and real-world dense 3D models are reconstructed from input spherical images, which are captured by an OMS from the ground during walking, by
VisualSFM [Wu13] and CMPMVS [JP11], as shown in Figure 5.19. Depth map
from each camera is generated from the reconstructed models. Real-scenes fromthe structured viewpoints are generated by per-pixel morphing [MD96] using thedepth map. The images are morphed with respect to each pixel of the depth
map. In our implementation, 56 structured viewpoints were designated on a line
at every 1m as shown in Figure 5.20. At each viewpoint, the process renders thevirtual objects as same manner as described in Section 4.3 (see Figure 5.21), as
well as the depth maps from the structured viewpoints. Note that, the application inputs LDR images; additional light sources were manually designated forthe rendering.
The online process generates the user’s view from the images at the two nearest structured viewpoints from the user’s viewpoint by the morphing, as similar
fashion as the oﬄine process. This process does not depend on the number ofpolygons, but rather depends on the number of pixels on the depth map. The
116

0

10

20

30

40

50
4

(a)

0
(b)
Captured position by OMS
Generated viewpoints

Structured viewpoints

(c)

-4
[m]

Figure 5.20. Top view of structured viewpoints where MR-world is rendered,capturing path of real-world spherical images, and online-generated viewpoints.

morphed textures are blended using a simple weight, i.e., the reciprocal ratioof the distances to the two neighboring viewpoints. The structured viewpoints
enable an easy selection of the neighboring views and O(1) calculation of theblending weight.
Results and discussions
Examples of augmented images, which were generated during the online process,are shown in Figure 5.22. The augmented images were successfully displayed,
including occlusion of virtual objects from real hedges. This online process was
performed at 30 fps with a GLSL implementation on the same desktop PC usedin the experiment described in Section 4.4.
As shown in Figure 5.22(a), the real-world scenes in this application includes
the appearance of pedestrians, which can be a cause of reducing the immersive
values. There are some methods to remove pedestrian figures from sphericalimages captured in a fixed point (e.g., [AHK+ 10]) as discussed in Section 2.1.3.
However, it is still a challenging problem for spherical image sequences captured
during motion, where this type of image sequences was used in this application
as well as Google Street View. A study by Flores et al. [FB10] tackles thisproblem using Google Street View images captured in environments consisting ofplanar objects; it can be a future direction to generalize the target scenes of suchapproaches to arbitrary geometry.
If this application is used around the actual location of the target environ117

(a) Real-world view at a structured viewpoint.

(b) MR-world view at a structured viewpoint.

Figure 5.21. Panoramic appearance of MR-world at a structured viewpoint for
virtual walk-through. The virtual models and the real environment are not semantically related; they are non-historical contents.

118

(a)

(b)

(c)

Figure 5.22. Virtual walk-through using 1D structure, whose viewpoints areillustrated in Figure 5.20, with and without superimposition of virtual objects.

119ment, it can provide a similar experience to a photorealistic AR. This type ofAR system, which displays pre-captured images as the background images for
AR as a substitute of the real-world images captured in real-time, is referred toas Indirect AR [WTA11, AOSY13]; however, current Indirect AR systems do notprovide changes in user viewpoints. The proposed framework in principle extends
Indirect AR to allow for a change in user viewpoint, and oﬀers a novel type ofAR application.

5.6. Summary
This chapter has described four applications that fully or partially use the proposed imaging and rendering frameworks: 1) an HDR immersive panorama, 2)
augmented-immersive panorama, 3) real-time AR, and 4) MR-world exploration.
HDR immersive panorama applications can be developed using full spherical HDR
images acquired by our imaging framework. We investigated tone-mapping methods of full spherical HDR images using a practical immersive panorama system,and the results show that a tone-mapping method that depends on the user’sviewing direction as well as an ordinary tone-mapping yields the best resultswith respect to critical human factors. A virtual historical tourism application
based on an augmented immersive panorama system was demonstrated for over
one-thousand people through a public demonstration, and we found that a highquality user interface, in addition to a photorealistic superimposition of virtual
objects, is demanded. We also showed that real-time AR applications can be developed based on the proposed rendering framework. Augmented scenes were successfully generated with high performance using appearance-based free-viewpoint
image generation, where the application performance was 10- to 100-times fasterthan the simple model-based rendering of virtual objects. Based on the implementation described in 4.3, we developed an application providing a change inviewpoint in a 3D space as a straightforward extension of the 2D application, as
well as a virtual walk-through application using spherical images captured during
walking on the ground.

120

Chapter 6
Conclusions
This thesis has described a novel framework for acquiring real-world scenes, as wellas the rendering both of the real and virtual worlds for photorealistic MR-world
exploration. To realize a photorealistic and immersive MR scene representation,
Chapter 2 illustrated that full spherical HDR images should be acquired throughan eﬀective and accurate acquisition of real-world images, and by using rendering
methods that can be utilized oﬄine with high-quality illumination through timeconsuming methods and a high number of manual operations.
For imaging issues, in particular, it has been diﬃcult to acquire full spherical
HDR images from the sky that can be used for virtual globe applications providing a bird’s-eye view; therefore, two approaches for capturing full sphericalaerial HDR images were described in Chapter 3. The first approach only requires
ordinary spherical aerial imaging equipment, which consists of an aerial vehicle
and an OMS mounted on the bottom. Spherical images captured by this equipment include missing areas, and therefore must be completed using a statisticalmodel of the sky luminance and radiance. The second approach requires very
special hardware configurations, that is, two OMSs mounted on the top and bottom of an aerial vehicle. The two OMSs cover a whole viewing direction, andfull spherical images are therefore generated by mutually completing the missing
areas appearing in the spherical images captured by each OMS. In addition, thetwo OMSs capture their own multi-exposure images for generating HDR images.
Our experiments demonstrated that the approach using two OMSs generated anotably higher intensity compared with a spherical image completion using a sky
121model.
To achieve a photorealistic superimposition of virtual objects onto a real-world
virtualization, we propose a novel framework for the photorealistic superimposition of static virtual objects into a virtualized real-world setting, as described in
Chapter 4. Oﬄine rendering and a free-viewpoint image generation are combined
to preserve the high quality of oﬄine rendering. The computational cost of theonline process is highly reduced using pre-generated structured viewpoints andpre-rendering virtual objects onto these viewpoints. We also introduced a practical implementation based on the proposed rendering framework, which inputs
spherical images captured using the proposed imaging approach. This framework provides a free configuration of the user’s viewpoint on a 2D plane througha VDTM-based free-viewpoint generation technique using pre-generated viewpoints at the grid points. Our implementation demonstrated a high performance
during an experiment conducted at a historical site.
Chapter 5 described four categories of applications that fully or partially
use the proposed imaging and rendering frameworks: 1) an HDR immersive
panorama, 2) augmented immersive panorama, 3) real-time AR, and 4) MR-world
exploration. We investigated tone-mapping methods for a practical immersive
panorama system using full spherical HDR images acquired using our imaging
approach, and the results indicate that a tone-mapping method that depends on
the user’s viewing direction, as well as an ordinary tone-mapping are promising. A virtual historical tourism application based on an augmented immersive
panorama system was demonstrated for over one-thousand people through a public demonstration, and we found that the participants demand a high-quality userinterface in addition to a photorealistic superimposition of virtual objects. We
also showed that real-time AR applications can be developed based on the proposed rendering framework. In an experiment on a real-time AR application
using appearance-based free-viewpoint image generation, augmented scenes weresuccessfully generated at a high performance 10- to 100-times faster than a simple model-based rendering of virtual objects. Two applications for MR-world
exploration are also developed: a fly-through application providing a change inviewpoint in a 3D space as a straightforward extension of the implementation
described in Section 4.3, and a virtual walk-through application using spherical122

images captured during walking on the ground.
Although the proposed framework can be employed in various applications,some issues can be further considered. The proposed imaging approaches do
not deal with a control method of the aerial vehicles used for image capturing.
To achieve a more eﬀective image acquisition, an automatic vehicle path planning method should be developed. In addition to our imaging framework, suchmethod may be an important issue for 3D reconstruction and free-viewpoint image generation. An important issue of the proposed rendering framework is to use
an enormous number of images. Applications using larger environment can be
developed by overcoming the problems, such as by using multi-viewpoint imagecompression approaches.

123

Acknowledgements
This work was completed under the supervision of Professor Naokazu Yokoya andAssociate Professor Masayuki Kanbara of the Graduate School of Information
Science at Nara Institute of Science and Technology.
The author would like to express his deepest appreciation to Professor Naokazu
Yokoya, who oﬀered continuing support and constant encouragement. Without
the encouragement, this dissertation would not have been possible. He would
particularly like to thank Associate Professor Masayuki Kanbara, who provided
him with numerous invaluable comments, suggestions, and ideas. Their extensive and constructive discussions, which would occasionally last for hours, were adriving force for his research.
The author is also deeply grateful to Professor Hirokazu Kato and Associate
Professor Tomokazu Sato, who, as members of the thesis committee, oﬀered him
insightful comments and suggestions.
The author is also benefited greatly by having the opportunity to meet andwork with a number of people during his doctoral work at the Nara Instituteof Science and Technology. Secretaries Mio Takahashi, Mina Nakamura, Chika
Kijima, and Yumi Ishitani of the Vision and Media Computing Laboratory gave
him invaluable support in daily aﬀairs. He would like to express his appreciation
to Assistant Professors Norihiko Kawai and Yuta Nakashima of the Vision andMedia Computing Laboratory, who gave him constructive comments and suggestions. The author would also like to thank all of the members of the Vision andMedia Computing Laboratory of the Graduate School of Information Science at
the Nara Institute of Science and Technology for their friendship and support, as
well as their cooperation with his experiments.
The author would like to oﬀer his special thanks to Professor Petri Pulli of the124University of Oulu, who provided him with an opportunity to visit the Universityof Oulu for three months. He appreciates the support of Assistant Professor
Goshiro Yamamoto for his research activity and daily life in Oulu. He would
also like to express his gratitude to the Japan Student Services Organization, theJapan Society for the Promotion of Science, and the Nara Institute of Scienceand Technology for their financial support for the visit.
The author would like to express his gratitude to the Japan Society for the
Promotion of Science for their financial support through JSPS Research Fellow
DC2. He would also like to acknowledge his appreciation for the courtesy extended by Toppan Printing Co., Ltd. for the use of their CG models of Heijo
Palace described in Chapter 4.
Finally, the author wants to particularly thank his family for their tirelessunderstanding and support during his Ph.D. studies.

125References[ABB+ 01]

Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner,
Simon Julier, and Blair MacIntyre. Recent advances in augmented
reality. IEEE Computer Graphics and Applications, vol. 21, no. 6,
pp. 34–47, 2001.

[ADF+ 10]

Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh,
Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, andJosh Weaver. Google Street View: Capturing the world at street
level. IEEE Computer Magazine, vol. 43, no. 6, pp. 32–38, 2010.

[AHK+ 10]Ismail Arai, Maiya Hori, Norihiko Kawai, Yohei Abe, Masahiro
Ichikawa, Yusuke Satonaka, Tatsuki Nitta, Tomoyuki Nitta,
Harumitsu Fujii, Masaki Mukai, Soichiro Hiromi, Koji Makita,
Masayuki Kanbara, Nobuhiko Nishio, and Naokazu Yokoya. Pano
UMECHIKA: A crowded underground city panoramic view system.
Proc. Int’l Symp. on Distributed Computing and Artificial Intelligence (DCAI’10), pp. 173–180, Valencia, Spain, 2010.

[ALCS03]

Kusuma Agusanto, Li Li, Zhu Chuangui, and Ng Wan Sing. Photorealistic rendering for augmented reality using environment illumination. Proc. Second IEEE/ACM Int’l Symp. on Mixed and Augmented
Reality (ISMAR’03), pp. 208–216, Tokyo, Japan, 2003.

[AOSY13]

Takayuki Akaguma, Fumio Okura, Tomokazu Sato, and Naokazu
Yokoya. Mobile AR using pre-captured omnidirectional images.
Proc. ACM SIGGRAPH Asia’13 Symp. on Mobile Graphics and Interactive Applications, pp. 26:1–26:4, Hong Kong, China, 2013.
126

[AS02]

Shai Avidan and Amnon Shashua. Novel view synthesis by cascading trilinear tensors. IEEE Trans. on Visualization and Computer
Graphics, vol. 4, no. 4, pp. 293–306, 2002.

[ASS+ 09]

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, andRichard Szeliski. Building Rome in a day. Proc. 12th IEEE Int’l
Conf. on Computer Vision (ICCV’09), pp. 72–79, Kyoto, Japan,
2009.

[ASSS10]Sameer Agarwal, Noah Snavely, Steven M. Seitz, and Richard
Szeliski. Bundle adjustment in the large. Proc. 11th European Conf.
on Computer Vision (ECCV’10), pp. 29–42, Crete, Greece, 2010.

[Azu97]

Ronald Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355–385, 1997.

[BBM+ 01]

Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler,
and Michael Cohen. Unstructured lumigraph rendering. Proc. ACM
SIGGRAPH’01, pp. 425–432, Los Angeles, CA, 2001.

[BL06]

Andrea Bottino and Aldo Laurentini. What’s NEXT? An interactive
next best view approach. Pattern Recognition, vol. 39, no. 1, pp.
126–132, 2006.

[BMOI08]

Atsuhiko Banno, Tomohito Masuda, Takeshi Oishi, and Katsushi
Ikeuchi. Flying laser range sensor for large-scale site-modeling and
its applications in Bayon digital archival project. Int’l Journal ofComputer Vision, vol. 78, no. 2, pp. 207–222, 2008.

[Bra97]

Richard Bradley. Rock Art and the Prehistory of Atlantic Europe:
Signing the Land. Psychology Press, New York, NY, 1997.

[But06]

Declan Butler. Virtual globes: The web-wide world. Nature, vol.
439, no. 7078, pp. 776–778, 2006.

[CCWG88] Michael F. Cohen, Shenchang Eric Chen, John R. Wallace, and Donald P. Greenberg. A progressive refinement approach to fast radiosity

127

image generation. Proc. ACM SIGGRAPH’88, pp. 75–84, Atlanta,
GA, 1988.
[CDSHD13] Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, andGeorge Drettakis. Depth synthesis and local warps for plausible
image-based navigation. ACM Trans. on Graphics, vol. 32, no. 3,
pp. 30:1–30:12, 2013.
[Che95]

Shenchang Eric Chen. Quicktime VR: An image-based approach
to virtual environment navigation. Proc. ACM SIGGRAPH’95, pp.
29–38, Los Angeles, CA, 1995.

[Cho09]Aidan Chopra. Google SketchUp 7 for Dummies. Wiley, Indianapolis, IN, 2009.

[Cro77]Franklin C. Crow. Shadow algorithms for computer graphics. Proc.
ACM SIGGRAPH’77, pp. 242–248, San Jose, CA, 1977.

[CSD11]

Gaurav Chaurasia, Olga Sorkine, and George Drettakis. Silhouetteaware warping for image-based rendering. Computer Graphics Forum, vol. 30, no. 4, pp. 1223–1232, 2011.

[Deb98]Paul Debevec. Rendering synthetic objects into real scenes: Bridging
traditional and image-based graphics with global illumination andhigh dynamic range photography. Proc. ACM SIGGRAPH’98, pp.
189–198, Orlando, FL, 1998.

[DLD12]

Abe Davis, Marc Levoy, and Fredo Durand. Unstructured lightfields. Computer Graphics Forum, vol. 31, no. 2pt1, pp. 305–314,
2012.

[DM97]Paul E. Debevec and Jitendra Malik. Recovering high dynamic range
radiance maps from photographs. Proc. ACM SIGGRAPH’97, pp.
369–378, Los Angeles, CA, 1997.

[DT01]Paul
Debevec
andChris
http://www.hdrshop.com/, 2001.

128

Tchou.

HDR

shop.

[DTM96]

Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. Modelingand rendering architecture from photographs: A hybrid geometryand image-based approach. Proc. ACM SIGGRAPH’96, pp. 11–20,
New Orleans, LA, 1996.

[DYB98]Paul Debevec, Yizhou Yu, and George Borshukov. Eﬃcient viewdependent image-based rendering with projective texture-mapping.
Proc. Ninth Eurographics Workshop on Rendering (EGWR’98), pp.
105–116, Vienna, Austria, 1998.

[FB81]

Martin A Fischler and Robert C Bolles. Random sample consensus:
a paradigm for model fitting with applications to image analysis andautomated cartography. Communications of the ACM, vol. 24, no. 6,
pp. 381–395, 1981.

[FB10]

Arturo Flores and Serge Belongie. Removing pedestrians from google
street view images. Proc. First IEEE Int’l Workshop on Mobile
Vision, pp. 53–58, San Francisco, CA, 2010.

[FBS06]

Jan Fischer, Dirk Bartz, and Wolfgang Straßer. Enhanced visual realism by incorporating camera image eﬀects. Proc. Fifth IEEE/ACM
Int’l Symp. on Mixed and Augmented Reality (ISMAR’06), pp. 205–
208, Santa Barbara, CA, 2006.

[FGR93]

Alain Fournier, Atjeng S. Gunawan, and Chris Romanzin. Common
illumination between real and computer generated scenes. Proc.
Graphics Interface’93, pp. 254–262, Toronto, Canada, 1993.

[FYS+ 08]

Takanori Fukao, Akito Yuzuriha, Takafumi Suzuki, Takeshi Kanzawa, Takashi Oshibuchi, Koichi Osuka, Takashi Kohno, Masahiro
Okuyama, Yasuhito Tomoi, and Masaaki Nakadate. Inverse optimal
velocity field control of an outdoor blimp robot. Proc. 17th IFAC
World Congress, pp. 4374–4379, Seoul, Korea, 2008.

[GB08]

Payam Ghadirian and Ian D. Bishop. Integration of augmented
reality and GIS: A new approach to realistic landscape visualisation.
Landscape and Urban Planning, vol. 86, pp. 226–232, 2008.
129

[GCHH03]

Simon Gibson, Jon Cook, Toby Howard, and Roger Hubbold. Rapid
shadow generation in real-world lighting environments. Proc. 14th
Eurographics Symp. on Rendering (EGSR’03), pp. 219–229, Leuven,
Belgium, 2003.

[GDS10]Thomas Gierlinger, Daniel Danch, and André Stork. Rendering techniques for mixed reality. Journal of Real-Time Image Processing,vol. 5, no. 2, pp. 109–120, 2010.

[GGSC96]

Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, andMichael F. Cohen. The lumigraph. Proc. ACM SIGGRAPH’96,
pp. 43–54, New Orleans, LA, 1996.

[GH97]

Michael Garland and Paul S. Heckbert. Surface simplification using
quadric error metrics. Proc. ACM SIGGRAPH’97, pp. 209–216, Los
Angeles, CA, 1997.

[Gla89]Andrews Glassner. An Introduction to Ray Tracing. Morgan Kaufmann, San Francisco, CA, 1989.

[GM00]

Simon Gibson and Alan Murta. Interactive rendering with realworld illumination. Proc. 11th Eurographics Workshop on Rendering(EGWR’00), pp. 365–376, Brno, Czech Republic, 2000.

[GMG+ 08] Michael A. Goodrich, Bryan S. Morse, Damon Gerhardt, Joseph L.
Cooper, Morgan Quigley, Julie A. Adams, and Curtis Humphrey.
Supporting wilderness search and rescue using a camera-equippedmini UAV. Journal of Field Robotics, vol. 25, no. 1, pp. 89–110,
2008.
[GN03]

Michael D. Grossberg and Shree K. Nayar. High dynamic range
from multiple images: Which exposures to combine? Proc. IEEE
Workshop on Color and Photometric Methods in Computer Vision(CPMCV), pp. 1–8, Beijing, China, 2003.

[Gro05]Thorsten Grosch. PanoAR: Interactive augmentation of omnidirectional images with consistent lighting. Proc. Computer Vi130

sion/Computer Graphics Collaboration Techniques and Applications(Mirage’05), pp. 25–34, Rocquencourt, France, 2005.
[Gro06]

Thorsten Grosch. Fast and robust high dynamic range image generation with camera and object movement. Proc. 2006 Vision Modelingand Visualization (VMV’06), pp. 277–284, Aachen, Germany, 2006.

[GRTS12]Lukas Gruber, Thomas Richter-Trummer, and Dieter Schmalstieg.
Real-time photometric registration from arbitrary geometry. Proc.
11th IEEE Int’l Symp. on Mixed and Augmented Reality (ISMAR’12), pp. 119–128, Atlanta, GA, 2012.

[GWWH03] Nolan Goodnight, Rui Wang, Cliﬀ Woolley, and Greg Humphreys.
Interactive time-dependent tone mapping using programmablegraphics hardware. Proc. 14th Eurographics Symp. on Rendering(EGSR’03), pp. 26–37, Leuven, Belgium, 2003.
[HDF10]Samuel W. Hasinoﬀ, Frédo Durand, and William T. Freeman.
Noise-optimal capture for high dynamic range photography. Proc.
2010 IEEE Conf. on Computer Vision and Pattern Recognition(CVPR’10), pp. 553–560, San Francisco, CA, 2010.

[HDH03]

Michael Haller, Stephan Drab, and Werner Hartmann. A real-timeshadow approach for an augmented reality application using shadow
volumes. Proc. 10th ACM Symp. on Virtual Reality Software andTechnology (VRST’03), pp. 56–65, Osaka, Japan, 2003.

[HJD+ 04]

Stanley Herwitz, Lee Johnson, Steve Dunagan, Robert Higgins, Don
Sullivan, Jian Zheng, Bradley Lobitz, Joe Leung, Bruce Gallmeyer,
Michio Aoyagi, Robert Slye, and James Brass. Imaging from an unmanned aerial vehicle: Agricultural surveillance and decision support. Computers and Electronics in Agriculture, vol. 44, pp. 49–61,
2004.

[HJSL04]

Emmanuel Hygounenc, Il-Kyun Jung, Philippe Soueres, and Simon
Lacroix. The autonomous blimp project of LAAS-CNRS: Achieve-131

ments in flight control and terrain mapping. Int’l Journal of Robotics
Research, vol. 23, no. 4–5, pp. 473–511, 2004.
[HKY09]Maiya Hori, Masayuki Kanbara, and Naokazu Yokoya. MR telepresence system with inertial force sensation using a motion platform
and an immersive display. Proc. IEEE Symp. on 3D User Interfaces(3DUI’09), pp. 133–134, Lafayette, LA, 2009.

[HKY10]Maiya Hori, Masayuki Kanbara, and Naokazu Yokoya. Arbitrary
stereoscopic view generation using multiple omnidirectional image
sequences. Proc. 20th IAPR Int’l Conf. on Pattern Recognition(ICPR’10), pp. 286–289, Istanbul, Turkey, 2010.

[HLHS03]

Jean-Marc Hasenfratz, Marc Lapierre, Nicolas Holzschuch, andFrançois X. Sillion. A survey of real-time soft shadows algorithms.
Computer Graphics Forum, vol. 22, no. 4, pp. 753–774, 2003.

[HS03]

Stefan Hrabar and Gaurav S. Sukhatme. Omnidirectional vision foran autonomous helicopter. Proc. IEEE Int’l Conf. on Robotics andAutomation (ICRA’03), vol. 1, pp. 558–563, Taipei, Taiwan, 2003.

[IHA02]

Michal Irani, Tal Hassner, and P. Anandan. What does the scene
look like from a scene point? Proc. Seventh European Conf. on
Computer Vision (ECCV’02), pp. 883–897, Copenhagen, Denmark,2002.

[IKH+ 11]

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux,Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges,
Dustin Freeman, and Andrew Davison. KinectFusion: Real-time
3D reconstruction and interaction using a moving depth camera.
Proc. 24th ACM Symp. on User Interface Software and Technology(UIST’11), pp. 559–568, Santa Barbara, CA, 2011.

[IKMN04]

Norio Igawa, Yasuko Koga, Tomoko Matsuzawa, and Hiroshi Nakamura. Models of sky radiance distribution and sky luminance distribution. Solar Energy, vol. 77, pp. 137–157, 2004.

132

[ISKY04]

Sei Ikeda, Tomokazu Sato, Masayuki Kanbara, and Naokazu Yokoya.
An immersive telepresence system with a locomotion interface using high-resolution omnidirectional movies. Proc. 17th Int’l Conf.
on Pattern Recognition (ICPR’04), vol. 4, pp. 396–399, Cambridge,
UK, 2004.

[ISY03]Sei Ikeda, Tomokazu Sato, and Naokazu Yokoya. High-resolution
panoramic movie generation from video streams acquired by an omnidirectional multi-camera system. Proc. IEEE Int’l Conf. on Multisensor Fusion and Integration for Intelligent System (MFI’03), pp.
155–160, Tokyo, Japan, 2003.

[Jen96]Henrik Wann Jensen. Global illumination using photon maps. Proc.
Seventh Eurographics Workshop on Rendering (EGWR’96), pp. 21–30, Porto, Portugal, 1996.

[JLW08]Katrien Jacobs, Celine Loscos, and Greg Ward. Automatic highdynamic range image generation for dynamic scenes. IEEE Computer Graphics and Applications, vol. 28, no. 2, pp. 84–93, 2008.

[JND12]Jan Jachnik, Richard A. Newcombe, and Andrew J. Davison. Realtime surface light-field capture for augmentation of planar specularsurfaces. Proc. 11th IEEE Int’l Symp. on Mixed and Augmented
Reality (ISMAR’12), pp. 91–97, Atlanta, GA, 2012.

[JP11]

Michal Jancosek and Tomás Pajdla. Multi-view reconstruction preserving weakly-supported surfaces. Proc. 2011 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’11), pp. 3121–3128,
Colorado Springs, CO, 2011.

[KCSC10]

Johannes Kopf, Billy Chen, Richard Szeliski, and Michael Cohen.
Street slide: Browsing street level imagery. ACM Trans. on Graphics(Proc. ACM SIGGRAPH’10), vol. 29, no. 4, pp. 96:1–96:8, 2010.

[KK12]

Peter Kán and Hannes Kaufmann. High-quality reflections, refractions, and caustics in augmented reality and their contribution to133

visual coherence. Proc. 11th IEEE Int’l Symp. on Mixed and Augmented Reality (ISMAR’12), pp. 99–108, Atlanta, GA, 2012.
[KLS+ 13]

Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard
Szeliski, and Michael Goesele. Image-based rendering in the gradient domain. ACM Trans. on Graphics (Proc. ACM SIGGRAPH
ASIA’13), vol. 32, no. 6, pp. 199:1–199:9, 2013.

[KMSY10]

Norihiko Kawai, Kotaro Machikita, Tomokazu Sato, and Naokazu
Yokoya. Video completion for generating omnidirectional video without invisible areas. IPSJ Trans. on Computer Vision and Applications, vol. 2, pp. 200–213, 2010.

[KOKY12]Masaki Kitaura, Fumio Okura, Masayuki Kanbara, and Naokazu
Yokoya. Tone mapping for HDR images with dimidiate luminanceand spatial distributions of bright and dark regions. Proc. SPIEElectronic Imaging, vol. 8292, pp. 829205:1–829205:11, San Francisco, CA, 2012.

[KSA00]Sing Bing Kang, Richard Szeliski, and P. Anandan. The geometryimage representation tradeoﬀ for rendering. Proc. 2000 IEEE Int’l
Conf. on Image Processing (ICIP’00), vol. 2, pp. 13–16, Vancouver,BC, 2000.

[KSY09]Norihiko Kawai, Tomokazu Sato, and Naokazu Yokoya. Image inpainting considering brightness change and spatial locality of textures and its evaluation. Proc. Third Pacific-Rim Symp. on Image and Video Technology (PSIVT’09), pp. 271–282, Tokyo, Japan,
2009.

[KTM+ 10]

Martin Knecht, Christoph Traxler, Oliver Mattausch, Werner Purgathofer, and Michael Wimmer. Diﬀerential instant radiosity for
mixed reality. Proc. Ninth IEEE Int’l Symp. on Mixed and Augmented Reality (ISMAR’10), pp. 99–107, Seoul, Korea, 2010.

[KTSY10]

Hideyuki Kume, Takafumi Taketomi, Tomokazu Sato, and Naokazu
Yokoya. Extrinsic camera parameter estimation using video images
134and GPS considering GPS positioning accuracy. Proc. 20th IAPR
Int’l Conf. on Pattern Recognition (ICPR’10), pp. 3923–3926, Istanbul, Turkey, 2010.
[KUWS03] Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard
Szeliski. High dynamic range video. ACM Trans. on Graphics (Proc.
ACM SIGGRAPH’03), vol. 22, no. 3, pp. 319–325, 2003.
[KY02]

Masayuki Kanbara and Naokazu Yokoya. Geometric and photometric registration for real-time augmented reality. Proc. First
IEEE/ACM Int’l Symp. on Mixed and Augmented Reality (ISMAR’02), pp. 279–280, Darmstadt, Germany, 2002.

[KZP+ 13]

Changil Kim, Henning Zimmer, Yael Pritch, Alexander SorkineHornung, and Markus Gross. Scene reconstruction from high spatioangular resolution light fields. ACM Trans. on Graphics (Proc. ACM
SIGGRAPH’13), vol. 32, no. 4, pp. 73:1–73:11, 2013.

[LA09]

Manolis I. A. Lourakis and Antonis A. Argyros. SBA: A software
package for generic sparse bundle adjustment. ACM Trans. on Mathematical Software, vol. 36, no. 1, pp. 2:1–2:30, 2009.

[LB12]

Philipp Lensing and Wolfgang Broll. Instant indirect illumination
for dynamic mixed reality scenes. Proc. 11th IEEE Int’l Symp. on
Mixed and Augmented Reality (ISMAR’12), pp. 109–118, Atlanta,
GA, 2012.

[LH96]

Marc Levoy and Pat Hanrahan. Light field rendering. Proc. ACM
SIGGRAPH’96, pp. 31–42, New Orleans, LA, 1996.

[Lhu11]

Maxime Lhuillier. Fusion of GPS and structure-from-motion using
constrained bundle adjustments. Proc. 2011 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’11), pp. 3025–3032,
Colorado Springs, CO, 2011.

[LLW04]Ping-Man Lam, Chi-Sing Leung, and Tien-Tsin Wong. A compression method for a massive image data set in image-based rendering.
135

Signal Processing: Image Communication, vol. 19, no. 8, pp. 741–
754, 2004.
[MAW+ 07] Paul Merrell, Amir Akbarzadeh, Liang Wang, Philippos Mordohai,
Jan-Michael Frahm, Ruigang Yang, David Nistér, and Marc Pollefeys. Real-time visibility-based fusion of depth maps. Proc. 11th
IEEE Int’l Conf. on Computer Vision (ICCV’07), pp. 1–8, Rio deJaneiro, Brazil, 2007.
[MB95]

Leonard McMillan and Gary Bishop. Plenoptic modeling: An imagebased rendering system. Proc. ACM SIGGRAPH’95, pp. 39–46, Los
Angeles, CA, 1995.

[MCMO10] Iván F Mondragón, Pascual Campoy, Carol Martinez, and Miguel
Olivares. Omnidirectional vision applied to unmanned aerial vehicles(UAVs) attitude and heading estimation. Robotics and Autonomous
Systems, vol. 58, no. 6, pp. 809 – 819, 2010.
[MD96]

Russell A. Manning and Charles R. Dyer. View morphing. Proc.
ACM SIGGRAPH’96, pp. 21–30, New Orleans, LA, 1996.

[Min80]Marvin Minsky. Telepresence. Omni, vol. 2, no. 9, pp. 45–51, 1980.

[Miy64]

Kenro Miyamoto. Fish eye lens. Journal of the Optical Society ofAmerica, vol. 54, no. 8, pp. 1060–1061, 1964.

[MK94]Paul Milgram and Fumio Kishino. A taxonomy of mixed reality
visual displays. IEICE Trans. on Information Systems, vol. E77-D,
no. 9, pp. 1321–1329, 1994.

[MN99]

Tomoo Mitsunaga and Shree K. Nayar. Radiometric self calibration.
Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR’99), vol. 1, pp. 374–380, Fort Collins, CO,1999.

[Moe97]

Saied Moezzi. Special issue on immersive telepresence. IEEE Multimedia, vol. 4, no. 1, pp. 17–56, 1997.

136

[MRG03]

Marcus Magnor, Prashant Ramanathan, and Bernd Girod. Multiview coding for image-based rendering using 3-D scene geometry.
IEEE Trans. on Circuits and Systems for Video Technology, vol. 13,
no. 11, pp. 1092–1106, 2003.

[MTUK94] Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino.
Augmented reality: A class of displays on the reality virtuality continuum. Proc. SPIE Telemanipulator and Telepresence Technologies,
vol. 2351, no. 34, pp. 282–292, San Francisco, CA, 1994.
[Nal96]

Vic Nalwa. A true omnidirectional viewer.
BL0115500-960115-01, Bell Laboratories, 1996.

[Nay88]

Shree K. Nayar. Sphereo: Determining depth using two specularspheres and a single camera. Proc. SPIE Optics, Illumination, andImage Sensing for Machine Vision III, pp. 245–254, San Francisco,CA, 1988.

[Nay97]

Shree K. Nayar. Catadioptric omnidirectional camera. Proc. 1997
IEEE Computer Society Conf. on Computer Vision and Pattern
Recognition (CVPR’97), pp. 482–488, San Juan, PR, 1997.

[NCS+ 09]

Masahiko Nagai, Tianen Chen, Ryosuke Shibasaki, Hideo Kumagai,
and Afzal Ahmed. UAV-borne 3-D mapping system by multisensor integration. IEEE Trans. on Geoscience and Remote Sensing,vol. 47, no. 3, pp. 701–708, 2009.

[NHIN86]

Eihachiro Nakamae, Koichi Harada, Takao Ishizaki, and Tomoyuki
Nishita. A montage method: The overlaying of the computer generated images onto a background photograph. Proc. ACM SIGGRAPH’86, pp. 207–214, Dallas, TX, 1986.

[NSD94]Jeﬀry S. Nimeroﬀ, Eero Simoncelli, and Julie Dorsey. Eﬃcient rerendering of naturally illuminated environments. Proc. Fifth Eurographics Workshop on Rendering (EGWR’94), pp. 359–373, Darmstadt, Germany, 1994.

137

Technical Report

[NTKH97]

Takeshi Naemura, Takahide Takano, Masahide Kaneko, and Hiroshi Harashima. Ray-based creation of photo-realistic virtualworld. Proc. Third Int’l Conf. on Virtual Systems and Multimedia(VSMM’97), pp. 59–68, Geneva, Switzerland, 1997.

[OKY10]Fumio Okura, Masayuki Kanbara, and Naokazu Yokoya. Augmented
telepresence using autopilot airship and omni-directional camera.
Proc. Ninth IEEE Int’l Symp. on Mixed and Augmented Reality (ISMAR’10), pp. 259–260, Seoul, Korea, 2010.

[OUSY13]

Fumio Okura, Yuko Ueda, Tomokazu Sato, and Naokazu Yokoya.
Teleoperation of mobile robots by generating augmented freeviewpoint images. Proc. 2013 IEEE/RSJ Int’l Conf. on Intelligent
Robots and Systems (IROS’13), pp. 665–671, Tokyo, Japan, 2013.

[OYTY98]

Yoshio Onoe, Kazumasa Yamazawa, Haruo Takemura, and Naokazu
Yokoya. Telepresence by real-time view-dependent image generationfrom omnidirectional video streams. Computer Vision and Image
Understanding, vol. 71, no. 2, pp. 154–165, 1998.

[PARJ+ 06] Ely Carneiro de Paiva, José Raul Azinheira, Josué G. Ramos Jr,
Alexandra Moutinho, and Samuel Siqueira Bueno. Project AURORA: Infrastructure and flight control experiments for a robotic
airship. Journal of Field Robotics, vol. 23, no. 2–3, pp. 201–222,
2006.
[PDG05]

Damien Porquet, Jean-Michel Dischler, and Djamchid Ghazanfarpour. Real-time high-quality view-dependent texture mapping using per-pixel visibility. Proc. Third Int’l Conf. on Computer Graphics and Interactive Techniques in Australasia and South East Asia
(GRAPHITE’05), pp. 213–220, Dunedin, New Zealand, 2005.

[PH10]

Matt Pharr and Greg Humphreys. Physically Based Rendering:
From Theory to Implementation. Morgan Kaufmann, Burlington,
MA, 2010.

138

[PHGD+ 12] Alistair WG Pike, DL Hoﬀmann, Marcos Garcı́a-Diez, Paul B Pettitt, Jose Alcolea, Rodrigo De Balbin, César González-Sainz, Carmen de las Heras, Jose A Lasheras, Ramón Montes, and J Zilhäo.
U-series dating of paleolithic art in 11 caves in Spain. Science, vol.
336, no. 6087, pp. 1409–1413, 2012.
[Pit99]Richard Pito. A solution to the next best view problem for automated surface acquisition. IEEE Trans. on Pattern Analysis andMachine Intelligence, vol. 21, no. 10, pp. 1016–1030, 1999.

[PNF+ 08]

Marc Pollefeys, David Nistér, Jan-Michael Frahm, Amir Akbarzadeh, Philippos Mordohai, Brian Clipp, Chris Engels, David
Gallup, S-J Kim, and Paul Merrell. Detailed real-time urban 3D reconstruction from video. Int’l Journal of Computer Vision, vol. 78,
no. 2, pp. 143–167, 2008.

[Rei02]Erik Reinhard. Parameter estimation for photographic tone reproduction. Journal of Graphics Tools, vol. 7, no. 1, pp. 45–51, 2002.

[RHD+ 10]

Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. High Dynamic Range
Imaging: Acquisition, Display, and Image-Based Lighting. Morgan
Kaufmann, San Francisco, CA, 2010.

[RKB04]

Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut:
Interactive foreground extraction using iterated graph cuts. ACM
Trans. on Graphics (Proc. ACM SIGGRAPH’04), vol. 23, no. 3, pp.
309–314, 2004.

[RSSF02]

Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda.
Photographic tone reproduction for digital images. ACM Trans. on
Graphics (Proc. ACM SIGGRAPH’02), vol. 21, no. 3, pp. 267–276,
2002.

[SCK07]

Heung-Yeung Shum, Shing-Chouw Chan, and Sing Bing Kang.
Image-Based Rendering. Springer, New York, NY, 2007.

139

[SHK+ 05]

Imari Sato, Morihiro Hayashida, Fumiyo Kai, Yoichi Sato, and Katsushi Ikeuchi. Fast image synthesis of virtual objects in a real scene
with natural shadings. Systems and Computers in Japan, vol. 36,
no. 14, pp. 102–111, 2005.

[SIY04]

Tomokazu Sato, Sei Ikeda, and Naokazu Yokoya. Extrinsic camera parameter recovery from multiple image sequences captured byan omni-directional multi-camera system. Proc. European Conf. on
Computer Vision (ECCV’04), vol. 2, pp. 326–340, Prague, Czech
Republic, 2004.

[SKC03]

Heung-Yeung Shum, Sing Bing Kang, and Shing-Chow Chan. Surveyof image-based representations and compression techniques. IEEE
Trans. on Circuits and Systems for Video Technology, vol. 13, no. 11,
pp. 1020–1037, 2003.

[SKG+ 12]

Sudipta N Sinha, Johannes Kopf, Michael Goesele, Daniel
Scharstein, and Richard Szeliski. Image-based rendering for scenes
with reflections. ACM Trans. on Graphics (Proc. ACM SIGGRAPH’12), vol. 31, no. 4, pp. 100:1–100:10, 2012.

[SSI02]

Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. Acquiring a radiancedistribution to superimpose virtual objects onto a real scene. IEEE
Trans. on Visualization and Computer Graphics, vol. 5, no. 1, pp.
1077–2626, 2002.

[SSI03]

Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. Illumination fromshadows. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 25, no. 3, pp. 290–300, 2003.

[SSS06]Noah Snavely, Steven M. Seitz, and Richard Szeliski. Photo tourism:
Exploring photo collections in 3D. ACM Trans. on Graphics (Proc.
ACM SIGGRAPH’06), vol. 25, no. 3, pp. 835–846, 2006.

[ST94]

Jianbo Shi and Carlo Tomasi. Good features to track. Proc. 1994
IEEE Computer Society Conf. on Computer Vision and Pattern
Recognition (CVPR’94), pp. 593–600, Seattle, WA, 1994.
140

[Ste92]

Jonathan Steuer. Defining virtual reality: Dimensions determining
telepresence. Journal of Communication, vol. 42, no. 4, pp. 73–93,
1992.

[STJ+ 04]

Jessi Stumpfel, Chris Tchou, Andrew Jones, Tim Hawkins, Andreas
Wenger, and Paul Debevec. Direct HDR capture of the sun andsky. Proc. Third Int’l Conf. on Computer Graphics, Virtual Reality,
Visualization and Interaction in Africa (AFRIGRAPH’04), pp. 145–
149, Stellenbosch, South Africa, 2004.

[STYY00]

Jun Shimamura, Haruo Takemura, Naokazu Yokoya, and Kazumasa
Yamazawa. Construction and presentation of a virtual environment
using panoramic stereo images of a real scene and computer graphics
models. Proc. 15th Int’l Conf. on Pattern Recognition (ICPR’00),
vol. 4, pp. 463–467, Barcelona, Spain, 2000.

[Sut65]

Ivan E Sutherland. The ultimate display. Proc. IFIP Congress,
vol. 65, no. 2, pp. 506–508, 1965.

[Sut68]

Ivan E Sutherland. A head-mounted three dimensional display. Proc.
Fall Joint Computer Conf., vol. 33, pp. 757–764, 1968.

[SY10]

Tomokazu Sato and Naokazu Yokoya. Eﬃcient hundreds-baseline
stereo by counting interest points for moving omni-directional multicamera system. Journal of Visual Communication and Image Representation, vol. 21, no. 5–6, pp. 416–426, 2010.

[TKIS00]

Takuji Takahashi, Hiroshi Kawasaki, Katsushi Ikeuchi, and Masao
Sakauchi. Arbitrary view position and direction rendering for largescale scenes. Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR’00), vol. 2, pp. 2296–2303, Hilton Head, SC,
2000.

[TMHF99]

Bill Triggs, Philip F. McLauchlan, Richard I. Hartley, and Andrew W. Fitzgibbon. Bundle adjustment—a modern synthesis. Proc.
Int’l Workshop on Vision Algorithms, pp. 298–372, Corfu, Greece,
1999.
141

[UCK+ 04]

Matthew Uyttendaele, Antonio Criminisi, Sing Bing Kang, Simon
Winder, Richard Szeliski, and Richard Hartley. Image-based interactive exploration of real-world environments. IEEE Computer
Graphics and Applications, vol. 24, no. 3, pp. 52–63, 2004.

[VCL+ 11]

Peter Vangorp, Gaurav Chaurasia, Pierre-Yves Laﬀont, Roland W
Fleming, and George Drettakis. Perception of visual artifacts
in image-based rendering of façades. Computer Graphics Forum,
vol. 30, no. 4, pp. 1241–1250, 2011.

[Vin07]

Luc Vincent. Taking online maps down to street level. IEEE Computer Magazine, vol. 40, no. 12, pp. 118–120, 2007.

[VRC+ 13]

Peter Vangorp, Christian Richardt, Emily A. Cooper, Gaurav
Chaurasia, Martin S. Banks, and George Drettakis. Perception ofperspective distortions in image-based rendering. ACM Trans. on
Graphics (Proc. ACM SIGGRAPH’13), vol. 32, no. 4, pp. 58:1–
58:11, 2013.

[WACS11]

Changchang Wu, Sameer Agarwal, Brian Curless, and Steven M.
Seitz. Multicore bundle adjustment. Proc. 2011 IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR’11), pp. 3057–
3064, Colorado Springs, CO, 2011.

[War94]

Greg J. Ward. The RADIANCE lighting simulation and renderingsystem. Proc. ACM SIGGRAPH’94, pp. 459–472, Orlando, FL,
1994.

[War03]

Greg J. Ward. Fast, robust image registration for compositing highdynamic range photographs from hand-held exposures. Journal ofGraphics Tools, vol. 8, no. 2, pp. 17–30, 2003.

[Wol08]Robert M. Wolk. Utilizing Google Earth and Google Sketchup tovisualize wind farms. Proc. 2008 IEEE Int’l Symp. on Technology
and Society (ISTAS’08), pp. 1–8, Fredericton, NB, 2008.

142

[WPF90]

Andrew Woo, Pierre Poulin, and Alain Fournier. A survey of shadow
algorithms. IEEE Computer Graphics and Applications, vol. 10,
no. 6, pp. 13–32, 1990.

[WTA11]

Jason Wither, Yun-Ta Tsai, and Ronald Azuma. Indirect augmented
reality. Computers and Graphics, vol. 35, no. 4, pp. 810–822, 2011.

[Wu13]

Changchang Wu. Towards linear-time incremental structure frommotion. Proc. 2013 Int’l Conf. on 3D Vision (3DV’13), pp. 127–
134, Seattle, WA, 2013.

[XBF+ 09]

Hanwei Xu, Rami Badawi, Xiaohu Fan, Jiayong Ren, and Zhiqiang
Zhang. Research for 3D visualization of digital city based on
SketchUp and ArcGIS. Proc. SPIE Int’l Symp. on Spatial Analysis, Spatial-Temporal Data Modeling, and Data Mining, vol. 7492,
pp. 74920Z:1–74920Z:12, San Francisco, CA, 2009.

[XT97]

Yalin Xiong and Kenneth Turkowski. Creating image-based VR using a self-calibrating fisheye lens. Proc. 1997 IEEE Computer Society
Conf. on Computer Vision and Pattern Recognition (CVPR’97), pp.
237–243, San Juan, PR, 1997.

[Yag99]Yasushi Yagi. Omnidirectional sensing and its applications. IEICE
Trans. on Information and Systems, vol. 82, no. 3, pp. 568–579,
1999.

[YISY06]

Yuji Yokochi, Sei Ikeda, Tomokazu Sato, and Naokazu Yokoya.
Extrinsic camera parameter estimation based-on feature tracking
and GPS data. Proc. Seventh Asian Conf. on Computer Vision(ACCV’06), vol. 1, pp. 369–378, Hyderabad, India, 2006.

[YK90]

Yasushi Yagi and Shinjiro Kawato. Panorama scene analysis withconic projection. Proc. 1990 IEEE/RSJ Int’l Conf. on Intelligent
Robots and Systems (IROS’90), pp. 181–187, Ibaraki, Japan, 1990.

[YYY93]Kazumasa Yamazawa, Yasushi Yagi, and Masahiko Yachida. Omnidirectional imaging with hyperboloidal projection. Proc. 1993
143

IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS’93),
vol. 2, pp. 1029–1034, Tokyo, Japan, 1993.
[ZC04]

Cha Zhang and Tsuhan Chen. A survey on image-based rendering–
representation, sampling and compression. Signal Processing: Image
Communication, vol. 19, no. 1, pp. 1–28, 2004.

[ZFPW03]

Assaf Zomet, Doron Feldman, Shmuel Peleg, and Daphna Weinshall.
Mosaicing new views: The crossed-slits projection. IEEE Trans. on
Pattern Analysis and Machine Intelligence, vol. 25, no. 6, pp. 741–
754, 2003.

144

List of Publications
Journal Papers
1. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Augmented telepresence using recorded aerial omnidirectional videos captured from unmanned
airship. Trans. of Virtual Reality Society of Japan, Vol. 16, No. 2, pp. 127–
138, Jun 2011 [VRSJ Outstanding Paper Award] (in Japanese). (Chapter
3, Chapter 4, Chapter 5)
2. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Aerial HDR video
generation of full spherical views using two omnidirectional camera units.
Trans. of Virtual Reality Society of Japan, Vol. 17, No. 3, pp. 139–149,
Sep 2012 (in Japanese). (Chapter 3).
3. Fumio Okura, Yuko Ueda, Tomokazu Sato, Naokazu Yokoya. Free-viewpoint
mobile robot teleoperation interface using view-dependent geometry andtexture. ITE Trans. on Media Technology and Applications, Vol. 2, No. 1,
pp. 82–93, Jan 2014.

International Conferences
1. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Augmented telepresence from the sky: AR using autopilot airship and omni-directional camera.
Proc. 3rd Korea-Japan Workshop on Mixed Reality (KJMR’10), pp. 190–
200, Apr 2010. (Chapter 3).
2. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Augmented telepres145

ence using autopilot airship and omni-directional camera. Proc. 9th IEEE
Int’l Symp. on Mixed and Augmented Reality (ISMAR’10), pp. 259–260,
Oct 2010. (Chapter 3).
3. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Fly-through Heijo
Palace Site: Augmented telepresence using aerial omnidirectional videos.
Proc. ACM SIGGRAPH’11 Posters, Article No. 78, Aug 2011. (Chapter
3, Chapter 4, Chapter 5).
4. Masaki Kitaura, Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Tone
mapping for HDR images with dimidiate luminance and spatial distributions of bright and dark regions. Proc. SPIE Electronic Imaging, Vol. 8292,
pp. 829205-01–829205-11, Jan 2012. (Chapter 5).
5. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Full spherical highdynamic range imaging from the sky. Proc. IEEE Int’l Conf. on Multimedia
and Expo (ICME’12), pp. 325–332, Jul 2012. (Chapter 3).
6. Fumio Okura. Spacetime freeview generation using image-based rendering,
relighting, and augmented telepresence. Proc. ACM Multimedia (MM’12)
Doctoral Symposium, pp. 1437–1440, Oct 2012. (Chapter 3).
7. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Fly-through Heijo
Palace Site: Historical tourism system using augmented telepresence. Proc.
ACM Multimedia (MM’12) Technical Demo, pp. 1283–1284, Oct 2012.
(Chapter 3, Chapter 4, Chapter 5).
8. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Interactive explorationof augmented aerial scenes with free-viewpoint image generation from prerendered images. Proc. 12th IEEE Int’l Symp. on Mixed and Augmented
Reality (ISMAR’13), pp. 279–280, Oct 2013. (Chapter 4).
9. Fumio Okura, Yuko Ueda, Tomokazu Sato, Naokazu Yokoya. Teleoperationof mobile robots by generating augmented free-viewpoint images. Proc.
2013 IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS’13),
pp. 665–671, Nov 2013.

146

10. Takayuki Akaguma, Fumio Okura, Tomokazu Sato, Naokazu Yokoya. Mobile AR using pre-captured omnidirectional images. Proc. ACM SIGGRAPH Asia 2013 Symp. on Mobile Graphics and Interactive Applications,
Article No. 26, Nov 2013.

Domestic Conferences
1. Naoko Nitta, Yukifumi Okada, Nozomu Kasuya, Yusuke Utsuno, Makoto
Fujigaki, Shinnosuke Tokumoto, Kenichiro Fuji, Fumio Ogawa, Toru Kawasaki,
Takuma Maruyama, Fumio Okura. PRMU Algorithm Contest 2009 : “FindClones!” summary report and prize winning algorithms. Technical Reportof IEICE, PRMU2009-155, Dec 2009 (in Japanese).
2. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Aerial imaging system
by automatic control of unmanned airship. Proc. IEICE General Conference 2010, D-12-84, Mar 2010 (in Japanese).
3. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Augmented telepresence from the sky: Augmented reality using autopilot airship and omnidirectional camera. Proc. Meeting on Image Recognition and Understanding (MIRU’10), pp. 1183–1189, Jul 2010 (in Japanese). (Chapter 3).
4. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Fly-through augmented
telepresence using recorded aerial omnidirectional videos. Proc. 15th Virtual Reality Society of Japan Annual Conference, pp. 394–397, Sep 2010
(in Japanese). (Chapter 3, Chapter 4, Chapter 5).
5. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Fly-through MR Heijokyo: Augmented telepresence using recorded aerial omnidirectional videos
captured from unmanned airship. Technical Report of IEICE, MVE2010-58,
Oct 2010 (in Japanese). (Chapter 3, Chapter 4, Chapter 5).
6. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Geometric and photometric registration for augmented telepresence using recorded aerial omnidirectional videos. Proc. Meeting on Image Recognition and Understanding147

(MIRU’11), pp. 1177–1184, Jul 2011 (in Japanese). (Chapter 3, Chapter
4, Chapter 5).
7. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Aerial HDR imaging ofcomplete spherical images using two omnidirectional cameras. Proc. Forumon Information Technology (FIT’11), Vol. 3, pp. 187–190, Sep 2011 (inJapanese). (Chapter 3).
8. Yoshiki Tanaka, Fumio Okura, Maiya Hori, Masayuki Kanbara, Naokazu
Yokoya. Omnidirectional image acquisition support system using recommendation degree map for telepresence with image-based rendering. Proc.
16th Virtual Reality Society of Japan Annual Conference, pp. 640–641, Sep
2011 (in Japanese).
9. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Aerial HDR imaging
of full spherical views without missing areas using two omnidirectional cameras. Technical Report of IEICE, MVE2011-80, Jan 2012 [MVE Award] (inJapanese). (Chapter 3).
10. Masaki Kitaura, Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Tonemapping for HDR images divided with dimidiate luminance and spatial distributions of bright and dark regions. Technical Report of IEICE, MVE201174, Jan 2012 (in Japanese). (Chapter 5).
11. Yoshiki Tanaka, Fumio Okura, Maiya Hori, Masayuki Kanbara, Naokazu
Yokoya. Image acquisition support system using recommendation degree
map for telepresence with image-based rendering. Technical Report of IEICE, MVE2011-128, Mar 2012 (in Japanese).
12. Masaki Kitaura, Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Tonemapping using region segmentation for HDR images with dimidiate luminance and spatial distributions. Proc. Meeting on Image Recognition andUnderstanding (MIRU’12), Aug 2012 (in Japanese). (Chapter 5).
13. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Displaying sphericalaerial HDR images for telepresence system. Technical Report of IEICE,
MVE2012-42, Sep 2012 (in Japanese). (Chapter 5).
148

14. Yuko Ueda, Fumio Okura, Tomokazu Sato, Naokazu Yokoya. Teleoperationinterface of mobile robots providing freely configurable viewpoint. Proc.
2012 Kansai Joint Convention of Institutes of Electrical Engineering, pp.
463–464, Dec 2012 (in Japanese).
15. Yuko Ueda, Fumio Okura, Tomokazu Sato, Naokazu Yokoya. Mobile robot
control interface using augmented free-viewpoint image generation. Technical Report of IEICE, MVE2012-73, Jan 2013 (in Japanese).
16. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya. Photorealistic superimposition of virtual objects onto virtualized real-world using pre-renderingand free-viewpoint image generation. Proc. Meeting on Image Recognition
and Understanding (MIRU’13), SS4-29, Jul 2013 (in Japanese). (Chapter
4).
17. Takayuki Akaguma, Fumio Okura, Tomokazu Sato, Naokazu Yokoya. Mobile AR using pre-rendered images. Proc. 18th Virtual Reality Society ofJapan Annual Conference, pp. 513–516, Sep 2013 (in Japanese).
18. Fumio Okura, Masayuki Kanbara, Naokazu Yokoya.Photorealistic augmented
reality based on free-viewpoint image generation using pre-rendered images.
Technical Report of VRSJ, Vol. 18, No. CS-3, pp. 11–16, Sep 2013 (inJapanese). (Chapter 4).
19. Fumio Okura, Yuko Ueda, Tomokazu Sato, Naokazu Yokoya. Teleoperationinterface of mobile robots using free-viewpoint image generation. Proc.
2013 Kansai Joint Convention of Institutes of Electrical Engineering, pp.
456–457, Nov 2013 (in Japanese).
20. Takayuki Akaguma, Fumio Okura, Tomokazu Sato, Naokazu Yokoya. AR
using pre-captured images considering illumination of real scene. Technical
Report of IEICE, PRMU2013-161, Feb 2014 (in Japanese).
21. Naoya Inoue, Norihiko Kawai, Tomokazu Sato, Fumio Okura, Yuta Nakashima,
Naokazu Yokoya. Removal of moving objects from omnidirectional video
taken by a moving camera. Proc. IEICE General Conference 2014, D-1143, Mar 2014 (in Japanese).
149

Awards
1. Jury’s Special Award, IEICE PRMU Algorithm Contest 2009, Sep 2009.
2. 2011 NAIST Top Scholarship Program, Jun 2011.
3. MVE Award, IEICE Technical Group on Multimedia and Virtual Environment (MVE), Sep 2012.
4. VRSJ Outstanding Paper Award, The Virtual Reality Society of Japan,
Sep 2012.

150

Appendix A
Spherical HDR Imaging on the
Ground Using OMS
Spherical HDR images can be acquired simply from the ground using an OMS,
although there seems to be a lack of knowledge accumulated for such imaging
methods. In the same fashion as HDR imaging using ordinary monocular cameras, multi-exposure images are first captured using an OMS. Unless capturing
in motion is particularly required, the OMS should be mounted on a tripod to
avoid a misalignment of the multi-exposure images. The auto exposure determination used for our aerial HDR imaging described in Section 3.3.2 can also be
employed for multi-exposure imaging on the ground. The pedestrian subtraction
method [AHK+ 10] can be applied for each multi-exposure image sequence captured at a fixed position. HDR images are generated from a set of multi-exposure
images using the method described by Devebec and Malik [DM97].
Figure 2.5 in Section 2.1.3 shows an example of an HDR spherical image captured in an underground city using the method mentioned above after pedestrianremoval [AHK+ 10]. Through this same method, outdoor scenes can also be captured as spherical HDR images. Figure A.1 shows multi-exposure images takenusing an auto exposure determination and pedestrian removal that were acquiredin an outdoor environment. A spherical HDR image was generated from themulti-exposure images, as shown in Figure A.2.

151

(a) 0.08 ms.

(b) 0.3 ms.

(c) 1.2 ms.

(d) 4.8 ms.

Figure A.1. Multi-exposure images whose exposure values are determined using
the method described in Section 3.3.2. Pedestrians in the images were removed
using the method described in [AHK+ 10].

Figure A.2. A spherical HDR image captured in outdoor environment.

152