Arbitrary Viewpoint Rendering from Multiple Omnidirectional Images
for Interactive Walkthroughs
Kaname Tomite, Kazumasa Yamazawa and Naokazu Yokoya
Graduate School of Information Science, Nara Institute of Science and Technology,
8916-5 Takayama, Ikoma, Nara 630-0101, Japan,{kaname-t, yamazawa, yokoya}@is.aist-nara.ac.jp

Abstract
Recently, the interactive walkthrough which enables us
to look around a virtualized real world has been widelyinvestigated. The walkthrough system can be applied tosimulation, telepresence, and so on. This paper describes
a method of interactive walkthrough which takes a realscene and enables us to look around in the virtualized
scene. Our method is based on acquiring omnidirectional
images at multiple points in the real world by using anomnidirectional camera. The method generates an omnidirectional image at an arbitrary location from captured
omnidirectional images, and presents a user a convertedview-dependent perspective image. In the experiment, werealized the interactive walkthrough using indoor omnidirectional images and confirmed the feasibility of themethod.

1. Introduction
Recently, many researches have been made to reconstruct walkthroughs in virtualized real scenes such as cityscenes, building rooms and natural environments[1][2][3][4][7][8][9]. These can be applied to virtual tour,
driving simulation, disaster simulation, and so on.
Our objective is to develop a walkthrough system
which enables a user to look around at arbitrary viewpoints widely using a limited number of input images. For
this purpose, the approach is based on using an omnidirectional camera HyperOmni Vision [10]. Omnidirectional images at arbitrary viewpoints are computed from alimited number of omnidirectional images captured atdiscrete positions. This differs from previous works[1][8][9] that require dense image sequences captured bya moving omnidirectional camera.
This paper is structured as follows. First, we describe amethod of generating omnidirectional images at arbitrary
viewpoints from sparsely acquired omnidirectional images. Next, we present experiments to reconstruct walkthroughs in an indoor scene from four omnidirectional
images. Finally, we summarize the present work.

Fig.1 Arbitrary novel viewpoint and multiple positions ofomnidirectional camera.

2. Generating Omnidirectional Image at Arbitrary Viewpoint2.1. Outline of the method
In our method, we take omnidirectional images at multiple positions with the omnidirectional camera HyperOmni Vision as illustrated in Fig.1. Interactive walkthrough is essentially accomplished by generating anomnidirectional image at an arbitrary position from threeomnidirectional images that are acquired around the target
position.
The flow of the method is shown in Fig.2. The present
work is based on two assumptions: (1) Positions of HyperOmni Vision are known; (2) Partial correspondencesamong captured omnidirectional images are given. First,
the positions of corresponding points in a novel omnidirectional image are computed. Second, the triangles aregenerated from the computed corresponding points. Third,
the novel omnidirectional image is generated by blending
corresponding triangular parts of captured omnidirectional
images. Blending weights are determined based on theposition of the target viewpoint in blending triangularparts. Finally, a common perspective image computedfrom the generated omnidirectional image is presented tothe user.

Input / Output

to camera axes as input data for reconstructing walkthroughs. As mentioned earlier, the camera positions areassumed to be known.

Processing

Positions ofcorresponding featurepoints in captured
omnidirectional images

Compute positions ofcorresponding points
in omnidirectional imageof arbitrary viewpoint

OM

Position ofarbitrary viewpoint

Triangulate imagecCommon
perspective imagex

where c = a 2 + b 2 .
A ray going from the point P( X , Y , Z ) in 3D toward
the inner focal point OM is reflected by the mirror andpasses through the outer focal point OC intersecting theimage plane at the point p(x, y ) . Therefore, the projectionof HyperOmni Vision is not common planar perspective,
but satisfies the single viewpoint constraint. The relationship between P( X , Y , Z ) and p(x, y ) is given by:(),(b + c )(Z − c ) − 2bc X + Y + (Z − c )f (b − c )Yy=.
(b + c )(Z − c ) − 2bc X + Y + (Z − c )222222222

22fy

P(X,Y,Z)p(x,y)

Image Plane
Camera CenterWe use HyperOmni Vision [10] (still version) as anomnidirectional camera in the present work. HyperOmni
Vision is composed of a hyperboloidal mirror and a digital still camera as illustrated in Fig.3. The camera acquires
an omnidirectional scene reflected by the hyperboloidal
mirror. The hyperboloidal mirror is constructed of a hyperboloid of two sheets of revolution, which has two focal
points ( OM and OC ). The camera lens center is fixed at thefocal point OC . Given a world coordinate ( X , Y , Z ) andan image coordinate (x, y ) as shown in the Fig.3(b), theshape of hyperboloidal mirror and the two focal points arerepresented as follows.
X 2 +Y 2 Z 2Mirror Shape :− 2 = −1 ,a2b(0,0,+c ) , (1)Inner focal point OM :Outer focal point OC (Camera lens center): (0,0,−c ) ,f b −c X

cOC

2.2. Omnidirectional camera: HyperOmni Vision

2YO

Fig.2 Flow chart of proposed method.

x=b

Blend images
X2

Focal Point
Hyperboloidal MirrorZ(2)2By using the above equation, an omnidirectional inputimage can be converted to a common perspective image atthe viewpoint OM [6].
In this study, omnidirectional images are obtained atsparsely spaced positions on a single plane perpendicular

(a)

(b)

Fig.3 HyperOmni Vision (still version).

2.3. Estimation of 3D position of corresponding
points
We estimate a 3D coordinate T ( X,Y,Z ) from corresponding points
p1 (x1 , y1 ) ,p 2 (x 2 , y 2 ) , andp 3 (x 3 , y 3 ) in three omnidirectional images for generating a novel omnidirectional view. The method is actuallya variant of omnidirectional trinocular stereo.
Figure 4(a) shows a bird’s-eye view of straight lines
l1 , l 2 , and l3 connecting centers of projection and thepoints p1 (x1 , y1 ) , p 2 (x 2 , y 2 ) , and p 3 (x 3 , y 3 ) , respectively. The lines l1 , l 2 , and l3 do not always cross at asingle point because of quantization errors. We first determine a 3D point T ′ by minimizing the sum of distances between the point T ′ and the lines l1 , l 2 , andl 3 . We then determine the points T1 , T2 , and T3 which
are the nearest to the point T ′ on the lines l1 , l 2 , andl 3 (see Fig.4(b)). Finally, we compute the 3D point
T ( X,Y,Z ) by solving the following equations as shown inFig.4(c).
Pv = w1 P1 + w2 P2 + w3 P3 ,w1 + w2 + w3 = 1,(3)T = w1T1 + w2T2 + w3T3 .
Note that points P1 ( X 1 ,Y1 ) , P2 ( X 2 ,Y2 ) , and P3 ( X 3 ,Y3 )represent omnidirectional camera positions and point
Pv ( X v , Yv ) is a novel viewpoint. Parameters w1 , w2

and w3 are weights to generate the same image as thecaptured image when the viewpoint Pv ( X v , Yv ) is at oneof the original camera positions. The point p v (x v , y v ) in
the omnidirectional image at the novel viewpoint can beobtained from the point T ( X,Y,Z ) using Eq. (2).

Input image 3l2

l1

p3 ( x3 , y3 )

l3

T3l2

T2 l1

P3 ( X 3 , Y3 )p 2 ( x2 , y 2 )yl3

p1 ( x1 , y1 )P2 ( X 2 , Y2 )T′T1

P1 ( X 1 , Y1 )Input image 1x

Input image 2(a)

(b)

T3

T

l2

l3

w3 T2 l1
w2
w1
T1(c)
Fig.4 Estimation of 3D point T from points p1 , p2 ,and p3 ; (a) Relationship between omnidirectional camera positions and estimated point T ′ , (b) Enlarged version of (a), (c) 3D point T estimated based on weights
w1 , w2 , and w3 .

2.4. Generating omnidirectional image at arbitrary viewpoint
We generate triangulated patches in a novel viewpoint
omnidirectional image based on two-dimensional coordinates of computed points p v using Delaunay’s triangulation [5]. It should be noted that these triangle patches
are generated supposing that the novel viewpoint is located at the center of triangle constructed of viewpoints ofinput omnidirectional images. This reduces the discontinuity in view changes when the novel viewpoint moves
continuously. The omnidirectional image at the novel
viewpoint is generated by blending the parts of input
omnidirectional images which correspond to the generated triangles. The blending is achieved by computing theweights w1 , w2 , and w3 at each viewpoint. A viewdependent common perspective image can be easily generated from the blended omnidirectional image usingEq.(2) [6].

3. Experiments
We took four omnidirectional images (1024x768 pixels) using the HyperOmni Vision composed of a digitalstill camera (Nikon CoolPix 990). Omnidirectional cam-era positions are illustrated in Fig.5. Figure 5(a) shows aroom to be captured, and Figure 5(b) shows the enlarged
version of the area containing camera positions. We divided the area into triangles (I,II,IV) and (I,III,IV). Next,
we manually selected 120 points such as corners of objects in each omnidirectional image as corresponding
points. In the experiment, the viewpoint is assumed tomove from the point A through E.
Figure 6(a) shows a sampled sequence of generated
omnidirectional images at the points from A to E. User’sview-dependent perspective images are also shown in
Fig.6, in which stereoscopic images are generated at thepoints B, C, and D. It took about 0.1 second to generateeach view-dependent common perspective image fromthree captured omnidirectional images using a PC (CPU:Dual Pentium4 Xeon 1.7GHz, Mem: 2GB RIMM). Therefore, we can confirm that our method realized the interactive walkthrough.
However, we have encountered some problems in theexperiments; for example, objects close to the viewpoint
are not correctly reconstructed, and the discontinuouschange is observed at the point C.
omnidirectional camera positions
monitorⅠⅡⅢ

ⅣA
B
C

bookshelfPC(a)
Fig.5

1m

DⅢE
EntranceⅡⅠ1m

Ⅳ(b)
Experimental environment.

4. Conclusions
We have proposed a new method of interactive walkthrough which takes the omnidirectional images at discrete positions and enables us to look around at arbitrary
viewpoints. Our method uses positions of HyperOmni
Vision, omnidirectional images and partial correspondences among omnidirectional images as input data. The
omnidirectional images at arbitrary viewpoints are generated by blending captured images. User’s view-dependentcommon perspective images computed from generated
omnidirectional images are presented to the user. The
proposed method is characterized by easy acquisition ofimages and capability of looking around the virtualized
scene at arbitrary viewpoints. Automatic finding of correspondences among original omnidirectional images
should be investigated in future work.

References[1] D. G. Aliga and I. Carlbom. Plenoptic Stitching: A Scalable
Method for Reconstructing 3D Interactive Walkthroughs. In
SIGGRAPH 2001, pages 443-450, 2001.
[2] S. E. Chen. QuickTime VR - An Image-based Approach toVisual Environment Navigation. In SIGGRAPH 1995,
pages 29-38, 1995.
[3] S. E. Chen and L. Williams. View Interpolation for Image
Synthesis. In SIGGRAPH 1993, pages 279-288, 1993.
[4] T. Endo, A. Katayama, H. Tamura, M. Hirose, T. Tanikawa, and M. Saito. Image-based Walk-through System forLarge-scale Scenes. In Proc. Virtual Systems and Multimedia, Vol. 1, pages 269-274, 1998.
[5] P. Heckbert Ed. Graphics Gems IV, pages 47-59. AcademicPress, 1994.
[6] Y. Onoe, K. Yamazawa, H. Takemura, and N. Yokoya.
Telepresence by Real-Time View-Dependent Image Generation from Omnidirectional Video Streams. Computer Vi-sion and Image Understanding, Vol. 71, No.2, pages 154165, 1998.
[7] J. Shimamura, H. Takemura, N. Yokoya, and K. Yamazawa. Construction and Presentation of a Virtual Environment Using Panoramic Stereo Images of a Real Scene andComputer Graphics Models. In Proc. Int. Conf. on PatternRecognition, volume Vol. 4, pages 463-467, 2000.
[8] T. Takahashi, H. Kawasaki, K. Ikeuchi, and M. Sakauchi.
Arbitrary View Position and Direction Rendering for LargeScale Scenes. In Proc. Conf. on Computer Vision and Pattern Recognition, Vol. 2, pages 296-303, 2000.
[9] K. Yamaguchi, H. Takemura, K. Yamazawa, and N. Yokoya. Real-time Generation and Presentation of Viewdependent Binocular Stereo Images Using a Sequence ofOmnidirectional Images. In Proc. Int. Conf. on Pattern Recognition, Vol. 4, pages 589-593, 2000
[10] K. Yamazawa, Y. Yagi, and M. Yachida. Obstacle Detection with Omnidirectional Image Sensor HyperOmni Vision. In Proc. Int. Conf. on Robotics and Automation, pages
1062-1067, 1995.

AB

C

D

E
(a)

(b)

(c)

Fig.6 Experimental results with an indoor scene; (a) Generated omnidirectional images, (b) Generated common perspective images of front view, (c) Generated common perspective images of right side view.