Panoramic Movie Generation Using
an Omnidirectional Multi-camera System
for Telepresence
Sei IKEDA, Tomokazu SATO, and Naokazu YOKOYA

Graduate School of Information Science, Nara Institute of Science and Technology,
8916-5 Takayama, Ikoma, 630-0101 Nara, Japanfsei-i, tomoka-s, yokoyag@is.aist-nara.ac.jp
http://yokoya.aist-nara.ac.jp/~sei-i/research/

Abstract. Telepresence systems using an omnidirectional image sensor

enable us to experience remote sites with rich sensation. An omnidirectional multi-camera system is more useful to acquire high-resolution omnidirectional images of outdoor scenes than a monocular camera system.
However, exact calibration of the camera system is necessary to generatea panoramic movie. In this paper, we describe a panoramic movie generation method based on geometric and photometric calibration of theomnidirectional multi-camera system. Some prototypes of telepresence
system are also shown as applications of generated panoramic movies.
These high-resolution telepresence systems have been proven to enable
us to experience remote sites with high presence.
1 Introduction
Telepresence systems using an omnidirectional image sensor enable us to experience remote sites with rich sensation. These systems are expected to be used in
the elds of education and entertainment. A curved mirror [1] and a sh eye lens[2] have been used for these telepresence systems because they can be attached
to a standard camera. However, such methods have some problems concerning
the shortage and position-dependency of image resolution, because these sensors
capture an omnidirectional scene by a single camera. Though a higher resolution telepresence system is also implemented by using an HD camera and acurved mirror [3], it does not solve the problem of position-dependency of imageresolution.
On the other hand, an omnidirectional multi-camera system can obtain ahigh-resolution image with almost uniform resolution. However, exact camera
calibration is necessary to generate a panoramic image from multiple images
acquired by such a system.
In this paper, we propose a camera calibration and panoramic movie generation method for the omnidirectional multi-camera system \Ladybug". In experiments, Ladybug is calibrated and a panoramic movie is generated based on
the proposed methods. As applications of the generated panoramic movie, three
prototypes of telepresence system are also demonstrated.

2 Camera Calibration for Omnidirectional Multi-camera
SystemThis section describes a method of geometric and photometric calibration foran omnidirectional multi-camera system[4]. As an omnidirectional multi-camera
system, we use Ladybug made by Point Grey Research Inc. shown in Figure 1.
The camera unit consists of six cameras (Figure 1 (left)): Five radially con gured on a horizontal ring and one pointing vertically. Figure 1 (right) shows astorage unit, which consists of four hard disks. The camera system can collectsynchronized movies at 15fps covering more than 75% of the full spherical view
with almost the same apparent point of view.

2.1 Geometric Calibration
In the geometric calibration, intrinsic parameters and extrinsic parameters ofeach camera should be estimated. The extrinsic parameters are estimated in auni ed coordinate system to obtain the relationship among six cameras. In orderto estimate camera parameters accurately, spatial arrangement of many markers
of known 3D position is required.
First, arrangement of markers are described. In our method, grid points ofa lattice pattern on a calibration board are used as markers. 3D coordinates
of four corners of the calibration board are measured by a totalstation and all
3D positions of the markers on the board are calculated by linear interpolation
among its corners. The calibration board is arranged and measured at several
di erent positions for acquiring a large number of makers' 3D positions and 2D
positions on images as shown in Figure 2. In our method, the totalstation must
be xed on the same position while the whole geometric calibration procedure isperformed. Consequently, 3D positions of all markers captured by all the cameras
can be represented in a single coordinate system made by the totalstation.
Next, the camera parameters of each camera are estimated. The intrinsicparameters (focal length, lens distortion parameters, center of distortion, aspectratio) are estimated by using the Tsai's method[5]. Note that the distortion
parameter 2 in Tsai's method is also considered because input images from thecameras are highly distorted.
In this paper, extrinsic parameters (position, orientation) of each camerac (c = 1; 2; :::; 6) are estimated by minimizing the sum of re-projection error
Ec = m (um 0 vm )2 [6], which is the sum of squared distances on the image
between the projected position vm of measured 3D position and the detected
2D position um of the marker m.

P

2.2 Photometric Calibration
In the photometric calibration, the limb darkening and color balances are corrected. First, the correction of the limb darkening is described. The limb darkening is a gradually decreasing e ect of brightness in peripheral regions in images

zyCalibration
BoardLadybug
Totalstation

xFig. 1. Camera head (left) and storage unit Fig. 2. Arrangement of calibration(right) of Ladybug
board in spaceacquired by a wide-angle camera. It is known that the cos4  phenomenon isproper to a wide-angle lens. Thus we treat the cos4  phenomenon and correct
image intensities using the estimated focal length and a lens diameter of a design
parameter of a camera [7].
Next, color balances among cameras are corrected by capturing the same
scene by all the cameras. Generally it is known that the irradiance of the image and the radiance of the surface have a linear relation. Therefore, the color
balances between the cameras are linearly adjusted. The linear transformation
parameters of colors are determined by equalizing the RGB histograms of all theimages.

3 Generating Omnidirectional Panoramic Movies
In generation of a panoramic image, the limb darkening and the color balance of
input images are corrected in advance. Then, corrected images are projected on
a spherical or cylindrical projection surface by using the intrinsic and extrinsic
parameters obtained by the geometric calibration. In this section we describe aprojection method of the corrected images.
The method is based on re-projecting calibrated input images on a sphericalor cylindrical virtual image surface. Since the centers of projection of multiple cameras of the omnidirectional multi-camera system are di erent from eachother, the single viewpoint perspective projection model is not applicable for thissystem. However, when the distance of a target from the system is suciently
large, the centers of projection can be considered as the same. Therefore, weassume that the target scene is far enough from the system and set the sphericalor cylindrical projection surface S far enough from the camera system. The center of the projection surface S is set at the center of gravity of all the camera's
center of projection. A panoramic image is generated by projecting all the pixelsof all the images onto the projection surface S . Note that a blending technique isused for generating a smooth panoramic image, when a point on the projection
surface S is projected from multiple images of di erent cameras.

4 Experiments
We have calibrated the Ladybug system and have generated a panoramic movie.
This section describes some experimental results concerning geometric and photometric calibration and panoramic movie generation. Three prototypes of telepresence system are also developed as applications of high-resolution panoramicmovies.

4.1 Generation of Panoramic movies(1) Calibration of Ladybug
In the geometric calibration, 3D positions of grid points on the calibration board
were measured by a totalstation LEICA TCR1105 XR. The 2D positions on input
images are detected at sub-pixel resolution from the calibration board images
captured by each camera of Ladybug xed on a tripod. The calibration board
was arranged and captured at di erent depth from the xed camera system threetimes. Consequently, more than 500 pairs of 3D and 2D positions of grid points
were obtained and used for calibration for each camera.
Figure 3 shows the result of reducing geometric distortion of an input image using the estimated intrinsic parameters. We can con rm that the intrinsic
parameters are fairly good. Figure 4 shows the result of correcting the limb darkening. The left is an input image. The right is a corrected image for the shortageof brightness in peripheral regions of the input image. Figure 5 shows a partof panoramic images before and after the color balance adjusting. In the rightimage, the boundary lines do not stand out compared with the non-corrected
image in the left.
(2) Generation of a Panoramic Movie
A panoramic movie was generated by using the result of calibration describedin the previous section. The input movie was obtained by setting Ladybug put
on a moving car. Figure 6 shows an example set of input images (resolution:76821024 each). A panoramic image (resolution: 384021920) generated fromthese input images is also shown in Figure 7. This panoramic image is unfoldedin polar coordinates from the spherical coordinate representation. A black partat the bottom of the panoramic image is the part of no input images. Since any
boundary lines between di erent camera images are hardly found in Figure 7,
we can con rm that the geometrical and photometrical calibration is successfully achieved. We can not perceive any boundary lines in the panoramic image
except objects very close to the camera system.

(a) input(b) corrected
Fig. 3. Lens distortion correction(a) input(b) corrected
Fig. 4. limb darkening correction(a) non-corrected (b) corrected
Fig. 5. Color adjustment
4.2 Prototype Telepresence Systems
This section brie y describes three prototype telepresence systems developed as
applications of high-resolution panoramic movies. Three telepresence systems areequipped with di erent immersive displays as shown in Figure 8: (a) a sphericaldisplay (Elumens VisionStation), (b) a face mounted display (Olympus EyeTrek) and (c) cylindrical display (CYLINDRA), which provide users di erent
usage and interaction. Our systems enable users to interactively view an omnidirectional scene by projecting a speci ed part of a panoramic movie in realtime.

(a) Spherical display: A user can observe almost the half of omnidirectional
panoramic view in the display as shown in Figure 8 (a). The direction ofthe view eld can be controlled by a game controller (Microsoft SideWinder
Game Pad Pro).
(b) Face mounted display: A user can view an omnidirectional view eld
naturally by turning his head as shown in Figure 8 (b). The direction ofuser's head is acquired by a magnetic sensor (Polhemus Fastrack): a source
located on a table and a receiver attached on the face mounted display.

Fig. 6. Input images: upper camera (right end) and horizontal cameras (others)

Fig. 7. One frame of spherical panoramic movie

(c) Cylindrical display: In this system, the full horizontal range of omnidirectional movie is projected to a large cylindrical screen by six projectors.
Multiple users can simultaneously view di erent directions as shown in Figure 8 (c).
The frame rate of displaying images is 15fps on all the systems, which is the sameas the image capturing rate by Ladybug. We have con rmed that our telepresence
systems exhibit high-presence of a remote site in respect of resolution, eld ofview, and interactive viewing.

5 Conclusion
In this paper, we have described a method for generating a high-resolution spherical panoramic movie based on the result of geometric and photometric calibration for an omnidirectional multi-camera system Ladybug. In the geometric calibration, a large number of markers were virtually arranged around the system
by using a calibration board and a totalstation for improving the accuracy ofcalibration. We have also prototyped three telepresence systems which provide
users with an omnidirectional eld of view. Experiments have exhibited that
telepresence systems using high-resolution panoramic movies provide us withthe feeling of high presence in remote sites. In future work, a better lens distortion model for wide-angle lens will be investigated for obtaining higher accuracy
in generation of panoramic movies.

References

1. Y. Onoe, K. Yamazawa, H. Takemura, N. Yokoya: Telepresence by Real-time Viewdependent Image Generation from Omnidirectional Video Streams. Computer Vision and Image Understanding 71(2):154{165, 1998
2. K. Miyamoto: Fish Eye Lens. Journal of Optical Society of America 54(8):1060{
1061, 1964
3. K. Yamazawa, H. Takemura, N. Yokoya: Telepresence system with an Omnidirectional HD Camera. Proc. 5th Asian Conf. on Computer Vision (ACCV2002)
2:535{538, 2002
4. S. Ikeda, T. Sato, N. Yokoya: A Calibration Method for an Omnidirectional Multicamera System. Proc. SPIE Electronic Imaging 5006, 2003
5. R. Y. Tsai: A Versatile Camera Calibration Technique for High-accuracy 3D Machine Vision Metrology Using O -the-shelf TV Cameras and Lenses. IEEE Journalof Robotics and Automation RA-3(4):323{344, 1987
6. T. Sato, M. Kanbara, N. Yokoya, H. Takemura: Dense 3D Reconstruction of anOutdoor Scene by Hundreds-Baseline Stereo Using a Hand-Held Video Camera.
Int. Journal of Computer Vision 47(1-3):110{129, 2002
7. B. K. P. Horn. In: Robot Vision. MIT Press :206{209, 1986

(a) spherical display(b) face mounted display(c) cylindrical display
Fig. 8. Appearance of telepresence systems