High-resolution Panoramic Movie Generation from Video Streams
Acquired by an Omnidirectional Multi-camera System
Sei IKEDA, Tomokazu SATO and Naokazu YOKOYA
Graduate School of Information Science,
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, 630-0192 Japan
Email: {sei-i, tomoka-s, yokoya}@is.aist-nara.ac.jp
Abstract
Telepresence systems using an omnidirectional image sensor enable us to experience remote sites with rich sensation. An omnidirectional multi-camera system is more useful to acquire high-resolution omnidirectional images of outdoor scenes than a monocular camera system. However, exact calibration of the camera system is necessary to generatea panoramic movie. In this paper, we describe a panoramic
movie generation method based on geometric and photometric calibration of the omnidirectional multi-camera system.
A prototype of telepresence system is also shown as applications of generated panoramic movies. This high-resolution
telepresence system has been proven to enable us to experience remote sites with rich presence.

1

Introduction

Telepresence systems using an omnidirectional image sensor enable us to experience remote site. These systems are
expected to be used in the fields of education and entertainment. A curved mirror [6] and a fish eye lens [7] have been
widely used for these telepresence systems because they canbe attached to a standard camera. However, these methods
have problems of shortage and position-dependency of image
resolution, because these sensors capture an omnidirectional
scene by a single camera.
On the other hand, an omnidirectional multi-camera system can obtain a high-resolution image with uniform resolution, although exact camera calibration is necessary to generate a panoramic image from multiple images accurately. In
related work, some calibration methods for omnidirectional
multi-camera systems are proposed, which are based on using markers of known 3D positions[2]. In these methods,
grid points of iterative patterns are used for markers and areprojected on a large-sized immersive projection screen. However, it is difficult to obtain highly accurate calibration using
such approaches, because the distribution of the markers islimited on a projection screen. Further more, there is no discussion about the precision of generated panoramic images.

In this paper, we propose a camera calibration andpanoramic movie generation method for the omnidirectional
multi-camera system “Ladybug”. Figure 1 shows the multicamera system Ladybug made by Point Grey Research Inc.
The camera unit consists of six cameras (Figure 1 (left)): Fiveconfigured in a horizontal ring and one pointing vertically.
Figure 1 (right) shows a storage unit, which consists of four
hard disks. The camera system can collect movies covering
more than 75% of the full spherical view with almost the same
apparent point of view.
This paper is constructed as follows. In Section 2, we describe a method of geometric and photometric calibration forthe omnidirectional multi-camera system Ladybug. Section
3 describes a method of generating a spherical panoramic
movie using the calibration result. The effect of disparity
among multiple cameras is also discussed. In Section 4,
the Ladybug camera system is actually calibrated geometrically and photometrically by using the proposed method. The
spherical panoramic movie composed of six input images isalso computed. A telepresence system is also prototyped inorder to confirm that the panoramic movie can be used fortelepresence well. In addition, we evaluate the discontinuity
in computed panoramic images.

2

Camera Calibration for Omnidirectional
Multi-camera System

This section describes a method of geometric and photometric calibration for an omnidirectional multi-camera system. In the geometric calibration, intrinsic and extrinsic parameters of each camera are estimated using a calibration
board and a laser measurement system called total station.
In the photometric calibration, limb darkening and color balances among multiple cameras are corrected.

2.1

Geometric Calibration

In the geometric calibration, intrinsic parameters (focal
length f, lens distortion parameters (κ1 , κ2 ), center of distortion (Cx , Cy ), aspect ratio sx ) and extrinsic parameters (position Tc , orientation Rc ) of each camera c should be esti-zyCalibration
Board

Ladybug
Figure 1. Camera head (left) and storage unit
(right) of Ladybug.

Totalstationx

Figure 2. Arrangement of calibration board inspace.
mated. The extrinsic parameters are estimated in a unifiedcoordinate system to obtain the relationship among six cameras.
In camera calibration, spatial arrangement of many markers of known 3D position is required to estimate camera parameters accurately. First, arrangement of markers are described. In our method, grid points of a lattice pattern ona calibration board are used as markers. 3D coordinates offour corners of the calibration board are measured by the total station and all 3D positions of the markers on the board are
calculated by linear interpolation among its corners. The calibration board is arranged and measured at several different
positions for acquiring a large number of makers’ 3D positions and 2D positions on images as shown in Figure 2. In
our method, the total station must be fixed on the same position while the whole geometric calibration procedure is performed. Consequently, 3D positions of all markers captured
by all the cameras can be represented in a single coordinate
system made by the total station. Grid points on images aredetected at sub-pixel resolution by calculating intersection of
two lines.
Next, the camera parameters of each camera are estimated. The intrinsic parameters are estimated by Tsai’s
method[4]. In Tsai’s method, the undistorted image coordinates (Xu , Yu ) of markers are transformed to the distorted
coordinates (Xd , Yd ) by using following equations:

In this paper, extrinsic parameters of each camera are represented as a world-to-camera transformation matrix Mc using position Tc (t1 , t2 , t3 ) and orientation Rc (r1 , r2 , r3 ) of
each camera c (c = 1, 2, ..., 6) on the world coordinate system:


Rc Tc
,
(5)
Mc =
0
1


c1 c3 + s1 s2 s3 s1 c2 −c1 s3 + s1 s2 c3 t1
 −s1 c3 + c1 s2 s3 c1 c2 s1 s3 + c1 s2 c3 t2  (,6)
=

c2 s3
−s2
c2 c3
t3 
0
0
0
1
where,
s1 = sin r1 , s2 = sin r2 , s3 = sin r3 ,c1 = cos r1 , c2 = cos r2 , c3 = cos r3 .

(7)

Xu = sx (Xd − Cx )(1 + κ1 r 2 + κ2 r 4 ),(1)

Yu = (Yd − Cy )(1 + κ1 r 2 + κ2 r 4 ),
r = Xd2 + Yd2 .

(2)

Estimating Mc for six parameters (t1 , t2 , t3 , r1, r2 , r3 ) is a
non-liner minimization problem. Since there exist problems
concerning calculation cost and local minima, we first estimate Mc for twelve parameters by minimizing Ec linearly.
Then, the estimated camera parameter Mc is linearly adjusted
to reduce the degree of freedom to six by assuming that the direction of optical axis is correctly estimated. Finally, the sumof re-projection error Ec is minimized by a gradient method
for optimizing Mc .

(3)

2.2

Note that the distortion parameter κ2 is also considered because input images from the cameras are highly distorted. The
extrinsic parameters of each camera are estimated by minimizing the sum of re-projection errors Ec of makers[5]. The
re-projection error of a marker m is defined as a squared distance on the image between the projected position vm of measured 3D position and the detected 2D position um of themarker m. The measure Ec is computed by:Ec =(um − vm )2 .
(4)

Photometric Calibration

In the photometric calibration, first the limb darkening andcolor balances are corrected. First, the correction of the limb
darkening is described. The limb darkening is a gradually decreasing effect of brightness in peripheral regions in images.
Generally, images from an omnidirectional multi-camera system suffer from the limb darkening effect problem because
wide-angle lenses are equipped. The cos4 θ phenomenon [1]and the vignetting [3] are well known as limb darkening effects. It is known that the cos4 θ phenomenon is proper to awide-angle lens and the vignetting is proper to a zoom lens.

m2

input images

Thus we only treat the cos4 θ phenomenon and correct image
intensities using the following equation.
πl2 cos4 θ
I =
I,
f2


Generating
Movie

Omnidirectional(8)

camera parameters

panoramic image
Figure 3. Flow diagram of generating apanoramic image.

when a point s on the spherical surface S is projected fromimage surfaces of different cameras. The intensity IS (s) of
the point s is determined by the following equation.
c∈C (s) γc Ic (uc ),(10)
IS (s) =c∈C (s) γc
where uc is the point in the image of the camera c which is
projected on the point s in the spherical surface S, γ is the
distance between uc and border of the image c, and C(s) is aset of cameras to which the point s in the S can be projected.

Panorama

3.2This section describes a method of generating omnidirectional panoramic images. The method is based on reprojecting calibrated input images on a spherical virtual image surface. We also discuss the effect of disparity between
multiple cameras which do not satisfy the single viewpoint
constraint.

3.1

distortion correction &image projection to S

limb darkening correction

where l is a lens diameter, f is a focal length, I is a radianceof surface of an object in the direction toward the lens, I  is anirradiance of the image and θ is an angle between the incident
light and the optical axis.
Next, we describe a method for correcting color balancesamong cameras. Generally it is known that the irradiance of
the image I  and the radiance of the surface I have a linear relation shown as I  = aI + b. The color balances are different
due to the difference in color transform parameters a and b
among cameras. In this paper, the color balance between two
cameras c and c is adjusted by estimating parameters ac andbc in the linear transformation Ic = ac Ic + bc. The parameters ac and bc are determined by equalizing the normalized
RGB histograms hc (i) and hc (i) where i is intensity of animage pixel. The following error function e(ac , bc) is minimized for equalizing the histograms hc (i) and hc (i):2i − bc1e(ac , bc) =hc (i) − hc
.
(9)
ac
ac
i

3

color balance correction

calibration results
focal length &lens diametercolor balanceparameters

Effect of Disparity

We discuss the effect of disparity in camera position. As
shown in Figure 4, when the target x is captured at uc anduc on the images of the two cameras c and c , they are projected to different position sc and sc on the spherical surface
S in the proposed method. Since sc and sc are not consistent, a panoramic image doubly shifted is generated. If thecircumference of the spherical surface S is composed of N
pixels, the effect of disparity is limited within one pixel if thecondition  sc Gsc < 2π
N is satisfied. When the radius of thespherical surface S is assumed to be infinite,  sc Gsc is approximated by  sc xsc < 2π
. Consequently, the conditionN sc Gsc < 2π is approximated by  Tc xTc < 2π .
NNWhen we assume that the distance between the two cameras is d and the distance from the each camera to the target
x is the same, the condition  Tc xTc < 2π
N is simplified thatthe distance λ of the target x from the center point of the twocamera is represented as follows:

Generation of a Panorama Movie

In generation of a panoramic image, limb darkening andthe color balance of input images are corrected as a pretreatment, as shown in Figure 3. Then, corrected images are projected on a spherical projection surface by using the intrinsic
and the extrinsic parameters of the geometric calibration result. In this section we describe a projection method of thecorrected images.
Since the centers of projection of multiple cameras of theomnidirectional multi-camera system are different from eachother, the single viewpoint perspective projection model is notapplicable for this system. However, when the distance of atarget from the system is sufficiently large, the centers of projection can be considered as the same. Therefore, we assume
that the target scene is far enough from the system and set thespherical projection surface S far enough from the camera
system. The center of the spherical projection surface is set ata center of gravity of all the cameras’ center of projection. A
panoramic image is generated by projecting all the pixels of
all the images onto the spherical surface S. Note that a blending method is used for generating a smooth panoramic image,

λ>

4

d
π .
2 tan N

(11)

Experiment

We have calibrated Ladybug system and have generated apanoramic movie. This section describes some experimental results concerning geometric and photometric calibration,
panoramic movie generation, and telepresence application.
3

Spherical projection surface
Camera

S

sc ′
sc

x

c

Tc

uc

G

Center of gravity

uc′

Tc′
(a) input image

Camerac′(b) corrected image

Figure 5. Lens distortion correction.

Figure 4. Effect of disparity.

4.1

Calibration of Ladybug

In the geometric calibration, the calibration board was captured by each camera of Ladybug fixed on a tripod. The calibration board was arranged and captured at different depth
from fixed cameras three times. For each of five horizontal
cameras, 187 grid points on the calibration board were used,then totally 561 set of 3D position and 2D position on captured images were obtained. For a vertical camera, 170 grid
points on the calibration board were used, then totally 510
set of 3D position and 2D position on captured images wereobtained. The 3D position was measured by a total station
LEICA TCR1105 XR.
Figure 5 shows the result of reducing distortion of an input image using the estimated intrinsic parameters. We canconfirm that the intrinsic parameters are fairly good. Figure 6
shows the result of correcting the limb darkening. The left is
an input image. The right is a corrected image for the shortageof brightness in peripheral regions of the input image. Figure
7 shows a part of panoramic images before and after the color
balance adjusting. In the right image, the boundary lines do
not stand out compared with the non-corrected image in the
left.

4.2

(a) input image(b) corrected image

Figure 6. Limb darkening correction.

(a) non-corrected image(b) corrected image

Figure 7. Color adjustment.

Generation of a Panorama Movie

A panoramic movie was generated by using the resultof calibration described in the previous section. The input
movie was obtained by setting Ladybug put on a moving car.
Figure 8 shows an example set of input images (resolution:
768×1024). A panoramic image (resolution: 3840×1920)
generated from these input images is also shown in Figure
9. This panoramic image is unfolded by polar coordinatesfrom the spherical surface. A black part at the bottom of thepanoramic image is the part of no input images. Since any
boundary lines are hardly found in Figure 9, we can confirm
that geometrical and photometrical calibration is successfully
achieved.

Figure 10 gives a panoramic image generated without theblend method blending for comparing it with the blended one.
We can find some boundaries in the non-blended panoramic
image; close objects and sky region containing saturated pixels due to direct rays of sun. On the other hand, we can not
perceive any boundary lines in the blended panoramic image
except very close objects.

4

1

2

3

4

5

6

Figure 8. Input images: upper camera (right) and horizontal camera (others).

(a) blending(b) no blending

Figure 10. Blending effect.
Figure 9. One frame of panoramic movie.

4.3

Prototype Telepresence System

A telepresence system was prototyped in order to confirm
that the panoramic movie can be used for telepresence application well. The system is composed of a spherical immersive display, a controller and a personal computer as shownin Table 1. Our system enables users to interactively view anomnidirectional view field scene projecting a specified partof a panoramic movie in real time. The system performanceof generating images is 20fps, which is faster than the performance 15fps of image capturing by Ladybug. Due to theresolution limitation 1024×768 of the immersive display, theresolution of a panoramic movie was resized to 2048×1024 in
the experiment. Figure 11 shows user’s appearance in telepresence experiment. We have confirmed that our telepresence
system exhibits high-presence in respect of resolution, fieldof view, and interaction.

Figure 11. Appearance of telepresence system.

4.4

Quantitative Evaluation

We have evaluated errors in panoramic image generation.
The error is defined as an angle between two lines connecting
the spherical projection center G and the two corresponding
projected points of the same point in space on the sphericalprojection surface.
There are common regions in input images of adjacentcameras of Ladybug as shown in Figure 12. A circle marker iscaptured in these regions to specify the corresponding pointsof adjacent two camera images. The center of gravity of thisTable 1. Components of telepresence system.
Spherical Display Elumens VisionStaion
Controller
Microsoft SideWinder Game Pad
Pro
PC
CPU:Intel Pentium4 1.7GHz,
Memory:1GB
Graphics Card
Nvidia Geforce4

5

around the system by using a calibration board and a total
station for improving the accuracy of calibration. We havealso prototyped a telepresence system having omnidirectionalview of field. Experiments have exhibited that panoramic
movies can be used for telepresence application. Finally, projected errors generating a panoramic image was evaluated.
We have found that the average of the error is about 3 pixels on an input image. In future work, a better lens distortion
model for wide-angle lens will be investigated for obtaining
higher accuracy in generation of panoramic movie.
Figure 12. Common regions in input images.

References[1] B. K. P. Horn. Robot Vision :206–209. Mit Press, 1986.
[2] J. Shimamura, H. Takemura, N. Yokoya, and K. Yamazawa.
Construction of an Immersive Mixed Environment Using an
Omnidirectional Stereo Image Sensor. Proc. IEEE Workshop
on Omnidirectional Vision :62–69, 2000.
[3] N. Asada, A. Amano, and M. Baba. Photometric Calibrationof Zoom Lens Systems. Proc. Int. Conf. Pattern Recognition,
A:186–190, 1996.
[4] R. Y. Tsai. A Versatile Camera Calibration Technique for Highaccuracy 3D Machine Vision Metrology Using Off-the-shelf TV
Cameras and Lenses. IEEE Journal of Robotics and Automation, RA-3(4):323–344, 1987.
[5] T. Sato, M. Kanbara, N. Yokoya, and H. Takemura. Dense
3D Reconstruction of an Outdoor Scene by Hundreds-Baseline
Stereo Using a Hand-Held Video Camera. Int. Journal of Computer Vision, 47(1-3):110–129, 2002.
[6] Y. Onoe, K. Yamazawa, H. Takemura, and N. Yokoya. Telepresence by Real-time View-dependent Image Generation from
Omnidirectional Video Streams. Computer Vision and Image
Understanding, 71(2):154–165, 1998.
[7] Z. L. Cao, S. J. Oh, and E. L. Hall. Dynamic Omnidirectional
Vision for Mobile Robots. Journal Robotic Systems, 3(1):5–17,
1986.

marker is calculated with sub-pixel accuracy. In this experiment, the angle  sc Gsc shown in Figure 4 was evaluated as
error , where uc and uc are centers of gravity of these markers. Because the distance decreasing the effect of disparity
within one pixel on the panoramic image is about 24m underassumptions of d = 4cm and N = 3840, the marker was located about 30m away from Ladybug. More than one hundred
corresponding points were obtained for each camera.
Table 2 shows maximum and average values of error  ingeneration a panoramic image. The number 6 represents thevertical camera and the others represent horizontal cameras.
The average value of the error  is 0.0063rad which is approximately equivalent to 3 pixels on an input image. We have also
found that points having large errors tend to distribute at theborders of input images.

5

Conclusion

In this paper, we have described a method of generating ahigh-resolution spherical panoramic movie based on the result of geometric and photometric calibration for an omnidirectional multi-camera system Ladybug. In the geometric calibration, a large number of markers were virtually arranged

Table 2. Error  of a generated panoramic image[rad].
Camera Number
1-2
2-3
3-4
4-5
Number of the data
227
258
229
216
Maximum value
0.0214 0.0378 0.0258 0.0227
Average value
0.0072 0.0057 0.0054 0.0048
Camera Number
Number of the data
Maximum value
Average value

5-1
305
0.01980.00416-1
176
0.01780.0058

Camera Number
Number of the data
Maximum value
Average value

6-4
123
0.01070.00636-5
154
0.01120.00526-2
131
0.04980.01446-3
1270.02080.00956