THE INSTITUTE OF ELECTRONICS,
INFORMATION AND COMMUNICATION ENGINEERSTECHNICAL REPORT OF IEICE.

Camera Recovery of an Omnidirectional Multi-camera System
Using GPS Positions
Sei IKEDA† , Tomokazu SATO† , and Naokazu YOKOYA†† Graduate School of Information Science, Nara Institute of Science and Technology
Takayama 8916–5, Ikoma, Nara, 630–0192, Japan
E-mail: †{sei-i,tomoka-s,yokoya}@is.naist.jp
Abstract This paper describes a novel method for estimating positions and postures of an omnidirectional multi–
camera system from its multiple image sequences and sparse position data acquired by GPS. The proposed method
is based on a structure-from-motion technique which is enhanced by using multiple image sequences as well as GPS
position data. Moreover, the position data are also used to remove mis-tracked features. The proposed method
allows us to estimate position and posture without accumulative errors and annoying eﬀects due to moving objects
in outdoor environments. The validity of the method is demonstrated through experiments using both synthetic
and real outdoor scenes.
Key words Camera Recovery, Omnidirectional Multi-camera System, Global Positioning System,

1. Introduction

data without accumulative errors. Highly accurate calibration and synchronization among sensors is needed but thisThis paper addresses a problem of estimating position andproblem is hardly treated. Giving a kind of prior knowl-

posture of an omnidirectional multi-camera system (OMS)

edge also resolves the accumulative error problem. As priorfrom its multiple image sequences and GPS position data.

knowledge about surroundings, manually selected known 3D-

This problem is especially important in acquisition of im-

positions [7], [8] (called feature landmarks) and manually cre-ages of outdoor environments and its geometric information.

ated 3D models [9], [10] are sometimes used in addition to fea-

A typical instance is construction of a virtual reality envi-

ture tracking. However, these are hardly applicable to largeronment using real images covering user’s ﬁeld of view [1]〜scale environments because much cost is required to obtain[3]. In this application, precise estimation of position andthis information.

posture of the camera is required to show a user virtual ob-

The most hopeful solution for the accumulative error prob-

jects superimposed seamlessly on presented real images in

lem is a combination of images and GPS positions [11]〜the environment. To use real images acquired outdoors for[13]. In this paper, we propose a camera recovery method

such kind of applications, camera recovery algorithm should

for an omnidirectional multi-camera system (OMS) using

possess two characteristics : (i) errors in estimating positions

both video sequences and GPS position data. The proposed

and postures are not accumulated; (ii) the algorithm is strong

method extends our previous works [11] for OMS whose ﬁeld

for non-rigid motion of video such as moving objects and in-of view is wide and is hardly ﬁlled with moving objects.

cidence of light of the sun. Camera recovery problem can

Our method is based on structure-from-motion with featurebe theoretically solved only from image information by using

tracking and parameter optimization using GPS positions

feature tracking based methods [4], [5]. However, algorithms

and video frames. In feature tracking, tentative parameters

using only images acquired with a general single camera do

are estimated from GPS position data. They are used to

not satisfy the characteristics above. Some kinds of sensors

avoid mismatching and to obtain correspondences of featureor some kinds of prior knowledge about surroundings should

points among diﬀerent cameras. In the optimization, a new

be used in order to satisfy them.

error function deﬁned by using GPS position data and re-

The simplest method is to measure the required informa-projection error is minimized to determine position and pos-tion directly by using any appropriate sensors. Guven [6] in-ture parameters of the OMS. In our method, the following

tegrated an RTK-GPS (Real Time Kinematic GPS), a mag-conditions are assumed: (i) OMS and GPS are correctly syn-netometer and a gyro sensor to obtain position and posturechronized; (ii) the geometrical relation between all the cam-

eras and the GPS receiver is always ﬁxed; (iii) the distance
between the GPS receiver and the representative camera of
the OMS is known, and the direction of GPS receiver in camera coordinate system is unknown. In this paper, it is also
assumed that OMS has been calibrated in advance and the
intrinsic camera parameters (including lens distortion, focal
length and aspect ratio) of each element camera are known.
In the remainder of this paper, we ﬁrst formulate the camera recovery problem of an OMS using GPS positions in
Section 2. The implementation of the proposed method isthen described in Section 3. In Section 4, the validity of themethod is demonstrated through experiments for both synthetic and real outdoor scenes. Finally, we give conclusion
Figure 1 A sampled frame of an acquired omnidirectional video.

and future work in Section 5.

Right bottom is an image of vertical element camera.

2. Formulation of Camera Recovery of
Omnidirectional Multi-camera System

Others are horizontal ones.

[

]
Ric

tic

parameters of an OMS and a direction of GPS receiver from0

1

camera when multiple video frames and GPS positions arewhere tic and Ric represent the translation and the rotation

The goal of this study is to obtain position and posture−1

Nic = Mc (M0 )

Ni0 =

,

(1)

given. The main topic described in this section is how tofrom the world coordinate system of the i-th frame to theintegrate GPS position data to the structure-from-motion

camera coordinate system of the camera c. This problem

problem. In the proposed method, the general structure-is treated as estimation of position (Ri = Ri0 ) and posturefrom-motion algorithm is enhanced to treat GPS position(ti = ti0 ) of the representative camera (c=0).

information.
In the general structure-from-motion algorithm,

2. 2 Error Function for Optimization Processre-projection error that is observation error is minimized toobtain the parameters. First, we make it clear what theparameters are. Second, as one of observation errors in our
problem, the re-projection error is brieﬂy explained. The error concerning GPS, which is another observation error, isthen modeled by using geometric relation between camera
and GPS. Finally, we describe a new error function combining re-projection error and the error function concerning

Re-projection Error
Re-projection error is generally used for camera recovery
based on feature tracking. The method for minimizing thesum of squared re-projection error is usually referred to as
bundle adjustment. The re-projection error Φij is deﬁned as|qij − q̂ij | for the feature j in the i-th frame, where q̂ represents the 2D projected position of the feature’s 3D positionand q represents the detected position of the feature in theimage.

GPS.
2. 1 Position and Posture Parameters of Omnidirectional Multi-camera System

Error of GPS positions
Generally, if GPS positions and estimated parameters do

Omnidirectional multi-camera system is constructed of anot contain any errors, the following equation is satisﬁed in

set of cameras such as Ladybug (Point Grey Research) which

the i-th frame among the parameters (position ti , posture

can obtain omnidirectional videos as shown in Figure 1. As

Ri ), GPS position gi and the position of GPS receiver d inmentioned in the previous section, we assume that position

the camera coordinate system.

and posture relations among element cameras are known andﬁxed in this paper. The positions and postures of all the cam-Ri gi + ti = d (i ∈ F ),(2)eras can be expressed as a pair of position and posture of awhere F denotes a set of frames in which GPS positions are

representative camera．In the i-th frame, the transformation

obtained. However, if GPS position gi and the parametersfrom the world coordinate system to the camera coordinate

ti and Ri contain some errors, we must introduce an error

system of each element camera c can be expressed by thevector ni .

following matrix Nic by using the transformation Mc fromthe world coordinate system of a calibration process to thecamera coordinate system of the camera c (= 0, 1, 2, 3...n).

Ri gi + ti = d + ni .

(3)

In this paper, we introduce an error function Ψi related toGPS receiver using the length of the error vector n: Ψi = |ni |.
This function means the distance between the measured position of the GPS receiver and the predicted one. Next, we
describe a new error function E which is a combination of
the error function Ψij related to GPS receiver and the reprojection error Φ.
Error Function Concerning Feature and GPS
The new error function E is deﬁned as follows:
E=

ω ∑ 2
Ψi
|F|
i∈F

∑∑

1

+∑ ∑
i

c

Figure 2 Overview of the proposed algorithm.

|Sic |

i

µi

c

∑

wj Φ2ijc ,matically detected by using the Harris operator for limiting
(4)

j∈Sic

where ω means a weight for Ψi , and Si denotes a set of featurefeature position candidates in the images. In the next process (b), every feature in the (i − 1)-th frame is tentatively
matched with a candidate feature point in the i-th frame bypoints detected in the i-th frame. The coeﬃcients µi and wj

using a standard template matching. In the third process (c),mean the conﬁdences for frame and feature, respectively. wj

tentative parameters are then estimated by selecting correct

represents the conﬁdence coeﬃcient of feature point j, whichmatches using a RANSAC approach [15]. In the ﬁnal process

is computed as an inverse variance of re-projection error Φij .

(d), every feature is re-tracked within a limited searching

The coeﬃcient µi denotes the conﬁdence of the i-th frame.

area in image frames of all the element cameras, which can

Two terms in the right-hand side in Eq. (4) is normalized

be computed by the tentative parameters and 3D positions

by |F | and∑ ∑i|Sic | so as to set ω as a constant valuecof the features.

independent of the number of features and GPS positioning(B) Initial parameter estimation :points.

This processes computes 3D positions of feature points and3. Implementation of Camera Recovery Method of Omnidirectional Multicamera System

position and posture parameters which minimize the sum ofsquared re-projection errors. In this process, the parametersof the current frame i are computed by using the tracked feature points. The error function Einit deﬁned by Eq. (5) is

The proposed method basically consists of feature tracking
and optimization of camera parameters as shown in Figure

minimized to optimize both the parameters ti and Ri of all
the frames and 3D positions of all the feature points.

2. First, two processes of (A) feature tracking and (B) initial
parameter estimation are performed in order. At constantframe intervals, the narrow range optimization process (C)Einit =∑

wj Φ2ijc .

(5)

j∈Sic

is then carried out to reduce accumulative errors. Finally,(C) Narrow range optimization :estimated parameters are reﬁned using many tracked feature

In this process, the frames from the (i − (k + 2l) + 1)-th topoints in the wide range optimization process (D). In thethe current frame are used to reﬁne the camera parameters

processes (C) and (D), a common optimization is performed.

from the (i − (k + 2l) + 1) to the (i − l)-th frames, as illus-

The diﬀerence in both processes is the range of optimized

trated in Figure 3. This process is designed to use featureframes. In the process (C), the range of optimization is apoints and GPS positions obtained in the frames around thesmall part of the input frames because future data cannot

updated frames. To reduce computational cost, this process

be treated in sequential process. On the other hand, in theis performed every k frames. Note that the estimation result

process (D), a large number of frames are optimized and up-

is insensitive to the value of l if it is large enough. The con-dated.

stant l is set as tens of frames to use a suﬃcient number of(A) Feature tracking :feature points reconstructed in the process (B). The constant

The purpose of this process is to determine corresponding

k is set as several frames, which is empirically given so as notpoints between the current frame i and the previous frameto accumulate errors in the initial parameters estimated in(i − 1). The main strategy to avoid mismatching in this pro-the process (B).

cess is that feature points are detected at corners of edges by

It is diﬃcult to obtain a global minimum solution because

Harris operator [14] and detected feature points are tracked

there are a large number of local minima in the error func-

robustly with a RANSAC [15] approach.

tion E. In order to avoid this problem, we currently adopt

In the ﬁrst process (a), natural feature points are auto-

a method to change the weight µi in the iteration of the op-updated frame
representative camerai-(k+l)+1

l framesi-lk framesil frameGPS position

optimized frames
Figure 3 Optimization in the process (C).

timization, which is experimentally derived from computer
simulations. In this method, the weight is changed when-(a) Errors of the estimated center of projection.

ever optimization process is converged. However, it should
be note that there is a room for improvement because thepresent method is found just experimentally. This implementation is used in the next process (D)(D) Wide range optimization :The optimization in the process (C) dose not provide suﬃcient accuracy for a ﬁnal output because it is performed fora part of frames and GPS positions for feedback to feature
tracking process (A). The purpose of this process is to reﬁne parameters by using tracked features and GPS positions
in wide range frames. The algorithm of this process is thesame as the narrow optimization process (C) when l and k
are set as several hundred frames except that divided ranges(b) Errors of estimated optical axes.
Figure 4 Error of the estimated position and posture.

are independent of each other.

4. Experimentvirtual camera takes 900 frames by moving in the virtual environment. The intrinsic parameters of the element camera

This section describes experiments for both synthetic andreal outdoor scenes. First, the experiment for synthetic datais carried out to evaluate the accuracy of the position and posture parameters of OMS estimated by the proposed method
when the correspondences of feature points are given. The
experiment for real data is then demonstrated to conﬁrm thevalidity of the whole proposed method.
Note that some parameters used in the optimization process (C) and (D) were set as follows. The weight coeﬃcientω in the error function E deﬁned by Eq. (4) was set as 10−9 .
The weight µi of the corresponding frame was always set as
1.0, when a GPS position was obtained, When it was not
obtained, 1.0 and 2.0 were alternately set as the weight µi
whenever the optimization step was converged. In the narrow optimization process (C), we set the number of updated
frames k = 5 and the number of optimized frames 49 (l = 22).
4. 1 Synthetic Data
The purpose of this simulation is to evaluate the parameters ti and Ri estimated in the wide optimization process(D). In addition, the validity of the proposed method is conﬁrm by comparison with the conventional method [8]. We
gave a point set as a virtual environment that was used togenerate 2D feature positions in synthetic input images. The

in the virtual OMS are set the same as the real camera described in the next section. The position of GPS receiver inthe representative camera coordinate system is set as (60,150,250)[mm]. We added errors to input data as follows.
The GPS positions with Gaussian noise (σ =30 mm) are
given every 15 frames. The feature points are projected to
the virtual camera, and detected with Gaussian noise (σ =1.6
pixel) and quantization error. The initial parameters are generated by adding Gaussian noise (position: σ = 500 mm, posture: σ = 0.020 rad) to the ground truth. In the comparedmethod [8], all the frames is set as key frames in which morethan 15 feature landmarks appear. The landmarks are given
as feature points whose conﬁdence coeﬃcient is set as large
enough, and the 2D positions of the landmarks in each frame
are given without any errors.
Position and posture errors in the simulation result forthe synthetic data are shown in Figure 4. In the comparedmethod, the position error is 47.5 mm, and the postures error is 0.0019 rad on average. In the proposed method, theposition error is 30.7 mm, and the posture error is 0.0023 radon average.
These results indicate that the proposed method enables
us to obtain position and posture parameters of OMS in theFigure 5 Omnidirectional multi-camera system and GPS receiver.

same order of precision as the conventional method without
any manual acquisitions of surrounding information. The difference in the accuracy between the proposed method and the

Figure 6 Horizontal trace of omnidirectional multi-camera systemon the environment map.

compared one can be caused by the diﬀerence of the eﬀectof the given absolute position information such as GPS positions and landmarks. Concretely, the reason why the postureerrors are smaller than the position ones is that landmark position information obtained from images is more sensitive tothe estimation of postures than GPS position information.
4. 2 Real Scene
The purpose of this experiment with real data is to conﬁrm the validity of the proposed method which includes thequence. The ranges of divided sequences are shown in Figure
9. The diﬀerence of almost all of frames are smaller than theaccuracy 30.7mm shown in the previous section 4. 1. This
result indicates that we can deal with more long sequencesby dividing the sequences if the range parameters k′ and l′
are set adequately.

5. Conclusion

feature tracking and the error models of feature point detection. In this section, we ﬁrst describe the condition of theexperiment. After that, two kinds of experimental results
are shown.
In this experiment, we used Ladybug and a GPS receiver(Nikon LogPakII, horizontal accuracy ±3.0 cm, vertical accuracy ±4.0 cm) ﬁxed on an electric mortar vehicle (see Figure5). We acquired 7200 frames and GPS positions while the vehicle was moving 1.0km distance at 7.6km/h. The acquired
frames and GPS positions were manually synchronized. Ladybug was calibrated by using the method developed in ourprevious work [16]. The distance between OMS and GPS receiver is 300 mm which is manually measured.
Figure 6 shows the estimated positions of Ladybug after thewide range optimization process (D). In this ﬁgure, the camera path is smoothly recovered and lies around the road. The
match move using the estimated parameters is also demonstrated in Figure 7. The virtual cube in Figure 7 seem tobe located at the same position in the real environment inmost part of the input sequence. We have conﬁrmed that
estimated parameters do not contain fatal errors from theseresults.
In the last experiment, we conﬁrm the result of optimization of divided sequences of the wide range optimization process (D). Figure 8 indicates the diﬀerences in estimated positions of the OMS between divided sequences and single se-

In this paper, we have proposed a method to estimate positions and postures of an omnidirectional multi-camera system without accumulative errors from image data and coordinated GPS positions. In the proposed method, GPS position
information is used for both feature tracking and optimization of position and posture parameters of the omnidirectional multi-camera system.
We have conﬁrmed that the proposed method allows us
to obtain extrinsic parameters in the same order precision
as the conventional shape-from-motion method using a large
number of landmarks if GPS positions are obtained well. We
will investigate the case that the number of GPS positions
including large errors is more than current experiments.
References[1] S. Chen, “QuickTime VR: An image-based approach to virtual environment navigation,” Proc. SIGGRAPH ’95, pp.29–
38, 1995.
[2] C. J. Taylor, “VideoPlus: A method for capturing thestructure and appearance of immersive environment,” IEEE
Trans. Visualization and Computer Graphics, vol.8, no.2,
pp.171–182, 2002.
[3] M. Uyttendaele, A. Criminisi, S. B. Kang, S. Winder, R.
Hartley, and R. Szeliski, “High-quality image-based interactive exploration of real-world environments,” IEEE Computer Graphics and Applications, 2004.
[4] A.W. Fitzgibbon and A. Zisserman, “Automatic camera recovery for closed or open image sequences,” Proc. 5th European Conf. on Computer Vision, pp.311 – 326, 1998.
[5] M. Pollefeys, R. Koch, M. Vergauwen, B. Deknuydt, and610th frame

630th frame

650th frame

670th frame

690th frame

2720th frame

2740th frame

2760th frame

2780th frame

3000th frame

Figure 7 Match move using estimated position and posture parameters of Ladybug.

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]
[15]

[16]

L.V. Gool, “Three-dimentional scene reconstruction from images,” Proc. SPIE, pp.215–226, 2000.
S. Güven and S. Feiner, “Authoring 3D hypermedia for wearable augmented and virtual reality,” Proc. 7th IEEE Int.
Symp. on Wearable Computers, pp.118–126, 2003.
A.J. Davison, “Real-time simultaneous localisation and mapping with a single camera,” Proc. 9th IEEE Int. Conf. on
Computer Vision, pp.1403–1410, 2003.
T. Sato, M. Kanbara, N. Yokoya, and H. Takemura, “Dense
3-D reconstruction of an outdoor scene by hundreds-baseline
stereo using a hand-held video csamera,” Int. Jour. of Computer Vision, vol.47, no.1-3, pp.119–129, 2002.
A.I. Comport, É. Marchand, and F. Chaumette, “A realtime tracker for markerless augmented reality,” Proc. 2nd
ACM/IEEE Int. Symp. on Mixed and Augmented Reality,
pp.36–45, 2003.
L. Vacchetti, V. Lepetit, and P. Fua, “Combining edge andtexture information for real-time accurate 3D camera tracking,” Proc. 3rd IEEE and ACM Int. Symp. on Mixed andAugmented Reality, pp.48–57, 2004.
Y. Yokochi, S. Ikeda, T. Sato, and N. Yokoya, “Extrinsic
camera parameter estimation based on feature tracking and
GPS data,” Proc. Asian Conf. on Computer Vision, pp.369–
378, 2006.
D. Nistér, O. Naroditsky, and J. Bergen, “Visual odometry,”
Proc. IEEE Computer Society Conf. on Computer Vision and
Pattern Recognition, pp.964–971, 2004.
Z. Hu, U. Keiichi, H. LU, and F. Lamosa, “Fusion of vision,
3D gyro and GPS for camera dynamic registration,” Proc.
17th Int. Conf. on Pattern Recognition, pp.351–354, 2004.
C. Harris and M. Stephens, “A combined corner and edge
detector,” Proc. Alvey Vision Conf., pp.147–151, 1988.
M.A. Fischler and R.C. Bolles, “Random sample consensus:
A pradigm for model ﬁtting with applications to image analysis and automated cartography,” Communications of theACM, vol.24, no.6, pp.381–395, 1981.
S. Ikeda, T. Sato, and N. Yokoya, “High-resolution

Figure 8 Diﬀerence of the estimaged positions from between divided sequences and single sequence.
1950

0
0

1949
1750

1950

1950

5850

7800
frame index

2150
3899
3700 3900

3200
5849

6050

5850
7799
updated fame
5650
optimized frame

Figure 9 Range of optimization of divided sequence.

panoramic movie generation from video streams acquired byan omnidirectional multi-camera system,” Proc. IEEE Int.
Conf. on Multisensor Fusion and Integration for Intelligent
System, pp.155–160, 2003.