Bundle adjustment using aerial images
with two-stage geometric verification
Hideyuki Kumea,∗, Tomokazu Satoa , Naokazu Yokoyaa
a Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara, Japan

Abstract
In this paper, a new pipeline of Structure-from-Motion for ground-view images
is proposed that uses feature points on an aerial image as references for removing
accumulative errors. The challenge here is to design a method for discriminating correct matches from unreliable matches between ground-view images andan aerial image. If we depend on only local image features, it is not possible in principle to remove all the incorrect matches, because there frequently
exist repetitive and/or similar patterns, such as road signs. In order to overcome this diﬃculty, we employ geometric consistency-verification of matches
using the RANSAC scheme that comprises two stages: (1) sampling-based local
verification focusing on the orientation and scale information extracted by afeature descriptor, and (2) global verification using camera poses estimated by
the bundle adjustment using sampled matches.
Keywords: Structure-from-motion, bundle adjustment, aerial image,RANSAC

1. Introduction
Structure-from-Motion (SfM) is one of the key techniques developed in thefield of computer vision and has been used in many applications, such as threedimensional reconstruction and image-based rendering. SfM became a widely∗ Corresponding author

Email addresses: hideyuki-k@is.naist.jp (Hideyuki Kume), tomoka-s@is.naist.jp(Tomokazu Sato), yokoya@is.naist.jp (Naokazu Yokoya)

Preprint submitted to Computer Vision and Image Understanding

August 22, 2016

5

used tool after implementations of the state-of-the-art SfM (Bundler [1], VisualSFM [2], etc.) were distributed by their authors. They are very useful forprocessing a short image sequence. However, one significant problem in SfM,
that is, the accumulation of estimation errors in a long image sequence with kmorder camera movement, remains to be solved. In this paper, to reduce accu-

10

mulative errors in SfM, we propose a sampling-based bundle adjustment (BA)
scheme using the aerial images that are already available for most outdoor scenes
as external references.
Although many types of external references, e.g., 3D models [3, 4, 5, 6, 7, 8],
GPS [9, 10], and road maps [11], have been used for reducing accumulative

15

errors in SfM, we focus primarily on aerial images owing to their availability
for outdoor environments. The existing methods using aerial images [12, 13,
14, 15, 16, 17, 18] are based on feature matching between given aerial images
and ground-view images taken by standard cameras. Unfortunately, existing
methods can handle only a short image sequence that does not include diﬃcult

20

situations. In this paper, we tackle more diﬃcult situations where a large number of similar/repetitive patterns exist and/or only a few texture patterns areavailable in a long image sequence, e.g., uniformly tiled ground or a road environment, where most of the available feature points are on uniform road signs
drawn on the ground surface. Even if we can approximately limit the search

25

area of feature points by using GPS, which is also commonly used as an externalreference in studies in the literature, if we depend on a local consistency check in
the feature matching stage, in principle, it is not possible to remove all incorrect
matches for a long image sequence because of the existence of repetitive and/or
similar patterns.

30

In order to overcome this problem, we remove incorrect matches caused byrepetitive/similar patterns by introducing a RANSAC framework [19] into boththe feature matching and BA stages that verifies the local and the global consistencies among estimated camera poses and matched features. Figure 1 shows
the flow of the proposed method. For local feature matching, in this study, we

35

assume that we can approximately limit the area used for feature matching, by2• Ground-view images• SfM result for ground-view images• Tracks of feature points• Relative camera poses• 3D positions of feature points• Approximate camera positions• Gravity directions
Feature matching between
ground-view and aerial images
Sampling-based
bundle adjustment
with external references• Refined camera poses• Refined 3D positions offeature points

Figure 1: Flow of the proposed method.

using, e.g., GPS embedded in mobile devices. Figure 2(a) shows an example of aconventional feature matching result obtained by using a common combinationof SIFT [20] and RANSAC for an aerial image containing many repetitive patterns. Even after limiting the search area and rectifying the ground-view image40

to facilitate matching, incorrect matches (blue dashed lines in the figure) areoften erroneously determined to be inliers. In this scene, SIFT finds 158 tentative matches of which only 5 are correct. In order to successfully determine theincorrect matches as outliers, we modify the verification process of RANSAC sothat it additionally checks the consistencies of matches according to the scale45

and orientation information from feature detectors and descriptors.
Although it is expected that most of the incorrect matches will be determined as outliers by this local verification method, some incorrect matches may
simultaneously satisfy the consistency in the position, scale, and orientation,
because the ground textures have similar structures. Figure 2(b) shows an ex-

50

ample in which incorrect matches (blue dashed lines in the figure) are foundeven when the consistency check for scale and orientation is used. To remove
these remaining incorrect matches, in the BA stage, we verify the global consistency of matches and poses for all the images using the RANSAC framework.
More precisely, camera poses are first estimated by the BA scheme using sam3

Correct matches

Image rectification
by homography

Ground truth position ofground-view image

Incorrect matches

Aerial image

Ground-view image(a) Local geometric verification. In this example, 158 tentative
matches, of which only 5 are correct, are found by SIFT [20].
Blue dashed lines are incorrect matches selected by conventional
RANSAC. Magenta solid lines are correct matches selected by
RANSAC with a scale and orientation check.
Estimated pose of -th image
from incorrect matches

Incorrect matches

Image rectification
by homography
Ground truth position of-th image

Estimated pose of -th image
Estimated camera path by BA

Aerial image-th ground-view image(b) Global geometric verification. Blue dashed lines are incorrect matches obtained by RANSAC with a scale and orientation
check. The red line and triangle represent the camera path andpose estimated by BA using sampled matches, respectively.
Figure 2: Two-stage geometric verification by RANSAC.

4

55

pled matches as external references (red line and triangle in the figure), andthe consistency between the estimated poses and each match is then checked.
After iterating sampling and estimation, the best samples that maximize thenumber of consistent frames are selected as inlier frames in which it is expectedthat incorrect matches will be excluded. When good feature matches have been

60

obtained through the two-stage verification, the camera poses are refined by the
BA scheme using the feature matches as external references to remove accumulative errors.
It should be noted that the proposed method assumes that an SfM result forground-view images is given as an initial guess for the BA. The camera model for65

an aerial image can be approximated by the orthographic camera model, and its
image plane is perpendicular to the gravity direction. In addition, approximate
positions and gravity directions of the ground-view images are given by externalsensors. An easy method for obtaining this information, which is employed inthis study, is to use the GPS and gyroscope sensors embedded in most recent

70

smartphones. It should be noted that this paper is an extended version of aprevious conference paper [21]. We have added experiments using a roadway
and an in-depth discussion in this version.

2. Related Work and our Contributions
To reduce accumulative errors in SfM, loop closing techniques [22, 23, 24]75

are sometimes employed. When the loops have been detected, the accumulative errors can be reduced in the BA stage. In an approach related to loop
closing, Cohen et al. [25] exploited symmetries, which often exist in man-madestructures, instead of loops. Although these techniques are eﬀective for some applications, it is essentially diﬃcult for these techniques to remove accumulative

80

errors for a general image sequence without either loops or symmetry.
To reduce accumulative errors in an image sequence captured by a moving
camera, several kinds of external references have been used. These externalreferences can be classified into 3D models [3, 4, 5, 6, 7, 8], GPS [9, 10], road5

maps [11], and aerial images [12]. In studies in the literature, many types
85

of 3D models, including 3D points [3, 4], wire-frame models [5], plane-basedmodels [6, 7], textured 3D models [8], and digital elevation models (DEM) [7]were employed as references. One disadvantage of 3D-model-based methods
for large outdoor environments is that time consuming manual intervention isrequired to create the 3D models. Although some models are already available

90in the GIS database [6, 7], the available areas are still limited to large cities.
One method to create 3D models without much manual intervention is to use
3D reconstruction techniques, e.g., SfM and multi-view stereo [26]. However,
the reconstructed models are also aﬀected by accumulative errors caused by SfM
itself.

95

In contrast to 3D models, GPS, road maps, and aerial images are alreadyavailable for most outdoor scenes around the world. Yokochi et al. [9] and Lhuillier [10] proposed extended-BA using GPS that minimizes the energy function
defined as the sum of reprojection errors and a penalty term of GPS. Thismethod can globally optimize camera poses and reduce accumulative errors by

100

updating poses so as to minimize the energy function. However, the accuracyof this method is directly aﬀected by errors in GPS positioning, which easily
grow to several tens of meters in urban areas when using the GPS embeddedin smartphones. Brubaker et al. [11] proposed a method that uses community developed road maps. This method can reduce accumulative errors by

105

matching the trajectory from SfM to road maps, unless there are ambiguities
in the matched trajectories (e.g., straight roads and Manhattan worlds). Pink
et al. [12] fused sparsely obtained camera poses from aerial images into SfM byusing the Kalman filter. However, unlike BA-based fusion, global optimization
is diﬃcult in the Kalman filter-based approaches.

110

There exist several methods that use aerial images as one of input data of
SfM [27] or RGBD-SLAM [28]. Shan et al. [27] employed oblique aerial images as additional inputs of SfM for reconstructing the regions that are notcovered by ground-view images. One challenge in our case is the employmentof top-view aerial images as an external reference in which common feature6

115

matching method used in [27] cannot give reasonable matches. For obtaining matches between top-view aerial images and ground-view images, Forster et
al. [28] utilized dense depth maps obtained from perspective aerial images in thefeature matching stage. Although feature matching methods from widely diﬀerent viewpoints [29, 30] also work in the case depth/3D information is available,

120

unfortunately, the information is not always easy to be obtained from commonly
available orthographic aerial images.
On the other hand, some methods estimate camera poses directly from aerial
images [13, 14, 15, 16, 17, 18]. There are two types of aerial images: perspective
and orthographic. Bansal et al. [13] proposed a method for estimating camera

125

poses by matching façades in the ground-view input image with perspective
aerial images. Although perspective aerial images are available on Google Maps
and Microsoft Bing Maps, the available areas are still limited to large cities.
Most methods using aerial images employ the orthographic aerial images.
These methods can be classified into learning- [14] and feature-matching-based [15,130

16, 17, 18]. Lin et al. [14] proposed a method based on the relationship of theappearance between ground-view and aerial images learned through community photos with position information. Although this method estimates camera
positions from large regions (1,600 [km2 ] in their experiments), camera positions can be estimated only approximately. Other methods match the building

135

edges [15, 16] or feature points [17, 18] of ground-view and aerial images. Oneof the diﬃculties of this approach is finding good matches for all the images of avideo sequence under severe conditions for feature matching. Toriya et al. [17]and Noda et al. [18] relaxed the problem by stitching multiple ground images
for feature matching. Toriya et al. [17] also proposed a robust feature match-140

ing procedure that compares the orientation and scale of each match with thedominant orientation and scale. Unfortunately, existing feature matching-based
methods, which are expected to achieve highly accurate pose estimation, do nothave the capability to handle a long image sequence because of the strong dependence on matching information given only for a local region. To the best of145

our knowledge, no method exists that handles feature matches between an aerial
7

image and ground-view images in the global optimization stage (BA stage).
As mentioned in the previous section, we tackle diﬃcult situations for feature
matching, where a large number of similar patterns exist in a large-scale outdoor
environment. The main contributions of this paper are summarized as follows:150• BA-based global optimization that uses feature matches between groundview and aerial images.
• Two-stage geometric verification for removing incorrect matches– Local verification that focuses on the transformation between aerial
image and each ground-view image and considers in particular the155

orientation and the scale information extracted by a feature descriptor,– Global verification that focuses on camera poses estimated using theBA scheme with sampled matches.

3. Feature Matching between Ground-view and Aerial Images
160

This section describes a method to find good matches between an aerial
image and each ground-view image with local verification. As shown in Fig. 3,
the method is composed of three processes: (1) ground-view image rectification,(2) feature matching, and (3) local geometric verification by RANSAC.
3.1. Ground-view image rectification and feature matching

165

Before finding matches, as in existing methods [17, 18], we rectify the groundview images so that the texture patterns are similar to those of the aerial image.
To achieve this, we use homography calculated from the gravity direction in thecamera coordinate system. More precisely, we map the ground image to a planethat is perpendicular to the gravity direction. To estimate the gravity direction,170

the vanishing points of parallel lines [31] or a gyroscope sensor can be used. Sinceeven a cheap gyroscope sensor provides an accurate gravity direction, we used agyroscope embedded in a smartphone in the experiment described below. Even
8

Ground-view image(1) Rectification

Aerial image(2) Feature matching(3) Local geometric verification
by RANSAC

Figure 3: Flow of the feature matching.

9

if the patterns cannot be perfectly rectified because of the irregularity of theground plane, it is expected that the chance of obtaining correct matches will
175

be increased by using this rectification process.
A region in an aerial image for feature matching is then determined. We first
determine a certain size of the region the center of which is the GPS position,
which includes measurement errors. In the experiment, the size is set to 50[m] × 50 [m]. Tentative matches are then found between the rectified ground180

image and the limited region in the aerial image using a feature detector and a
descriptor. Although we employed SIFT [20] in the experiment because of its
robustness, any feature operators that output scale and orientation information
can be employed in our framework. It should also be noted that a large GPS
error may result in correct feature points outside the limited region. Even in185

this case, incorrect matches are automatically excluded by applying two-stage
RANSAC with a geometric consistency check.
3.2. Local geometric verification
Tentative matches often include many incorrect matches. The rate of incorrect matches sometimes reached over 95% in our experiment, even if the190

search range for matching was correctly set. In order to decrease the number ofincorrect matches included in the tentative matches, we apply local geometricverification by using RANSAC with a consistency check of the orientation andscale of texture patterns, as shown in Fig. 4.
Although final camera poses are estimated in 6-DOF with BA, to achieve
stable matching between rectified ground-view images and the aerial image, we
use a 3-DOF similarity transform, which is composed of scale s, rotation θ, andtranslation τ , as the model in RANSAC. In a standard RANSAC procedure,tentative matches of minimum number required for estimating the similarity
transform (s, θ, τ ), which are two matches in this case, are randomly selected
first. The number of inlier matches that satisfy the following condition is thencounted.
|ak − (sR(θ)gk + τ )| < dth ,10(1)Correspondence

Consistency check

Similarity transform

Rectified ground-view
image

Aerial image

Figure 4: Criteria used in local geometric verification.

where ak and gk are the 2D positions of the k-th match in the aerial image
195

and the rectified ground-view image, respectively. R(θ) is the 2D rotation matrix with rotation angle θ, and dth is a threshold. After iterating the random
sampling process, the trial with the largest number of inlier matches is selected.
The problem here is that the distance-based single criterion described above
cannot successfully find correct matches when there exists a huge number ofincorrect matches. In order to achieve more robust matching, we modify thecriterion commonly used in RANSAC by adding a consistency check for orientation and scale information extracted from a feature descriptor. More precisely,
we select the matches that simultaneously satisfy Equation (1) and the following
two conditions as inliers in RANSAC procedure.
()
sgk · s sakmax,< sth ,
sak sgk · saad(θgk + θ, θak ) < θth ,

(2)
(3)

where (sak , sgk ) and (θak , θgk ) represent the scale and orientation of featurepoints for the k-th match on the aerial image and the rectified ground-view
11

200

image, respectively. The function ‘aad’ returns the absolute angle diﬀerencein the domain [0◦ , 180.0◦ ]. sth and θth are the thresholds for scale and angle,
respectively. By using the additional consistency check, the feature matches
are strictly verified, and it is expected that most of the incorrect matches will
be removed as outliers. It should be noted that even though we employ a 3-

205

DOF model in RANSAC in this stage, as shown in the experiment in Section 5,
feature points on slanted ground that violate the 3-DOF model are successfully
matched, since we can relax each threshold by simultaneously checking threecriteria in this stage.

4. Sampling-based Bundle Adjustment
210

As shown in Fig. 2(b), some frames contain incorrect matches even after
local geometric verification because of repetitive similar patterns. In this study,
as a global verification stage, we propose a new sampling-based BA scheme tofind the frames that contain incorrect matches.
4.1. Definition of energy function
In order to use the matches between ground-view and aerial images in BA
as external references, as shown in Fig. 5, the energy function E is newly
defined for this problem as the sum of reprojection errors for both ground-view(perspective) images Φ and the aerial (orthographic) image Ψ:E({Ri , ti }Ii=1 , {pj }Jj=1 ) = Φ({Ri , ti }Ii=1 , {pj }Jj=1 ) + ωΨ({pj }Jj=1 ),

215

(4)

where Ri and ti represent 3D rotation and translation from the world coordinate
system to the camera coordinate system for the i-th frame, respectively. pj is
a 3D position of the j-th feature point, I and J are the number of frames andfeature points, respectively, and ω is a weighting coeﬃcient that balances Φand Ψ. Since the energy function is non-linearly minimized in BA, good initial

220

values of parameters are required to avoid local minima. Before minimizing theenergy function, we fit the parameters estimated by SfM to the GPS positions

12

3D position of feature point
Reprojection error for
ground-view image

Reprojection error for
ground-view image(Altitude)(North): Projected 2D position
World
coordinate system: Detected 2D position(East)
Reprojection error for aerial image

Figure 5: Reprojection errors for ground-view (perspective) images and an aerial (orthographic) image.

using a 3D similarity transform. In the following, two energy terms associatedwith reprojection errors Φ and Ψ are given in detail.
4.1.1. Reprojection errors for ground-view images
In our method, camera poses and 3D positions of the feature points estimatedby BA dynamically move in the world coordinate system, which is set on theaerial image coordinate, because of the tension from the external references(matches on the aerial image). Because of this dynamic camera movement, the3D positions of the reference points on an aerial image frequently go behindthe camera. However, the commonly used reprojection errors for the pinhole
camera model cannot deal with projections from behind the camera. In thisstudy, instead of the commonly used squared distance errors on the image plane,
we employ the following angular reprojection error that is employed in SfM for

13

omnidirectional cameras.
Φ({Ri , ti }Ii=1 , {pj }Jj=1 ) = ∑I

1

I ∑
∑

i=1 |Pi | i=1 j∈Pi

Φij ,  
2
   2
xij
Xij
y
Y
 + ̸  ij , ij  ,
Φij = ̸  ,
fi
Zij
fi
Zij
(Xij , Yij , Zij )T = Ri pj + ti ,
225

(5)

(6)
(7)

where Pi is a set of feature points detected in the i-th frame. Function ̸ returns
an angle between two vectors, (xij , yij )T is a detected 2D position of the j-thfeature points in the i-th frame, and fi is the focal length of the i-th frame. By
this definition, the energy becomes large when projections behind the camera
occur.

230

Here, as mentioned in [32], the convergence of energy is very poor with an angular reprojection error Φ̂ij = ̸ ((xij , yij , fi )T , (Xij , Yij , Zij )T )2 . We then split
the angular reprojection error into xz and yz components in order to simplify
the Jacobian matrix of E required by non-linear least squares methods, such as
the Levenberg-Marquardt method. The first and second terms of Φij do not235

depend on the y and x components of ti in this definition. We experimentallyconfirmed that this splitting largely aﬀects the convergence performance.
4.1.2. Reprojection errors for aerial image
The reprojection errors for the aerial (orthographic) image are defined asΨ({pj }Jj=1) = ∑∑ ∑1i∈M |Ai |2|aj −pr(pj )| ,(8)i∈M j∈Ai

where M is a set of frames in which matches between ground-view and aerial
images are found, Ai is a set of feature points that are matched to the aerial
240

image in the i-th frame, and aj is the 2D position of the j-th feature point in theaerial image. The function ‘pr’ projects a 3D point onto the xy plane (aerial image coordinate system). Although the height of the 3D points is not aﬀected bythis term, the remaining 2D positions are constrained to their positions on the14aerial image. These constraints are eﬀective for reducing the accumulative er245

rors through simultaneously minimizing both the perspective and orthographic
reprojection errors in the BA.
4.2. Global geometric verification
This section describes a RANSAC scheme introduced into BA for global
geometric verification. Since the matches remaining after local verification are
consistent in each frame, we judge inliers in a frame-wise manner. First, we
randomly sample n frames from the frames that passed local geometric verification and execute BA using the matches in the sampled frames, i.e., using a
set of sampled frames M ′ instead of M in Equation (8). We then check theconsistency between the camera poses obtained by BA and each frame that includes feature matches. More precisely, we count the number of inlier framesthat satisfy the condition
average(αij ) < αth ,

(9)

j∈Ai

where αij is an angular reprojection error of the j-th feature point on the aerial
image coordinate system, as shown in Fig. 6, and αth is a threshold. Here, αij
is computed as
αij = ̸(())
aj −pr(−RiT ti ), pr RiT (xij , yij , fi )T .

(10)

After iterating the random sampling process at given times, the trial that has
the largest support is selected. Finally, camera poses are refined by executing
250

BA again using the feature matches in the selected inlier frames.
In the experiments described below, the threshold αth is experimentally determined. It should also be noted that biased sampling, where samples are close
to each other, frequently yields an unstable result in RANSAC. Thus, we modify the random sampling process of frames so that the distances between the255

average positions of matches on an aerial image are larger than threshold lth .

15-th camera coordinate system
Ray of -th feature pointin camera coordinate system:

Position of -th feature point on aerial image:

Projected ray on aerial image:(Altitude)-th camera position
on aerial image:(North)Worldcoordinate system(East)

Figure 6: Criterion used in global geometric verification.

5. Experiments
To validate the eﬀectiveness of the proposed method, we quantitatively evaluated the performance of the proposed BA with two-stage geometric verification
using two datasets: (1) data captured by a hand-held sensor unit on textured
260

ground for Experiment 1, and (2) data captured by a car-mounted sensor unit
on a roadway for Experiment 2. In the following, we first describe the setup
used for both the experiments. The results of each experiment are then detailed.
5.1. Experimental setup
We used an iPhone 5 (Apple) as a sensor unit including a camera, GPS,265

and a gyroscope. The GPS and gyroscope measured the position at 1 [Hz] andthe direction of gravity for every frame, respectively. We also used an RTKGPS (Topcon GR-3, 1 [Hz]; horizontal positioning accuracy in the specification
sheet is 0.01 [m]) to obtain the ground truth positions. The positions obtained
from the GPS data were assigned temporally to the nearest frame. As exter-270

nal references, we downloaded the aerial images covering the area used in the16experiments from Google Maps [maps.google.com], whose coordinate system isassociated with the metric scale.
To obtain the initial values for the BA, we employed VisualSFM [2] as a stateof-the-art SfM implementation. For non-linear optimization, we used Ceres275

Solver [33]. We experimentally set dth = 2 [pixel], sth = 2 and θth = 40 [◦ ] forthe feature matching, and ω = 10−5 and αth = 5.0 [◦ ] for the BA.
5.2. Quantitative evaluation using data captured on textured ground (Experiment 1)
In this experiment, we used video images (640 [pixel] × 480 [pixel], 2,471

280

frames, 494 [s]) captured by a hand-held sensor unit on a textured ground. As
shown in Fig. 2, a large number of similar patterns exist on the ground in
this environment. Figure 7 shows an aerial image covering the area used in theexperiments (approximately 1 [pixel] = 5.2 [cm]).
5.2.1. Eﬀect of local verification

285

In this experiment, we first evaluated the eﬀectiveness of the proposed feature matching process including local geometric verification by RANSAC using
the scale and orientation check described in Section 3. Here, we tested local verification with variable thresholds sth and θth . To count the number of correctly
matched frames, we first selected frames that had four or more inlier matches

290

after local verification. From these frames, we manually selected the frames
whose matches were correct.
Figure 8 shows the rate and the number of frames in which all the selected
matches were correct. It should be noted that sth = ∞ and θth = 180.0 [◦ ], whichmeans that the orientation check and scale check were disabled, respectively.

295

The results indicate that the rate was significantly improved through the scaleand orientation check. We can also confirm that small values of sth and θthtend to increase this rate. However, the number of correctly matched frames,
which is important for optimizing camera poses using BA, was decreased when17

Last frame

GPS on iPhone
RTK-GPS (Ground truth)
BA without references
BA with references without global verification
BA with references and global verification(proposed method)
Correctly matched feature point on aerial image(selected as inlier by global verification)
Incorrectly matched feature point on aerial image(selected as outlier by global verification)

First frame

50 [m]

50 [m]
Figure 7: Experimental environment and results (Experiment 1).

18

Rate of frames in which

all selected matches are correct

1.0

sth = 1.25
sth = 1.5

0.8

sth = 2.0

0.6

sth = 2.5

0.4

sth = 3.0
sth = ∞

0.2
0
0

20

40

60

80
100
θth [◦ ]

120

140

160

180

Number of frames in which

all selected matches are correct(a) Rates of frames in which all selected matches are correct

sth = 1.25

10

sth = 1.5

8

sth = 2.0
sth = 2.5

6

sth = 3.0
sth = ∞

4
2
0
0

20

40

60

80
100
θth [◦ ]

120

140

160

180

(b) Numbers of frames in which all selected matches are correct

Figure 8: Rates and numbers of frames in which all selected matches are correct (Experiment1).

19

using small thresholds. In the following experiments, we then employed feature300

matches with sth = 2 and θth = 40 [◦ ].
Figure 9 shows the eﬀects of the scale and orientation check for two sampled
images. In both cases, local verification without a scale and orientation check
could not select any correct matches, whereas the proposed local verification
with a scale and orientation check was able to do so. However, as shown in

305

Fig. 10, incorrect matches still remained even when we used both the scale andorientation check, because similar patterns exist.
5.2.2. Eﬀect of global verification
We then evaluated the eﬀectiveness of global geometric verification by samplingbased BA, as described in Section 4. In this stage, frames with GPS data were

310

sampled (650 out of 2,471 frames) and used for the BA to reduce the computational time. As external references, we used the frames and feature matches
selected through the orientation and scale check described in the previous section. Here, 10 out of 14 frames had correct matches.
We first investigated the influence of weight ω for balancing two types of315

reprojection errors in the energy function of the BA. Figure 11 shows the average
position errors produced by the BA with variable weight ω using all the correctly
matched frames. This result demonstrates that position errors did not largely
depend on weight ω, except when small values were applied. In the experiments
described next, we employed ω = 10−5 .

320

We next evaluated the proposed sampling-based global verification in termsof its capability to select frames with correct matches. Here, we experimentally
set n = 4 and lth = 25 [m], and tested 100 trials. Figure 12 shows the number of
inlier frames produced by global verification with variable threshold αth . The results demonstrate that incorrectly matched frames were selected as inliers when325

large values of αth were used and that the number of correctly matched frames
decreased when small values of αth were used. In the experiments described
next, we employed αth = 5.0 [◦ ].
We also checked the number of inlier frames selected in each trial with αth =20

Similarity transform
Angle [ ] Scale
325.56 0.06

Similarity transform
Angle [ ] Scale
158.71 1.19

SIFT keypoints
Angle [ ] Scale
334.02 0.85
332.84 0.91
330.88 0.84
352.56 0.43
156.73 1.06

SIFT keypoints
Angle [ ] Scale
159.92 1.11
154.96 0.89
154.24 1.25
148.45 1.12(a) Without orientation and scale check (b) With scale check, with/without orientation check

Similarity transform
Angle [ ] Scale
66.47 0.07

Similarity transform
Angle [ ] Scale
336.34 0.96

SIFT keypoints
Angle [ ] Scale
164.35 0.86
334.38 0.82
334.94 0.78
329.40 0.68
171.96 0.46

SIFT keypoints
Angle [ ] Scale
342.27 0.90
338.34 0.90
339.67 0.73
341.82 0.77
333.08 0.77(c) Without orientation and scale check (d) With orientation check, with/withoutscale check
Figure 9: Selected inliers for example images (Experiment 1). The solid and dashed lines represent correct and incorrect matches, respectively. The relative angle and scale of the matchedfeature points are shown in the bottom right hand table together with the corresponding line
colors. The green points are the ground truths of the camera positions. Note that local
verification with/without the orientation check for (b) and scale check for (d) gave the same
results.

21

Similarity transform
Angle [ ] Scale
3.87 0.89

Similarity transform
Angle [ ] Scale
169.41 1.07

SIFT keypoints
Angle [ ] Scale
343.16 1.63
19.02 0.80
12.64 1.22
13.32 0.93

SIFT keypoints
Angle [ ] Scale
169.48 0.99
168.87 1.15
163.06 1.17
170.35 1.04

(a)

(b)

Figure 10: Examples of incorrect matches by local verification using orientation and scale

Avg. horizontal position error [m]check (Experiment 1). The interpretations of the symbols are the same as in Fig. 9.

12

8

4

0
10−10

10−8

10−6

10−4

10−2

100

ω
Figure 11: Relationship between weight ω and average horizontal position error (Experiment1).

22

Number of frame

10
8
6
4
2
0
0

2

4

6

8

10
αth [◦ ]

12

14

16

18

20

Correctly matched frame selected as inlier
Incorrectly matched frame selected as inlier
Figure 12: Number of inlier frames with variable threshold αth (Experiment 1).

Number of trials

30
25
20
15
10
5
0
0

1

2

3
4
5
6
7
8
Number of inlier frames
Sampled frames without incorrect matches
Sampled frames with incorrect matches

910

Figure 13: Number of trials and inlier frames derived by each trial (Experiment 1).

5.0 [◦ ]. Figure 13 shows the number of trials and inlier frames derived by each330

trial. In this figure, it can be seen that the sampled frames without incorrect
matches tend to increase the number of inlier frames. This result demonstrates
that the criterion of global verification is eﬀective. We also confirmed that thetrials that derived the largest number of inlier frames successfully selected allof the correct matches.

335

In order to validate the eﬀectiveness of the global verification and the use of
external references on an aerial image, the results of the following three methods
were compared.

23

Horizontal position error [m]

60
50
40
30
20
10
0
0

1000

2000

Frame number
BA w/o references
BA w/ references w/o global verification

BA w/ references and global verification(proposed method)

Correctly matched frame(selected as inlier by global verification)

Incorrectly matched frame(selected as outlier by global verification)

Figure 14: Horizontal position error in each frame (Experiment 1).

• BA without references [2]• BA with references without global verification
340• BA with references and global verification (proposed method).
Figures 7 and 14 show the estimated camera positions and horizontal position
errors for each frame, respectively. Since the BA without references cannot estimate absolute camera poses, we fitted the camera positions estimated using SfM
to the ground truths through a similarity transform. These results demonstrate

345

that the camera positions estimated through the BA without references wereaﬀected by the accumulative errors. The BA without global verification wasaﬀected by the incorrect matches. The proposed BA with global verification
reduced the accumulative errors. It should be noted that, at the end of thesequence, the accumulative errors still remained, because the ground was not

350

level and thus no matches were found.

24

5.3. Quantitative evaluation using data captured on roadways (Experiment 2)
In this experiment, we used video images (640 [pixel] × 480 [pixel], 7,698
frames, 396 [s]) captured by a car-mounted sensor unit on a roadway. Figure 15
shows an aerial image covering the area used in the experiments (approximately355

1 [pixel] = 4.5 [cm]). It should be noted that we manually excluded frames
captured when the car was stopped at a traﬃc light.
We first applied the feature matching process, including local verification
with a scale and orientation check. After selecting the frames with 4 or moreinlier matches, we obtained 37 frames (28 frames without and 9 frames with360

incorrect matches). We then applied global verification using frames with GPS
data (739 out of 7,698 frames). Here, we experimentally set n = 7 and lth = 100[m]. After 100 trials, the trial that derived the largest number of inlier frames
selected 22 frames as inliers (19 frames without and 3 frames with incorrect
matches) and 15 frames as outliers (9 frames without and 6 frames with incor-

365

rect matches). Figures 16 and 17 show example frames selected as inliers andoutliers, respectively. As shown in Figure 16, the frames with incorrect matches
were selected as inlier frames by global verification because the positions of theincorrect matches on the aerial image were close to the correct positions. Figures
15 and 18 show the estimated camera positions and horizontal position errors for370

each frame, respectively. Although the frames with incorrect matches still remained even when using two-stage geometric verification, the proposed method
clearly reduced the accumulative errors. However, as can be seen in Fig. 18,
the accumulative errors are still large around the 6,000th frame because there
was only a small number of matches.

375

6. Discussion and Limitations
This section discusses the way of parameter setting and the limitations in
the feature matching process. The proposed method has some parameters whichshould be manually determined. Generally, it is not easy to find the best parameters for individual dataset. However, it should be noted that, as shown

25

Last frame

GPS on iPhone
RTK-GPS (Ground truth)
BA without references
BA with references without global verification
BA with references and global verification(proposed method)
Correctly matched feature point on aerial image(selected as inlier by global verification)
Correctly matched feature point on aerial image(selected as outlier by global verification)
Incorrectly matched feature point on aerial image(selected as inlier by global verification)
Incorrectly matched feature point on aerial image(selected as outlier by global verification)

First frame

100 [m]Figure 15: Experimental environment and results (Experiment 2).

26

100 [m]

(a)

(b)

(c)

(d)

Figure 16: Examples of frames selected as inliers by global verification (Experiment 2). The
solid and dashed lines represent correct and incorrect matches, respectively.

(a)

(b)

Figure 17: Examples of frames selected as outliers by global verification (Experiment 2). The
dashed lines represent incorrect matches.

27

Horizontal position error [m]

50

40

30

20

10

0
0

2000

4000

6000

Frame number
BA w/o references
BA w/ references w/o global verification
Correctly matched frame(selected as inlier by global verification)
Correctly matched frame(selected as outlier by global verification)

BA w/ references and global verification(proposed method)
Incorrectly matched frame(selected as inlier by global verification)
Incorrectly matched frame(selected as outlier by global verification)

Figure 18: Horizontal position error in each frame (Experiment 2).

380

in Figs. 8, 11 and 12, most of parameters in the proposed pipeline have wide
range of sweet spots where sub-optimal results can be obtained. We thus have
employed common values for most of parameters in two diﬀerent experiments
despite characteristics of the datasets are quite diﬀerent. These results imply
that parameter values used in the experiments may be valid for other datasets.

385

Parameters for which we did not use common values in the experiments are
the number of samples n and the minimum distance between samples lth usedin the global geometric verification. These parameters depend on the numberof matches, the ratio of incorrect matches, and the scale of input data, e.g.
moving distance, distance between camera and feature point, and number of390

images. With analyzing various experimental data, automatic determination
way for these two parameters is expected to be developed in the future.
One major limitation of the proposed method exists in the feature matchingprocess. Since the proposed method projects a ground image to a plane that is28

perpendicular to the gravity direction, it is not easy to find correct matches in395

situations where the ground is not level or not a plane. In fact, there exist along slope and steps around the last frame of the dataset 1 (top right of Fig. 7)
and thus our method could not find correct matches for these regions. Matchesalso cannot be obtained from texture-less ground such as roadways shown in
Section 5.3. However, as shown in these experiments, even if there exist re-400

gions where the proposed method cannot obtain good matches, the proposed
method can successfully reduce the accumulative errors only if there exist several regions where correct matches can be found. To find more good matcheseven for non-level ground, aﬃne and/or perspective invariant features, such as
ASIFT [34] and Ferns [35], can be used with finding better homography pa-

405

rameters in local verification process. However, it should be noted that simply
applying these feature operators to our pipeline will drop the accuracy for manycases due to the increment of degrees of freedom in feature matching process.
If the scene is expected to be level for most regions, standard feature matching
operators should give better result. The development of eﬀective way for em-

410

ploying aﬃne/perspective invariant feature operators into the proposed pipeline
is our next challenge for increasing the practicality.

7. Conclusion
In this paper, we proposed a method for removing accumulative errors in
SfM using aerial images that are already available for many places around the415

world as external references. To achieve this, we proposed a new BA schemethat uses feature matches between the ground-view and aerial images. In order
to discriminate the correct matches among unreliable matches, we introduced
local and global geometric verification procedures provided by RANSAC. The
local verification focuses on transformation between the aerial image and each420

ground-view image, considering in particular the orientation and the scale information extracted by a feature descriptor. The global verification focuses on theconsistency of matches and poses for all the images through a sampling-based29

BA. To the best of our knowledge, ours is the first method that uses aerial images
as external references in BA. We confirmed experimentally that the proposed425

method is eﬀective for estimating the camera poses of real video sequences taken
in outdoor environments. However, the accumulative errors still remain when
there are no available matches during a long period of time. To find matches
in situations where the ground is not level, aﬃne and/or perspective invariant
features, such as ASIFT [34] and Ferns [35], can be used with homography as

430

a geometric transformation in local verification. The proposed method requires
several tens of hours for global geometric verification. To reduce computationaltime, BA with incorrect matches should be determined and discontinued in an
early step of non-linear optimization.

Acknowledgements
435This research was partially supported by JSPS Grant-in-Aid for Scientific
Research Nos. 23240024 and 26330193.

References[1] N. Snavely, S. M. Seitz, R. Szeliski, Modeling the World from Internet
Photo Collections, Int. J. of Computer Vision 80 (2) (2008) 189–210.
440[2] C. Wu, VisualSFM: A Visual Structure from Motion System, http://ccwu.me/vsfm/ (2013).
[3] T. Sato, S. Ikeda, N. Yokoya, Extrinsic Camera Parameter Recovery from
Multiple Image Sequences Captured by an Omni-directional Multi-camera
System, in: Proc. European Conf. on Computer Vision, 2004, pp. 326–340.

445[4] N. Fioraio, L. D. Stefano, Joint Detection, Tracking and Mapping by Semantic Bundle Adjustment, in: Proc. IEEE Conf. on Computer Vision and
Pattern Recognition, 2013, pp. 1538–1545.

30

[5] G. Bleser, H. Wuest, D. Stricker, Online Camera Pose Estimation in Partially Known and Dynamic Scenes, in: Proc. IEEE and ACM Int. Symp.
450

on Mixed and Augmented Reality, 2006, pp. 56–65.
[6] P. Lothe, S. Bourgeois, E. Royer, M. Dhome, S. Naudet-Collette, Real-time
Vehicle Global Localisation with a Single Camera in Dense Urban Areas:
Exploitation of Coarse 3D City Models, in: Proc. IEEE Conf. on Computer
Vision and Pattern Recognition, 2010, pp. 863–870.

455[7] D. Larnaout, S. Bourgeois, V. Gay-Bellile, M. Dhome, Towards Bundle
Adjustment with GIS Constraints for Online Geo-localization of a Vehicle
in Urban Center, in: Proc. Int. Conf. on 3D Imaging, Modeling, Processing,Visualization and Transmission, 2012, pp. 348–355.
[8] M. Tamaazousti, V. Gay-Bellile, S. Naudet Collette, S. Bourgeois,

460

M. Dhome, NonLinear Refinement of Structure from Motion Reconstruction by Taking Advantage of a Partial Knowledge of the Environment, in:Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2011, pp.
3073–3080.
[9] Y. Yokochi, S. Ikeda, T. Sato, N. Yokoya, Extrinsic Camera Parameter

465

Estimation Based-on Feature Tracking and GPS Data, in: Proc. Asian
Conf. on Computer Vision, 2006, pp. 369–378.
[10] M. Lhuillier, Incremental Fusion of Structure-from-Motion and GPS using
Constrained Bundle Adjustments, IEEE Trans. on Pattern Analysis andMachine Intelligence 34 (12) (2012) 2489–2495.

470[11] M. A. Brubaker, A. Geiger, R. Urtasun, Lost! Leveraging the Crowd for
Probabilistic Visual Self-localization, in: Proc. IEEE Conf. on Computer
Vision and Pattern Recognition, 2013, pp. 3057–3064.
[12] O. Pink, F. Moosmann, A. Bachmann, Visual Features for Vehicle Localization and Ego-motion Estimation, in: Proc. IEEE Intelligent Vehicles

475

Symposium, 2009, pp. 254–260.
31

[13] M. Bansal, K. Daniilidis, H. Sawhney, Ultra-wide Baseline Facade Matchingfor Geo-localization, in: Proc. European Conf. on Computer Vision, 2012,
pp. 175–186.
[14] T.-Y. Lin, S. Belongie, J. Hays, Cross-view Image Geolocalization, in: Proc.
480

IEEE Conf. on Computer Vision and Pattern Recognition, 2013, pp. 891–
898.
[15] S. Kim, S. DiVerdi, J. S. Chang, T. Kang, R. Iltis, T. Höllerer, Implicit
3D Modeling and Tracking for Anywhere Augmentation, in: Proc. ACM
Symp. on Virtual Reality Software and Technology, 2007, pp. 19–28.

485[16] K. Y. K. Leung, C. M. Clark, J. P. Huissoon, Localization in Urban Environments by Matching Ground Level Video Images with an Aerial Image,in: Proc. IEEE Int. Conf. on Robotics and Automation, 2008, pp. 551–556.
[17] H. Toriya, I. Kitahara, Y. Ohta, A Mobile Camera Localization Method
using Aerial-view Images, in: Proc. IAPR Asian Conf. on Pattern Recog-

490

nition, 2013, pp. 49–53.
[18] M. Noda, T. Takahashi, D. Deguchi, I. Ide, H. Murase, Y. Kojima, T. Naito,
Vehicle Ego-localization by Matching In-vehicle Camera Images to an
Aerial Image, in: Proc. Computer Vision in Vehicle Technology: From
Earth to Mars, 2010, 10 pages.

495[19] M. A. Fischler, R. C. Bolles, Random Sample Consensus: A Paradigm
for Model Fitting with Applications to Image Analysis and AutomatedCartography, Communications of the ACM 24 (6) (1981) 381–395.
[20] D. G. Lowe, Distinctive Image Features from Scale-invariant Keypoints,
Int. J. of Computer Vision 60 (2) (2004) 91–110.

500[21] H. Kume, T. Sato, N. Yokoya, Sampling based Bundle Adjustment using
Feature Matches between Ground-view and Aerial Images, in: Proc. Int.
Conf. on Computer Vision Theory and Applications, Vol. 3, 2014, pp. 692–
698.
32

[22] A. Angeli, D. Filliat, S. Doncieux, J.-A. Meyer, Fast and Incremental
505

Method for Loop-closure Detection using Bags of Visual Words, IEEE
Trans. on Robotics 24 (5) (2008) 1027–1037.
[23] H. Strasdat, A. J. Davison, J. Montiel, K. Konolige, Double Window Optimisation for Constant Time Visual SLAM, in: Proc. IEEE Int. Conf. on
Computer Vision, 2011, pp. 2352–2359.

510[24] B. Williams, M. Cummins, J. Neira, P. Newman, I. Reid, J. Tardós, A
Comparison of Loop Closing Techniques in Monocular SLAM, Robotics
and Autonomous Systems 57 (12) (2009) 1188–1197.
[25] A. Cohen, C. Zach, S. N. Sinha, M. Pollefeys, Discovering and Exploiting 3D Symmetries in Structure from Motion, in: Proc. IEEE Conf. on

515

Computer Vision and Pattern Recognition, 2012, pp. 1514–1521.
[26] S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, R. Szeliski, Building Romein a Day, in: Proc. IEEE Int. Conf. on Computer Vision, 2009, pp. 72–79.
[27] Q. Shan, R. Adams, B. Curless, Y. Furukawa, S. M. Seitz, The Visual
Turing Test for Scene Reconstruction, in: Proc. Int. Conf. on 3D Vision,

520

2013, pp. 25–32.
[28] C. Forster, M. Pizzoli, D. Scaramuzza, Air-ground Localization and
Map Augmentation Using Monocular Dense Reconstruction, in: Proc.
IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2013, pp. 3971–
3978.

525

[29] C. Wu, B. Clipp, X. Li, J.-M. Frahm, M. Pollefeys, 3D Model Matching with
Viewpoint-invariant Patches (VIP), in: Proc. IEEE Conf. on Computer
Vision and Pattern Recognition, 2008, 8 pages.
[30] B. Zeisl, K. Köser, M. Pollefeys, Automatic Registration of RGB-D Scans
via Salient Directions, in: Proc. IEEE Int. Conf. on Computer Vision, 2013,

530

8 pages.

33[31] A. Criminisi, I. Reid, A. Zisserman, Single View Metrology, Int. J. of Computer Vision 40 (2) (2000) 123–148.
[32] E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, P. Sayd, Generic andReal-time Structure from Motion using Local Bundle Adjustment, Image535

and Vision Computing 27 (8) (2009) 1178–1193.
[33] S. Agarwal, K. Mierle, Others, Ceres Solver, https://code.google.com/
p/ceres-solver/ (2013).
[34] J.-M. Morel, G. Yu, ASIFT: A New Framework for Fully Aﬃne Invariant
Image Comparison, SIAM J. on Imaging Sciences 2 (2) (2009) 438–469.

540[35] M. Özuysal, M. Calonder, V. Lepetit, P. Fua, Fast Keypoint Recognition
using Random Ferns, IEEE Trans. on Pattern Analysis and Machine Intelligence 32 (3) (2010) 448–461.

34