NAIST-IS-DD1061008

Doctoral Dissertation

Camera Pose Estimation for an Image Sequencewith External References

Hideyuki Kume

March 13, 2014

Department of Information Systems
Graduate School of Information Science
Nara Institute of Science and Technology

A Doctoral Dissertation
submitted to Graduate School of Information Science,
Nara Institute of Science and Technology
in partial fulfillment of the requirements for the degree ofDoctor of ENGINEERING
Hideyuki Kume
Thesis Committee:
Professor Naokazu Yokoya(Supervisor)
Professor Hirokazu Kato
(Co-supervisor)
Associate Professor Tomokazu Sato (Co-supervisor)

Camera Pose Estimation for an Image Sequencewith External References∗
Hideyuki Kume

Abstract
Structure-from-Motion (SfM) is one of the key techniques developed in thefield of computer vision, and has been used in many applications such as threedimensional reconstruction, robot navigation, and augmented reality. The mostsignificant problem in SfM is the accumulation of estimation errors in a long image
sequence. Although many types of methods for reducing accumulative errors havebeen proposed, SfM essentially cannot be free from accumulative errors unlesscertain external references (e.g., GPS, aerial images, or feature landmarks) are
given.
To treat various scenes for which the appropriate external references are notunique, this thesis proposes camera pose estimation methods employing the following three types of references: (1) GPS, (2) aerial images, and (3) a 3D point
database created by SfM. GPS and aerial images are already available for mostoutdoor scenes around the world. To reduce the accumulative errors in SfM, we
propose bundle adjustment (BA)-based methods that can globally optimize thecamera poses using GPS and aerial images. Some applications of robot navigation and augmented reality require estimating the camera poses along a previously
taken route. For such applications, we propose an online camera pose estimation
method using a 3D point database created by SfM from previously captured
images.
In methods using GPS, extended BA that fuses SfM and GPS data has beenpreviously proposed, and it works properly if the GPS data are acquired accurately. However, because existing methods do not consider the confidence of GPS∗

Doctoral Dissertation, Department of Information Systems, Graduate School of Information
Science, Nara Institute of Science and Technology, NAIST-IS-DD1061008, March 13, 2014.

i

positioning, the accuracy of estimated camera positions depends largely on theconfidence of the GPS positioning data. To solve this problem, we add weighting
coeﬃcients depending on the GPS positioning confidence to the energy function
for extended BA.
No existing method uses aerial images as external references in BA. We propose a new SfM pipeline that uses feature matches between ground-view andaerial images. To find proper matches from unreliable matches, we newly propose RANSAC-based outlier elimination methods for both the feature matching
and BA stages.
As a method using a 3D point database created by SfM, we estimate thecamera poses online from the 2D positions of the feature points in the current
image and their 3D positions obtained from the database. The challenge here is
how to accurately obtain the 3D-2D correspondences. To this end, the proposed
method identifies the database image that is most similar to the current image
by considering both topological information and image features.
The usefulness of the proposed methods was quantitatively confirmed throughexperiments using data obtained in real environments.

Keywords:
camera pose estimation, structure-from-motion, bundle adjustment, GPS, aerial
image, 3D point databaseii

Contents
1 Introduction
1.1. Camera Pose Estimation without External References . . . . . . .
1.1.1 Structure-from-Motion . . . . . . . . . . . . . . . . . . . .
1.1.2 Techniques to Reduce Accumulative Errors . . . . . . . . .
1.2. Camera Pose Estimation with External References . . . . . . . . .
1.2.1 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.2 Fiducial Markers . . . . . . . . . . . . . . . . . . . . . . .
1.2.3 Image Databases . . . . . . . . . . . . . . . . . . . . . . .
1.2.4 3D Models . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.5 3D Point Databases . . . . . . . . . . . . . . . . . . . . . .
1.2.6 Aerial Images . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.7 Road maps . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3. Contributions of this Thesis . . . . . . . . . . . . . . . . . . . . .
1.3.1 Camera Pose Estimation without a Pre-Measurement of theTarget Environments . . . . . . . . . . . . . . . . . . . . .
1.3.2 Camera Pose Estimation along a Previously Taken Route .
1.4. Organization of this Thesis . . . . . . . . . . . . . . . . . . . . . .

1
4
4
7
9
10
13
14
16
17
19
21
21
22
24
24

2 Extended Bundle Adjustment using GPS Positioning and Its
Confidence25
2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2. Energy Function Considering GPS Positioning Confidence . . . . 26
2.2.1 Reprojection Errors . . . . . . . . . . . . . . . . . . . . . . 28
2.2.2 Penalty Term for GPS Positioning . . . . . . . . . . . . . . 28
2.3. Optimization by Minimizing Energy Function . . . . . . . . . . . 30
iii

2.3.1 Range of Optimization . . . . . . . . . . . . . . . . . . . .
2.3.2 Parameter Fitting to GPS Positions . . . . . . . . . . . . .
2.4. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . .
2.4.2 Determination of Weighting Coeﬃcients Depending on GPS
Positioning Confidence . . . . . . . . . . . . . . . . . . . .
2.4.3 Quantitative Evaluation using Data Including Many LowConfidence GPS Positions (Experiment 1) . . . . . . . . .
2.4.4 Quantitative Evaluation using Data Including a Long GPS
Outage (Experiment 2) . . . . . . . . . . . . . . . . . . . .
2.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30
31
33
33
37
37
42
44

3 Sampling-Based Bundle Adjustment using Feature Matches Between Ground-View and Aerial Images48
3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2. Feature Matching Between Ground-View and Aerial Images . . . 49
3.2.1 Image Rectification using Homography . . . . . . . . . . . 51
3.2.2 Feature Matching . . . . . . . . . . . . . . . . . . . . . . . 51
3.2.3 RANSAC with Orientation and Scale Check . . . . . . . . 51
3.3. Sampling-Based Bundle Adjustment . . . . . . . . . . . . . . . . . 52
3.3.1 Definition of Energy Function . . . . . . . . . . . . . . . . 53
3.3.2 RANSAC for Bundle Adjustment . . . . . . . . . . . . . . 57
3.4. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . 59
3.4.2 Quantitative Evaluation using Data Captured on Textured
Ground (Experiment 1) . . . . . . . . . . . . . . . . . . . 59
3.4.3 Quantitative Evaluation using Data Captured on Roadways(Experiment 2) . . . . . . . . . . . . . . . . . . . . . . . . 67
3.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4 Online Camera Pose Estimation using 3D Point Database Created from Structure-from-Motion
74
4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.2. Oﬄine Creation of 3D Point Database . . . . . . . . . . . . . . . 75
iv

4.3. Online Camera Pose Estimation . . . . . . . . . . . . . . . . . . .
4.4. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . .
4.4.2 Quantitative Evaluation . . . . . . . . . . . . . . . . . . .
4.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

77
78
78
79
84

5 Conclusions
5.1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2. Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . .

85
85
86

Acknowledgements

88

References

90

List of Publications

104v

List of Figures
1.1 Applications of camera pose estimation. . . . . . . . . . . . . . .
2
1.2 SfM from community photos [41]. . . . . . . . . . . . . . . . . . .
7
1.3 Example of the eﬀectiveness of BA [45]. . . . . . . . . . . . . . . .
8
1.4 Loop closing for the trajectory around a courtyard [55]. . . . . . .
9
1.5 BA exploiting symmetry [56]. . . . . . . . . . . . . . . . . . . . . 10
1.6 Sensor examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.7 Examples of fiducial markers. . . . . . . . . . . . . . . . . . . . . 14
1.8 Example of an image database [88]. . . . . . . . . . . . . . . . . . 15
1.9 Examples of 3D models. . . . . . . . . . . . . . . . . . . . . . . . 16
1.10 3D point database [100]. Camera poses are estimated by matching
the feature points between the input images and 3D point database. 18
1.11 Examples of aerial images. . . . . . . . . . . . . . . . . . . . . . . 19
1.12 Example road maps [121]. . . . . . . . . . . . . . . . . . . . . . . 21
1.13 Characteristics of external references in terms of the amount ofmanual intervention and the extent of the applicable environments. 22
2.1

2.2
2.3
2.4
2.5
2.6
2.7

Flow diagram of the proposed method using GPS, where f is theframe index, G is a set of frames in which GPS data are obtained,
and Grecovered is a set of frames in which GPS positioning is recovered after a GPS outage. . . . . . . . . . . . . . . . . . . . . . . .
Reprojection error. . . . . . . . . . . . . . . . . . . . . . . . . . .
Energy term with respect to GPS positioning. . . . . . . . . . . .
Parameter fitting to GPS positions. . . . . . . . . . . . . . . . . .
Camera and RTK-GPS mounted on the roof of a vehicle. . . . . .
Example input images. . . . . . . . . . . . . . . . . . . . . . . . .
Ground truth GPS positions. . . . . . . . . . . . . . . . . . . . .
vi

27
28
29
32
34
35
36

2.8 Input GPS positions (experiment 1). . . . . . . . . . . . . . . . .
2.9 Estimated GPS positions (experiment 1). . . . . . . . . . . . . . .
2.10 Position errors in each frame (experiment 1). . . . . . . . . . . . .
2.11 Relationship between weight cfloat and average position errors (experiment 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.12 Input GPS positions (experiment 2). . . . . . . . . . . . . . . . .
2.13 Estimated GPS positions after a GPS outage (experiment 2). . . .
2.14 Position errors in each frame (experiment 2). . . . . . . . . . . . .
2.15 Change in energy during sequential process (experiment 2). . . . .
3.1
3.2
3.3

Flow of the proposed method using aerial images. . . . . . . . . .
Flow of the feature matching. . . . . . . . . . . . . . . . . . . . .
Examples of road signs in an aerial image from Google Maps
[maps.google.com]. . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Reprojection errors for ground-view (perspective) images and an
aerial (orthographic) image. . . . . . . . . . . . . . . . . . . . . .
3.5 Reprojection error for ground-view (perspective) image. . . . . . .
3.6 Reprojection error for an aerial (orthographic) image. . . . . . . .
3.7 Criterion used in RANSAC for BA. . . . . . . . . . . . . . . . . .
3.8 Example input ground-view images (experiment 1). . . . . . . . .
3.9 Experimental environment and results (experiment 1). . . . . . .
3.10 Rates and numbers of frames in which all selected matches arecorrect (experiment 1). . . . . . . . . . . . . . . . . . . . . . . . .
3.11 Selected inliers for example images (experiment 1). The solid anddashed lines represent correct and incorrect matches, respectively.
The relative angle and scale of the matched feature points areshown in bottom-right table along with the corresponding line colors. The green points are the ground truths of the camera positions. Note that RANSAC with/without orientation check for (b)and scale check for (d) gave the same results. . . . . . . . . . . . .
3.12 Examples of incorrect matches by RANSAC using orientation andscale check (experiment 1). The interpretations of the symbols are
the same as in Figure 3.11. . . . . . . . . . . . . . . . . . . . . . .

vii

39
40
41
42
43
45
46
46
49
50
53
54
55
57
58
60
61
63

64

65

3.13 Relationship between weight ωΩ and average horizontal position
error (experiment 1). . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.14 Number of inlier frames with variable threshold αth (experiment 1). 66
3.15 Number of trials and inlier frames derived by each trial (experiment 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.16 Horizontal position error in each frame (experiment 1). . . . . . . 68
3.17 Example input ground-view images (experiment 2). . . . . . . . . 69
3.18 Experimental environment and results (experiment 2). . . . . . . 70
3.19 Examples of frames selected as inliers by RANSAC during the BA
stage (experiment 2). The solid and dashed lines represent correctand incorrect matches, respectively. . . . . . . . . . . . . . . . . . 71
3.20 Examples of frames selected as outliers by RANSAC during the BA
stage (experiment 2). The dashed lines represent incorrect matches. 71
3.21 Horizontal position error in each frame (experiment 2). . . . . . . 72
4.1
4.2
4.3

4.4

4.5

Flow of the proposed method using a 3D point database createdusing SfM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Evaluation vehicle. . . . . . . . . . . . . . . . . . . . . . . . . . .
Examples of input images (right), the database images identified
through topometric localization (left), and the results of feature
matching between these images (red line, inlier; blue line, outlier).
The vehicle poses (red) and 3D positions of the feature points(gray) for the database images, the vehicle poses estimated by the
proposed method (blue), and the reference vehicle poses (green). .
Histograms of the errors. . . . . . . . . . . . . . . . . . . . . . . .

viii

76
78

80

82
83

List of Tables
2.1
2.2
2.3
2.4

Specifications of the RTK-GPS receiver. . . . . . . . . . . . . . .
Comparison of position errors (experiment 1) [m]. . . . . . . . . .
Relationship between l and average position errors (experiment 1)[m]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Comparison of position errors (experiment 2) [m]. . . . . . . . . .

41
44

4.1

Computation time of the proposed method [ms]. . . . . . . . . . .

84

ix

34
41

Chapter 1
Introduction
Camera pose estimation for an image sequence has been widely investigated andused for many computer vision and virtual reality applications, including thefollowing.
• Augmented reality [1, 2]• Robot navigation [3]• Match move [4]• 3D reconstruction [5–7]• Free-viewpoint image generation [8, 9]• Super-resolution [10]• Video stabilization [11]Figure 1.1 shows some example applications. For augmented reality and match
move, which are applied in navigation systems, educational materials, and film
making, camera pose estimation is required for superimposing virtual objects
into geometrically correct positions. To virtualize a real environment, 3D reconstruction and free-viewpoint image generation techniques have been thoroughly
investigated, and are based on the camera pose information used to integratemany input images. One of the requirements of robot navigation is the ability
to determine the robot’s location, which can be realized through camera pose1(a) Augmented reality [1](b) 3D reconstruction [7]Free-viewpoint image

Novel camera position (red)(c) Free-viewpoint image generation [9]Input image

Super-resolved image(d) Super-resolution [10]Figure 1.1: Applications of camera pose estimation.

2

estimation. In this manner, camera pose estimation is one of the fundamental
tasks in computer vision, virtual reality, and robotics applications.
For estimating camera poses, Structure-from-Motion (SfM) methods, which
estimate camera poses and 3D positions of feature points from an image sequence,are frequently used. The most significant problem of SfM is the accumulation of
estimation errors in a long image sequence. Although many types of methods
for reducing accumulative errors have been proposed, SfM methods essentiallycannot be free from accumulative errors unless certain external references (e.g.,
GPS, aerial images, or feature landmarks) are given. In addition, SfM without
external references can only be used to estimate relative camera poses and not
absolute camera poses, which are required for certain applications such as robot
navigation.
To treat various scenes for which the appropriate external references are notunique, this thesis proposes camera pose estimation methods employing the following three types of references: (1) GPS, (2) aerial images, and (3) a 3D point
database created by SfM. GPS and aerial images are already available for mostoutdoor scenes around the world. To reduce accumulative errors in SfM, we
propose bundle adjustment (BA)-based methods that can globally optimize camera poses using GPS and aerial images. Some applications of robot navigation
and augmented reality require estimating the camera poses along a previously
taken route. For these applications, we propose an online camera pose estimation method using a 3D point database created by SfM from previously captured
images.
Various methods for estimating camera poses have been proposed. In thischapter, we first review existing camera pose estimation methods with and without external references. We then describe the contributions of this thesis against
these existing methods. Finally, we describe the remaining organization of thisthesis.

3

1.1. Camera Pose Estimation without External
References
This section describes SfM methods that estimate camera poses and 3D positionsof feature points from the correspondences of feature points among input images. We first review the SfM methods in terms of their estimation approaches.
Techniques used to reduce accumulative errors in SfM are then detailed.

1.1.1 Structure-from-Motion
Many SfM methods have been proposed in the fields of computer vision androbotics [12–14]. These methods can be classified as follows based on their estimation approaches.
• Epipolar geometry• Factorization• Filtering• Local bundle adjustment
In the following, we describe the characteristics of these methods.
Epipolar geometry
The epipolar geometry represents the geometric relationship among two, three,and four cameras [12]. Because the epipolar geometry for three and four cameras
can be considered as combination of the epipolar geometry for two cameras, theepipolar geometry for two cameras has been thoroughly studied.
The relationship of the 2D positions of the corresponding points between
two cameras is represented using a fundamental matrix. If there are eight ormore correspondences, the fundamental matrix can be computed using the eightpoint algorithm [15]. Hartley [16] proposed the normalized eight-point algorithm
to achieve a robust estimation against image noises. Because the rank of thefundamental matrix is two, a fundamental matrix can be computed through sevencorrespondences using a rank constraint [17].
4If the intrinsic camera parameters are known, a fundamental matrix can beconverted into an essential matrix, from which the relative camera pose can becomputed. In addition, methods that directly compute an essential matrix fromfive correspondences have been proposed [18–21]. If the intrinsic camera parameters are unknown, self-calibration methods that simultaneously estimate therelative pose and intrinsic parameters are required. A simple method [12] calculates the focal length from a fundamental matrix. Moreover, methods that
estimate the relative pose and focal length from six correspondences have also
been proposed [20, 22].
These epipolar-geometry-based techniques are useful for certain applications
such as 3D reconstruction from two images. More importantly, these methods
can be used as initialization procedures in the filter-based and local-BA-based
methods described later in this thesis.
Factorization
Factorization [23] is a method for linearly estimating the camera poses and 3D
positions of feature points from the 2D correspondences of the feature points
among the input images. Factorization was originally developed for orthographic
cameras [23], and was later extended to paraperspective cameras [24], projective
cameras [25] and perspective cameras [26].
One disadvantage of factorization methods is that they require a complete setof 2D correspondences, i.e., each point must be visible in each frame. Althoughmethods that can deal with missing components of correspondences have been
proposed [27, 28], they require iterations to solve this problem. Moreover, it
is diﬃcult for factorization methods to handle incorrect correspondences, which
often occur for long video sequences.
FilteringTo estimate the camera poses and 3D positions of feature points sequentially for
an image sequence, stochastic filters are sometimes used. Davison et al. [3] proposed a method based on the extended Kalman filter (EKF) that can deal with
the non-linearity of perspective projections in the observation model. Civera
et al. [29] proposed RANSAC for EKF-based SfM to achieve robust estimation
5

against incorrect correspondences of 2D feature points. Eade and Drummond [30]proposed a method based on FastSLAM using the Rao-Blackwellized particle filter, which is more accurate than EKF SLAM in the field of robotics. One problemof filter-based SfM is that it has diﬃculty handling large-scale environments withmany feature points. To solve this problem, Eade and Drummond [31] proposeda method for dividing large-scale environments into smaller sub-spaces.
These filter-based methods tend to be employed in real-time applications suchas robot navigation and augmented reality because they can estimate camera
poses with low computational cost. However, Strasdat et al. [32] showed that the
local-BA-based methods described later in this thesis can achieve a more accurate
estimation than filter-based methods.
Local Bundle Adjustment
BA optimizes the camera poses and 3D positions of feature points so as to nonlinearly minimize the sum of the reprojection errors. BA was originally used as
the final optimization process for SfM methods owing to its high computationalcosts.
Thanks to recent advances in the computational power of PCs, some methods[1, 33–40] have employed BA in the incremental processing of an image sequenceby limiting the range of BA, which is called local BA. These methods first estimate
the initial camera poses and 3D positions of the feature points using the epipolargeometry. When each new frame is input, these methods then (1) estimate thecamera pose by solving the PnP problem, (2) estimate the 3D positions of thefeature points through triangulation, and (3) apply local BA.
As an initial attempt, Nister et al. [33] proposed a real-time method that
employs this incremental framework except for applying local BA. Zhang andShan [34] and Engels et al. [35] employed local BA that uses only a small numberof the most recent images. Mouragnon et al. [36] and Klein and Murray [1]employed key-frames that are spatio-temporally distant from each other, insteadof the most recent frames. To deal with large-scale environments, Holmes andMurray [37] proposed a method for dividing the target environments into small
sub-spaces. In addition, methods that consider environmental changes [38,39] anda rolling shutter camera [40] have been proposed. As mentioned in the previous
6

(a)

(b)

Figure 1.2: SfM from community photos [41].
section, Strasdat et al. [32] showed that local-BA-based methods can achieve amore accurate estimation than filter-based methods.
Snavely et al. [41] proposed a local-BA-based method that can deal with
unordered images and applied their method to community photos downloaded
from the Internet. Figure 1.2 shows example results of SfM from community
photos. Because community photos include a significant number of images, thecomputational cost becomes quite expensive. To solve this problem, as with keyframe approaches, Snavely et al. [42] selected a smaller number of images for theoriginal estimation and folded in the remaining images at the very end. Wu [43]developed VisualSFM, a state-of-the-art implementation of local-BA-based SfM
that can handle unordered images as well as image sequences.

1.1.2 Techniques to Reduce Accumulative Errors
Since SfM methods essentially suﬀer from accumulative errors owing to theirincremental manner, techniques to reduce accumulative errors have also been
investigated and applied. Such techniques are based on BA and loop closing.
Bundle Adjustment
BA is a procedure for optimizing camera poses and 3D positions of feature points
so as to non-linearly minimize the sum of squared reprojection errors. BA is themost accurate way to address the SfM problem because BA can be consideredas the maximum likelihood estimation when errors in the 2D positions of the7(a) Image of scene(b) Before BA(c) After BA

Figure 1.3: Example of the eﬀectiveness of BA [45].
correspondences can be assumed to be normally-distributed [44]. Figure 1.3 shows
an example of the eﬀectiveness of BA.
BA requires solving the large non-linear least squares problem and is computationally expensive. To reduce the computational cost, some methods [44–47]
exploit the sparsity of BA, i.e., each reprojection error depends only on one camera pose and one 3D position of a point.
On the other hand, Hedborg et al. [40] proposed BA considering a rolling
shutter model. To improve the robustness against the outliers of the correspondences, Dai et al. [48] proposed BA using the L1 norm instead of the squared(L2) norm. Implementations of BA have been recently made available [49–52].
Loop Closing
As shown in Figure 1.4, loop closing is a technique for reducing accumulative errors by detecting loops, i.e., camera returns to a position where the camera passes
before. Once the loops are detected, the accumulative errors can be reduced by
the BA. The problem then is how to detect the loops. For oﬄine SfM methods,
it is easy to detect loops by matching the images with each other. In contrast,
for online real-time SfM methods, this is relatively diﬃcult owing to the real-time
constraint. For eﬃcient loop detection, Angeli et al. [53] employed bag-of-visualwords, which are often used for image retrieval, and Strasdat et al. [54] employeda key-frame approach. In addition, Williams et al. [55] compared loop closing
techniques and summarized their characteristics.
As an approach related to loop closing, Cohen et al. [56] exploited symmetries that often exist in man-made structures, e.g., two identical wings of a large
8(a) Before loop closing(b) After loop closing(c) Aerial view of courtyard

Figure 1.4: Loop closing for the trajectory around a courtyard [55].
building complex. They employed BA by considering the detected symmetries toreduce accumulative errors (Figure 1.5).
Although these loop closing techniques can reduce accumulative errors, theapplicable environments and movements of a camera are limited.

1.2. Camera Pose Estimation with External References
As described in the previous section, while many SfM methods have been proposed, they suﬀer from accumulation of estimation errors in a long image sequence. Although loop closing techniques can reduce accumulative errors, theapplicable environments and camera movements are limited, and SfM methods
cannot be free from accumulative errors unless certain external references areprovided.
This section reviews camera pose estimation methods using external references. These external references can be classified into the following categories.
• Sensors• Pre-knowledge of the target environments
9(a) SfM reconstruction(b) Estimated similarity transformations(c) Result

Figure 1.5: BA exploiting symmetry [56].
– Fiducial markers– Image databases– 3D models– 3D point databases– Aerial images– Road maps
In the following, we describe the characteristics and problems of methods using
external references.

1.2.1 Sensors
Sensors used for estimating camera poses can be classified into two types: (1) sensors that measure relative poses, such as odometry, and (2) sensors that measureabsolute poses, such as RFID and GPS.
10

Measuring Relative Poses
Sensors that measure relative poses, such as odometry, can estimate accurate
poses in a short period of time. Eudes et al. [57] and Michot et al. [58] proposed
methods fusing odometry estimates into SfM to reduce the accumulation of errors
in the scale parameters. Michot et al. [58] also proposed a method using a gyroscope to reduce the accumulation of errors in the posture parameters. However,sensors that measure relative poses also suﬀer from accumulative errors over long
periods of time. Thus, it is diﬃcult for these sensors to remove accumulative
errors in a long image sequence.
Measuring Absolute Poses
To measure absolute poses, GPS and other kinds of infrastructure sensors suchas RFID and infrared beacon can be used.
Some methods estimate camera poses directly from sensors. Tenmoku etal. [59] used RFID and infrared beacons (Figure 1.6(a)) to measure absolute
poses in an indoor environment. They also used a pedometer to estimate therelative poses in cases when the RFID and infrared beacons cannot be used.
Piekarski et al. [60] used a backpack with GPS and an Inertial Measurement
Unit (IMU) (Figure 1.6(b)) to obtain camera poses in an outdoor environment.
Kourogi et al. [61] proposed a method using RFID, GPS, and odometry to handle
both indoor and outdoor environments. Methods for estimating camera poses
directly from sensors are robust against rapid camera movements. However, forvision applications such as 3D reconstruction and augmented reality, it is diﬃcult
for these methods to achieve pixel-level registration owing to calibration errors
between sensors and the camera.
Some methods fuse sensors into SfM to reduce the accumulative errors. Ramachandran et al. [62] proposed an optimization procedure based on reprojection
errors and the direction of gravity measured by inertial sensors. Although thismethod produces better solutions than ordinary BA, it is diﬃcult to reduce theaccumulation of errors in the position and scale parameters.
In outdoor environments, some methods use GPS together with SfM owing
to its availability. These methods can be classified in terms of their fusion typeas follows.
11Infrared beacons

RFID tags and a tag reader(a) Positioning infrastructures [59](b) Backpack including GPS and IMU [60]Figure 1.6: Sensor examples.
• Epipolar-geometry-based method [63]• Fitting-based methods [64–66]• Filter-based methods [67–69]• BA-based methods [70–73]
Carceroni et al. [63] proposed a method for estimating the essential matrices
from the correspondences of the 2D points and given camera positions using GPS.
Like the SfM methods without references described in the previous section, thismethod is useful for the initialization of BA-based methods.
Fitting-based methods [64–66] fuse GPS data into SfM by fitting the camera
positions estimated by SfM to those by GPS using a similarity transform. Although Bok et al. [65] and Wei et al. [66] employed local fitting, the accumulative12

errors cannot be removed through a similarity transform because errors are notuniformly accumulated in SfM.
Filter-based methods [67–69] tend to be employed for real-time applications
because GPS and vision data can instantly be fused from the previous state andthe current measurement, for example, using the Kalman filter. One problem offilter-based methods is the diﬃculty of global optimization owing to the sequential
updating strategy of the filter design.
Some methods [70–73] employ extended BA that minimizes the energy function defined as the sum of reprojection errors and the penalty term of the GPS.
The extended BA can globally optimize the camera poses, which works fine if
GPS data are accurately acquired. However, the accuracy of estimated camera
positions depends largely on the confidence of GPS positioning data because existing extended-BA methods do not consider the GPS positioning confidence, andthe error of GPS positioning easily grows to a level of 10 [m] in urban areas.

1.2.2 Fiducial Markers
Fiducial markers are artificial 2D objects whose appearances are designed to beeasily determined through image processing techniques. Once fiducial markers
are determined, camera poses in the marker coordinate system can be estimatedby solving the PnP problem. Kato and Billinghurst [74] proposed one of themost famous camera pose estimation methods based on fiducial markers called
ARToolKit (Figure 1.7(a)). This method is simple and useful, especially foraugmented reality applications. However, to estimate the camera poses in largescale environments using fiducial markers, many markers should be installed in
the environment, which requires significant manual intervention and results in a
disturbance in the scenery. To avoid disturbing the scenery, invisible markers [75](Figure 1.7(b)) and markers designed to blend in with the scenery as wall paper[76] (Figure 1.7(c)) and posters [77] (Figure 1.7(d)) were proposed. However, theproblem of manual intervention still remains.

13Capturing without a flash(a) ARToolKit marker [74]Capturing with a flash(b) Invisible markers [75](c) Wall paper [76](d) Posters [77]Figure 1.7: Examples of fiducial markers.

1.2.3 Image Databases
As shown in Figure 1.8, the image database consists of images and their camera poses estimated beforehand by sensors or SfM. Methods utilizing an image
database [78–89] estimate camera poses by identifying the database image thathas the most similar appearance to the current image. These methods can beclassified into methods for a single image and methods for an image sequence.
For methods using a single image, Wang et al. [78] proposed a method using
vocabulary trees [79], which is an eﬃcient technique for image retrieval, to identify
the most similar image. After identifying the most similar database image, somemethods [78, 80, 81] use epipolar geometry to estimate the relative poses between
the current image and the identified database image. Through recent advances inimage retrieval techniques and community photos whose camera poses are tagged
14

(a) Positions of images(b) Examples of images

Figure 1.8: Example of an image database [88].
by sensors, this approach can handle environments from the city scale [82, 83]
up to the entire planet [84]. In addition, for community photos without pose
information, Kalantidis et al. [85] proposed a method to automatically determine
landmarks using Wikipedia [wikipedia.org].
For methods using an image sequence, the methods using a single image described above can be used by applying the particular method to each image ofthe image sequence. However, estimated camera poses may change discontinuously between successive images because each image is treated independently.
Therefore, the methods for an image sequence consider spatio-temporal information between successive images. Yagi et al. [86] created a route panorama
from database images. The camera poses are then continuously estimated bytracking the image pattern on the route panorama that is similar to the current
image through the use of the active contour model. Some methods [87–89] use
a topological graph to identify the database image that is the most similar tothe current image by considering the spatio-temporal connection of the database
images. Badino et al. [88] fused metric-scale information into a topological graph.
Vaca-Castano et al. [89] used street-view images available on the Internet, such as
from Google Street View [maps.google.com/streetview], as database images.
These methods utilizing an image database [78–89] require the image database
to be prepared beforehand, which is costly. Although community photos andstreet-view images can be used as a database, the available areas are still limited
15

Comport et al. [90]Oikawa et al. [94](a) Wire-frame models(b) Textured models [92]Figure 1.9: Examples of 3D models.
to famous landmarks and large cities. However, preparing a database is acceptable for certain applications in which the camera iteratively passes along thesame route. Additionally, these methods are eﬃcient for a camera that follows apreviously taken route by considering the spatio-temporal information.

1.2.4 3D Models
Three-dimensional models of scenes such as wire-frame models (Figure 1.9(a)) andtextured models (Figure 1.9(b)) are used as external references. Some methods[90–94] estimate camera poses directly from 3D models by matching the features
between input images and 3D models. Comport et al. [90] used wire-frame models
of scenes. To improve the robustness of the matching, texture information is also
used [91–93] along with wire-frame models. On the other hand, Oikawa et al. [94]16

addressed the problem of tracking textureless rigid curved objects using quadrics
to approximate the object contours.
Some methods fuse 3D models into SfM [95–98]. These methods can reduce
the accumulative errors in SfM using the following 3D models.
• Wire-frame models [95]• Plane-based models [96, 97]• Textured 3D models [98]• Digital elevation models (DEM) [97]Approaches fusing 3D models into SfM can be classified into switching [95, 96]
and extended BA [97, 98]. Bleser et al. [95] and Lothe et al. [96] simply switch
the SfM and model-based estimation depending on the availability of the models.
As with methods using GPS with SfM, some methods [97, 98] have employedextended BA because it can globally optimize the camera poses using sparsely
acquired information on the camera poses estimated from the models.
One disadvantage of 3D-model-based methods is that the manual intervention
required to create the 3D models is costly. Although some models are alreadyavailable in the GIS database [96, 97], the available areas are still limited to large
cities.

1.2.5 3D Point Databases
As shown in Figure 1.10, some methods [99–108] employ a database that consistsof 3D points and their image features and estimate camera poses by matching thefeature points between input images and the database. Since 3D points can be
estimated from images using SfM methods, less manual intervention is required
than in 3D-model-based methods. The main problem of this approach is how
to accurately and eﬃciently estimate the feature matches. Approaches to thisproblem depend on whether the methods estimate the camera pose for a single
image or for an image sequence.
Regarding single-image-based methods, an earlier method [99] simply foundfeature matches using a brute force search. Irschara et al. [100] clustered 3D points
17

Figure 1.10: 3D point database [100]. Camera poses are estimated by matching
the feature points between the input images and 3D point database.
by generating virtual views and projecting 3D points with image features into
these views. The search space is then limited using an image retrieval techniquethat identifies a virtual view including the image features most similar to thoseof the current image. Li et al. [101] introduced priorities of feature points toaccelerate the search process. When a match is found, the priorities increase forthe points that are visible with the matched point in a database image. Sattler et
al. [102] matched feature points between the input image and the database using
a vocabulary tree [79].
In image-sequence-based methods, temporal information is helpful to limit
the search space. Arth et al. [103] limited the search space by clustering 3D
points using the visibility of the feature points. Wientapper et al. [104] estimateda tentative camera pose through the use of the EKF with an inertial sensor.
Matches are then searched for only among those feature points that are projected
onto the tentative camera’s field of view. Taketomi et al. [105] also estimated atentative camera pose by tracking the feature points temporally. Lim et al. [106]also tracked the feature points temporally and searched only images in which the
tracked feature points are detected.
As with the methods using other references, some methods [107, 108] fusethe 3D point database into SfM. These methods employ BA-based optimization
because it can globally optimize the camera poses.
These methods utilizing a 3D point database require the database to be prepared in advance, which is costly even if an SfM method can be used. However,18

(a) Perspective image [109](b) Orthogonal image [115]Figure 1.11: Examples of aerial images.
as with the methods utilizing an image database, preparing the database is admissible for certain applications in which the camera iteratively travels along thesame route.

1.2.6 Aerial Images
Aerial images have recently become available for most outdoor scenes aroundthe world, such as Google Maps [maps.google.com] and Microsoft Bing Maps[bing.com/maps]. Focusing on this availability, camera pose estimation methods
using aerial images have been proposed. There are two types of aerial images:
perspective images (Figure 1.11(a)) and orthogonal images (Figure 1.11(b)).
Bansal et al. [109] proposed a method for estimating camera poses by matchingfaçades in the ground-view input image with perspective aerial images. Althoughperspective aerial images are available on Google Maps and Microsoft Bing Maps,
the available areas are still limited to large cities.
Most methods using aerial images employ orthogonal aerial images that areavailable for most outdoor scenes and correspond to absolute 2D positions, i.e.,
latitudes and longitudes. These methods can be classified into learning-based[110], edge-based [111, 112], and feature-point-based [113–115] methods.
Lin et al. [110] proposed a method using community photos with position
information and aerial images. Their method first searches for community photos
that are similar and dissimilar to input ground-view images. Then, using aerial
19images at positions in the identified community photos, a support vector machine(SVM) is trained and applied to sliding windows over aerial images to estimate thecamera position of the input ground-view image. Although this method estimatesthe camera positions from large regions (1,600 [km2 ] in their experiments), only
rough positions of the cameras can be estimated.
Kim et al. [111] and Leung et al. [112] proposed methods for estimating camera
poses by matching building edges in the ground-view images to those in the aerial
images. As a related approach, Cham et al. [116] estimated camera poses for anomnidirectional image using a 2D building map instead of aerial images.
Other methods [113–115] match the feature points between input ground-view
images and aerial images. However, it is not easy to find good matches for all
images in a long video sequence, especially for scenes where unique landmarks
cannot be observed. Mills [117] and Toriya et al. [113] proposed robust featurepoint matching procedures that compare the orientation and scale of the matches,
which are from feature descriptors such as SIFT [118] and SURF [119], with thedominant orientation and scale identified through a histogram analysis. However,this does not work well when a very large number of outliers exist. Toriya et
al. [113] and Noda et al. [114] relaxed the problem by generating mosaic images of
the ground from multiple images for feature matching. However, the accumulative
errors in a mosaic image are not considered in these methods. To resolve thisproblem, Pink et al. [115] fuse sparsely obtained camera poses estimated fromaerial images into SfM using the Kalman filter. However, it is diﬃcult for the
Kalman filter to globally optimize the camera poses.
Methods based on matches of the feature points first estimate the homography matrices, and the camera poses are then extracted from the homography.
It should be noted that a homography can represent the relationship between
ground-view and aerial images only when the ground is flat. To overcome thislimitation, Sekii et al. [120] treated feature points on aerial images as 3D points
whose horizontal 2D positions are known and altitudes are unknown. Camera
poses are then estimated from these 3D points and their 2D positions on the
ground-view image.

20

(a) OpenStreetMap (OSM)(b) Road map extracted from OSM

Figure 1.12: Example road maps [121].

1.2.7 Road maps
In addition to aerial images, road maps are also available for most outdoor scenes
around the world. Brubaker et al. [121] proposed a method that employs SfM
with OpenStreetMap [openstreetmap.org], which is a community-developed roadmap (Figure 1.12). Although this method can reduce accumulative errors bymatching the trajectory from SfM to the road maps, there are ambiguities forcertain types of scenes such as straight roads or Manhattan worlds.

1.3. Contributions of this Thesis
In the previous sections, existing camera pose estimation methods were reviewed;
in addition, we indicated that external references should be used to remove accumulative errors. Many kinds of external references have been proposed, and theyshould be selected depending on the particular situation or application. Figure
1.13 summarizes the characteristics of external references in terms of the amountof manual intervention and the extent of the applicable environments. In thisthesis, we focus on the following two situations.
• Situations in which it is necessary to estimate the camera poses without apre-measurement of the target environments. This type of situation applies
21Less manual intervention
GPS, Aerial images, Road maps
Already
available

Community photos
Image databases, 3D point databases
Fiducial markers
3D models
Indoor Outdoor

Large-scale
environment

Figure 1.13: Characteristics of external references in terms of the amount ofmanual intervention and the extent of the applicable environments.
to a 3D reconstruction, match move, free-viewpoint image generation, andso on.
• Situations in which it is necessary to estimate the camera poses along apreviously taken route. This type of situation applies to robot navigation,
augmented reality, and so on.
In the following, we detail the contributions of the present study for each situation
by selecting the appropriate external references.

1.3.1 Camera Pose Estimation without a Pre-Measurementof the Target Environments
As shown in Figure 1.13, the following external references are already available
without a pre-measurement of the target environments by users.
• GPS

22

• Pre-knowledge of a target environment created from community photos
and street-view images such as image databases, 3D models, or 3D point
databases• Aerial images• Road maps
Although community photos and street-view images have been increasing in number, the available areas are still limited to famous landmarks and large cities. In
contrast, GPS, aerial images, and road maps are available for most outdoor scenes
around the world. Road-map-based methods suﬀer from ambiguities in certain
types of scenes such as straight roads or Manhattan worlds. Therefore, we focus
on GPS and aerial images as external references.
It is not easy to estimate camera poses for all images in a long video sequence
directly from GPS and aerial images owing to the sampling rate and matchingdiﬃculty. We thus propose techniques that fuse SfM with GPS and aerial images
using BA because BA-based methods can globally optimize camera poses using
sparsely acquired information on camera poses from external references.
In methods using GPS, extended BA that fuses SfM and GPS data has beenpreviously proposed [70–73] and it works fine if the GPS data are acquired accurately. However, the accuracy of the estimated camera position depends largelyon the confidence of the GPS positioning data because such methods do not consider the GPS positioning confidence. To solve this problem, we add weighting
coeﬃcients depending on the GPS positioning confidence to the energy function
for extended BA.
No existing method uses aerial images as external references in the BA. We
propose a new SfM pipeline that uses feature matches between ground-view andaerial images. To find good matches from unreliable matches, we newly propose
RANSAC-based [122] outlier elimination methods in both feature matching and
BA stages.

23

1.3.2 Camera Pose Estimation along a Previously Taken
Route
Some robot navigation and augmented reality applications require estimating thecamera poses along a previously taken route. For these applications, most of
the references described in the previous section can be used because they can beprepared beforehand. We focus on methods using a 3D point database and an
image database because, as shown in Figure 1.13, such databases can be createdfor large-scale environments with relatively less human eﬀort using SfM methods.
As mentioned in Section 1.2.5, the main problem of methods using a 3D point
database is how to obtain matches between the input images and the database.
To limit the search space, existing methods track the feature points temporally.
However, tracking sometimes fails owing to occlusions and a rapid camera movement. On the other hand, without tracking the feature points, methods using animage database can eﬃciently identify the database image that is most similar tothe current image by considering a spatio-temporal connection. We thus propose
a method using a 3D point database that employs an image-database method tolimit the search space.

1.4. Organization of this Thesis
The rest of this thesis is organized as follows. Chapter 2 describes a camera
pose estimation method that fuses SfM and GPS data using extended BA whileconsidering the confidence of GPS positioning. Chapter 3 describes a new SfM
pipeline that uses feature matches between ground-view and aerial images, witha robust feature matching procedure employing a two-stage RANSAC. Chapter
4 describes a method for estimating camera poses online from 2D positions of
the feature points in the current image and their 3D positions obtained from thedatabase by considering both the topological information and the image features.
Finally, Chapter 5 summarizes this thesis.

24Chapter 2
Extended Bundle Adjustmentusing GPS Positioning and Its
Confidence
2.1. Introduction
This chapter describes a camera pose estimation method using GPS positioning
data as external references that are already available in most outdoor scenes
around the world. As described in Chapter 1, the most significant problem in
SfM is the accumulation of estimation errors in a long image sequence. Althoughmany kinds of methods that reduce accumulative errors have been proposed,
SfM methods essentially cannot be free from accumulative errors unless certain
external references are given.
In this study, to fuse GPS data into SfM, we employ the framework of extended
BA that can globally optimize camera poses using sparsely acquired camera positions from GPS. Although extended-BA methods using GPS data have beenpreviously proposed [70–73] and work fine if GPS data are accurately acquired,existing methods have the following problems.
• The accuracy of the estimated camera position depends largely on the confidence of the GPS positioning data because such methods do not consider
the GPS positioning confidence.
25• The solution often converges to a local minimum when GPS data cannot
be acquired for a long period of time.
To resolve these problems, we add weighting coeﬃcients depending on the GPS
positioning confidence to the energy function for extended BA. To avoid the local
minima, camera positions estimated without GPS data are fitted to the GPS
positions prior to the optimization.
The proposed method basically follows the framework of existing extendedBA method [70]. As shown in Figure 2.1, the camera pose estimation (A) and 3D
position estimation of the feature points (B) are repeated sequentially for eachframe, from the first frame to the last [5]. In this repetition, the local optimization
process (D) is applied for each frame in which GPS data are obtained to reduce
the accumulative errors. Here, to avoid the local minima, parameter fitting to the
GPS positions (C) is applied for frames in which the GPS positioning is recovered
after a GPS outage. After estimating the initial camera poses using processes(A) through (D), the estimated poses are globally refined (E). In processes (D)and (E), a common energy function is minimized. In the following, the energyfunction is first defined using reprojection errors and the penalty terms for the
GPS positioning. The optimization process with parameter fitting (C) is thendetailed.

2.2. Energy Function Considering GPS Positioning Confidence
To fuse GPS data into SfM, as with existing extended-BA methods [70–73], theenergy function is minimized. The energy function Egps is defined using the sumof reprojection errors Φ and the penalty term of GPS Ψ as follows:
Egps ({Ri , ti }Ii=1 , {pj }Jj=1 ) = Φ({Ri , ti }Ii=1 , {pj }Jj=1 ) + ωΨ Ψ({Ri , ti }Ii=1 ),(2.1)

where Ri and ti represent the rotation and translation from the world coordinate
system to the camera coordinate system for the i-th frame, respectively; pj is the
3D position of the j-th feature point; I and J are the numbers of frames andfeature points, respectively, and ωΨ is a weight that balances Φ and Ψ.
26

(A) Camera pose estimation by feature tracking(B) 3D position estimation of feature points

NYNY(C) Parameter fitting to GPS positions(D) Local optimization using GPS positions(E) Global optimization using GPS positions
Figure 2.1: Flow diagram of the proposed method using GPS, where f is theframe index, G is a set of frames in which GPS data are obtained, and Grecoveredis a set of frames in which GPS positioning is recovered after a GPS outage.

27

3D position of -th feature point:-th camera coordinate system
Projected 2D position :Reprojection error
Detected 2D position :Figure 2.2: Reprojection error.
Here, to treat unreliable GPS data, we newly add weighting coeﬃcients depending on the GPS positioning confidence to the penalty term of GPS Ψ. In thefollowing, the energy associated with the reprojection error Φ and the penalty
energy for GPS positioning Ψ are both detailed.

2.2.1 Reprojection Errors
As shown in Figure 2.2, the reprojection error is the distance between the detected
2D position of the feature point and the projected position of the corresponding
3D feature point. The reprojection error has often been used in SfM. In this study,
the energy term associated with the reprojection error Φ is defined as follows:( )  Xijc  2
I ∑
∑
xij
1
Zc
Φ({Ri , ti }Ii=1 , {pj }Jj=1 ) = ∑I
µj
(2.2)
−  Yijijc  ,
y
|P
|
ij
c
i
i=1
i=1 j∈P
Zi(Xijc , Yijc , Zijc )T = Ri pj + ti ,

ij

(2.3)

where Pi is a set of the feature points detected in the i-th frame, (xij , yij )T is thedetected 2D position of the j-th feature point in the i-th frame, and µj represents
the confidence of the j-th feature point, which is computed from the reprojection
errors in the sequential process [5].

2.2.2 Penalty Term for GPS Positioning
As shown in Figure 2.3(a), existing extended-BA methods using GPS [70–73]
define the penalty term for GPS positioning through the distance between the28

-th camera
coordinate
system,GPS
coordinate
systemPosition of GPS receiver incamera coordinate system䠖

Observed GPS position in
GPS coordinate system䠖

Estimated GPS position in
GPS coordinate system䠖

(a) Distance between observed and estimated GPS positions

Estimated GPS position in
GPS coordinate system䠖

Observed GPS position in
GPS coordinate system䠖
Case of high confidence
GPS positioning

Case of low confidence
GPS positioning(b) Weighting coeﬃcient depending on the GPS positioning confidence

Figure 2.3: Energy term with respect to GPS positioning.

29

observed and estimated GPS positions as follows:Ψ̂({Ri , ti }Ii=1 ) =1 ∑ gps|g − RiT (dc − ti )|2 ,|G| i∈G i(2.4)

where G is a set of frames in which the GPS positioning data are obtained, gigps isthe observed GPS position for the i-th frame in the GPS coordinate system, anddc indicates the position of the GPS receiver in the camera coordinate system.
It should be noted that Lhuillier [72] and Larnaout et al. [73] set dc = 0.
As shown in Figure 2.3(b), we newly define the weighting coeﬃcients depending on the GPS positioning confidence ci . The penalty term for GPS positioning
is defined as follows:()2
1 ∑ |gigps − RiT (dc − ti )|
I
.
(2.5)
Ψ({Ri , ti }i=1 ) =|G| i∈Gci
Here, ci should be determined depending on the dilution of precision (DOP), a
type of solution in RTK-GPS (RTK-fix, RTK-float), and other error factors. In
the experiments described later, ci is determined by observing the GPS positioning data at a fixed point for a long time period.

2.3. Optimization by Minimizing Energy Function
2.3.1 Range of Optimization
In processes (D) and (E) shown in Figure 2.1, to optimize the camera poses and3D positions of the feature points, the energy function Egps defined in Equation(2.1) is non-linearly minimized. The diﬀerence in processes (D) and (E) is therange of optimized frames. In process (D), to reduce accumulative errors in thesequential process, the parameters from the (f − l)-th frame to the current frame(f -th frame) are refined. In process (E), the parameters for all frames are refined
to globally optimize the camera poses.

30

2.3.2 Parameter Fitting to GPS Positions
During a long GPS outage, the local optimization process (D) does not work well.
Therefore, in the optimization process after GPS positioning is recovered from a
GPS outage, the energy often converges to a local minimum because the initial
parameters include large accumulative errors (Figure 2.4(a)). Concretely, cameras
and feature points around only a frame in which the GPS positioning is recovered
are drawn to the recovered GPS position, and most cameras and feature points
remain at the original positions (Figure 2.4(b)). In this study, to avoid the local
minima, the consistency between the estimated camera poses and the recovered
GPS position is improved before the optimization by fitting the camera positions
and 3D positions of the feature points to the recovered GPS position. Thisparameter fitting process is applied except when the confidence of the recovered
GPS position is significantly low because confidence of the recovered GPS position
is usually low and the next GPS outage may occur before the high-confidence GPS
position is obtained. It should be noted that if a high-confidence GPS position is
obtained after a GPS outage, the influence of the low-confidence GPS position issuppressed by the optimization when considering the GPS positioning confidence.
For parameter fitting, a similarity transform is not appropriate because reprojection errors become significantly large in frames around the start of the GPS
outage. We therefore fit the parameters to gradually decrease the changes fromthe end of the GPS outage to the start. Concretely, the 3D positions of thefeature points and camera poses are corrected through the following steps.
1. As shown in Figure 2.4(c), the 3D position of the j-th feature point observed
during the GPS outage is updated as follows:
pj ← pj + bj (gigps − RiT (dc − ti )), mj −fs ; m − f > 0
j
s
f −fs
bj =
,
0
; otherwise

(2.6)
(2.7)

where fs is the starting frame of the GPS outage, and mj represents themedian of the frame indexes in which the j-th feature point is observed.
2. As shown in Figure 2.4(d), for the frames during the GPS outage, camera
31

Feature
point
CameraEstimated GPS
Observed GPSTime(a) Before optimization(b) After optimization without parameter
fitting(c) Parameter fitting 1: Translating points (d) Parameter fitting 2: Estimating camera poses

Figure 2.4: Parameter fitting to GPS positions.

32

poses are estimated by solving the PnP problem using the 3D positions of
the feature points updated by the previous step.
Note that the range of local optimization is set to whole frames during the GPS
outage (from the fs -th frame to the f -th frame) when this fitting process isapplied.

2.4. Experiments
To validate the eﬀectiveness of introducing weighting coeﬃcients depending on the
GPS positioning confidence and parameter fitting, we conducted two experiments
using two datasets: (1) data including many low-confidence GPS positions, and(2) data including a long GPS outage. The accuracy of the proposed method iscompared with that of the existing extended-BA method [70] quantitatively using
a real video sequence. In the following, we first describe the common aspects ofthe two experiments and how to determine the weighting coeﬃcients dependingon the GPS positioning confidence. The results of each experiment are thendetailed.

2.4.1 Experimental Setup
In the experiments, camera poses were estimated for an image sequence (1,600[pixel] × 1,200 [pixel], 2,756 frames, and 194 [s]) captured by a moving video
camera (Point Grey Research Grasshopper2) mounted on the roof of a vehicle,
as shown in Figure 2.5. Figure 2.6 shows examples of the captured images.
An RTK-GPS receiver (TOPCON GR-3) was attached to the camera, and thepositioning data were acquired at 1 [Hz] during the video capture. Table 2.1
shows the specifications of the RTK-GPS receiver. Approximately 91% of the
GPS positioning data were acquired as high-confidence GPS solutions (RTK-fix)
and used as the ground truth data. Figure 2.7 shows the positions of the groundtruth data. From these data, two datasets including low-confidence GPS solutions(RTK-float) and a GPS outage were generated by masking the GPS satellite datausing post-process software (TOPCON Tools), and the generated data were usedas the input in the experiments. The details of datasets are described in Sections
33

RTK-GPS

CameraFigure 2.5: Camera and RTK-GPS mounted on the roof of a vehicle.
2.4.3 and 2.4.4. It should be noted that the GPS signals used were actuallyacquired in the experimental environment and included the GPS error sources
such as multipaths from buildings and signal decay from trees.
The other conditions were as follows. The intrinsic camera parameters andthe position of the GPS receiver in the camera coordinate system were calibrated
in advance, and these parameters were fixed during the video capture. The video
frames and GPS input were manually synchronized. To fix the accuracy of thefeature tracking, we used feature tracks obtained by Sato et al. [5] for all methods
Table 2.1: Specifications of the RTK-GPS receiver.
Model

GR-3
GPS/GLONASS L1/L2/L5
SignalC/A and P Code & Carrier
Horizontal accuracy (RTK)
10 mm + 1 ppm
Vertical accuracy (RTK)
15 mm + 1 ppm
34

(a) 1st frame(b) 400th frame(c) 800th frame(d) 1,200th frame(e) 1,600th frame(f) 2,000th frame(g) 2,400th frame(h) 2,756th frame

Figure 2.6: Example input images.
35

200

North-South [m]1000First frame

-100

-200
0

100

200
300
East-West [m]400(a) Horizontal 2D positions

5
0
Altitude [m]

-5
-10
-15
-20
-25
-30
0

500

1000
1500
Frame number

2000(b) Altitudes

Figure 2.7: Ground truth GPS positions.
36

2500

compared. Position errors were calculated by comparing the ground truth GPS
positions with the estimated GPS positions RiT (dc −ti ). We set the range of local
optimization to l = 200, and the weight of the penalty term for GPS positioning
to ωΦ = 10−9 according to Yokochi et al. [70]. For a non-linear minimization of
the energy function, we used sparseLM [50].

2.4.2 Determination of Weighting Coeﬃcients Depending
on GPS Positioning Confidence
In our experiments, we used RTK-GPS whose accuracy depends largely on thetypes of solutions (RTK-fix, RTK-float). Thus, using these solution types, wedefined the weighting coeﬃcients depending on the GPS positioning confidence
ci as follows:
1; i ∈ Gfix
ci =
,
(2.8)
cfloat ; otherwise
where Gfix is a set of frames in which RTK-fix solution data are obtained. Here,the weight cfloat for the RTK-float solution data was determined experimentally.
Concretely, we first calculated the RMS for both RTK-fix (rms fix ) and RTK-float(rms float ) by observing the GPS positioning data at a fixed point for a long time
period. Then, cfloat was experimentally determined as cfloat = rms float /rms fix =107.4.

2.4.3 Quantitative Evaluation using Data Including Many
Low-Confidence GPS Positions (Experiment 1)
To validate the eﬀectiveness of introducing weighting coeﬃcients depending on
the GPS positioning confidence, as described in Section 2.2.2, the accuracy of thecamera positions estimated through the following methods was compared.
Method A: The existing method [70] in which the solution types of GPS
data are not considered.
Method B: The existing method [70] using only RTK-fix solution data.
37

Method C: The proposed method considering the solution types of GPS
data.
Method D: An SfM method that does not use GPS data.
Because SfM cannot estimate the absolute camera poses, we fitted the camera
positions estimated from Method D to the ground truths using a similarity transform.
In this experiment, to generate the GPS positioning data through a simulation, we assumed city environments where the confidence of the GPS positioning is occasionally low owing to the occlusion of GPS signals from buildings ortrees. As shown in Figure 2.8, the data generated consist of sparsely obtained
high-confidence (RTK-fix) GPS positions and many (86% of GPS positions) lowconfidence (RTK-float) GPS positions.
Comparison of Position Errors with the Existing Method
Figures 2.9 and 2.10 show the estimated GPS positions and position errors in eachframe, respectively. Table 2.2 shows the statistics of the position errors. These
results demonstrate that the vision-based method (Method D) was aﬀected by
the accumulative errors even when the estimated positions were fitted to theground truths through a similarity transform. Methods A, B, and C reduced
the accumulative errors using GPS data. However, Method A, which does notconsider GPS positioning confidence, was aﬀected by RTK-float solution data.
In Method B, which used only RTK-fix solution data, the errors were large while
the RTK-float solution data were obtained (500-2,000th frames). The proposed
method (Method C) obtained the most accurate positions, as shown in Table 2.2,
using the weighting coeﬃcients depending on the GPS positioning confidence.
Discussion about Influence of Parameters
Table 2.3 shows the average position errors from the proposed method with avariable range of local optimization, l, which demonstrates that the optimization
within a short range (l = 50) did not reduce the accumulative errors. Except
for a short range, position errors did not largely depend on the range of local
optimization.
38

200

RTK-fix
RTK-float

North-South [m]1000First frame

-100

-200
0

100

200
300
East-West [m]400(a) Horizontal 2D positions

5

RTK-fix
RTK-float

0
Altitude [m]

-5
-10
-15
-20
-25
-30
0

500

1000
1500
Frame number

2000(b) Altitudes

Figure 2.8: Input GPS positions (experiment 1).
39

2500

200

RTK-fix (ground truth)
RTK-fix (input)
RTK-float (input)
Method A
Method B
Method C
Method D

North-South [m]1000First frame

-100

-200
0

100

200
300
East-West [m]400(a) Horizontal 2D positions

5

RTK-fix (ground truth)
RTK-fix (input)
RTK-float (input)
Method A
Method B
Method C
Method D

0
Altitude [m]

-5
-10
-15
-20
-25
-30
0

500

1000
1500
Frame number

2000

2500

(b) Altitudes

Figure 2.9: Estimated GPS positions (experiment 1).
40

60

12
10
8

40

6
4

20

2
0

Error (Method D) [m]Error (Except for method D) [m]

14

0
0

500

RTK-fix (input)
RTK-float (input)
Method A

1000
1500
2000
Frame number
Method B
Method C
Method D

2500

Figure 2.10: Position errors in each frame (experiment 1).

Table 2.2: Comparison of position errors (experiment 1) [m].
MethodAverage Std. dev. Max
RTK-fix
0.003
0.003 0.010
RTK-float
1.844
1.093 6.641
A
1.553
1.182 6.326
B
4.298
3.944 14.041
C
1.217
1.070 4.073
D
37.18913.033 69.388

Table 2.3: Relationship between l and average position errors (experiment 1) [m].
lAvg. error

50
13.431

100
1.284

41

200
1.217

400
1.232

800
1.219

Average error [m]

1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0
0

50

100
cf loat150

200

Figure 2.11: Relationship between weight cfloat and average position errors (experiment 1).
Figure 2.11 shows the average position errors from the proposed method withvariable weighting coeﬃcients for low-confidence GPS positioning around cfloat =107.4. Note that cfloat = 1 indicates Method A, which does not consider the
GPS positioning confidence. The results demonstrate that position errors do notlargely depend on cfloat , and that cfloat obtained from a fixed point observation is
not the best value but is still suﬃcient.

2.4.4 Quantitative Evaluation using Data Including a LongGPS Outage (Experiment 2)
To validate the eﬀectiveness of the parameter fitting to avoid the local minima
described in Section 2.3.2, the accuracy of the estimated camera positions fromthe following methods was compared.
Method C: The proposed method.
Method C’: The proposed method without parameter fitting.
In this experiment, to generate the GPS positioning data through a simulation,
we assumed city environments in which GPS outages occasionally occur owing to
occlusions from buildings and trees. As shown in Figure 2.12, the data generated
include a long GPS outage (139 [s], 315-2,271th frames, and 72 % GPS positioning). We employed parameter fitting even when the confidence of the recovered
42

200

RTK-fix (input)
RTK-float (input)

North-South [m]1000First frame

-100

-200
0

100

200
300
East-West [m]400Altitude [m](a) Horizontal 2D positions

20
15
10
5
0
-5
-10
-15
-20
-25
-30

RTK-fix (input)
RTK-float (input)

0

500

1000
1500
Frame number

2000(b) Altitudes

Figure 2.12: Input GPS positions (experiment 2).
43

2500

GPS positioning was RTK-float because the errors in the RTK-float positioning
are usually smaller than the accumulative errors of SfM.
Figure 2.13 shows the estimated GPS positions when the GPS positioning wasrecovered. For Method C, the results before optimization (right after parameter
fitting) and after optimization are shown. The results demonstrate that cameras
around only the frame in which the GPS positioning was recovered were drawn
to the obtained GPS position in Method C’, which does not apply parameter
fitting. In Method C, the positions close to the ground truth positions wereobtained through parameter fitting and optimization.
Figures 2.14 and 2.15 show position errors in each frame after global optimization and a change in energy during the sequential process, respectively. Table 2.4
shows the statistics of the position errors. These results show that the errors from
Method C’ were large during a GPS outage because the energy converged to a
local minimum and optimization did not work well. The errors from Method C’
were also large at around the last frame because an inconsistency between the 3D
positions of the feature points and the camera poses arose from the optimization.
In Method C, the accumulative errors were reduced using GPS positioning aroundthe first and last frames. However, a comparison of the results from experiment
1 shows that the accumulative errors were still large during a GPS outage.

2.5. Conclusions
In this chapter, we proposed a method using images and GPS data for accurately
estimating camera poses even when the GPS positioning accuracy drops to a
Table 2.4: Comparison of position errors(experiment 2) [m].
MethodAverage Std. dev.
Max
RTK-fix
0.006
0.009
0.038
RTK-float
1.315
0.989
2.594
C
3.593
3.617 14.935
C’
60.29967.829 277.530

44

200

North-South [m]100RTK-fix (ground truth)
RTK-fix (input)
RTK-float (input)
Method C
Method C (before optimization)
Method C’

0First frame

-100

-200
0

100

200
300
East-West [m]400Altitude [m](a) Horizontal 2D positions

20
15
10
5
0
-5
-10
-15
-20
-25
-30

RTK-fix (ground truth)
RTK-fix (input)
RTK-float (input)
Method C
Method C (before optimization)
Method C’

0

500

1000
1500
Frame number

2000

2500

(b) Altitudes

Figure 2.13: Estimated GPS positions after a GPS outage (experiment 2).
45

300
RTK-fix (input)
RTK-float (input)
Method C
Method C’

Error [m]

250
200
150
100
50
0
0

500

1000
1500
Frame number

2000

2500

GPS outage
Figure 2.14: Position errors in each frame (experiment 2).

101

Method C
Method C’

100

Energy

10−1
10−2
10−3
10−4
10−5
0

500

1000
1500
Frame number

2000

2500

GPS outage
Figure 2.15: Change in energy during sequential process (experiment 2).
46

low level by introducing weighting coeﬃcients depending on the GPS positioning confidence for extended BA. We also introduced parameter fitting using GPS
positions to avoid the local minima during optimization after a GPS outage. We
confirmed experimentally that the proposed method provides more accurate camera positions compared with an existing extended-BA method. For more practicaluse of the proposed method, an automatic weight determination method should
be investigated because appropriate weights are dependent on the environment.
In addition, combining other sensors such as an IMU is important to reduce
accumulative errors during a long GPS outage.

47

Chapter 3
Sampling-Based Bundle
Adjustment using Feature
Matches Between Ground-View
and Aerial Images
3.1. Introduction
This chapter describes a camera pose estimation method using aerial images asexternal references that are already available for most outdoor scenes around theworld. As mentioned in Chapter 1, the most significant problem in SfM is theaccumulation of estimation errors during a long image sequence. To reduce theaccumulative errors, we employed the framework of extended BA to the SfM
problem using an aerial image as an external reference. Although many kindsof methods using aerial images have been proposed [109–115, 120], to the best of
our knowledge, ours is the first method using aerial images as external referencesin BA. For the successful use of an aerial image as a reference in SfM, successful
matching between the aerial image and the ground-view image is very important.
To find good matches from unreliable matches, in addition to the use of GPS andgyroscope sensors embedded in most recent smartphones, we use two new methods: (1) RANSAC-based [122] outlier elimination in the feature matching stage

48

Sensor unit
(e.g., smartphones)
GPS data

Ground-view Images

Ground-view Images
Gyroscope data
GPS data(b) Feature matching between
ground-view and aerial images(a) Structure-from-Motion
Tracks of feature points
Camera poses
3D positions of feature points

Aerial image

Tentative matches(c) Sampling based
bundle adjustment

Refined camera poses
Refined 3D positions of feature points

Figure 3.1: Flow of the proposed method using aerial images.
by focusing on the consistency of the orientation and scale extracted from theimages by feature descriptors such as SIFT [118], and (2) RANSAC-based outlier
elimination in the BA stage using the consistency of the estimated geometry andmatches.
As shown in Figure 3.1, the proposed method consists of three processes:(a) SfM, (b) feature matching between the ground-view and aerial images, and(c) sampling-based BA. For process (a), any SfM method can be employed. In
the experiments described later, we employed VisualSFM [43] as a state-of-theart SfM implementation. In the following, we describe processes (b) and (c), thefeature matching between the ground-view and aerial images, and sampling-based
BA, respectively.

3.2. Feature Matching Between Ground-View and
Aerial Images
In this section, we propose a robust method for finding matches between groundview and aerial images. As shown in Figure 3.2, the method is composed of(1) ground-view image rectification using homography, (2) feature matching, and49

Ground-view image(1) Rectification

Aerial image(2) Feature matching(3) RANSAC withorientation and scale check

Figure 3.2: Flow of the feature matching.
50

(3) RANSAC. Here, to achieve robust matching, we propose new criteria for
RANSAC using a consistency check of the orientation and scale from a featuredescriptor. It should be noted that matching for all input frames is not necessary
in our pipeline. Even if we can find only several matched frame candidates, they
can be eﬀectively used as references during the BA stage.

3.2.1 Image Rectification using Homography
Before calculating the feature matches using a feature detector and a descriptor,
as shown in Figure 3.2, we rectify the ground-view images so that the texture
patterns are similar to those of the aerial image. In most cases, aerial images
are taken very far away from the ground and can thus be assumed to have been
captured by an orthographic camera whose optical axis is directed toward the
direction of gravity. To rectify the ground-view images, we also assume thatthese images contain the ground plane whose normal vector is directed toward
the direction of gravity. We then compute a homography matrix using the gravity direction in the camera coordinate system which can be estimated from thevanishing points of parallel lines in the ground-view images or from a gyroscope.

3.2.2 Feature Matching
The feature matches between the rectified ground-view images and the aerial
image are calculated. Here, we use GPS data corresponding to the ground-view
images to limit the search area in the aerial image. More concretely, we selectthe region whose center is the GPS position and size is ι × ι. In the experiments
described later, ι was set to 50 [m]. The feature matches are then calculated using
a feature detector and a descriptor. We employed SIFT [118] in our experiments
because of its robustness to changes in scale, rotation and illumination.

3.2.3 RANSAC with Orientation and Scale Check
As shown in Figure 3.2, the results of feature matching often include many incorrect matches. To remove these matches, we use RANSAC with a consistency
check of the orientation and scale parameters.

51

For matches between the rectified ground-view images and the aerial image, we
can use a similarity transform composed of scale s, rotation θ, and translation τ .
During the RANSAC procedure, we randomly sample two matches (the minimum
number required to estimate the similarity transform) to compute the similarity
transform (s, θ, τ ). Here, we count the number of inlier matches that satisfy|ak − (sR(θ)γk + τ )| < dth ,(3.1)

where ak and γk are the 2D positions of the k-th match in the aerial image andrectified ground-view image, respectively, R(θ) is the 2D rotation matrix withrotation angle θ, and dth is the threshold. After repeating the random sampling
process, the sampled matches with the largest number of inliers are selected.
The problem here is that the distance-based criterion described above cannot
successfully find correct matches when a very large number of incorrect matches
exist. To achieve more robust matching, we modify the criterion of RANSAC bychecking the consistency of the orientation and scale from a feature descriptor.
Concretely, we count the number of inliers that simultaneously satisfy Equation(3.1) and the following two conditions.
()
sgk · s sak
max
(3.2)
,
< sth ,
sak sgk · saad(θgk + θ, θak ) < θth ,(3.3)

where (sak , sgk ) and (θak , θgk ) are the scale and orientation of the feature points forthe k-th match on the aerial image and rectified ground-view image, respectively.
The function aad returns the absolute angle diﬀerence in the domain [0◦ , 180.0◦ ].
Additionally, sth and θth are the thresholds for the scale and angle, respectively.

3.3. Sampling-Based Bundle Adjustment
Even using the modified RANSAC proposed in the previous section, it is notpossible to remove all incorrect matches in principle because repetitive and/or
similar patterns may exist, e.g., road signs in actual environments, as shown in
Figure 3.3. To overcome this diﬃculty, we also employ RANSAC for the BA stage
by focusing on the consistency between the feature matches and the estimatedcamera poses from the images.
52

Figure 3.3: Examples of road signs in an aerial image from Google Maps
[maps.google.com].

3.3.1 Definition of Energy Function
To consider the matches between the ground-view and aerial images, the energyfunction is newly defined. As shown in Figure 3.4, since we deal with perspective
ground-view images and an orthographic aerial image, two types of reprojection
errors should be considered. The energy function Eaerial is defined using the
reprojection errors for the ground-view (perspective) images Φ̂, and the aerial(orthographic) image Ω, as follows:
Eaerial ({Ri , ti }Ii=1 , {pj }Jj=1 ) = Φ̂({Ri , ti }Ii=1 , {pj }Jj=1 ) + ωΩ Ω({pj }Jj=1 ),(3.4)

where Ri and ti represent the rotation and translation from a world coordinate
system into a camera coordinate system for the i-th frame, respectively; pj is
the 3D position of the j-th feature point; I and J are the numbers of frames andfeature points, respectively; and ωΩ is the weight that balances Φ̂ and Ω. Because
the energy function is non-linearly minimized in BA, good initial parameter values
53

3D position of feature point
Reprojection error for
ground-view image

Reprojection error for
ground-view image(Altitude)(North): Projected 2D position
World
coordinate system: Detected 2D position(East)
Reprojection error for aerial image

Figure 3.4: Reprojection errors for ground-view (perspective) images and an aerial(orthographic) image.
are required to avoid the local minima. Before minimizing the energy function, we
fit the parameters estimated by SfM to the GPS positions using a 3D similarity
transform. In the following, the energy associated with the reprojection errors Φ̂and Ω is detailed.
Reprojection Errors for Ground-view Images
The commonly used reprojection errors are defined by employing a pinhole camera model that cannot deal with projections from behind the camera. Such behindprojections often occur in BA with external references owing to dynamic movements of the camera poses caused by these references. Here, as shown in Figure
3.5, instead of common squared distance errors on the image plane, we employ

54

3D position of -th feature point:-th camera coordinate system
Projected 2D position :Angular reprojection error
Detected 2D position :(a) Angular reprojection error

2D position of -th feature point:

Detected 2D position:
Projected 2D position:

X-Z plane of-th camera coordinate system(b) Split angular reprojection error

Figure 3.5: Reprojection error for ground-view (perspective) image.

55

reprojection errors using the angles of rays as follows:Φ̂({Ri , ti }Ii=1 , {pj }Jj=1 ) = ∑I

1

I ∑(
∑

i=1 |Pi | i=1 j∈Pi(( ) ( ))
Xijc
xij
̸
Φ̂xz
=
,
ij
Zijc
1
(( ) ( ))
Yijc
yij
yz
Φ̂ij = ̸
,
Zijc
1

yz 2
2
(Φ̂xz
ij ) + (Φ̂ij )

)
(3.5)

(3.6)
(3.7)

(Xijc , Yijc , Zijc )T = Ri pj + ti

(3.8)

where Pi is a set of feature points detected in the i-th frame. Function ̸ returns
an angle between two vectors. Note that (xij , yij )T is the detected 2D position of
the j-th feature points in the i-th frame.
Here, as mentioned in [36], convergence of energy is very poor with angular
reprojection error (Φ̂ij )2 = ̸ ((xij , yij , fi )T , (Xij , Yij , Zij )T )2 . Then, as shown in
Figure 3.5(b), we split the angular reprojection error into the xz and yz components to simplify the Jacobian matrix of Eaerial required by non-linear least squares
methods such as the Levenberg-Marquardt method. In this definition, Φ̂xz
ij andyz
Φ̂ij do not depend on the y and x components of ti , respectively. We confirmed
experimentally that this splitting largely aﬀects the convergence performance.
Reprojection Errors for an Aerial Image
As shown in Figure 3.6, the reprojection errors for an aerial (orthographic) image
can be defined as follows:Ω({pj }Jj=1 ) = ∑1∑∑i∈M |Ai | i∈M j∈A2aj − (Xjw , Yjw )T ,

(3.9)

i

(Xjw , Yjw , Zjw )T = pj

(3.10)

where M is a set of frames in which the feature matches between the ground-view
and aerial images are obtained, Ai is a set of feature points matched to the aerial
image in the i-th frame, and aj is the 2D position of the j-th feature point in theaerial image.

56

3D position of -th feature point:

Detected 2D position:

Projected 2D position:(Altitude)(North)Worldcoordinate system(East)
Reprojection error for aerial image:

Figure 3.6: Reprojection error for an aerial (orthographic) image.

3.3.2 RANSAC for Bundle Adjustment
The RANSAC scheme is introduced in BA using the consistency between thefeature matches and the estimated camera poses from the images. First, we
randomly sample n frames from the matched frame candidates and apply BA
using feature matches included in the sampled frames, i.e., using a set of selected
frames M ′ instead of M in Equation (3.9). We then count the number of inlier
frames that satisfy the following condition.
average(αij ) < αth(3.11)

j∈Ai

where αij is an angular reprojection error of the j-th feature point in the aerial
image coordinate system, as shown in Figure 3.7, and αth is the threshold. Here,αij can be computed as follows:))((
αij = ̸ aj − prxy (−RiT ti ), prxy RiT (xij , yij , 1)T ,(3.12)
57

-th camera coordinate system
Ray of -th feature pointin camera coordinate system:

Position of -th feature point on aerial image:

Projected ray on aerial image:(Altitude)-th camera position
on aerial image:(North)Worldcoordinate system(East)

Figure 3.7: Criterion used in RANSAC for BA.
where prxy is a function projecting a 3D point onto the x-y plane (the aerial image
coordinate system).
After repeating the random sampling process at the given times, sampled
frames with the largest number of inlier frames are selected. Finally, camera
poses are refined by reapplying BA using the feature matches with the selected
inlier frames as references.

3.4. Experiments
To validate the eﬀectiveness of the proposed method, we quantitatively evaluated the performances of the sampling-based BA as well as the feature matchingprocess using two datasets: (1) data captured by a hand-held sensor unit on the
textured ground, and (2) data captured by a car-mounted sensor unit on a roadway. In the following, we first describe the common setup for the two experiments.
The results of each experiment are then detailed.

58

3.4.1 Experimental Setup
We used an iPhone 5 (Apple) as a sensor unit including a camera, GPS, anda gyroscope. The GPS and gyroscope measured the position at 1 [Hz] and the
direction of gravity for every frame, respectively. We also used an RTK-GPS(Topcon GR-3, 1 [Hz], horizontal positioning accuracy in specification sheet is0.01 [m]) to obtain the ground truth positions. The positions from the GPS data
were assigned temporally to the nearest frame. As the external references, we
downloaded the aerial images covering the area used in the experiments from
Google Maps [maps.google.com], whose coordinate system is associated with themetric scale.
To obtain the initial values for the BA, we employed VisualSFM [43] as astate-of-the-art SfM implementation. For non-linear minimization, we used CeresSolver [52]. We experimentally set dth = 2 [pixel], sth = 2 and θth = 40 [◦ ] for thefeature matching, and ωΩ = 10−5 and αth = 5.0 [◦ ] for the BA. We evaluated theaccuracy of the proposed method by comparing the following methods.
• BA without references [43]• BA with references without RANSAC, using all the matches obtained by
the feature matching process• BA with references and RANSAC
Since the BA without references cannot estimate absolute camera poses, we fitted the camera positions estimated using SfM to the ground truths through asimilarity transform.

3.4.2 Quantitative Evaluation using Data Captured on Textured Ground (Experiment 1)
In this experiment, we used video images (640 [pixel] × 480 [pixel], 2,471 frames,
494 [s]) captured by a hand-held sensor unit on a textured ground. Figures 3.8
and 3.9 show example input ground-view images and an aerial image (19.2 [pixel]= 1 [m].)59

(a) 1st frame(b) 350th frame(c) 700th frame(d) 1,050th frame(e) 1,400th frame(f) 1,750th frame(g) 2,100th frame(h) 2,450th frame

Figure 3.8: Example input ground-view images (experiment 1).
60

Last frame

GPS on iPhone
RTK-GPS (Ground truth)
BA without references
BA with references without RANSAC
BA with references and RANSAC
Correctly matched feature point on aerial image(selected as inlier by RANSAC)
Incorrectly matched feature point on aerial image(selected as outlier by RANSAC)First frame

50 m

50 m

Figure 3.9: Experimental environment and results (experiment 1).
61Quantitative Evaluation of Feature Matching
In this experiment, we first evaluated the eﬀectiveness of the proposed feature
matching process including RANSAC using the scale and orientation check described in Section 3.2. Here, we tested RANSAC with variable thresholds sth andθth . To count the number of correctly matched frames, we first selected framesthat have four or more inlier matches after RANSAC. From these frames, we
manually counted frames whose matches were correct.
Figure 3.10 shows the rate and number of frames in which all selected matches
were correct. Note that sth = ∞ and θth = 180.0 [◦ ] mean that the orientation
check and scale check were disabled, respectively. The results indicate that therate was significantly improved through the scale and orientation check. We canalso confirm that small values of sth and θth tend to increase this rate. However, the number of correctly matched frames, which is important for optimizing
the camera poses using BA, was decreased when using small thresholds. In thefollowing experiments, we employed feature matches with sth = 2 and θth = 40 [◦ ].
Figure 3.11 shows the eﬀects of the scale and orientation check for two sampled images. In both cases, RANSAC without scale and orientation check could
not select any correct matches, whereas the proposed RANSAC with scale andorientation check was able to do so. However, as shown in Figure 3.12, incorrect
matches still remain even when we used both scale and orientation check because
similar patterns exist.
Quantitative Evaluation of Bundle Adjustment
In this experiment, we evaluated the eﬀectiveness of BA with RANSAC, as described in Section 3.3. In this stage, frames with GPS data were sampled (650
out of 2,471 frames) and used to reduce the computational time. As externalreferences, we used the frames and feature matches selected through the orientation and scale check described in the previous section. Here, ten out of fourteenframes had correct matches.
We first investigated the influence of weight ωΩ for balancing two types ofreprojection errors in the energy function of the BA. Figure 3.13 shows the average
position errors from the BA with variable weight ωΩ using all of the correctly
matched frames. This result demonstrates that position errors did not largely
62all selected matches are correct

Rate of frames in which

1.0

sth = 1.25
sth = 1.5

0.8

sth = 2.0

0.6

sth = 2.5

0.4

sth = 3.0
sth = ∞

0.2
0
0

20

40

60

80
100
θth [◦ ]

120

140

160

180

all selected matches are correct

Number of frames in which(a) Rate of frames in which all selected matches are correct
sth = 1.25

10

sth = 1.5

8

sth = 2.0
sth = 2.5

6

sth = 3.0
sth = ∞

4
2
0
0

20

40

60

80
100
θth [◦ ]

120

140

160

180

(b) Number of frames in which all selected matches are correct

Figure 3.10: Rates and numbers of frames in which all selected matches arecorrect (experiment 1).

63

Similarity transform
Angle [ ] Scale
325.56 0.06

Similarity transform
Angle [ ] Scale
158.71 1.19

SIFT keypoints
Angle [ ] Scale
334.02 0.85
332.84 0.91
330.88 0.84
352.56 0.43
156.73 1.06

SIFT keypoints
Angle [ ] Scale
159.92 1.11
154.96 0.89
154.24 1.25
148.45 1.12(a) Without orientation and scale check (b) With scale check, with/without orientation check

Similarity transform
Angle [ ] Scale
66.47 0.07

Similarity transform
Angle [ ] Scale
336.34 0.96

SIFT keypoints
Angle [ ] Scale
164.35 0.86
334.38 0.82
334.94 0.78
329.40 0.68
171.96 0.46

SIFT keypoints
Angle [ ] Scale
342.27 0.90
338.34 0.90
339.67 0.73
341.82 0.77
333.08 0.77(c) Without orientation and scale check (d) With orientation check, with/withoutscale check

Figure 3.11: Selected inliers for example images (experiment 1). The solid anddashed lines represent correct and incorrect matches, respectively. The relative
angle and scale of the matched feature points are shown in bottom-right table
along with the corresponding line colors. The green points are the ground truthsof the camera positions. Note that RANSAC with/without orientation check for(b) and scale check for (d) gave the same results.

64Similarity transform
Angle [ ] Scale
3.87 0.89

Similarity transform
Angle [ ] Scale
169.41 1.07

SIFT keypoints
Angle [ ] Scale
343.16 1.63
19.02 0.80
12.64 1.22
13.32 0.93

SIFT keypoints
Angle [ ] Scale
169.48 0.99
168.87 1.15
163.06 1.17
170.35 1.04

(a)

(b)

Avg. horizontal position error [m]Figure 3.12: Examples of incorrect matches by RANSAC using orientation andscale check (experiment 1). The interpretations of the symbols are the same as
in Figure 3.11.

12

8

4

0
10−10

10−8

10−6

10−4

10−2

100

ωΩ

Figure 3.13: Relationship between weight ωΩ and average horizontal position
error (experiment 1).

65

Number of frame

10
8
6
4
2
0
0

2

4

6

8

10
αth [◦ ]

12

14

16

18

20

Correctly matched frame selected as inlier
Incorrectly matched frame selected as inlier

Figure 3.14: Number of inlier frames with variable threshold αth (experiment 1).
depend on weight ωΩ except when small values were applied. In the following
experiments, we employed ωΩ = 10−5 .
We next evaluated the proposed RANSAC in terms of its capability to select frames with correct matches. Here, we experimentally set n = 4 and tested
100 trials. To eﬃciently optimize the camera poses using feature matches between ground-view and aerial images, we modified the random sampling processof frames in RANSAC so that the distances between average positions of matches
on an aerial image were 25 [m] or more. Figure 3.14 shows the numbers of inlier
frames from RANSAC with variable threshold αth . The results demonstrate thatincorrectly matched frames were selected as inliers by large values of αth , andthat the numbers of correctly matched frames decreased by small values of αth .
In the following experiments, we employed αth = 5.0 [◦ ].
We also checked the number of inlier frames selected in each trial with αth =5.0 [◦ ]. Figure 3.15 shows the number of trials and inlier frames derived by eachtrial. From this figure, we can see that the sampled frames without incorrect
matches tend to increase the number of inlier frames. This result demonstrates
that the criterion of RANSAC in the BA stage works successfully. We also confirmed that the trials that derived the largest number of inlier frames successfully66

Number of trials

30
25
20
15
10
5
0
0

1

2

3
4
5
6
7
8
Number of inlier frames
Sampled frames without incorrect matches
Sampled frames with incorrect matches

910

Figure 3.15: Number of trials and inlier frames derived by each trial (experiment 1).
selected all of the correct matches.
Figures 3.9 and 3.16 show the estimated camera positions and horizontal position errors for each frame, respectively. These results demonstrate that theestimated camera positions from BA without references were aﬀected by theaccumulative errors. The BA without RANSAC was aﬀected by the incorrect
matches. The proposed BA with RANSAC reduced the accumulative errors. It
should be noted that, at the end of the sequence, the accumulative errors still
remained because the ground was not level, and no matches were therefore found.

3.4.3 Quantitative Evaluation using Data Captured on Roadways (Experiment 2)
In this experiment, we used video images (640 [pixel] × 480 [pixel], 7,698 frames,
396 [s]) captured by a car-mounted sensor unit on a roadway. Figures 3.17 and3.18 show example input ground-view images and an aerial image (approximately
22.3 [pixel] = 1 [m]). Note that we manually excluded frames captured when the
car was stopped at a traﬃc light.
We first applied the feature matching process including RANSAC throughscale and orientation check. After selecting frames with four or more inlier

67

Horizontal position error [m]

60
50
40
30
20
10
0
0

1000

2000

Frame number
BA w/o references

BA w/ references and RANSAC

BA w/ references w/o RANSAC
Correctly matched frame(selected as inlier by RANSAC)

Incorrectly matched frame(selected as outlier by RANSAC)

Figure 3.16: Horizontal position error in each frame (experiment 1).
matches, we obtained 37 frames (28 frames without incorrect matches and nineframes with incorrect matches). We then applied RANSAC during the BA stage
using frames with GPS data (739 out of 7,698 frames). Here, we experimentally
set n = 7. To eﬃciently optimize the camera poses using feature matches between ground-view and aerial images, we modified the random sampling processof frames in RANSAC so that the distances between average positions of matches
on an aerial image were 100 [m] or more. After 100 trials, the trial that derived
the largest number of inlier frames selected 22 frames as inliers (nineteen frames
without incorrect matches and three frames with incorrect matches) and fifteen
frames as outliers (nine frames without incorrect matches and six frames with incorrect matches). Figures 3.19 and 3.20 show example frames selected as inliers
and outliers, respectively. As shown in Figure 3.19, the frames with incorrect
matches were selected as inlier frames by RANSAC because the positions of theincorrect matches on the aerial image were close to the correct positions. Figures
3.18 and 3.21 show the estimated camera positions and horizontal position errors
68

(a) 1st frame(b) 1,100th frame(c) 2,200th frame(d) 3,300th frame(e) 4,400th frame(f) 5,500th frame(g) 6,600th frame(h) 7,698th frame

Figure 3.17: Example input ground-view images (experiment 2).
69

Last frame

GPS on iPhone
RTK-GPS (Ground truth)
BA without references
BA with references without RANSAC
BA with references and RANSAC
Correctly matched feature point on aerial image(selected as inlier by RANSAC)
Correctly matched feature point on aerial image(selected as outlier by RANSAC)
Incorrectly matched feature point on aerial image(selected as inlier by RANSAC)
Incorrectly matched feature point on aerial image(selected as outlier by RANSAC)First frame

100 m

Figure 3.18: Experimental environment and results (experiment 2).

70

100 m

(a)

(b)

(c)

(d)

Figure 3.19: Examples of frames selected as inliers by RANSAC during the BA
stage (experiment 2). The solid and dashed lines represent correct and incorrect
matches, respectively.

(a)

(b)

Figure 3.20: Examples of frames selected as outliers by RANSAC during the BA
stage (experiment 2). The dashed lines represent incorrect matches.
71

Horizontal position error [m]

50

40

30

20

10

0
0

2000

4000

6000

Frame number
BA w/o references

BA w/ references and RANSAC

BA w/ references w/o RANSAC
Correctly matched frame(selected as inlier by RANSAC)
Correctly matched frame(selected as outlier by RANSAC)

Incorrectly matched frame(selected as inlier by RANSAC)
Incorrectly matched frame(selected as outlier by RANSAC)

Figure 3.21: Horizontal position error in each frame (experiment 2).
for each frame, respectively. Although frames with incorrect matches still remained even when using a two-stage RANSAC, the proposed BA with RANSAC
reduced the accumulative errors. However, at around the 6,000th frame, theaccumulative errors are still large because there were only a small number ofmatches.

3.5. Conclusions
In this chapter, we proposed a method for removing accumulative errors in SfM
using aerial images as external references that already exist for many places
around the world. To this end, we proposed BA that uses feature matches between the ground-view and aerial images. To find correct matches from unreliablematches, we introduced new RANSAC schemes to both the feature matching and72bundle adjustment stages. In the experiments, we confirmed that the proposed
method is eﬀective for estimating the camera poses of real video sequences taken
in outdoor environments. However, the accumulative errors still remain when
there are no available matches during a long period of time. To find matches
where the ground is not level, aﬃne and/or perspective invariant features such
as ASIFT [123] and Ferns [124] can be used with homography as a geometrictransformation in RANSAC.

73

Chapter 4
Online Camera Pose Estimation
using 3D Point Database Createdfrom Structure-from-Motion
4.1. Introduction
Some robot navigation and augmented reality applications require estimating thecamera poses along a previously taken route. In this chapter, we propose an online
camera pose estimation method for such applications using a 3D point databasecreated using SfM from previously captured images. Unlike the methods based on
SfM described in Chapters 2 and 3, the method proposed in this chapter estimatesthe camera poses directly from a database. Errors are therefore not accumulated
during the online camera pose estimation. Although the database created by SfM
is aﬀected by accumulative errors, the methods that reduce accumulative errors,e.g., loop closing, can be applied during the oﬄine stage.
As mentioned in Section 1.2.5, many methods based on a 3D point database
have been previously proposed [99–106]. The main problem of methods using a
3D point database is how to obtain matches between the input images and the
database. To limit the search space, existing methods track the feature points
temporally [104–106]. However, tracking sometimes fails owing to occlusions andrapid movement of the cameras. On the other hand, methods using an image
database, as described in Section 1.2.3, can eﬃciently identify the database image
74

that is the most similar to the current image by considering the spatio-temporalconnections between the database and the input images. By combining these
two approaches, we propose a method utilizing a 3D point database that employs an image-database method to limit the search space. For a state-of-the-art
image-database method, we employ topometric-localization [88], which considers
topological information such as the spatio-temporal connections between sequential images at the metric scale.
As an application, we focus in particular on vehicle navigation for autonomous
driving. The proposed method estimates the camera pose related to the vehiclepose by assuming that the transformation between the camera coordinate systemand the vehicle coordinate system is known. As shown in Figure 4.1, the proposed
method consists of two stages:
Oﬄine creation of 3D point database: A 3D point database is created fromimages that are captured when a vehicle drives along a route for the first time.
This database consists of images, 3D positions of the feature points estimated by
SfM [43], and a topological graph for topometric localization [88].
Online localization: The vehicle is localized using a 3D point database and thecurrent image through the following three steps. First, we identify the database
image that is the most similar to the current image using topometric localization [88]. Next, we estimate the 2D-2D correspondences of the feature points
between the current image and the identified database image to obtain 3D-2D
correspondences of the feature points for the current image. The vehicle pose is
finally estimated from these 3D-2D correspondences by solving the PnP problem.

4.2. Oﬄine Creation of 3D Point DatabaseIn our method, the database is created from images captured when the vehicle
drives along a route for the first time. The database consists of images, the 3D
positions of the feature points estimated using SfM, and a topological graph for
topometric localization.
Structure-from-motion: We use VisualSFM [43], which is a state-of-the-art
SfM implementation, to obtain the 3D positions of the feature points and camera

75Offline creation of 3D point database
OdometerCameraImage sequence
Distance

Structure-from-motion- Camera poses- 3D positionsof feature points

CameraScaling
Metric camera poses
Creation of graph for
topometric localization

Online localization

Graph

Current image
Topometric localization
Identified
database image
Feature matching
2D-2D
correspondences

Metric 3D positions of feature points

Estimation of camera
pose from 3D-2D
correspondences
Estimated camera pose

Figure 4.1: Flow of the proposed method using a 3D point database created using
SfM.

76

poses. Since SfM cannot estimate the camera pose at the metric scale, which
is necessary to estimate the vehicle pose at the metric scale for online localization, we measure the total driving distance using an odometer, which is standard
equipment in ordinary vehicles. Specifically, the scale is determined by adjusting
the total driving distance obtained using SfM based on the distance obtained
from the odometer. In this way, metric 3D positions of the feature points andcamera poses are obtained.
Creation of graph for topometric localization: We prepare a graph for topometric localization, which is used in the online process to identify the database
image that is the most similar to the current image. The graph is created as described in [88], except that the camera positions are estimated using SfM insteadof GPS. The edges of graph represent spatio-temporal connections of the images,
and the nodes represent the images with their SURF features [119].

4.3. Online Camera Pose Estimation
The vehicle is localized using a 3D point database and the current image throughthe following three steps.
Topometric localization: We first identify the database image that is the mostsimilar to the current image using topometric localization [88], which considers
both the image features and spatio-temporal connections between sequential images represented as the graph.
Feature matching: We next estimate the 2D-2D correspondences of the featurepoints between the current image and the identified database image to obtainthe 3D-2D correspondences of the feature points for the current image. We use
SiftGPU [125], a GPU implementation of the SIFT feature, to achieve real-timeprocessing.
Estimation of camera pose from 3D-2D correspondences: In the feature
matching stage, 2D-2D correspondences between the current image and the identified database image are obtained. We also have the 3D positions of the featurepoints for the database image estimated during the oﬄine stage. By combining
them, we can obtain the 3D-2D correspondences of the feature points for the cur77

Figure 4.2: Evaluation vehicle.
rent image. We then estimate the camera pose from these 3D-2D correspondencesby solving the PnP problem. To do so, we simply use a solver implemented in
OpenCV [126], which non-linearly minimizes the reprojection errors. We also
apply RANSAC to reject the incorrect feature matches. If the number of inlier
matches is smaller than the threshold, the estimated camera pose is ignored asa failure. In our experiment, the threshold was set to 6, which is the minimum
number needed to linearly solve the PnP problem.

4.4. Experiments
To test the eﬀectiveness of the proposed method, we evaluated its accuracy quantitatively using image sequences captured in an indoor parking lot.

4.4.1 Experimental Setup
To evaluate our proposed method, we used a vehicle equipped with a sensor suite.
As shown in Figure 4.2, a camera was mounted on the roof of the vehicle, andwas oriented approximately 45 [◦ ] to the right of the straightforward direction,and configured to acquire 1,024 [pixel] × 768 [pixel] images. The vehicle was able
to output its driving distance like an ordinary vehicle.
Two image sequences were captured in an indoor parking lot on diﬀerent days78

by manually driving along a route that makes a loop between the entrance anda parking space. One of the sequences was used for creating the database, andthe other was used as input for online localization. The reference poses used
to evaluate the accuracy of the estimated poses were estimated using an oﬄine
SfM [43]. This oﬄine SfM used the feature matches among all of the images andtook a long time to achieve an accurate estimation.

4.4.2 Quantitative Evaluation
Figure 4.3 shows examples of the input images, the database images identified
through topometric localization, and the results of feature matching between
these images. Figure 4.4 shows the vehicle poses and 3D positions of the featurepoints for the database images, the vehicle poses estimated using the proposed
method, and the reference vehicle poses. Except for the last quarter, the estimated vehicle poses were almost the same as the reference vehicle poses along
the entire route, despite changes in the illumination (Figure 4.3(b)) and the environment (Figure 4.3(c)). Figure 4.5 shows histograms of the errors by comparing
the estimated vehicle poses with the reference vehicle poses. Table 4.1 shows theaverage computation time of the proposed method, which was obtained using a
PC with a 3.40 [GHz] Intel Core i7-2600k CPU and an NVIDIA GeForce GTX
580 GPU. From these results, it was confirmed that the proposed method estimated the vehicle pose at 8 [Hz] within a position error of 0.1 [m] and a postureerror of 0.3 [◦ ] in approximately 70% of the input images.
We investigated the causes of the errors for frames whose position errors arelarger than 0.1 [m] or posture errors are larger than 0.3 [◦ ]. Approximately 51%(54/104) of the frames were aﬀected by the feature matches of the vehicle parked
in diﬀerent places in the database and the input images (Figure 4.3(d)). The otherframes can be aﬀected by the errors of SfM during the oﬄine database creation.
We also investigated the causes of the failures. Approximately 24% (19/78) of theframes were aﬀected by the errors in the topometric localization (Figure 4.3(e)),
and the other frames were aﬀected by significant changes in the illumination andthe environment (Figure 4.3(f)). Although the proposed method occasionallyprovides unstable estimates and failures owing to changes in the environment,
it can be used in autonomous vehicles by combining sensors providing relative
79

(a)

(b)

(c)

Figure 4.3: Examples of input images (right), the database images identified
through topometric localization (left), and the results of feature matching between these images (red line, inlier; blue line, outlier). (1/2)
80

(d)

(e)

(f)

Figure 4.3. Continued. (2/2)

81Entrance

Parking space(start and goal)
10 m

10 m

Figure 4.4: The vehicle poses (red) and 3D positions of the feature points (gray)for the database images, the vehicle poses estimated by the proposed method(blue), and the reference vehicle poses (green).
82

83

0.1

0
0.6 - 0.7

0.5 - 0.6

0.4 - 0.5

0.3 - 0.4

0.2 - 0.3

0.1 - 0.2

0.0 - 0.1

Failure0.2

Failure

0.3
1.0+

0.4

1.0+

0.5
0.9 - 1.0

0.6

0.9 - 1.0

0.7
0.8 - 0.9

0.8

0.8 - 0.9(a) Position
0.7 - 0.8

Error [m]0.7 - 0.8

0.6 - 0.7

0.5 - 0.6

0.4 - 0.5

0.3 - 0.4

0.2 - 0.3

0.1 - 0.2

0.0 - 0.1

Probability

Probability
0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

Error [◦ ](b) Posture

Figure 4.5: Histograms of the errors.

Table 4.1: Computation time of the proposed method [ms].
Topometric Feature
Solving
localization matching PnP problem
8.8
47.8
67.6

Total
124.2

measurements such as odometry and IMU.

4.5. Conclusions
In this chapter, we proposed a method for localizing a vehicle along a previously driven route using a 3D point database created in advance. The proposed
method identifies the database image that is the most similar to the current image
through topometric localization, and estimates the vehicle poses from the 3D-2D
correspondences of the feature points between the database and the current image. Our experiment showed that the method can estimate a vehicle pose withina position error of 0.1 [m] and a posture error of 0.3 [◦ ] in approximately 70%of the input images captured in an indoor parking lot. However, the proposed
method sometimes provides unstable estimates and failures owing to changes inthe environment. To overcome this problem and achieve an autonomous vehicle, sensors such as odometry and IMU should be combined with the proposed
method.

84

Chapter 5
Conclusions
5.1. Summary
This thesis proposed methods for estimating camera poses without accumulative errors by utilizing external references. For situations in which camera poses
should be estimated without a pre-measurement of the target environments, weemployed GPS and aerial images as external references by focusing on their availability in outdoor scenes. For situations in which cameras iteratively pass along
the same route, we employed a 3D point database, which can be created by SfM,
as an external reference.
For the method using GPS, we proposed extended BA by considering the
GPS positioning confidence to achieve an accurate estimation even when the
GPS positioning confidence is low. For the RTK-GPS, we introduced weighting
coeﬃcients depending on the solution types (RTK-fix or RTK-float). We also
introduced parameter fitting to avoid the local minima in the extended BA after
a long GPS outage. We confirmed experimentally that the proposed method canobtain more accurate camera positions than an existing extended-BA method that
does not consider the GPS positioning confidence. However, the accumulative
errors are still large during a long GPS outage.
For the method using aerial images, we proposed BA using feature matches
between ground-view images and an aerial image. To the best of our knowledge,
ours is the first method that uses aerial images as external references in BA. To
this end, we proposed a robust feature matching method using RANSAC for both85

the feature matching and BA stages. We confirmed experimentally that two-stage
RANSAC selects correct matches, and BA using feature matches achieves a moreaccurate estimation than ordinary BA. However, accumulative errors still remain
when there are no available matches for a long period of time.
For the method using a 3D point database, to eﬃciently limit the search
space of the feature matches, we employed the topometric localization [88] thatcan eﬃciently identify the database image most similar to the current image.
We confirmed experimentally that the proposed method can estimate the camera
pose at 8 [Hz] within a position error of 0.1 [m] and a posture error of 0.3 [◦ ] inapproximately 70% of the input images using a PC with a 3.40 [GHz] Intel Core
i7-2600k CPU and an NVIDIA GeForce GTX 580 GPU. However, the proposed
method provides some unstable estimates and failures owing to changes in theenvironment.

5.2. Future Directions
In this thesis, we proposed three methods employing three types of external references to estimate camera poses without accumulative errors. As mentioned in
Section 1.2, other types of external references such as community photos from theInternet and road maps are also available for certain environments. Therefore, toreduce accumulative errors in various environments and improve the robustnessof the estimation, combining various types of external references depending on
their availability and confidence is important. If more than one type of externalreference is available, the confidence of the external references can be mutuallyestimated.
To more accurately estimate a camera pose, improving the quality of eachexternal reference is also important. Although the resolution of aerial images
is increasing, such images are still insuﬃcient for certain areas. In addition,
a number of moving objects, e.g., cars and pedestrians, hide the ground in roadenvironments. Since the proposed method using aerial images can robustly matchground-view images with aerial images, it is possible to increase the quality of anaerial image using high-resolution ground-view images without moving objects.
This is useful not only for estimating camera poses but also for ordinary map
86

applications such as Google Maps. For a 3D point database, it is possible toiteratively update the database using images for online camera pose estimation.

87

Acknowledgements
The research described in this thesis was carried out during my Ph.D. course
at the Graduate School of Information Science, Nara Institute of Science andTechnology.
First, I would like to express my deepest gratitude to my supervisor and thechair of my thesis committee, Professor Naokazu Yokoya of the Graduate Schoolof Information Science, Nara Institute of Science and Technology, for his patient
instruction, encouragement, and valuable comments throughout this research. He
also provided me with a number of opportunities and invaluable experiences.
I would also like to express my appreciation to Professor Hirokazu Kato of the
Graduate School of Information Science, Nara Institute of Science and Technology, who served on my thesis committee and provided many insightful suggestions
and critical comments regarding this thesis.
In addition, I am deeply grateful to Associate Professor Tomokazu Sato of the
Graduate School of Information Science, Nara Institute of Science and Technology, who also served on my thesis committee. Without his continuous encouragement, countless lessons, and constructive advice, I would never have completed
this thesis.
I would particularly like to thank Professor Takeo Kanade of the Robotics
Institute, Carnegie Mellon University, who suggested the research direction during
my stay there. The precious lessons he provided me beyond the research haveguided and supported every aspect of my life.
Regarding the research on camera pose estimation using GPS, my sincere
appreciation goes to Dr. Nobuo Kochi and Mr. Tetsuji Anai of the R&D Center,
Topcon Corporation. This research is a joint work with them, and their innovative
suggestions guided this research to its successful completion.
88

I would also like to thank all the past and present members of the Vision
and Media Computing Laboratory at the Graduate School of Information Science, Nara Institute of Science and Technology. They oﬀered warm encouragement and friendship that gave me strength through diﬃcult times. Professor
Kazumasa Yamazawa of the Faculty of Information Engineering, Fukuoka Institute of Technology, Associate Professor Masayuki Kanbara, Assistant Professor
Norihiko Kawai, Assistant Professor Takafumi Taketomi, and Assistant Professor
Yuta Nakashima of the Graduate School of Information Science, Nara Institute of
Science and Technology, provided helpful comments and invaluable discussions.
Secretaries Ms. Mio Takahashi, Ms. Mina Nakamura, Ms. Chika Kijima, and
Ms. Yumi Ishitani of the Vision and Media Computing Laboratory gave me
administrative support.
I also benefited greatly by meeting and working with a number of people
during my stay at the Robotics Institute, Carnegie Mellon University. I would
particularly like to thank Mr. Arne Suppé with whom I have cooperatively completed papers involving this thesis. I also thank Ms. Yukiko Kano who helped
me as an administrative staﬀ member.
Finally, I wish to thank my family for their love and support.

89

References[1] G. Klein and D. Murray, “Parallel Tracking and Mapping for Small AR
Workspaces,” Proc. IEEE and ACM Int. Symp. on Mixed and Augmented
Reality, pp.225–234, 2007.
[2] Z. Li, Y. Wang, J. Guo, L.-F. Cheong, and S.Z. Zhou, “Diminished Reality
using Appearance and 3D Geometry of Internet Photo Collections,” Proc.
IEEE Int. Symp. on Mixed and Augmented Reality, pp.11–19, 2013.
[3] A.J. Davison, I.D. Reid, N.D. Molton, and O. Stasse, “MonoSLAM: RealTime Single Camera SLAM,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.29, no.6, pp.1052–1067, 2007.
[4] F. Okura, M. Kanbara, and N. Yokoya, “Fly-Through Heijo Palace Site:
Augmented Telepresence using Aerial Omnidirectional Videos,” Proc. ACM
SIGGRAPH Posters, p.78, 2011.
[5] T. Sato, M. Kanbara, N. Yokoya, and H. Takemura, “Dense 3-D Reconstruction of an Outdoor Scene by Hundreds-Baseline Stereo using a Hand-Held
Video Camera,” Int. J. of Computer Vision, vol.47, no.1-3, pp.119–129,
2002.
[6] S. Agarwal, N. Snavely, I. Simon, S.M. Seitz, and R. Szeliski, “Building
Rome in a Day,” Proc. IEEE Int. Conf. on Computer Vision, pp.72–79,
2009.
[7] M. Jancosek and T. Pajdla, “Multi-View Reconstruction Preserving
Weakly-Supported Surfaces,” Proc. IEEE Conf. on Computer Vision and
Pattern Recognition, pp.3121–3128, 2011.
90

[8] S. Knorr, M. Kunter, and T. Sikora, “Super-Resolution Stereo- and MultiView Synthesis from Monocular Video Sequences,” Proc. Int. Conf. on 3-D
Digital Imaging and Modeling, pp.55–64, 2007.
[9] G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis, “Depth
Synthesis and Local Warps for Plausible Image-Based Navigation,” ACM
Trans. on Graphics, vol.32, no.3, pp.30:1–30:12, 2013.
[10] Y. Awatsu, N. Kawai, T. Sato, and N. Yokoya, “Spatio-Temporal SuperResolution using Depth Map,” Proc. Scandinavian Conf. on Image Analysis,
pp.696–705, 2009.
[11] F. Liu, M. Gleicher, H. Jin, and A. Agarwala, “Content-Preserving Warpsfor 3D Video Stabilization,” ACM Trans. on Graphics, vol.28, no.3, pp.44:1–
44:10, 2009.
[12] R.I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2004.
[13] R. Szeliski, Computer Vision: Algorithms and Applications, Springer, 2010.
[14] A. Torii, T. Okatani, and S. Nobuhara, “Recent Research Trends in MultiView Three-Dimensional Reconstruction,” IPSJ SIG Technical Report,
vol.CVIM-176, no.1, pp.1–22, 2011 (in Japanese).
[15] H.C. Longuet-Higgins, “A Computer Algorithm for Reconstructing a Scenefrom Two Projections,” Nature, vol.293, pp.133–135, 1981.
[16] R.I. Hartley, “In Defense of the Eight-Point Algorithm,” IEEE Trans. on
Pattern Analysis and Machine Intelligence, vol.19, no.6, pp.580–593, 1997.
[17] R.I. Hartley, “Projective Reconstruction and Invariants from Multiple Images,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.16,
no.10, pp.1036–1041, 1994.
[18] D. Nistér, “An Eﬃcient Solution to the Five-Point Relative Pose Problem,”IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.26, no.6,
pp.756–770, 2004.
91

[19] H. Li and R. Hartley, “Five-Point Motion Estimation Made Easy,” Proc.
IAPR Int. Conf. on Pattern Recognition, pp.630–633, 2006.
[20] Z. Kukelova, M. Bujnak, and T. Pajdla, “Polynomial Eigenvalue Solutions
to Minimal Problems in Computer Vision,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.34, no.7, pp.1381–1393, 2012.
[21] V. Lui and T. Drummond, “An Iterative 5-pt Algorithm for Fast and Robust Essential Matrix Estimation,” Proc. British Machine Vision Conference, pp.127.1–127.11, 2013.
[22] H. Stewénius, D. Nistér, F. Kahl, and F. Schaﬀalitzky, “A Minimal Solution for Relative Pose with Unknown Focal Length,” Image and Vision
Computing, vol.26, no.7, pp.871–877, 2008.
[23] C. Tomasi and T. Kanade, “Shape and Motion from Image Streams under
Orthography: A Factorization Method,” Int. J. of Computer Vision, vol.9,
no.2, pp.137–154, 1992.
[24] C.J. Poelman and T. Kanade, “A Paraperspective Factorization Method
for Shape and Motion Recovery,” IEEE Trans. on Pattern Analysis andMachine Intelligence, vol.19, no.3, pp.206–218, 1997.
[25] P. Sturm and B. Triggs, “A Factorization Based Algorithm for Multi-Image
Projective Structure and Motion,” Proc. European Conf. on Computer Vision, pp.709–720, 1996.
[26] S. Christy and R. Horaud, “Euclidean Shape and Motion from Multiple
Perspective Views by Aﬃne Iterations,” IEEE Trans. on Pattern Analysis
and Machine Intelligence, vol.18, no.11, pp.1098–1104, 1996.
[27] T. Okatani, T. Yoshida, and K. Deguchi, “Eﬃcient Algorithm for Low-Rank
Matrix Factorization with Missing Components and Performance Comparison of Latest Algorithms,” Proc. IEEE Int. Conf. on Computer Vision,
pp.842–849, 2011.

92

[28] Y. Dai, H. Li, and M. He, “Projective Multiview Structure and Motion
from Element-Wise Factorization,” IEEE Trans. on Pattern Analysis andMachine Intelligence, vol.35, no.9, pp.2238–2251, 2013.
[29] J. Civera, O.G. Grasa, A.J. Davison, and J.M.M. Montiel, “1-PointRANSAC for Extended Kalman Filtering : Application to Real-Time Structure from Motion and Visual Odometry,” J. of Field Robotics, vol.27, no.5,
pp.609–631, 2010.
[30] E. Eade and T. Drummond, “Scalable Monocular SLAM,” Proc. IEEE
Conf. on Computer Vision and Pattern Recognition, pp.469–476, 2006.
[31] E. Eade and T. Drummond, “Monocular SLAM as a Graph of Coalesced
Observations,” Proc. IEEE Int. Conf. on Computer Vision, 8 pages, 2007.
[32] H. Strasdat, J.M.M. Montiel, and A.J. Davison, “Visual SLAM: Why Filter?,” Image and Vision Computing, vol.30, no.2, pp.65–77, 2012.
[33] D. Nistér, O. Naroditsky, and J. Bergen, “Visual Odometry,” Proc. IEEE
Conf. on Computer Vision and Pattern Recognition, pp.652–659, 2004.
[34] Z. Zhang and Y. Shan, “Incremental Motion Estimation through Modified
Bundle Adjustment,” Proc. IEEE Int. Conf. on Image Processing, pp.343–
346, 2003.
[35] C. Engels, H. Stewénius, and D. Nistér, “Bundle Adjustment Rules,” Proc.
Photogrammetric Computer Vision, pp.266–271, 2006.
[36] E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd, “Genericand Real-Time Structure from Motion using Local Bundle Adjustment,”
Image and Vision Computing, vol.27, no.8, pp.1178–1193, 2009.
[37] S.A. Holmes and D.W. Murray, “Monocular SLAM with Conditionally Independent Split Mapping,” IEEE Trans. on Pattern Analysis and MachineIntelligence, vol.35, no.6, pp.1451–1463, 2013.
[38] A. Kundu, K.M. Krishna, and C.V. Jawahar, “Realtime Multibody Visual
SLAM with a Smoothly Moving Monocular Camera,” Proc. IEEE Int. Conf.
on Computer Vision, pp.2080–2087, 2011.
93

[39] W. Tan, H. Liu, Z. Dong, G. Zhang, and H. Bao, “Robust Monocular
SLAM in Dynamic Environments,” Proc. IEEE Int. Symp. on Mixed andAugmented Reality, pp.209–218, 2013.
[40] J. Hedborg, P.-E. Forssén, M. Felsberg, and E. Ringaby, “Rolling Shutter
Bundle Adjustment,” Proc. IEEE Conf. on Computer Vision and Pattern
Recognition, pp.1434–1441, 2012.
[41] N. Snavely, S.M. Seitz, and R. Szeliski, “Modeling the World from Internet
Photo Collections,” Int. J. of Computer Vision, vol.80, no.2, pp.189–210,
2008.
[42] N. Snavely, S.M. Seitz, and R. Szeliski, “Skeletal Graphs for Eﬃcient Structure from Motion,” Proc. IEEE Conf. on Computer Vision and Pattern
Recognition, 8 pages, 2008.
[43] C. Wu, “VisualSFM: A Visual Structure from Motion System,” http://
ccwu.me/vsfm/, 2013.
[44] B. Triggs, P. Mclauchlan, R. Hartley, and A. Fitzgibbon, “Bundle Adjustment - A Modern Synthesis,” Proc. Int. Workshop on Vision Algorithms,
pp.298–372, 1999.
[45] Y. Jeong, D. Nistér, D. Steedly, R. Szeliski, and I.-S. Kweon, “Pushing
the Envelope of Modern Methods for Bundle Adjustment,” IEEE Trans.
on Pattern Analysis and Machine Intelligence, vol.34, no.8, pp.1605–1617,
2012.
[46] Y.-D. Jian, D.C. Balcan, and F. Dellaert, “Generalized Subgraph Preconditioners for Large-Scale Bundle Adjustment,” Proc. IEEE Int. Conf. on
Computer Vision, pp.295–302, 2011.
[47] A. Kushal and S. Agarwal, “Visibility Based Preconditioning for Bundle
Adjustment,” Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp.1442–1449, 2012.

94

[48] Z. Dai, F. Zhang, and H. Wang, “Robust Maximum Likelihood Estimation
by Sparse Bundle Adjustment using the L1 Norm,” Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, pp.1672–1679, 2012.
[49] M.I.A. Lourakis and A.A. Argyros, “SBA: A Software Package for GenericSparse Bundle Adjustment,” ACM Trans. on Mathematical Software,vol.36, no.1, pp.2:1–2:30, 2009.
[50] M.I.A. Lourakis, “Sparse Non-Linear Least Squares Optimization for Geometric Vision,” Proc. European Conf. on Computer Vision, pp.43–56, 2010.
[51] C. Wu, S. Agarwal, B. Curless, and S.M. Seitz, “Multicore Bundle Adjustment,” Proc. IEEE Conf. on Computer Vision and Pattern Recognition,
pp.3057–3064, 2011.
[52] S. Agarwal, K. Mierle, and Others, “Ceres Solver,” https://code.google.
com/p/ceres-solver/, 2013.
[53] A. Angeli, D. Filliat, S. Doncieux, and J.-A. Meyer, “Fast and Incremental
Method for Loop-Closure Detection using Bags of Visual Words,” IEEE
Trans. on Robotics, vol.24, no.5, pp.1027–1037, 2008.
[54] H. Strasdat, A.J. Davison, J.M.M. Montiel, and K. Konolige, “Double Window Optimisation for Constant Time Visual SLAM,” Proc. IEEE Int. Conf.
on Computer Vision, pp.2352–2359, 2011.
[55] B. Williams, M. Cummins, J. Neira, P. Newman, I. Reid, and J. Tardós, “A
Comparison of Loop Closing Techniques in Monocular SLAM,” Robotics
and Autonomous Systems, vol.57, no.12, pp.1188–1197, 2009.
[56] A. Cohen, C. Zach, S.N. Sinha, and M. Pollefeys, “Discovering and Exploiting 3D Symmetries in Structure from Motion,” Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, pp.1514–1521, 2012.
[57] A. Eudes, S. Naudet-Collette, M. Lhuillier, and M. Dhome, “Weighted
Local Bundle Adjustment and Application to Odometry and Visual SLAM
Fusion,” Proc. British Machine Vision Conference, pp.25.1–25.10, 2010.

95

[58] J. Michot, A. Bartoli, and F. Gaspard, “Bi-Objective Bundle Adjustment
with Application to Multi-Sensor SLAM,” Proc. Int. Symp. on 3D Data
Processing, Visualization and Transmission, 8 pages, 2010.
[59] R. Tenmoku, M. Kanbara, and N. Yokoya, “A Wearable Augmented Reality
System using Positioning Infrastructures and a Pedometer,” Proc. IEEE
Int. Symp. on Wearable Computers, pp.110–117, 2003.
[60] W. Piekarski, R. Smith, and B.H. Thomas, “Designing Backpacks for High
Fidelity Mobile Outdoor Augmented Reality,” Proc. IEEE and ACM Int.
Symp. on Mixed and Augmented Reality, pp.280–281, 2004.
[61] M. Kourogi, N. Sakata, T. Okuma, and T. Kurata, “Indoor/Outdoor Pedestrian Navigation with an Embedded GPS/RFID/Self-Contained Sensor
System,” Proc. Int. Conf. on Artificial Reality and Telexistence, pp.1310–
1321, 2006.
[62] M. Ramachandran, A. Veeraraghavan, and R. Chellappa, “A Fast Bilinear
Structure from Motion Algorithm using a Video Sequence and Inertial Sensors,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.33,
no.1, pp.186–193, 2011.
[63] R. Carceroni, A. Kumar, and K. Daniilidis, “Structure from Motion with
Known Camera Positions,” Proc. IEEE Conf. on Computer Vision and
Pattern Recognition, pp.477–484, 2006.
[64] M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B.
Clipp, C. Engels, D. Gallup, S.-J. Kim, P. Merrell, C. Salmi, S. Sinha,
B. Talton, L. Wang, Q. Yang, H. Stewénius, R. Yang, G. Welch, and H.
Towles, “Detailed Real-Time Urban 3D Reconstruction from Video,” Int.
J. of Computer Vision, vol.78, no.2-3, pp.143–167, 2008.
[65] Y. Bok, Y. Jeong, D.-G. Choi, and I.S. Kweon, “Capturing Village-Level
Heritages with a Hand-Held Camera-Laser Fusion Sensor,” Int. J. of Computer Vision, vol.94, no.1, pp.36–53, 2011.

96

[66] L. Wei, C. Cappelle, Y. Ruichek, and F. Zann, “GPS and StereovisionBased Visual Odometry: Application to Urban Scene Mapping and Intelligent Vehicle Localization,” Int. J. of Vehicular Technology, vol.2011, 17
pages, 2011.
[67] M. Agrawal and K. Konolige, “Real-Time Localization in Outdoor Environments using Stereo Vision and Inexpensive GPS,” Proc. IAPR Int. Conf.
on Pattern Recognition, pp.1063–1068, 2006.
[68] D. Schleicher, L.M. Bergasa, M. Ocaña, R. Barea, and M.E. López, “RealTime Hierarchical Outdoor SLAM Based on Stereovision and GPS Fusion,”
IEEE Trans. on Intelligent Transportation Systems, vol.10, no.3, pp.440–
452, 2009.
[69] D. Dusha and L. Mejias, “Error Analysis and Attitude Observability of a
Monocular GPS/Visual Odometry Integrated Navigation Filter,” Int. J. of
Robotics Research, vol.31, no.6, pp.714–737, 2012.
[70] Y. Yokochi, S. Ikeda, T. Sato, and N. Yokoya, “Extrinsic Camera Parameter
Estimation Based-on Feature Tracking and GPS Data,” Proc. Asian Conf.
on Computer Vision, pp.369–378, 2006.
[71] S. Ikeda, T. Sato, K. Yamaguchi, and N. Yokoya, “Construction of Feature
Landmark Database using Omnidirectional Videos and GPS Positions,”
Proc. Int. Conf. on 3-D Digital Imaging and Modeling, pp.249–256, 2007.
[72] M. Lhuillier, “Incremental Fusion of Structure-from-Motion and GPS using
Constrained Bundle Adjustments,” IEEE Trans. on Pattern Analysis andMachine Intelligence, vol.34, no.12, pp.2489–2495, 2012.
[73] D. Larnaout, V. Gay-Bellile, S. Bourgeois, and M. Dhome, “Vehicle 6-DoF
Localization Based on SLAM Constrained by GPS and Digital Elevation
Model Information,” Proc. IEEE Int. Conf. on Image Processing, pp.2504–
2508, 2013.
[74] H. Kato and M. Billinghurst, “Marker Tracking and HMD Calibration fora Video-Based Augmented Reality Conferencing System,” Proc. IEEE andACM Int. Workshop on Augmented Reality, pp.85–94, 1999.
97

[75] Y. Nakazato, M. Kanbara, and N. Yokoya, “Localization System for Large
Indoor Environments using Invisible Markers,” Proc. ACM Symp. on Virtual Reality Software and Technology, pp.295–296, 2008.
[76] S. Saito, A. Hiyama, T. Tanikawa, and M. Hirose, “Indoor Marker-Based
Localization using Coded Seamless Pattern for Interior Decoration,” Proc.
IEEE Virtual Reality, pp.67–74, 2007.
[77] R. Tenmoku, A. Nishigami, F. Shibata, A. Kimura, and H. Tamura, “Balancing Design Freedom and Constraints in Wall Posters Masquerading as
AR Tracking Markers,” Proc. Human-Computer Interaction International,
pp.263–272, 2009.
[78] J. Wang, H. Zha, and R. Cipolla, “Coarse-to-Fine Vision-Based Localization by Indexing Scale-Invariant Features,” IEEE Trans. on Systems, Man,
and Cybernetics, Part B: Cybernetics, vol.36, no.2, pp.413–422, 2006.
[79] D. Nistér and H. Stewénius, “Scalable Recognition with a Vocabulary Tree,”Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp.2161–
2168, 2006.
[80] C. Valgren and A.J. Lilienthal, “SIFT, SURF & Seasons: AppearanceBased Long-Term Localization in Outdoor Environments,” Robotics andAutonomous Systems, vol.58, no.2, pp.149–156, 2010.
[81] A.C. Murillo, C. Sagüés, J.J. Guerrero, T. Goedemé, T. Tuytelaars, andL. Van Gool, “From Omnidirectional Images to Hierarchical Localization,”
Robotics and Autonomous Systems, vol.55, no.5, pp.372–382, 2007.
[82] G. Schindler, M. Brown, and R. Szeliski, “City-Scale Location Recognition,” Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 7
pages, 2007.
[83] D.M. Chen, G. Baatz, K. Köser, S.S. Tsai, R. Vedantham, T. Pylvänäinen,
K. Roimela, X. Chen, J. Bach, M. Pollefeys, B. Girod, and R. Grzeszczuk,“City-Scale Landmark Identification on Mobile Devices,” Proc. IEEE Conf.
on Computer Vision and Pattern Recognition, pp.737–744, 2011.
98

[84] J. Hays and A.A. Efros, “IM2GPS : Estimating Geographic Informationfrom a Single Image,” Proc. IEEE Conf. on Computer Vision and Pattern
Recognition, 8 pages, 2008.
[85] Y. Kalantidis, G. Tolias, Y. Avrithis, M. Phinikettos, E. Spyrou, P. Mylonas, and S. Kollias, “VIRaL: Visual Image Retrieval and Localization,”
Multimedia Tools and Applications, vol.51, no.2, pp.555–592, 2010.
[86] Y. Yagi, K. Imai, K. Tsuji, and M. Yachida, “Iconic Memory-Based Omnidirectional Route Panorama Navigation,” IEEE Trans. on Pattern Analysis
and Machine Intelligence, vol.27, no.1, pp.78–87, 2005.
[87] J. Košecká, F. Li, and X. Yang, “Global Localization and Relative Positioning Based on Scale-Invariant Keypoints,” Robotics and Autonomous
Systems, vol.52, no.1, pp.27–38, 2005.
[88] H. Badino, D. Huber, and T. Kanade, “Real-Time Topometric Localization,” Proc. IEEE Int. Conf. on Robotics and Automation, pp.1635–1642,
2012.
[89] G. Vaca-Castano, A.R. Zamir, and M. Shah, “City Scale Geo-Spatial Trajectory Estimation of a Moving Camera,” Proc. IEEE Conf. on Computer
Vision and Pattern Recognition, pp.1186–1193, 2012.
[90] A.I. Comport, E. Marchand, M. Pressigout, and F. Chaumette, “Real-Time
Markerless Tracking for Augmented Reality: The Virtual Visual Servoing
Framework,” IEEE Trans. on Visualization and Computer Graphics, vol.12,
no.4, pp.615–628, 2006.
[91] G. Reitmayr and T.W. Drummond, “Going Out: Robust Model-Based
Tracking for Outdoor Augmented Reality,” Proc. IEEE and ACM Int.
Symp. on Mixed and Augmented Reality, pp.109–118, 2006.
[92] C. Cappelle, M.E. El Najjar, F. Charpillet, and D. Pomorski, “Virtual 3D
City Model for Navigation in Urban Areas,” J. of Intelligent & Robotic
Systems, vol.66, no.3, pp.377–399, 2012.

99

[93] G. Baatz, K. Köser, D. Chen, R. Grzeszczuk, and M. Pollefeys, “Leveraging
3D City Models for Rotation Invariant Place-of-Interest Recognition,” Int.
J. of Computer Vision, vol.96, no.3, pp.315–334, 2012.
[94] M.A. Oikawa, T. Taketomi, G. Yamamoto, M. Fujisawa, T. Amano, J.
Miyazaki, and H. Kato, “A Model-Based Tracking Framework for Textureless 3D Rigid Curved Objects,” SBC J. on 3D Interactive Systems, vol.3,
no.2, pp.2–15, 2012.
[95] G. Bleser, H. Wuest, and D. Stricker, “Online Camera Pose Estimation in
Partially Known and Dynamic Scenes,” Proc. IEEE and ACM Int. Symp.
on Mixed and Augmented Reality, pp.56–65, 2006.
[96] P. Lothe, S. Bourgeois, E. Royer, M. Dhome, and S. Naudet-Collette, “RealTime Vehicle Global Localisation with a Single Camera in Dense Urban
Areas: Exploitation of Coarse 3D City Models,” Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp.863–870, 2010.
[97] D. Larnaout, S. Bourgeois, V. Gay-Bellile, and M. Dhome, “Towards Bundle Adjustment with GIS Constraints for Online Geo-Localization of a Vehicle in Urban Center,” Proc. Int. Conf. on 3D Imaging, Modeling, Processing,Visualization and Transmission, pp.348–355, 2012.
[98] M. Tamaazousti, V. Gay-Bellile, S. Naudet Collette, S. Bourgeois, and M.
Dhome, “NonLinear Refinement of Structure from Motion Reconstruction
by Taking Advantage of a Partial Knowledge of the Environment,” Proc.
IEEE Conf. on Computer Vision and Pattern Recognition, pp.3073–3080,
2011.
[99] I. Skrypnyk and D.G. Lowe, “Scene Modelling, Recognition and Tracking
with Invariant Image Features,” Proc. IEEE and ACM Int. Symp. on Mixed
and Augmented Reality, pp.110–119, 2004.
[100] A. Irschara, C. Zach, J.-M. Frahm, and H. Bischof, “From Structure-fromMotion Point Clouds to Fast Location Recognition,” Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, pp.2599–2606, 2009.

100

[101] Y. Li, N. Snavely, D. Huttenlocher, and P. Fua, “Worldwide Pose Estimation using 3D Point Clouds,” Proc. European Conf. on Computer Vision,
pp.15–29, 2012.
[102] T. Sattler, B. Leibe, and L. Kobbelt, “Improving Image-Based Localization
by Active Correspondence Search,” Proc. European Conf. on Computer
Vision, pp.752–765, 2012.
[103] C. Arth, D. Wagner, M. Klopschitz, A. Irschara, and D. Schmalstieg, “WideArea Localization on Mobile Phones,” Proc. IEEE and ACM Int. Symp. on
Mixed and Augmented Reality, pp.73–82, 2009.
[104] F. Wientapper, H. Wuest, and A. Kuijper, “Composing the Feature Map
Retrieval Process for Robust and Ready-to-Use Monocular Tracking,” Computers & Graphics, vol.35, no.4, pp.778–788, 2011.
[105] T. Taketomi, T. Sato, and N. Yokoya, “Real-Time and Accurate Extrinsic
Camera Parameter Estimation using Feature Landmark Database for Augmented Reality,” Computers & Graphics, vol.35, no.4, pp.768–777, 2011.
[106] H. Lim, S.N. Sinha, M.F. Cohen, and M. Uyttendaele, “Real-Time ImageBased 6-DOF Localization in Large-Scale environments,” Proc. IEEE Conf.
on Computer Vision and Pattern Recognition, pp.1043–1050, 2012.
[107] T. Sato, S. Ikeda, and N. Yokoya, “Extrinsic Camera Parameter Recovery
from Multiple Image Sequences Captured by an Omni-Directional MultiCamera System,” Proc. European Conf. on Computer Vision, pp.326–340,
2004.
[108] N. Fioraio and L.D. Stefano, “Joint Detection, Tracking and Mapping by
Semantic Bundle Adjustment,” Proc. IEEE Conf. on Computer Vision and
Pattern Recognition, pp.1538–1545, 2013.
[109] M. Bansal, K. Daniilidis, and H. Sawhney, “Ultra-Wide Baseline Facade
Matching for Geo-Localization,” Proc. European Conf. on Computer Vision, pp.175–186, 2012.

101

[110] T.-Y. Lin, S. Belongie, and J. Hays, “Cross-View Image Geolocalization,”
Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp.891–
898, 2013.
[111] S. Kim, S. DiVerdi, J.S. Chang, T. Kang, R. Iltis, and T. Höllerer, “Implicit 3D Modeling and Tracking for Anywhere Augmentation,” Proc. ACM
Symp. on Virtual Reality Software and Technology, pp.19–28, 2007.
[112] K.Y.K. Leung, C.M. Clark, and J.P. Huissoon, “Localization in Urban Environments by Matching Ground Level Video Images with an Aerial Image,”
Proc. IEEE Int. Conf. on Robotics and Automation, pp.551–556, 2008.
[113] H. Toriya, I. Kitahara, and Y. Ohta, “A Mobile Camera Localization
Method using Aerial-View Images,” Proc. IAPR Asian Conf. on Pattern
Recognition, pp.49–53, 2013.
[114] M. Noda, T. Takahashi, D. Deguchi, I. Ide, H. Murase, Y. Kojima, and T.
Naito, “Vehicle Ego-Localization by Matching In-Vehicle Camera Images
to an Aerial Image,” Proc. Computer Vision in Vehicle Technology: From
Earth to Mars, 10 pages, 2010.
[115] O. Pink, F. Moosmann, and A. Bachmann, “Visual Features for Vehicle
Localization and Ego-Motion Estimation,” Proc. IEEE Intelligent Vehicles
Symposium, pp.254–260, 2009.
[116] T.-J. Cham, A. Ciptadi, W.-C. Tan, M.-T. Pham, and L.-T. Chia, “Estimating Camera Pose from a Single Urban Ground-View Omnidirectional
Image and a 2D Building Outline Map,” Proc. IEEE Conf. on Computer
Vision and Pattern Recognition, pp.366–373, 2010.
[117] S. Mills, “Relative Orientation and Scale for Improved Feature Matching,”
Proc. IEEE Int. Conf. on Image Processing, pp.3484–3488, 2013.
[118] D.G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,”
Int. J. of Computer Vision, vol.60, no.2, pp.91–110, 2004.

102[119] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust
Features (SURF),” Computer Vision and Image Understanding, vol.110,
no.3, pp.346–359, 2008.
[120] T. Sekii, T. Sato, H. Kume, and N. Yokoya, “6-DOF Camera Pose Estimation using Reference Points on an Aerial Image without Altitude Information,” IPSJ Trans. on Computer Vision and Applications, vol.5, pp.134–142,
2013.
[121] M.A. Brubaker, A. Geiger, and R. Urtasun, “Lost! Leveraging the Crowd
for Probabilistic Visual Self-Localization,” Proc. IEEE Conf. on Computer
Vision and Pattern Recognition, pp.3057–3064, 2013.
[122] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm
for Model Fitting with Applications to Image Analysis and Automated Cartography,” Communications of the ACM, vol.24, no.6, pp.381–395, 1981.
[123] J.-M. Morel and G. Yu, “ASIFT: A New Framework for Fully Aﬃne Invariant Image Comparison,” SIAM J. on Imaging Sciences, vol.2, no.2,
pp.438–469, 2009.
[124] M. Özuysal, M. Calonder, V. Lepetit, and P. Fua, “Fast Keypoint Recognition using Random Ferns,” IEEE Trans. on Pattern Analysis and MachineIntelligence, vol.32, no.3, pp.448–461, 2010.
[125] C. Wu, “SiftGPU: A GPU Implementation of Scale Invaraint Feature Transform (SIFT),” http://cs.unc.edu/~ccwu/siftgpu, 2007.
[126] “OpenCV,” http://opencv.org/.

103

List of Publications
Journal Papers
1. T. Sekii, T. Sato, H. Kume, and N. Yokoya, “6-DOF Camera Pose Estimation using Reference Points on an Aerial Image without Altitude Information,” IPSJ Trans. on Computer Vision and Applications, vol.5, pp.134–
142, 2013.
2. H. Kume, T. Anai, T. Sato, T. Taketomi, N. Kochi, and N. Yokoya, “Extrinsic Camera Parameter Estimation from Video Images Considering GPS
Positioning Confidence,” J. of Institute of Image Electronics Engineers ofJapan, vol.43, no.1, pp.35–43, 2014 (in Japanese). (Chapter 2).

International Conferences
1. H. Kume, T. Taketomi, T. Sato and N. Yokoya, “Extrinsic Camera Parameter Estimation using Video Images and GPS Considering GPS Positioning
Accuracy,” Proc. IAPR Int. Conf. on Pattern Recognition, pp.3923–3926,
2010.
2. H. Kume, A. Suppé, and T. Kanade, “Vehicle Localization along a Previously Driven Route using an Image Database,” Proc. IAPR Conf. on
Machine Vision Applications, pp.177–180, 2013. (Chapter 4).
3. T. Kanatani, H. Kume, T. Taketomi, T. Sato, and N. Yokoya, “Detectionof 3D Points on Moving Objects from Point Cloud Data for 3D Modeling104

of Outdoor Environments,” Proc. IEEE Int. Conf. on Image Processing,
pp.2163–2167, 2013.
4. H. Kume, T. Sato and N. Yokoya, “Sampling Based Bundle Adjustmentusing Feature Matches Between Ground-View and Aerial Images,” Proc.
Int. Conf. on Computer Vision Theory and Applications, pp.692–698,
2014. (Chapter 3).

Domestic Conferences
1. H. Kume, T. Taketomi, T. Sato and N. Yokoya, “Camera Pose Estimationfrom an Image Sequence and GPS Data Considering GPS Positioning Confidence,” Proc. JSPRS Autum Conference, pp.127–128, 2009 (in Japanese).
2. H. Kume, T. Taketomi, T. Sato and N. Yokoya, “Consideration of GPS
Positioning Accuracy for Camera Parameter Estimation from GPS andVideo Images,” Proc. Symp. on Pattern Measurement, pp.33–38, 2009
(in Japanese).
3. H. Kume, T. Taketomi, T. Sato and N. Yokoya, “Extrinsic Camera Parameter Estimation using Image Sequence and GPS Considering GPS Positioning Accuracy,” Proc. Meeting on Image Recognition and Understanding,
pp.1198–1205, 2010 (in Japanese).
4. H. Kume, T. Taketomi, T. Sato and N. Yokoya, “Extrinsic Camera Parameter Estimation using Video Images and GPS Considering GPS Positioning
Confidence and Outlier,” IPSJ SIG Technical Report, vol.2010–CVIM–173,
no.36, pp.1–8, 2010 (in Japanese).
5. Y. Babaguchi, H. Kume, T. Sato and N. Yokoya, “Cyclic-Motion Reproduction for Moving Object in Telepresence System,” Proc. ITE Winter Annual
Convention, no.1–1, 1 page, 2010 (in Japanese).
6. T. Kanatani, H. Kume, T. Taketomi, M. Kanbara, and N. Yokoya, “Determination of Moving Objects for 3-D Modeling of Outdoor Environments,”
Proc. IEICE General Conference, no.D–12–31, p.134, 2011 (in Japanese).
105

7. Y. Babaguchi, H. Kume, T. Sato and N. Yokoya, “Viewpoint Varying Video
Textures Preserving Temporal and Spatial Continuity of Moving Objects,”
IEICE Technical Report, PRMU2010-262, pp.139–144, 2011 (in Japanese).
8. H. Kume, T. Taketomi, T. Sato and N. Yokoya, “Camera Pose Estimation from an Image Sequence and GPS Data by Bundle Adjustment Considering Position Continuity,” Proc. Meeting on Image Recognition andUnderstanding, pp.825–830, 2011 (in Japanese).
9. T. Sekii, H. Kume, T. Sato and N. Yokoya, “Camera Pose Estimation using
Reference Points on an Aerial Image without Altitude Information by Solving PnP Problem,” Proc. Kansai-Section Joint Convention of Institutes ofElectrical Engineering, no.30P2–33, 2 pages, 2011 (in Japanese).
10. T. Sekii, H. Kume, T. Sato and N. Yokoya, “Camera Position and Posture
Estimation for a Single Ground-View Image using Aerial Images,” IPSJ SIG
Technical Report, vol.2012–CVIM–180, no.5, pp.1–8, 2012 (in Japanese).
11. T. Kanatani, H. Kume, T. Taketomi, T. Sato, and N. Yokoya, “Detection of
3D Points on Moving Objects from Point Cloud Data Based on Luminance
Variation for 3D Modeling of Outdoor Environments,” IEICE Technical
Report, ITS2011–42, pp.153–158, 2012 (in Japanese).
12. T. Sekii, H. Kume, T. Sato and N. Yokoya, “Camera Pose Estimation froma Ground-View Image using Reference Points on an Aerial Image without
Altitude Information,” Proc. Meeting on Image Recognition and Understanding, 8 pages, 2012 (in Japanese).
13. H. Kume, A. Suppé, and T. Kanade, “Vehicle Pose Estimation along a
Previously Driven Route using an Image Database,” IPSJ SIG Technical
Report, vol.2013–CVIM–186, no.13, pp.1–7, 2013 (in Japanese). (Chapter
4).
14. A. Oko, H. Kume, T. Sato, T. Machida, N. Sano, and N. Yokoya, “FreeViewpoint Image Rendering for Evaluating Image Processing Algorithms
for In-Vehicle Camera System,” Proc. Forum on Information Technology,
vol.3, pp.141–142, 2013 (in Japanese).
10615. A. Oko, T. Sato, H. Kume, T. Machida, and N. Yokoya, “Evaluation of
Image Processing Algorithms on In-Vehicle Camera System using FreeViewpoint Image Rendering,” IEICE Technical Report, ITS2013–54, pp.141–
146, 2014 (in Japanese).

Commentaries
1. H. Kume, A. Suppé, and T. Kanade, “Vehicle Localization using an Image
Database,” Image Laboratory, vol.24, no.12, pp.1–7, 2013 (in Japanese).
(Chapter 4).

Awards
1. NAIST Top Scholarship Program, 2010.

107