Project AutoVision: Localization and 3D Scene Perception for an
Autonomous Vehicle with a Multi-Camera System

arXiv:1809.05477v2 [cs.RO] 5 Mar 2019

Lionel Heng1 , Benjamin Choi1 , Zhaopeng Cui2 , Marcel Geppert2 , Sixing Hu3 , Benson Kuan1 , Peidong Liu2 ,Rang Nguyen3 , Ye Chuan Yeo1 , Andreas Geiger4 , Gim Hee Lee3 , Marc Pollefeys2,5 , and Torsten Sattler6

Abstract— Project AutoVision aims to develop localizationand 3D scene perception capabilities for a self-driving vehicle.
Such capabilities will enable autonomous navigation in urban
and rural environments, in day and night, and with cameras as
the only exteroceptive sensors. The sensor suite employs manycameras for both 360-degree coverage and accurate multi-view
stereo; the use of low-cost cameras keeps the cost of this sensor
suite to a minimum. In addition, the project seeks to extend
the operating envelope to include GNSS-less conditions which
are typical for environments with tall buildings, foliage, andtunnels. Emphasis is placed on leveraging multi-view geometry
and deep learning to enable the vehicle to localize and perceive
in 3D space. This paper presents an overview of the project,and describes the sensor suite and current progress in the areasof calibration, localization, and perception.

I. INTRODUCTION
The three DARPA Grand Challenges in the last decade
set off a wave of disruption in the automotive industry. Withwidespread belief that autonomous vehicles can revolutionize
logistics and mobility, automakers and technology companies
are racing with one another to put autonomous vehicles on
the road within the next few years. LiDAR sensors are theprimary sensing modality for a vast majority of autonomousvehicles; they generate highly accurate 3D point cloud datain both day and night, and enable localization and 3D scene
perception at all times of the day. In contrast, cameras requiresufficient ambient lighting, and do not directly provide 3Dpoint cloud data. However, cameras yield high-resolution
image data which better facilitates scene segmentation andunderstanding. In addition, we can leverage multi-view geometry techniques to infer depth data from multiple cameras
albeit with lower accuracy than depth data from LiDAR
sensors. Cameras can be fitted with either wide-field-of-view
or fisheye lenses, giving them a significantly larger verticalfield of view and higher vertical resolution compared to
LiDAR sensors. In Project AutoVision, we choose to focus
on cameras as the sole sensing modality for autonomousvehicles; we observe that much research remains to be donein realising robust visual localization and perception forautonomous vehicles.
Project AutoVision started in late 2016 with the goal
to develop localization and 3D scene perception algorithms
1 DSO National Laboratories
2 ETH Zürich
3 National University of Singapore
4 MPI-IS and University of Tübingen
5 Microsoft, Switzerland
6 Chalmers University of Technology, Sweden

for autonomous vehicles exclusively equipped with cameras.
Project AutoVision is similar to, but differs from Project
V-Charge [9, 31, 14] in the aspect that Project AutoVisionextends the operating envelope from parking lots and garages
to large-scale urban and rural environments with higher
driving speeds and widely varying illumination conditions.
Parallels can also be drawn between Project AutoVision and
AutoX, both of which only rely on cameras for localizationand perception. However, little is known about AutoX’s
localization and perception approaches due to commercial
interests.
We aim to localize in both mapped and unmapped areas
without relying on GNSS; we do not want to limit thevehicle’s operation to mapped areas in GNSS-less conditions. In addition, we work towards real-time 3D mapping
as a 3D geometric map can aid navigation of multi-levelstructures and higher-level scene perception tasks such as
terrain analysis and 3D scene understanding. We follow thetraditional approach [33, 3] of applying multi-view geometry
to localization and 3D geometric mapping, and machinelearning to cross-modal matching, object detection, and scene
segmentation. On the other hand, with the advent of deepneural networks, people [2] have used end-to-end learning forvision-based autonomous vehicles but with limited success.
We make the following contributions:
1) Real-time visual-inertial odometry with a multi-camera
system [25, 26].
2) Real-time GNSS-less visual localization in unmappedareas using geo-referenced satellite imagery and without GNSS, assuming that the initial global position andheading of the vehicle are known [19, 18].
3) Real-time GNSS-less visual localization with a multicamera system in mapped areas using a geo-referenced
sparse 3D map and without prior knowledge of thevehicle’s global pose [11].
4) Real-time 3D dense mapping with a multi-fisheyecamera system [5].
In this paper, we briefly describe these contributions whichhave been integrated into a single working system. The
reader can refer to our published work for more details on thealgorithms and experimental results. In addition, the paper
gives details of the hardware setup, the software architecture,and the automated methods used for calibrating the multisensor suite; such details are not found in our published workon individual localization and perception components.

Fig. 2. A close-up of the sensor suite on the AutoVision vehicle platform.
Fig. 1.

The AutoVision vehicle platform.

II. S YSTEM
In this section, we give an overview of the sensors on the
AutoVision vehicle platform, and the software architecturethat enables various software modules to work together toenable the vehicle to localize and perceive in 3D.
Our AutoVision vehicle platform is a Isuzu D-Max pickup truck which has been modified to include a drive-by-wiresystem for autonomous driving. Fig. 1 shows the vehicle
platform while Fig. 2 shows a close-up view of the sensors
on the vehicle roof. Four color cameras and twelve NIR
cameras are fitted with 180◦ -field-of-view fisheye lenses
and installed in a surround-view configuration on top of
the vehicle. All cameras output 2-megapixel images at 30
Hz, and are set to automatic exposure mode so that they
can adapt to changing lighting conditions. We only use 12bit grayscale images from the NIR cameras as input to all
localization and perception modules; color cameras are onlyused for visualization purposes. NIR cameras are more lightsensitive than color cameras, and can detect light in boththe visible and NIR wavelengths. In addition, NIR cameras
provide sharp and clean images unlike Bayer-encoded color
images that suffer from demosaicing artefacts. We use NIR
cameras in conjunction with NIR illuminators which can
improve low-light imaging quality and whose illumination
is invisible to and does not distract drivers on the road.
Fig. 4 shows examples of images captured at approximately
the same location in varying lighting conditions. Camera
enclosures provide cameras with IP67 protection from theweather elements. Fig. 3 shows the camera layout on thevehicle roof. The front side of the vehicle has the highest
number of NIR cameras; these 5 NIR cameras facilitate widebaseline stereo, and in turn, long-range perception whichis critical for autonomous vehicles moving forward at highspeeds. We exploit the dominantly longitudinal movement of
the vehicle by simulating a multi-baseline stereo system oneach side of the vehicle and which consists of 2 actual NIR
cameras and at least 1 virtual NIR camera.
A. Hardware
A dual-antenna GNSS/INS system with a tactical-grade
IMU is installed in the vehicle. Data from this GNSS/INS

Fig. 3. The sensor layout on the AutoVision vehicle platform. Each greencolored frustum and grey-colored cylinder represent a camera and LiDAR
sensor respectively. The origin of the three axes indicates the location of
the IMU. The red-colored x-axis points towards the front of the vehicle
while the green-colored y-axis points towards the left of the vehicle. The
extrinsic transformation between each sensor and the IMU was estimatedwith automated calibration tools.

system is post-processed offline to yield centimeter-levelground-truth position and attitude data which is used to evaluate localization accuracy. A 3D LiDAR sensor is mounted ateach of the four corners of the vehicle roof; fused point clouddata from all 4 LiDAR sensors is used to evaluate perception
accuracy. All sensor data is hardware-timestamped to submicrosecond precision. Such accurate time synchronization
is made possible through the use of a time server. This time
server synchronizes to the GNSS/INS system via PPS signals
and NMEA data. In turn, LiDAR sensors and cameras timesynchronize to the time server via PPS/NMEA and PTP
respectively. All sensors interface over a 10 GbE network
switch with multiple industrial-grade computers equippedwith GPUs. The 16-camera system yields 1.38 GB/s of image
data which we record simultaneously to 4 solid-state drives(SSDs); we configure each computer to have two SSDs, andthus, two computers are required for data recording.
B. Software
Our software stack is based on the ROS 2 software framework, and runs on Windows 10. We use RTI Connext DDS
for inter-process communications. Fig. 5 shows our softwarearchitecture; with the exception of the block representing
sparse 3D map reconstruction which runs offline, each block
represents a node which subscribes and publishes to topics.
GNSS and IMU measurements are only used by sparse 3D

(a) Day-time.
Fig. 4.

(b) Night-time with NIR illumination.

(c) Night-time without NIR illumination.

Images captured from a front left camera on the AutoVision vehicle and in varying lighting conditions.

map reconstruction and visual-inertial odometry respectively.
All nodes subscribe to image data.
III. C ALIBRATION
An accurate calibration is an essential prerequisite for
localization and perception to work well with a multi-camera
system. Our calibration pipeline is automated and involves
the following steps in order: intrinsic and extrinsic calibrationof a multi-camera system, extrinsic calibration between acalibrated multi-camera system and a GNSS/INS system,and extrinsic calibration between a LiDAR sensor and acalibrated multi-camera system.
We do intrinsic and extrinsic calibration of the multicamera system with the help of a fiducial target which is a
grid of AprilTag markers [32] with known dimensions, andis shown in Fig. 6. As each AprilTag marker has an unique
identifier, calibration will work even if multiple cameras
observe different parts of the target. This versatility comesin handy when calibrating pairs of cameras at the vehicle’s
corners and with perpendicular optical axes; it is difficult forsuch a pair of cameras to observe the entire target. For the
intrinsic calibration, we can choose from multiple camera
projection and distortion models. In this project, we use theunified projection model [12, 1] and the plumb bob distortion
model [4]. We also perform a photometric calibration of eachcamera using the method described in [7]. This photometriccalibration is useful for photometric-based matching between
images captured with different exposure times, for example,
in direct visual odometry and plane-sweeping stereo.
We obtain the extrinsic transformation between the calibrated multi-camera system and the GNSS/INS system byfollowing an approach similar to that of Heng et al. [16, 17].
Here, the reference frame of the GNSS/INS system coincides
with that of the IMU. We run semi-direct stereo visual
odometry (VO) [15] for a stereo pair on each side of thevehicle. Each instance of stereo VO yields a set of camera
poses and feature tracks. From hand-eye calibration using thereference camera’s poses from stereo VO and the GNSS/INS
system’s poses, we obtain an initial estimate of the extrinsic
transformation. Subsequently, we refine the extrinsic transformation by solving a non-linear least-squares problem inwhich we minimize the sum of squared reprojection errors
associated with feature tracks while keeping the GNSS/INS
system’s poses and inter-camera transformations fixed.

With the same fiducial target from intrinsic and extrinsic
calibration of the multi-camera system, we perform extrinsic
calibration between each LiDAR sensor and the calibrated
multi-camera system. Given a set of images captured simultaneously from the multi-camera system, we detect the fiducial
target in each image, and estimate its pose with respect tothe multi-camera system by minimizing the squared sumof reprojection errors across all images in which the target
was detected. At the same time, we identify the set ofpoints corresponding to the fiducial target in the LiDAR
scan by using plane segmentation [10], and estimate thetarget’s plane parameters with respect to the LiDAR sensor.
With repeated observations of the target in different orientations, we independently estimate the rotation and translation components of the extrinsic transformation between
the LiDAR sensor and the multi-camera system by doing
singular value decomposition and solving a linear system of
equations respectively. Subsequently, we refine the extrinsic
transformation by minimizing the sum of squared point-planeerrors; the points form part of the LiDAR scan identified ascorresponding to the target, and the plane parameters areinferred from the estimated pose of the target with respect
to the multi-camera system. Fig. 7 shows the projection of
LiDAR scan points into a camera image using the resultsof extrinsic calibration between the LiDAR sensor and themulti-camera system.
IV. L OCALIZATION
One goal of Project AutoVision is to enable an autonomous vehicle to localize in both unmapped andpremapped environments. A vehicle relying on map-based
localization is restricted to movement within the map. We
want to allow the vehicle to navigate beyond the map into
unmapped areas by leveraging satellite imagery. However, apremapped environment enables the vehicle to localize withhigher accuracy. Global pose estimates are susceptible topose jumps; smooth local pose estimates are required forstable path tracking and to build consistent 3D maps. For
this purpose, we use direct visual-inertial odometry whichruns at the frame rate of the multi-camera system.
A. Direct Visual-Inertial Odometry
Our direct visual-inertial odometry (VIO) implementation
for a multi-camera system [25, 26] estimates the local pose of
the vehicle at 30 Hz. Our direct VIO implementation contains

Fig. 5.

Our AutoVision software architecture. GNSS is only used for sparse 3D map reconstruction.

Fig. 6. The fiducial target used for calibration of the multi-sensor system.
Fig. 8. The positions estimated by VIO vs ground truth positions. One
front stereo pair and one rear stereo pair were used for VIO.

B. 3-DoF Localization with Satellite Imagery

Fig. 7.

Projeciton of LiDAR scan points into a camera image.

two threads: (1) the tracking thread estimates the local poseby minimizing photometric errors between the most recent
keyframe and the current frame, and (2) the mapping thread
initializes the depth of all sampled feature points using
plane-sweeping stereo, and uses a sliding window optimizer
to refine poses and structure jointly. Extensive experiments
described by Liu et al. [26] show our implementation towork robustly for a 4-stereo-camera configuration with lessthan 1% translational drift in day-time and night-time with
NIR illumination, and less than 2% translational drift innight-time without NIR illumination. Fig. 8 plots the pose
estimates from our VIO implementation against ground truth
data for a 8.2km route in a route covering both urban andrural environments.

In areas that have not been premapped, we rely on satellite
imagery to estimate the 3-DoF global pose. Specifically, we
estimate the (x, y) position and heading of the vehicle withrespect to the UTM coordinate frame. Assuming that theinitial position and heading are known from user input, we
use the particle filter approach in which we use local pose
data from VIO for particle propagation and output from a
deep network [19, 18] for particle weighting.
As shown in Fig. 9, the deep network called CVMNet is a Siamese network that takes satellite and groundlevel panoramic images as input. We obtain the panoramic
image by stitching the cylindrical projections of images
taken from four cameras: one camera on each side of thevehicle. For each image, we extract local features via fully
convolutional networks. Two aligned NetVLADs aggregate
local features from both images into global descriptors thatare in a common space for similarity comparison. The weight
for each particle is inversely proportional to the Euclidean
distance between the global descriptors corresponding to theground-level panoramic image and the satellite image patch
nearest the particle’s position.
We run two experiments with a 5km route in both anurban environment and a rural environment. Experimental

(a) Bird’s eye view.

Fig. 9. The architecture of our deep network, CVM-Net, for cross-view
matching [19, 18].

(b) Close-up view.
Fig. 11. The sparse 3D map of a mixed urban and rural environment andgenerated by our reconstruction pipeline.

Fig. 10. A visualization of our satellite-imagery-based localisation in an
urban environment.

results show that our satellite-imagery-based localizationachieves an average position error of 9.92m and an averageheading error of 0.32◦ over a 5km route in the urban
environment, and an average position error of 9.29m andan average heading error of 0.42◦ in the rural environment.
Fig. 10 visualizes our satellite-imagery-based localisation inan urban environment. The top left image shows the groundview panoramic image. The bottom left image shows thepaths estimated by our localization and GNSS/INS system
in green and red respectively on the bottom left. In the rightimage, particles are shown in blue on the right and interposed
against a likelihood map; the more red the pixel, the higher
the likelihood that the vehicle is located at that pixel.
C. Sparse 3D Map Reconstruction
Sparse 3D map reconstruction is required for map-based
6-DoF localization which is described in Section IV-D. Prior
to localization, we build a sparse 3D map in which each 3D
point is associated with one or more local SIFT features [27].
To minimize the time required for large-scale reconstruction, our approach does not reconstruct the scene fromscratch, and instead, uses reasonably accurate initial pose
estimates from a GNSS/INS system to initialize all camera
poses. In addition, we require a minimum amount of camera
motion between images used for mapping. Next, we perform
feature matching between nearby images, and use the feature

matches and initial poses to triangulate the scene. We thenrepeatedly optimize the scene structure and camera poses
using bundle adjustment followed by the merging of feature
tracks. This approach is implemented on top of the COLMAP
structure-from-motion (SfM) framework [30]. During bundle
adjustment, we enforce that the extrinsic parameters ofthe multi-camera system on the AutoVision vehicle remain
constant. Fig. 11 shows a sparse 3D map of a mixed urban
and rural environment.
D. 6-DoF Localization with a Sparse 3D Map
Given a sparse 3D map computed using the approachdescribed in Section IV-C and without prior knowledgeof the AutoVision vehicle’s global pose, we localize the
AutoVision vehicle by extracting local SIFT features [27]from the images captured by the cameras mounted on the
AutoVision vehicle and matching the descriptors of these 2D
features against the descriptors associated with the 3D points
in the map. From the 2D-3D matches, we apply a generalized
perspective-n-point pose solver [22, 23] inside a RANSAC
loop [8] to estimate the vehicle’s pose. With a sparse 3D
map of a large area which contains many 3D points, 2D3D matching is the main computational bottleneck in our
pipeline. In this case, we use a prioritized matching approachbased on Active Search [29] for improved matching efficiency. Geppert et al. [11] describe our localization approachin greater detail.
For our experiments, we use the same routes used for thesatellite-imagery-based localization experiments described in
Section IV-B. Our localization pipeline runs at around 2 Hzon the AutoVision vehicle. Fig. 12 shows the estimated andground truth positions for the urban and rural routes. For
the urban route, the mean and median errors of all reported

(a) Urban environment. In some parts (b) Rural environment. The errors atof the area, the localization fails con- the top end are likely caused by repetsistently to estimate a pose.
itive structures on both sides of theroad.
Fig. 12. The positions estimated by map-based localization vs. groundtruth positions for a 5km route in two different environments.

(a) Image from the front center cam- (b) Depth image generated by planeera.
sweeping stereo.

(c) Dynamic objects detected by theobject detector.
Fig. 13.

(d) Raycasted TSDF volume.

Output from the AutoVision perception modules [5].

poses are 3.31m and 1.84m for the position, and 2.6◦ and1.9◦ for the heading, respectively. For the rural route, themean and median errors of all reported poses are 3.48m and1.81m for the position, and 4.2◦ and 3.3◦ for the heading,respectively.
V. P ERCEPTION
Dense 3D mapping in real-time is a prerequisite for autonomous navigation in complex environments. Our perception pipeline begins with plane-sweeping stereo generating
depth images. In turn, depth images are fused into a truncatedsigned distance function (TSDF) volume. A 3D map isreconstructed from this TSDF volume via ray-casting. To
avoid dynamic objects from corrupting the 3D map via
trails of artefacts, we detect potentially dynamic objects notbelonging to the static environment, and remove their associated depth estimates from the depth images prior to depth
fusion. Fig. 13 visualizes the outputs from plane-sweeping
stereo, object detection, and TSDF-based 3D mapping. Moredetails of our perception pipeline can be found in [5].
A. Plane-Sweeping Stereo
Plane-sweeping stereo computes a depth image for multiple images with known camera poses by sweeping a setof planes through 3D space. Each plane represents a depth
hypothesis and defines a homography mapping from every
other view to the reference view. We estimate the depth foreach pixel in a reference image by using each plane to warp
each non-reference image to the reference image, evaluating
the image dissimilarity at that pixel, and choosing the planethat minimizes the image dissimilarity. We use the GPU
implementation of plane-sweeping stereo for fisheye cameras[13] which computes depth images directly from fisheye
images without the need for undistortion, allowing us to
use the full field-of-view of the cameras. On the AutoVisionvehicle, plane-sweeping stereo runs at an average of 15 Hz
for the 5 cameras at the front of the vehicle and with images
downsampled to half-resolution.
B. TSDF-based 3D Mapping
A single depth image may not contain sufficient geometric
information for up-stream modules such as 3D semantic
segmentation and motion planning. Hence, we need to fusedepth images estimated at different positions in time tocreate a dense 3D map. We use a standard fusion technique:the scene is represented via a set of voxels where eachvoxel stores a TSDF value [6]. Here, each voxel stores thesigned distance to the closest object surface (negative insideof objects, positive outside of objects, zero on surfaces),
truncated to a certain maximum / minimum value. Whenevera new depth image along with its camera pose becomes
available, we update the 3D model. We use the map fusion
pipeline in the InfiniTAM library [20, 21]. We also use thefast raycasting algorithm in [20, 21] to reconstruct the 3D
map in the current camera view. The pipeline runs at around
20 Hz on the AutoVision vehicle.
C. Dynamic Object Detection
Dynamic objects leave behind trails of artefacts in the 3D
map. We leverage 2D object detection to solve this problem.
Given a reference image, we detect dynamic objects, i.e.
humans and vehicles. In turn, for the corresponding depth
image, we mask out pixels located within the 2D boundingboxes of detected objects. This way, we avoid integrating
depth estimates associated with dynamic objects into the 3D
map. We use the YOLOv3 object detection network [28]trained on the Microsoft COCO dataset [24]. To improve
the inference performance with distorted grayscale images
from our NIR fisheye cameras, we fine-tune the network bytruncating the first and last layers, and retrain the network
using our labeled datasets.
VI. CONCLUSIONS
Project AutoVision has successfully demonstrated localization and 3D scene perception for autonomous vehicles
with multi-camera systems, in both urban and rural environments, and without GNSS. As Project AutoVision progresses, we will continue to enhance localization and perception capabilities, and add more modules to our software
stack. These modules include but are not limited to, change
detection for 3D maps, obstacle detection, dynamic object
tracking and classification, and semantic 3D mapping.

R EFERENCES[1] J. Baretto and H. Araujo. Issues on the geometry of central catadioptic image formation. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR),
2001.
[2] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner,
B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller,
J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end
learning for self-driving cars. CoRR, abs/1604.07316,
2016.
[3] A. Broggi, P. Cerri, S. Debattisti, M. C. Laghi,
P. Medici, D. Molinari, M. Panciroli, and A. Prioletti.
Proudpublic road urban driverless-car test. IEEE Transactions on Intelligent Transportation Systems, 16(6):3508–3519, 2015.
[4] D. C. Brown. Decentering distortion of lenses. Photometric Engineering, 32(3):444–462, 1966.
[5] Z. Cui, L. Heng, Y. C. Yeo, A. Geiger, M. Pollefeys,
and T. Sattler. Real-time dense mapping for self-driving
vehicles using fisheye cameras. In IEEE International
Conference on Robotics and Automation (ICRA), 2019.
[6] B. Curless and M. Levoy. A volumetric method forbuilding complex models from range images. In Annual Conference on Computer Graphics and Interactive
Techniques (SIGGRAPH), 1996.
[7] J. Engel, V. C. Usenko, and D. Cremers. A photometrically calibrated benchmark for monocular visual
odometry. CoRR, abs/1607.02555, 2016.
[8] M. A. Fischler and R. C. Bolles. Random sample
consensus: a paradigm for model fitting with applications to image analysis and automated cartography.
Communications of the ACM, 24:381395, 1981.
[9] P. Furgale, U. Schwesinger, M. Rufli, W. Derendarz,
H. Grimmett, P. Mühlfellner, S. Wonneberger, J. Timpner, S. Rottmann, B. Li, B. Schmidt, T. N. Nguyen,
E. Cardarelli, S. Cattani, S. Brüning, S. Horstmann,
M. Stellmacher, H. Mielenz, K. Köser, M. Beermann,
C. Häne, L. Heng, G. H. Lee, F. Fraundorfer, R. Iser,
R. Triebel, I. Posner, P. Newman, L. Wolf, M. Pollefeys,
S. Brosig, J. Effertz, C. Pradalier, and R. Siegwart.
Toward automated driving in cities using close-tomarket sensors: An overview of the v-charge project.
In IEEE Intelligent Vehicles Symposium (IV), 2013.
[10] A. Geiger, F. Moosmann, U. Car, and B. Schuster.
Automatic camera and range sensor calibration using
a single shot. In IEEE International Conference on
Robotics and Automation (ICRA), 2012.
[11] M. Geppert, P. Liu, Z. Cui, M. Pollefeys, and T. Sattler.
Efficient 2d-3d matching for multi-camera visual localization. In IEEE International Conference on Robotics
and Automation (ICRA), 2019.
[12] C. Geyer and K. Daniilidis. A unifying theory forcentral panaromic systems and practical implications.
In European Conference on Computer Vision (ECCV),
2000.
[13] C. Häne, L. Heng, G. H. Lee, A. Sizov, and M. Polle-

feys. Real-time direct dense matching on fisheye
images using plane-sweeping stereo. In International
Conference on 3D Vision (3DV), 2015.
[14] C. Häne, L. Heng, G. H. Lee, F. Fraundorfer, P. Furgale,
T. Sattler, and M. Pollefeys. 3d visual perception for
self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection.
Image and Vision Computing (IVC), 68:14–27, 2017.
[15] L. Heng and B. Choi. Semi-direct visual odometry
for a fisheye-stereo camera. In IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS),
2016.
[16] L. Heng, P. Furgale, and M. Pollefeys. Leveraging
image-based localization for infrastructure-based calibration of a multi-camera rig. Journal of Field Robotics(JFR), 32:775–802, 2015.
[17] L. Heng, G. H. Lee, and M. Pollefeys. Self-calibrationand visual slam with a multi-camera system on a micro
aerial vehicle. Autonomous Robots (AURO), 39:259–
277, 2015.
[18] S. Hu and G. H. Lee. Image-based geo-localization
using satellite imagery. International Journal of Computer Vision (IJCV), 2019.
[19] S. Hu, M. Feng, R. M. H. Nguyen, and G. H. Lee. Cvmnet: Cross-view matching network for image-based
ground-to-aerial geo-localization. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR),
2018.
[20] O. Kähler, V. A. Prisacariu, C. Y. Ren, X. Sun, P. Torr,
and D. Murray. Very high frame rate volumetricintegration of depth images on mobile devices. IEEE
Transactions on Visualization and Computer Graphics(TVCG), 21(11):1241–1250, 2015.
[21] O. Kähler, V. Prisacariu, J. Valentin, and D. Murray.
Hierarchical voxel block hashing for efficient integration of depth images. IEEE Robotics and Automation
Letters (RA-L), 1(1):192–197, 2016.
[22] L. Kneip, P. Furgale, and R. Siegwart. Using multicamera systems in robotics: Efficient solutions to thenpnp problem. In IEEE International Conference on
Robotics and Automation (ICRA), 2013.
[23] G. H. Lee, B. Li, M. Pollefeys, and F. Fraundorfer.
Minimal solutions for the multi-camera pose estimation
problem. International Journal of Robotics Research(IJRR), 34:837–848, 2015.
[24] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona,
D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft
coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[25] P. Liu, L. Heng, T. Sattler, A. Geiger, and M. Pollefeys.
Direct visual odometry for a fisheye-stereo camera.
In IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2017.
[26] P. Liu, M. Geppert, L. Heng, T. Sattler, A. Geiger,
and M. Pollefeys. Towards robust visual odometry
with a multi-camera system. In IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS),

2018.
[27] D. G. Lowe. Distinctive image features from scaleinvariant keypoints. International Journal of Computer
Vision (IJCV), 60(2):91110, 2004.
[28] J. Redmon and A. Farhadi. Yolov3: An incremental
improvement. CoRR, abs/1804.02767, 2018.
[29] T. Sattler, B. Leibe, and L. Kobbelt. Efficient & effective prioritized matching for large-scale image-based
localization. IEEE Transactions on Pattern Analysis
and Machine Intelligence (PAMI), 39:17441756, 2017.
[30] J. Schönberger and J. Frahm. Structure-from-motion
revisited. In IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2016.
[31] U. Schwesinger, M. Bürki, J. Timpner, S. Rottmann,
L. Wolf, L. M. Paz, H. Grimmett, I. Posner, P. Newman,
C. Häne, L. Heng, G. H. Lee, T. Sattler, M. Pollefeys, M. Allodi, F. Valenti, K. Mimura, B. Goebelsmann, W. Derendarz, P. Mühlfellner, S. Wonneberger,
R. Waldmann, S. Grysczyk, C. Last, S. Brüning,
S. Horstmann, M. Bartholomäus, C. Brummer, M. Stellmacher, F. Pucks, M. Nicklas, and R. Siegwart. Automated valet parking and charging for e-mobility. In
IEEE Intelligent Vehicles Symposium (IV), 2016.
[32] J. Wang and E. Olson. AprilTag 2: Efficient androbust fiducial detection. In IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS),
2016.
[33] J. Ziegler, P. Bender, M. Schreiber, H. Lategahn,
T. Strauss, C. Stiller, T. Dang, U. Franke, N. Appenrodt,
C. G. Keller, E. Kaus, R. G. Herrtwich, C. Rabe,
D. Pfeiffer, F. Lindner, F. Stein, F. Erbs, M. Enzweiler,
C. Knoppel, J. Hipp, M. Haueis, M. Trepte, C. Brenk,
A. Tamke, M. Ghanaat, M. Braun, A. Joos, H. Fritz,
H. Mock, M. Hein, and E. Zeeb. Making bertha drive- an autonomous journey on a historic route. IEEE
Intelligent Transportation Systems Magazine, 6(2):8–
20, 2014.