Towards Robust Visual Odometry with a Multi-Camera System
Peidong Liu1 , Marcel Geppert1 , Lionel Heng2 , Torsten Sattler1 , Andreas Geiger1,3 , and Marc Pollefeys1,4

Abstract— We present a visual odometry (VO) algorithm
for a multi-camera system and robust operation in challenging environments. Our algorithm consists of a pose tracker
and a local mapper. The tracker estimates the current poseby minimizing photometric errors between the most recent
keyframe and the current frame. The mapper initializes thedepths of all sampled feature points using plane-sweeping
stereo. To reduce pose drift, a sliding window optimizer isused to refine poses and structure jointly. Our formulation
is flexible enough to support an arbitrary number of stereo
cameras. We evaluate our algorithm thoroughly on five datasets.
The datasets were captured in different conditions: daytime,night-time with near-infrared (NIR) illumination and nighttime without NIR illumination. Experimental results show thata multi-camera setup makes the VO more robust to challenging
environments, especially night-time conditions, in which a single
stereo configuration fails easily due to the lack of features.

I. I NTRODUCTION
Visual odometry (VO) is a technique used to estimate
camera motion from images. As a fundamental block for
robot navigation, virtual reality, augmented reality, and so on,
VO has made significant progress over the past few decades.
Existing approaches range across conventional feature-based
methods [15], hybrid methods [8], and direct methods [7].
To make VO useful for real-world applications, we needto make VO work robustly under various conditions. For
example, to serve as a fundamental module for autonomous
driving, robust vehicle motion estimation is critical for
path/trajectory tracking, and environment perception modules. Motion estimation errors and failures would cause tragic
accidents, and limit wide-scale deployment of VO.
Several approaches have been proposed to improve therobustness of VO for specific environments. Alismail et al.
[1] propose a dense binary descriptor that can be integrated
within a multi-channel Lucas Kanade framework to improveillumination change robustness. Park et al. [21] performa systematic evaluation of real-time capable methods forillumination change robustness in direct visual SLAM. Zhang
et al. [26] propose an active exposure control method forrobust visual odometry in high dynamic range (HDR) environments. For each frame, they choose an exposure timethat maximizes an image quality metric. In [22], a direct
monocular SLAM algorithm based on the Normalized Information Distance (NID) metric is proposed. They show that
1 Computer Vision and Geometry Group, Department of Computer Science, ETH Zürich, Switzerland
2 Robotics Autonomy Lab, Manned - Unmanned Programme, Information
Division, DSO National Laboratories, Singapore
3 Autonomous Vision Group, MPI for Intelligent Systems, Tübingen,Germany
4 Microsoft, Redmond, USA

the information-theoretic NID metric provides robustness toappearance variations due to lighting, weather, and structuralchanges in the scene.
In this paper, we propose to improve the robustness of VO
by using a multi-camera system. A multi-camera system cancover a wide field-of-view and thus provide redundancy forpoorly textured environments. Our experimental evaluation
demonstrates that VO with a multi-camera system can still
estimate vehicle motion reliably in dark environments atnight; in contrast, a single stereo configuration is prone to
failure due to the lack of texture.
Our algorithm consists of a pose tracker and a local
mapper. The tracker estimates the current pose by minimizing photometric errors between the most recent keyframe
and the current frame across multiple cameras. The local
mapper consists of a sliding window optimizer and a twoview stereo matcher. Both vehicle poses and structure are
jointly optimized by the sliding window optimizer. This
optimization minimizes long-term pose drift. The depths of
newly sampled feature points are initialized by the two-view
stereo matcher. We thoroughly evaluate our algorithm undermultiple challenging conditions. In particular, we selectfive different datasets with different characteristics for ourevaluations. The datasets are captured in varying lighting
conditions, at different vehicle speeds, and over different
trajectory lengths.
The remainder of the paper is organized as follows.
Section II reviews related work. Section III introduces thenotation used throughout the paper. Section IV describes
our algorithm in detail. Section V provides an extensive
experimental evaluation of our method. Finally, Section VI
concludes the paper.
II. R ELATED W ORK
Many different camera configurations have been used in
the literature. The two most commonly used configurations
are the monocular and the stereo settings. Approaches which
use multiple cameras to build up a generalized camera systemhave also been proposed in the literature. We summarize
these papers below.
To the best of our knowledge, the first work in realtime monocular camera SLAM can be attributed to [3].
They proposed a filtering-based algorithm which runs at
30 Hz using a single thread on standard PC hardware. In
contrast to [3], Nister et al. [20] proposed a pure visual
odometry algorithm using an iterative optimization approach.
The proposed algorithm runs at video rates on a single
thread. A system to separate localization and mapping on
two parallel threads was proposed in [15], and became aCoordinate frames: There are three main coordinate frames
used throughout the algorithm derivations. They are a global
world coordinate frame FW , a vehicle-centric body coordinate frame FB , and a camera coordinate frame FCi foreach camera Ci . Frame FW is defined to be fixed relative
to the global earth coordinate frame in which the cameras
navigate. Furthermore, we define the initial vehicle-centric
body coordinate frame FB to coincide with FW , i.e., FB attimestamp 0.

Motionpredictor

Visual direct
imagealignment

Local mapper
RefineSliding window
optimizer
InsertIII. N OTATION
We use lower case letters (e.g. λ) for scalar variables, boldlower case letters (e.g. v) for vectors, and bold capital letters(e.g. T) for matrices. A coordinate frame x is denoted as Fx .
We define TW
Bk as the transformation matrix that transforms
vectors from frame FB to frame FW at timestamp k. We use
pWk to denote the vector variable p represented in frame FWat timestamp k. We use bk to denote the vector variable b
at timestamp k. Furthermore, we denote kxkΣ as a weighted
L2 norm, i.e., xT Σ−1 x.

Tracker

KF request

standard paradigm for subsequent VO algorithms. This separation enables the system to produce high-quality maps byusing more advanced but computational expensive inferencetechniques while simultaneously achieving real-time camera
pose tracking. However, the proposed system is mainly
designed for a small AR workspace. Mur-Artal et al. [19]extend the system to handle large-scale environments byincorporating re-localization and loop closure techniques into
the system. Concurrently, both Engel et al. [5] and Forster
et al. [8] proposed two novel monocular SLAM systems.
They also follow the two-thread paradigm, but with newfeature representations and inference techniques in contrast
to the aforementioned algorithms. In particular, instead ofusing conventional sparse corner features, [5] uses all highgradient pixels directly for pose estimation. The camera poseis estimated by minimizing the photometric intensity errors.
Recently, Engel et al. [7] proposed a new monocular VO system which has the same front-end as [5] but with a different
novel back-end for both pose and structure refinements. In
particular, they jointly optimize camera poses, 3D structuresand other variables by minimizing the photometric intensity
errors directly (aka. direct method).
One of the drawbacks of monocular VO is its inability torecover metric scale. Therefore, to make them more usefulfor robot applications, additional sensor modalities [25] orcameras [17] are typically employed to capture scale information. For most of the above mentioned monocular systems,
there exists a stereo version which allows for metric scale
recovery. For example, Mur-Artal and Tardós [18] extend
ORB-SLAM from [19] to stereo camera configurations. Both[5] and [8] have also been extended for stereo or multicamera configurations [6, 9].
For most robot applications, a wide field of view (FoV) or
even a 360◦ FoV is necessary for better situational awarenessand perception capabilities. For example, Harmat et al. [12],Heng et al. [14] successfully demonstrated a multi-camera
SLAM system for a micro aerial vehicle (MAV).

Local map

KF?

Two-view
stereoYes

Fig. 1: Schematic representation of our VO pipeline.

Vehicle state: We describe the motion state Sk at timestamp
k by its translation and rotation from frame FB to frame
FW . The position is denoted by pWk . The orientation isrepresented by a rotation matrix RW
Bk ∈ SO(3). We obtain thehomogeneous transformation matrix TW
Bk ∈ SE(3) from thevehicle body frame FB to the world frame FW at timestampk: W
RBk pWk
W
TBk =
.
(1)
01×3
1
We denote the image captured at timestamp k by cameraC
Cj as Ik j . We further denote the measured pixel intensity atCC
pixel coordinates u in Ik j as Ik j (u); the pixel intensity is ascalar value for grayscale images. For simplicity, we denoteCj
the image captured at the latest keyframe as IKF.
IV. A LGORITHM
In this section, we describe our VO algorithm for a multicamera system. We follow [7], which describes a VO framework for monocular cameras that uses direct image alignment
based on minimizing a photometric cost function for a sparseset of pixels. We extend the formulation to a multi-camera
system which we model as a generalized camera consistingof cameras with multiple centers of projection. As shownin Fig. 1, our algorithm consists of two threads: a tracker
and a local mapper. The tracker estimates the vehicle posein real-time using direct image alignment with respect tothe latest keyframe. The local mapper is mainly used tominimize long-term pose drift by refining both the vehiclepose and the estimated 3D point cloud. The local mapper
uses a computationally intensive batch optimization method
for both pose and structure refinement. Thus, it runs at amuch lower frame rate compared to the tracker and only
keyframes are processed. For a new keyframe, its newly
sampled 3D points are initialized by a two-view stereo
algorithm. We will describe the algorithm in detail in thefollowing.
A. Hardware Configurations
To make our algorithm work for different hardware configurations, we assume we have N cameras in total. Eachcamera can be configured either as a reference camera formotion tracking or as an auxiliary camera for static stereo
matching. For ease of reference, we denote using camera Ci

as a reference camera as r Ci . Without loss of generality, we
assume that we have Nr reference cameras and Na auxiliary
cameras where Nr + Na = N .
B. Tracker
As shown in Fig. 1, the tracker consists of two parts: amotion predictor and direct image alignment. Direct imagealignment is used to estimate the current vehicle pose withrespect to the latest keyframe. We use the vehicle pose
provided by the motion predictor to initialize the alignment
in order to avoid local minima.
Constant velocity motion prediction model: We use aconstant motion model to predict the current vehicle pose.
Let us define the current vehicle pose as TW
Bk which is thetransformation from the vehicle frame FB at timestamp kto the world frame FW . Similarly, we define the vehicle
poses corresponding to the two latest frames as TW
Bk−1 andTW
respectively.
Weassumethatthemotionvelocityfrom
Bk−2
timestamp k − 2 to k is constant, we thus can predict thecurrent vehicle pose as
B

B

(2)

−1 W
TBk−2
= (TW
TBk−1 .
Bk−2 )
k−1

(3)

k−1
k−2
W
TW
= TW
Bk = TBk−1 TBk
Bk−1 TBk−1 ,

where

B

Direct sparse tracker: Given the initial pose estimate
provided by the motion predictor, we use a direct sparsetracker for refinement. In contrast to the local mapper which
will be explained in a later section, only images fromreference cameras are used for motion tracking. In particular,kwe estimate the relative vehicle pose T̂B
BKF between thelatest keyframe and the current frame by minimizing thefollowing energy function:k
T̂B
BKF = argmin
B

TB k

KF

Nr
X

X

r

r

Ci
(u) − IkCi (û))2 .
(IKF

i=1 u∈Ω(Ir Ci )
KF

(4)
r
Ci
)
Here, Nr is the number of reference cameras and Ω(IKF
is the set of feature pixel points sampled from the keyframe
image of reference camera r Ci . For each feature point u in
the keyframe image, û is the corresponding pixel position in
the current frame, obtained as
Bk
Ci −1
i
û = π(TC· π −1 (u, d)) .
B · TBKF · (TB )

(5)

i
TC
B is the relative transformation from vehicle body frame to−1

camera frame. π and π are the camera projection function
and back projection function, respectively. d is the depthof the feature point u in the keyframe. The only unknownCikvariable is TB
BKF since TB can be obtained from offline
sensor calibration and d can be initialized by stereo matching
and refined by the joint optimization performed by the local
mapper.
The above formulation follows the standard forward compositional method [2] which requires the Hessians and Jacobians of the residuals to be computed in every iteration of the
optimization. For improved efficiency, we adopt the inverse

compositional method [2] to minimize the energy function.
To robustify the least squares estimator which is sensitive tooutliers, we further apply a robust loss function (Huber loss)
to all residuals.
Outlier removal: Besides the use of a robust loss function, a
specific outlier removal mechanism is also used to robustify
both the tracker and the local mapper. We detect outliers
based on the template matching scores between the reference
frame and the current frame. In our implementation, we use
the Zero Mean Normalized Cross-Correlation (ZNCC) score[24]. If the score is smaller than a predefined threshold, we
classify a feature match as an oulier and remove it fromthe optimization. The tracker already uses image patches
around the feature points and the warped image patches
are already computed during direct image alignment. Thus,
the computational overhead of the outlier removal step ismarginal.
Relationship with the tracker from [7]: In contrast to thetracker in the Direct Sparse Odometry (DSO) algorithm [7],we do direct sparse pose tracking. To do pose tracking, DSO
projects sampled feature points from all recent keyframes tothe current keyframe and then do semi-dense pose tracking.
Compared to semi-dense pose tracking, the computational
complexity of direct tracking with sparse feature points is
far smaller than that of semi-dense tracking. Furthermore,
an outlier removal step as described in the previous section
is included to make the tracking and mapping more stable.
C. Keyframe and Feature Selections
Keyframe selection: One of the implicit assumptions
behind motion tracking is that the scene difference between
the reference keyframe and the current frame is sufficiently
small. If the difference between the current frame and thereference keyframe becomes too large, we instantiate a new
keyframe. Intuitively, the difference between frames shouldbe measured based on changes in image content rather thanbased on absolute pose differences (since the former strongly
depends on the depth of the scene while the latter is oblivious
to it). Therefore, we use the mean square optical flow as oneof the criterions for detecting scene changes. More precisely,
we computen1X
kui − ûi k2 ,
(6)
f=
n i=1
where k·k2 is the Euclidean norm, ui is a feature point in thereference keyframe, and ûi is its corresponding feature pointin the current frame. If f is above a threshold, we create a
new keyframe.
Feature selection: Once a new keyframe is created, sparsefeature points are sampled from all reference cameras. We
sample N sparse features uniformly from each image basedon their gradient magnitudes. In particular, we divide theimage into a grid and select the point with the highest
gradient magnitude per grid cell. However, we will notsample a point from this grid cell if the highest gradientmagnitude of a grid is smaller than a pre-defined threshold.
Feature representation and depth initialization: As per
the implementation in [7], we sample all pixels by following
a predefined patch pattern (e.g., 5×5 pattern) for each sparsefeature. All these pixels are used for motion tracking andlocal mapping. In particular, the created visual residuals fromthe tracker and local mapper are per pixel instead of per
patch. Furthermore, pixels from the same patch are assumed
to lie on the same 3D plane, which can be parameterized
by its inverse plane depth and plane normal. The inverse
plane depths and plane normals are initialized by a stereo
matching algorithm [11]. The plane depths of the sampled
patch instead of ray depths of each pixel are refined during
the joint optimization carried out by the local mapper. Eachpixel from the same patch has different ray depths, but has
the same plane depth. It thus reduces the number of variables
to optimize and increases the computational efficiency of the
optimizer.
D. Local Mapper
Two-view stereo matching: Stereo matching is used toinitialize the depth of each sampled feature from the new
keyframe. Rather than performing disparity search along
epipolar lines, we use plane-sweeping stereo [11] to computethe depths. This allows us to directly operate on the fisheye
images, and thus, avoid a loss of field-of-view from having
to undistort and rectify the images.
Based on the implementation from [11], we use the GPU
to accelerate plane-sweeping stereo. We sweep planes in
two directions: fronto-parallel to the viewing direction of
the keyframe and parallel to the ground plane. For eachdirection, we generate 64 plane hypotheses with a constantdisparity step size between them. The planes cover the range[0.5 30] m in front of the camera. This results in a total number of 128 plane hypotheses that are evaluated. We compute
the ZNCC score between the 7×7 image patches from ICi
and warped image patches from ICj . The plane hypothesis
with the largest template matching score is selected for eachfeature point.
Sliding window optimizer: To minimize drift, the local
mapper jointly optimizes all vehicle states and the 3D
scene geometry. As the vehicle moves, the number of states
increases over time. To bound running time, we use statemarginalization to remove old states. We only keep a fixed
number of previous states, resulting in a constant computational complexity [4]. As shown in [23], a marginalizedstate will result in all remaining states connected to it tobe connected with each other after marginalization. The
resulting Hessian matrix is not sparse anymore. This inturn increases the run-time cost of computing the Schurcomplement during structure optimization. For efficiency, we
follow [16, 7] and only fill terms of the Hessian that do not
involve geometry terms. For further efficiency, optimizationis only performed on selected keyframes.

In particular, we define a sliding window containing k vehicle states. All states outside this sliding window are treatedas old states and are removed through partial marginalization[7].
We use S∗ to represent the set of all vehicle states within
the sliding window and they are represented in vector form.
The local mapper estimates the vehicle states S∗ inside thesliding window, and the set D∗ of inverse plane depths of
all the sampled features within the sliding window via
Ŝ∗ , D̂∗ = argmin E0 (S∗ ) + Evision (S∗ , D∗ ).

(7)S∗ ,D∗

Here, E0 (S∗ ) is either a prior energy term obtained frompartial marginalization or an initial prior and Evision (S∗ , D∗ )is the visual energy term between all keyframes within
the sliding window. Optimization is carried out using theGauss-Newton algorithm. In the following, we describe theindividual terms in more detail.
There are two types of prior terms. One is from the initialprior. In order to fix the unobservable degrees of freedom of
the VO problem, we need to fix the initial state such that all
subsequent states are estimated relative to it. Thus, they areusually formulated in the following form:
1,(8)Ŝ0 − S0
E0 (S0 ) =2
Σ0where Ŝ0 is the value of the initial state, S0 is the inital
state variable to estimate and Σ0 is the covariance matrixof the inital state. To fix S0 as equal to Ŝ0 , Σ0 is usually
selected to have infinitesimal diagonal terms. This prior term
only appears in the energy function when the initial state iswithin the sliding window. Once it leaves the window, it
will be marginalized out in the same way as all other energy
terms [23, 4].
The other prior term is a direct result of partial marginalization. Partial marginalization introduces priors to all remaining states which are connected to the marginalized states[23, 4]. The information from eliminated states after theirremoval is stored in prior Hessian and Jacobian matrices suchthat we can optimally remove them. Thus, we can have thefollowing prior energy term for the remaining states
1
E0 (S∗ ) = JTm (Ŝ∗0 − S∗ ) +Ŝ∗0 − S∗
,
(9)
2
Hm −1
where Ŝ∗0 is a set of prior vehicle states of S∗ estimatedfrom previous iterations, Jm and Hm are the Jacobian andHessian matrices respectively, and accumulated from statemarginalization.
The visual energy term contains the sum of all photometric error residuals and is modeled as
Evision (S∗ , D∗ )

=

Nr
k X
X

X

X

(10)

m=1 i=1 u∈Ω(Ir Ci ) n∈Θ(u)mX

kr(Sm , r Ci , Sn , Cj , ρ)kΣ .

j∈Φ(Sm ,u)Here, k again is the number of keyframes and Nr is thenumber of reference cameras. r Ci and Cj refer to the ith

r

and j th camera respectively. Ω(ImCi ) is the set of featurepoints sampled from the image of the ith reference camerain the mth keyframe. Θ(u) is the set of indices of thevehicle states that observe the feature point u. Φ(Sm , u)is the set of indices of the cameras in the mth keyframe thatobserve feature point u. ρ is the inverse plane depth of thefeature u. r(Sm , r Ci , Sn , Cj , ρ) is a pixel intensity residualthat measures intensity differences between two projectionsof the same 3D scene point (see below for a definition of theresidual). Furthermore, we use both inter-camera and intracamera irradiance residuals in our implementation. Finally, Σ
is the scalar variance of the photometric error residual which
is obtained from each camera’s photometric calibration.
r
Let ImCi be the image captured by the ith camera in themth keyframe. Consider a feature point u sampled from thatimage. Using plane sweep stereo, we know the inverse depth
ρ of its corresponding plane as well as the normal n of thatplane. The 3D point Pu corresponding to u is thus givenby:1
πr−1 (u) ,
(11)
Pu =
ρ cos θ Ci
where πr−1
Ci is the inverse projection function of the camera
and cos θ = −nT · πr−1
Ci (u) is the angle between the viewing
ray corresponding to u and the plane normal. The vehicleW
poses TW
Bm and TBn at timestamps m and n are initiallyestimated by the tracker while the relative transformationslTC
B between the vehicle frame FB and the camera frames
FCl can be estimated offline. We use these transformations
to compute the pixel û in the j th camera and in the nth
keyframe corresponding to the feature u by projecting Pu
into the image. Given u and û, we use their intensity
difference to define the residual r(Sm , r Ci , Sn , Cj , ρ):rjr(Sm , r Ci , Sn , Cj , ρ) = ImCi (p) − ICn (p̂) .

(12)

In order to minimize the visual energy term from Eq. 10,
we adjust the motion state parameters (namely the poseTWBm ) as well as the inverse plane depth of each feature.
Rather than looking at a single pixel per feature u, we
consider a pixel patch per feature. Each pixel in the patchis warped into the other images and contributes a residual
according to Eq. 12. In this setting, parameterizing the depthof u based on the plane has the advantage that we only
require a single parameter per patch.
V. E XPERIMENTAL E VALUATIONS
A. Hardware Setup
We have installed twelve fisheye cameras on our vehicle
platform which is shown in Fig. 2. In particular, five cameras
are installed at the front of the vehicle, two cameras are
installed on each of the left and right sides, and three cameras
are installed at the back. We select one stereo pair from eachside to evaluate our algorithm. In particular, the front stereo
pair has a baseline of 0.722m, the back stereo pair has abaseline of 0.755m, the left stereo pair has a baseline of
0.502m, and the right stereo pair has a baseline of 0.497m.
All cameras output 1024×544 gray scale images at 25 frames

per second, and are hardware-time-synchronized with the
GPS/INS system.
We calibrate the multi-camera system using a grid ofAprilTag markers. To compute the transformation between
the multi-camera system and the GPS/INS system, we runsemi-direct VO [13] for each stereo pair, obtain an initial estimate using hand-eye calibration, triangulate landmark points
using the GPS/INS poses and feature correspondences fromVO, and refine the initial estimate together with the landmark
points by minimising the sum of squared reprojection errors
while keeping the GPS/INS poses fixed. This transformation
allows direct comparison of visual odometry pose estimateswith the post-processed GPS/INS pose estimates which areused as ground truth.
B. Experiments and Discussions
Dataset selection: To avoid parameter overfitting, and, atthe same time, evaluate our algorithm thoroughly, we selectfive datasets with different characteristics for evaluation. The
first three datasets are collected in a car-park. The first one(Science-park-day) is collected in normal daylight conditions. The second one (Science-park-night-illum) is collected
at night with NIR illumination. The third one (Science-parknight) is collected at night without NIR illumination. The
other two datasets are collected from a public urban street.
One of them (West-coast-day) is collected in day light conditions. Another one (West-coast-night-no-illum) is collected atnight without near-infrared illumination. Four sample images
are shown in Fig. 3. Furthermore, the characteristics of thedatasets are summarized in Table I.
Science-park-day
Science-park-night-illum
Science-park-nightWest-coast-day
West-coast-night-no-illum

Length (m)547.448612.078
613.096
1179.131224.95

Max. speed (m/s)
3.941
3.863
3.685
10.68
10.23

TABLE I: Characteristics of datasets used for evaluation.
Evaluation metrics: We follow the metric used by the
KITTI benchmark [10] for accuracy evaluation. In particular, we compute the position and orientation drifts oversequences of length 200m, 400m, 600m and 800m for eachframe, and average drifts over all sequences. We take into
account the runtime of both the tracker and the mapper asa metric for evaluating efficiency. Furthermore, we observethat the ground truth heights for all science park datasets
are not reliable even though a highly accurate GPS/IMU
system was used. For example, for the Science-park-day
dataset, the closed-loop ground truth height drift is morethan 2.5 m. Since these errors introduce significant bias
into the evaluation, we only consider horizontal translationsof all three science park datasets for accuracy evaluations.
The remaining datasets are evaluated with 3-axis translation
errors.
Parameter settings: For all experiments, we sample a total

Fig. 2: Two sample images of the vehicle.

(a) Science-park-day(b) Science-park-night-illum(c) West-coast-night-no-illum

Fig. 3: Three sample images from our datasets, demonstrating the challenges of the datasets which include strong distortion
and absence of features at night.

of 800 5 × 5 feature patches. All features are uniformly
distributed in each camera. For example, if we consider four
stereo pairs, then each stereo pair has 200 features. A sliding
window with 5 keyframes is used for the local mapper. The
optical flow threshold for keyframe selection is 20 pixels.
For the Huber parameter, we use 30 for daylight conditions
and 10 for night-time conditions.
Baseline algorithm: We choose ORB-SLAM2 [18] as
the baseline algorithm for our comparisons. Since ORBSLAM2 does not support the fisheye camera models, we
undistort the fisheye images to generate pinhole images forall experiments.
Our algorithm is evaluated on a laptop with an Intel i7
CPU @ 2.8 GHz. We are able to run our tracking thread at
more than 30 Hz and the local mapping thread at around 2
Hz for the current configurations.
Ablation studies: We conduct thorough ablation studies
with respect to the hardware configurations. The results canbe found in Table II. It shows that a higher number ofstereo cameras improves both the accuracy and robustnessof our VO algorithm. In particular, both ORB-SLAM2 [18]and our single stereo configuration fail easily for all thenight sequences when using only a single or two stereo
pairs. Based on our observations, the main reason of the
failure is due to the lack of good features. However, using

a multi-camera setup which covers a larger field of view
can provide the required redundancy necessary for handlingpoorly textured night environments.
From Table II, we also observe that our algorithm performs better than ORB-SLAM2 [18]. We give two possible
reasons: (1) the direct method can do refinement with subpixel accuracy which improves the accuracy as long as thealgorithm is well-implemented, and (2) the image quality
degrades after undistortion and stereo rectification, and canaffect ORB-SLAM’s performance. In contrast, our method
is able to directly operate on the raw fisheye input images.
By doing horizontal comparisons (i.e. same dataset) of
the same algorithm for different hardware configurations,
we can observe that by using more cameras improves theaccuracy of our VO algorithm. We think that spatiallywell distributed features can give better constraints to the
optimization problem, which makes our VO algorithm moreaccurate.
Furthermore, by doing vertical comparisons (i.e. same
hardware configuration) of the same algorithm for different
light conditions, we can observe that night-time conditions
degrade the performance of VO algorithms. Besides thelack of good features, another reason is that the image
quality in night-time conditions is worse than that in daylight
conditions. When the vehicle moves fast at night, motion bluris inherent in the captured images, especially for the left andright stereo pairs.
Qualitative evaluations: Fig. 4 shows qualitative evaluation
results of our algorithm for all datasets with four pairsof stereo cameras. We plot the x-y trajectories estimatedby our VO algorithm against the ground truth trajectories.
A supplementary video can also be found on our project
website https://cvg.ethz.ch/research/visual-odometry/.
VI. C ONCLUSION
We present a direct sparse visual odometry algorithm fora multi-camera system and robust operation in challenging
environments. Our algorithm includes a direct sparse posetracker and a local mapper. The tracker tracks the current
camera pose in real-time. The local mapper jointly optimizes
both poses and structure within a sliding window. Insteadof minimizing re-projection errors, both the tracker andmapper directly minimize photometric errors. We evaluate
our algorithm extensively with five datasets which havedifferent characteristics. Experimental results show that a
multi-camera setup makes the VO more robust to challenging
night environments and also improves its accuracy.
R EFERENCES[1] H. Alismail, B. Browning, and S. Lucey. Robust
tracking in low light and sudden illumination changes.
In 3DV, 2016.
[2] S. Baker and I. Matthews. Lucas-Kanade 20 years
on: A Unifying framework. International Journal ofComputer Vision (IJCV), 56(3):221–255, 2004.
[3] A. J. Davison. Real-time simultaneous localisation andmapping with a single camera. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2003.
[4] T.-C. Dong-Si and A. I. Mourikis. Motion tracking with
fixed-lag smoothing: algorithm and consitency analysis.
In Proc. IEEE International Conf. on Robotics andAutomation (ICRA), 2011.
[5] J. Engel, T. Schöps, and D. Cremers. LSD-SLAM:
Large-scale direct monocular slam. In Proc. of theEuropean Conf. on Computer Vision (ECCV), 2014.
[6] J. Engel, J. Stückler, and D. Cremers. Large-scale direct
slam with stereo cameras. In International Conferenceon Intelligent Robots and Systems (IROS), 2015.
[7] J. Engel, V. Koltun, and D. Cremers. Direct sparse
odometry. In arXiv, 2016.
[8] C. Forster, M. Pizzoli, and D. Scaramuzza. SVO:
Fast semi-direct monocular visual odometry. In Proc.
IEEE International Conf. on Robotics and Automation(ICRA), 2014.
[9] C. Forster, Z. Zhang, M. Gassner, M. Werlberger, and
D. Scaramuzza. Svo: Semi-direct visual odometry formonocular and multi-camera systems. IEEE Transactions on Robotics, 33(2), 2017.
[10] A. Geiger, P. Lenz, and R. Urtasun. Are we readyfor autonomous driving? the kitti vision benchmark
suite. In Conference on Computer Vision and Pattern
Recognition (CVPR), 2012.

[11] C. Hane, L. Heng, G. H. Lee, A. Sizov, and M. Pollefeys. Real-time direct dense matching on fisheye
images using plane-sweep stereo. In International
Conference on 3D Vision (3DV), 2015.
[12] A. Harmat, M. Trentini, and I. Sharf. Multi-Camera
Tracking and Mapping for Unmanned Aerial Vehicles
in Unstructured Environments. Journal of Intelligent
and Robotic Systems, 2014.
[13] L. Heng and B. Choi. Semi-direct visual odometry
for a fisheye-stereo camera. In IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS),
2016.
[14] L. Heng, G. H. Lee, and M. Pollefeys. Self-Calibration
and Visual SLAM with a Multi-Camera System on a
Micro Aerial Vehicle. In Proc. Robotics: Science andSystems (RSS), 2014.
[15] G. Klein and D. Murray. Parallel tracking and mapping
for small ar workspaces. In International Symposiumon Mixed and Augmented Reality, 2007.
[16] S. Leutenegger, S. Lynen, M. Bosse, R. Siegwart, andP. T. Furgale. Keyframe-based visual–inertial odometry
using nonlinear optimization. International Journal of
Robotics Research (IJRR), 2015.
[17] P. Liu, L. Heng, T. Sattler, A. Geiger, and M. Pollefeys.
Direct visual odometry for a fisheye-stereo camera. In
Proc. IEEE International Conf. on Intelligent Robots
and Systems (IROS), 2017.
[18] R. Mur-Artal and J. D. Tardós. ORB-SLAM2: an opensource SLAM system for monocular, stereo and RGB-D
cameras. arXiv preprint arXiv:1610.06475, 2016.
[19] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós.
ORB-SLAM: A versatile and accurate monocular slamsystem. IEEE Transactions on Robotics, 31(5), 2015.
[20] D. Nister, O. Naroditsky, and J. Bergen. Visual Odometry. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2004.
[21] S. Park, T. Schöps, and M. Pollefeys. Illumination
change robustness in direct visual slam. In ICRA, 2017.
[22] G. Pascoe, W. Maddern, M. Tanner, P. Pinies, and
P. Newman. NID-SLAM: Robust monocular SLAM
using normalized information distance. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition(CVPR), 2017.
[23] G. Sibley, L. Matthies, and G. Sukhatme. Sliding
Window Filter with Application to Planetary Landing.
Journal of Field Robotics (JFR), 2010.
[24] L. D. Stefano, S. Mattoccia, and F. Tombari. ZNCCbased template matching using bounded partial correlation. Pattern Recognition Letters, 2005.
[25] J. Zhang and S. Singh. Visual-lidar odometry and mapping: Low-drift, robust, and fast. In IEEE International
Conference on Robotics and Automation (ICRA), 2015.
[26] Z. Zhang, C.Forster, and D. Scaramuzza. Active Exposure Control for Robust Visual Odometry in HDR
Environments. In Proc. IEEE International Conf. on
Robotics and Automation (ICRA), 2017.

Datasets
SD
SN1
SN2
WCD
WCN

Alg*
Ours
ORB*
Ours
ORB*
Ours
ORB*
Ours
ORB*
Ours
ORB*

F
0.449
1.168
x
3.24
x
x
2.33
2.34
x
x

B
1.266
1.565
3.7
x
x
x
1.32
1.86
x
x

L
1.185
x
x
x
x
x
2.17
x
x
x

R
1.194
4.203
x
x
x
x
4.35
x
x
x

FB
0.56
N.A.
x
N.A.
x
N.A.
0.48
N.A.
2.88
N.A.

FL
0.282
N.A.
2.05
N.A.
x
N.A.
1.79
N.A.
11.6
N.A.

FR
0.632
N.A.
x
N.A.
x
N.A.
2.02
N.A.
8.57
N.A.

BR
0.86
N.A.
1.01
N.A.
x
N.A.
3.47
N.A.
5.12
N.A.

BL
0.64
N.A.
1.544
N.A.
x
N.A.
0.96
N.A.
4.99
N.A.

LR
1.24
N.A.
2.265
N.A.
1.01
N.A.
1.53
N.A.
2.84
N.A.

BLR
0.511
N.A.
0.625
N.A.
1.47
N.A.
1.59
N.A.
3
N.A.

FLR
0.635
N.A.
0.735
N.A.
0.815
N.A.
1.97
N.A.
2.23
N.A.

FBL
0.838
N.A.
0.956
N.A.
1.544
N.A.
1.47
N.A.
2.33
N.A.

FBR
0.524
N.A.
1.193
N.A.
1.51
N.A.
0.87
N.A.
2.05
N.A.

FBLR
0.352
N.A.
0.691
N.A.
1.03
N.A.
0.9
N.A.
1.74
N.A.

TABLE II: Accuracy evaluations with offline datasets for different hardware configurations (Accuracy (translational drift) isin units of %, the smaller the better; SD: Science-park-day. SN1: Science-park-night-illum. SN2: Science-park-night. WCD:
West-coast-day. WCN: West-coast-night-no-illum. Alg*: Algorithms. ORB*: ORB-SLAM2. F: front stereo pair. B: backstereo pair. L: left stereo pair. R: right stereo pair. x: fails to complete whole sequence. N.A.: not available.)Fig. 4: Trajectories for five selected datasets (Top left: West-coast-day dataset. Top right: West-coast-night-illum dataset.
Bottom left: Science-park-day dataset. Bottom center: Science-park-night-illum dataset. Bottom right: Science-park-night
dataset).