VITAMIN-E: VIsual Tracking And MappINg
with Extremely Dense Feature Points

arXiv:1904.10324v1 [cs.CV] 23 Apr 2019

Masashi Yokozuka, Shuji Oishi, Thompson Simon, Atsuhiko Banno
Robot Innovation Research Center,
National Institute of Advanced Industrial Science and Technology (AIST), Japan{yokotsuka-masashi, shuji.oishi, simon.thompson, atsuhiko.banno}@aist.go.jp

Abstract
In this paper, we propose a novel indirect monocular
SLAM algorithm called “VITAMIN-E,” which is highly accurate and robust as a result of tracking extremely densefeature points. Typical indirect methods have difficulty inreconstructing dense geometry because of their careful feature point selection for accurate matching. Unlike conventional methods, the proposed method processes an enormous number of feature points by tracking the local extrema of curvature informed by dominant flow estimation.
Because this may lead to high computational cost during
bundle adjustment, we propose a novel optimization technique, the ”subspace Gauss–Newton method”, that significantly improves the computational efficiency of bundle adjustment by partially updating the variables. We concurrently generate meshes from the reconstructed points andmerge them for an entire 3D model. The experimental
results on the SLAM benchmark dataset EuRoC demonstrated that the proposed method outperformed state-of-theart SLAM methods, such as DSO, ORB-SLAM, and LSDSLAM, both in terms of accuracy and robustness in trajectory estimation. The proposed method simultaneously generated significantly detailed 3D geometry from the densefeature points in real time using only a CPU.

Figure 1. Dense geometry reconstruction with VITAMIN-E on
EuRoC V101. (https://youtu.be/yfKccCmmMsM)

mance, these methods are based on pure monocular SLAM;
hence, improving monocular SLAM is important.
Monocular SLAM methods can be classified into two
types: direct methods and indirect methods.
Direct methods: Direct methods estimate camera poses
and reconstruct the scene by minimizing the photometricerror defined as a sum of the intensity difference between
each pixel in the latest image and the reprojection of thecolor / monochrome 3D map. Direct methods, such as LSDSLAM[5], SVO[9], and DSO[6], process almost all pixels
in incoming images. They do not require exact pixel correspondences among multiple views unlike indirect methods,
which leads to denser map reconstruction. However, direct
methods are susceptible to image noise, luminance fluctuation, and lens aberration because they directly use pixel
intensities. To overcome this drawback, Bergmann et al.[1]proposed a normalization method against luminance fluctuation and a calibration method for lens aberration. As another approach, Zhang et al.[35] proposed an auto-exposure
method that is suitable for direct methods.
Indirect methods: Indirect methods minimize the geometric error between observed 2D feature points and re-

1. Introduction
Simultaneous localization and mapping (SLAM) is a key
technology for applications such as autonomous systems
and augmented reality. Whereas LiDAR-based SLAM[34,11, 25] is well established and widely used in autonomousvehicles, visual SLAM with a monocular camera does notprovide sufficient accuracy and robustness, particularly regarding dense map reconstruction, to replace LiDAR-based
SLAM. Although some visual SLAM algorithms that use
stereo cameras[7, 32], RGB-D cameras[12, 14, 33], and inertial sensors[20, 18, 2, 28, 19] have achieved high perfor1

Input image(a) Feature tracking(b) Localization and Mapping.

(c) Meshing & Denoising(d) Reconstruction by Mesh Merging*Colored by normal.

Figure 2. Overview of the proposed monocular SLAM.

projections of the corresponding 3D points. As a result of the use of feature descriptors, indirect methods
such as PTAM[15] and ORB-SLAM[21] are robust againstbrightness changes and image noise. Additionally, indirect
methods explicitly establish feature point correspondences;hence, outliers are easily removed using RANSAC[8] or
M-estimation[27]. This characteristic, however, can be a
drawback: Indirect methods carefully select stable featurepoints, thus the reconstructed 3D map tends to be sparse anddoes not provide detailed geometry. Densification methods,
such as PMVS[10] and the extension L-PMVS[26], might
be useful for obtaining dense geometry; however, they are
offline methods and not applicable in real time.
In this paper, we propose the novel VIsual Tracking And MappINg with Extremely dense feature points,“VITAMIN-E,” which is highly precise, robust, and dense
because of the tracking of a large number of feature points.
Indirect methods are inherently robust against noise, illumination change, and outliers as a result of the use of featuredescriptors. Retaining this advantage, we reconstruct detailed 3D maps by establishing dense point correspondence.
The contributions of this study are as follows: We first introduce a new dense feature point tracking algorithm basedon dominant flow estimation and curvature extrema tracing.
This allows VITAMIN-E to process an enormous number offeature points; however, the need to maintain them simultaneously might lead to a high computational cost. Therefore, we also introduce a novel optimization technique,called subspace Gauss–Newton method, for bundle adjustment. The optimization technique significantly improves

the efficiency of bundle adjustment by partially updating thevariables. Moreover, VITAMIN-E generates meshes fromthe reconstructed feature points and integrates them using
a truncated signed distance function (TSDF)[22, 30, 24].
Compared with not only conventional indirect methods but
also state-of-the-art direct methods, VITAMIN-E provides
highly detailed 3D geometry as shown in Figure 1 in realtime using only a CPU

2. Dense Feature Point Tracking
2.1. Feature Point Tracking
Indirect methods that use image descriptors can be unstable because of incorrect feature point correspondences.
They build feature point correspondences between multiple views by matching the descriptors. Extracting consistent descriptors over multiple frames, however, becomesdifficult because descriptors vary as the monocular camerachanges its pose. Methods such as the KLT tracker[29] that
continuously track feature points while updating the featuredescriptors might be useful for overcoming the problem.
However, because the tracked positions drift as a result of aminute change of feature descriptors, the correspondences
over multiple views tend to be incorrect. These problems
originate with the use of feature descriptors.
Rather than associating feature points based on descriptors, VITAMIN-E tracks the local extrema of curvature inincoming images. In the proposed method, feature points
denote the extrema of curvature on image intensities. Letf (x, y) be an image, then curvature κ of image f is as fol-

2.3. Curvature Extrema Trackinglows:κ = fy2 fxx − 2fx fy fxy + fx2 fyy ,(1)where fx represents the partial derivative of f with respect
to x, which can be obtained using a Sobel operator or similar technique. VITAMIN-E builds point correspondences
over multiple images by tracking the local maximum pointof curvature κ(x, y, t), which is the extension of κ to time
domain t. Figure 2(a) shows an example scene from whicha large number of extrema of curvature κ are extracted.
Whereas conventional indirect methods rely only on feature points with a large curvature to obtain stable correspondences, the proposed method tracks all detected extrema toreconstruct detailed geometry.

2.2. Dominant Flow Estimation
After detecting the extrema of curvature, the proposed
method estimates a dominant flow that represents the average of optical flow over the images, which provides agood initial value to extrema tracking and makes it significantly stable, as explained later. Specifically, we determine the corresponding feature pairs between current andprevious images using the BRIEF[4] feature. Because weonly have to identify coarse feature pairs over consecutive frames at this moment, feature matching is performedon low-resolution images, subsampled to 1/6 of the former
size.
Then, we fit the affine transformation model y = Ax+b
to the feature pairs. x and y denote the position of a feature point in the previous and current frame, respectively,
and A and b represent a matrix of 2 × 2 and a 2D translation, respectively. A and b are obtained by minimizing costfunction E using the Gauss–Newton method:E=NX

ρ (kyi − (Axi + b) k2 ) ,(2)i

VITAMIN-E tracks feature points by tracing the extremaof image curvature by making use of the dominant flow. Because it depends only on extrema instead of feature descriptors used in conventional indirect methods, VITAMIN-E isfree from the variation of feature descriptors caused by image noise or illumination changes, which makes VITAMINE highly robust.
According to the dominant flow represented by A and b,
we first predict a current position x̄t1 of tracking point xt0 :x̄t1 = Axt0 + b.

(4)

Next, prediction x̄t1 is corrected to xt1 by maximizing evaluation function F :F = κ (xt1 , t1 ) + λw kxt1 − x̄t1 k2 ,(5)where κ stores the curvature in each pixel, and w(x) =1 − ρ(x) and λ denote an evaluation function and weight
for the prediction, respectively. The maximization is performed using the hill climbing method, with x̄t1 as the initial position. Specifically, maximum point xt1 is obtainedby iterating the hill climbing method in eight neighboring
pixels at each step until it reaches the local maximum valueof F . Function w prevents the maximization process fromfalling into wrong extrema, thereby playing a regularizationrole.
Note that extrema tracking can easily fall into local solutions because there are many extrema in image curvatureand it is almost impossible to distinguish them without any
descriptors. However, the prediction according to the dominant flow boosts the accuracy of extrema tracking and enables it to approach the optimal solution.

3. Bundle Adjustment for Dense Tracking
3.1. Bundle Adjustment

where N and ρ denote the total number of corresponding
points and a kernel function for M-estimation, respectively.
The following Geman–McClure kernel with scale parameter σ is used in VITAMIN-E :ρ(x) =x2.
x2 + σ 2(3)

As a result of M-estimation, the dominant flow represented by A and b can be estimated stably, even for lowresolution images, and it allows us to roughly predict theposition of feature points in the next frame. Note that
VITAMIN-E does not rely on conventional feature matching in its core but only for prior information for dense extrema tracking, as described in the next section. Whereas
conventional feature matching has difficulty in making all
feature points couple correctly between consecutive frames,
affine transformation is easily obtained when at least threecorrespondences are given.

Bundle adjustment iteratively adjusts the reconstructed
map by minimizing reprojection errors. Given the i-th 3D
point pi , the j-th camera’s rotation Rj and translation tj ,and the 2D position uij of pi observed in the j-th camera
frame, the objective function is formulated as follows:
E=

N X
M
X
i

 
ρ kuij − φ RjT (pi − tj ) k2 ,(6)jwhere N and M are the numbers of feature points andcamera poses respectively, φ denotes the 3D-2D projection function, and ρ is a kernel function for M-estimation.
Specifically, optimal camera variables cj = (Rj , tj ) andpi are obtained by applying the Gauss–Newton method toEquation 6, which results in iteratively solving the following equations:Hδx = −g,x = x + δx,(7)where x = (c1 , · · · , cM , p1 , · · · , pN )T , and H and g represent the Hessian matrix and gradient around x of E, respectively. H and g can be represented by the camera variable cj block and the feature point variable pi block as follows: 
Hcc Hcp
gc
,
g
=
H=
.
(8)
T
Hcp
Hpp
gp
Hessian matrix H in bundle adjustment has a unique structure: Hcc and Hpp are sparse matrices with only diagonal
elements in block units, whereas Hcp is a dense matrix. Efficient solutions that focus on this unique structure are thekeys to developing highly precise and robust visual SLAM.
State-of-the-art monocular SLAM methods, such as
ORB-SLAM[21] and DSO[6], solve Equation 7 by decomposing it using the Schur complement matrix instead of directly solving it:−1−1
Hcc − Hcp Hpp
Hcp δxc = −gc + Hcp Hpp
gp , (9)TT
Hpp δxp = −gp − Hcp
δxc ,

(10)

where xc = (c1 , · · · , cM )T and xp = (p1 , · · · , pN )T .
The decomposition allows us to solve bundle adjustment
faster. The number of camera variables M is remarkably
smaller than that of feature point variables, and the inverse
matrix of Hpp can be easily calculated because it has only
diagonal components; thus, the Schur complement matrix−1 T
Hcp whose size is M ×M is significantly
Hcc − Hcp Hpptiny compared with the original H, and the inverse matrix
is rapidly computable.
Equation 9 is also called marginalization. When regard−1
−1 T
gp as
Hcp as a new H and gc − Hcp Hpping Hcc − Hcp Hppa new g, the decomposition is equivalent to eliminating all
feature point variables p from cost function E. State-of-theart SLAMs make themselves efficient using the marginalization technique to prevent the increase in computational
cost caused by a large number of variables.
However, in the case of maintaining thousands of feature points in every frame, as in the dense extrema tracking
in VITAMIN-E, the size of matrix H fundamentally cannot be made sufficiently small because variable elimination
is applicable only to old variables unrelated to the current
frame for stability. Moreover, the size of the Schur complement matrix is proportional to the number of feature points;
thus, the calculation cost of bundle adjustment over tens of
thousands points, where the size of H is 100,000 × 100,000
or more, becomes too high to run bundle adjustment in realtime.

3.2. Subspace Gauss–Newton Method
To deal with the explosion in the size of H, we propose
a novel optimization technique called the “subspace Gauss–
Newton method.” It partially updates variables rather than

updating all of them at once, as in Equations 9 and 10, by
decomposing these equations further as follows:i−1X
Hci ci δxci = − gci +Hcl ci δxcl +l=1
MX

Hci cr δxcr +

r=i+1

N
X


Hci pj δxpj , (11)

j=1


j−1
X
Hpj pj δxpj = − gpj +Hpl pj δxpl +l=1
NX

Hpi pr δxpr +

r=j+1

M
X

HcTi pj δxci. (12)

i=1

Equation 11 updates δxci of a camera variable, and Equation 12 δxpj of a feature point variable. Hci ci , Hci pj ,and Hpj pj are matrices of 6 × 6, 6 × 3, and 3 × 3, respectively. The subspace Gauss–Newton method iteratively
solves Equations 11 and 12 until δxci and δxpj converge,respectively.
These formulae are extensions of the Gauss–Seidel
method, which is an iterative method to solve a linear system of equations, and equivalent to solving the Gauss–
Newton method by fixing all variables except the variables
to be optimized. The advantage of the proposed subspace
Gauss–Newton method is that it does not require a large
inverse matrix, unlike Equation 9, but instead, only an inverse matrix of 6 × 6 at most. Additionally, as in ORBSLAM[21] and DSO[6], further speedup is possible by appropriately performing variable elimination that sets mostelements of Hcc and Hpp to zero. Because the proposed
optimization method limits the search space in the Gauss–
Newton method to its subspace, we call it the “subspace
Gauss–Newton method.” 1

4. Dense Reconstruction
A large number of accurate 3D points are generated inreal time with VITAMIN-E as a result of the dense extrema
tracking and subspace Gauss–Newton method described inprevious sections. This leads not only to point cloud generation but also allows further dense geometry reconstructionthat cannot be achieved by conventional indirect methods.
Meshing and Noise Removal: We first project the 3D
points onto an image and apply Delaunay triangulation to
generate triangular meshes. Then, We use NLTGV minimization proposed by Greene et al[23] to remove noise on
the meshes. NLTGV minimization allows us to smooth themeshes, thereby retaining local surface structures, unlike
1 Alternating optimization, such as our method, has been used in somecontexts[36][31]. See the supplementary information for details of thenovel aspect of our method.

typical mesh denoising methods such as Laplacian smoothing. Figure 2(c) shows example results of Delaunay triangulation and smoothing with NLTGV minimization.
Mesh Integration in TSDF: Finally, we integrate themeshes in a TSDF to reconstruct the entire geometry of thescene. The TSDF represents an object shape by discretizing
the space into grids that store the distance from the object
surface, and can merge multiple triangular meshes by storing the average value of distances from the meshes to eachgrid.

5. Experimental Results
5.1. Setup
We evaluated the performance of the proposed
VITAMIN-E on the visual SLAM benchmark EuRoC[3]. 2
The dataset was created using flying drones equipped witha stereo camera and an IMU in an indoor environment,
and provided ground truth of trajectories obtained by a
Leica MS 50 laser tracker and Vicon motion capture. EuRoC is also well-known for data variations, with different
difficulties ranked by the movement speed and lighting conditions. In this experiment, we compared results obtainedusing VITAMIN-E and other monocular SLAM methods,
DSO[6], ORB-SLAM[21], and LSD-SLAM[5], using
only the left images of the stereo camera in the EuRoC
dataset. Note that because VITAMIN-E and DSO[6] do
not include loop closing and relocalization, these functions
in ORB-SLAM[21] and LSD-SLAM[5] were disabled toevaluate performance fairly. Similar evaluations can befound in the papers on DSO[6] and SVO[9].
VITAMIN-E ran on a Core i7-7820 HQ without any
GPUs, threading each process for real time processing. Initialization was performed using essential matrix decomposition. Bundle adjustment is significantly sensitive to theinitial value of the variable. In VITAMIN-E, we initialized
the camera variables using P3P[17, 13] with RANSAC[8]and feature point variables using triangulation. Note thatthe proposed bundle adjustment ran so fast that we applied
it to every frame rather than each key frame as in ORBSLAM and DSO. To manage the TSDF, OpenChisel[16]was used in our implementation because of its ability tohandle a TSDF on a CPU.

5.2. Evaluation Criteria
The evaluation on EuRoC was performed according to
the following criteria:
Localization success rate: We defined the localization
success rate as NNs , where Ns and N denote the number ofimages that were successfully tracked and the images in theentire sequence, respectively. If the success rate was less2 See the supplementary information for experimental results on otherdatasets.

than 90%, we regarded the trial as a failure. When localization failed even once, the methods could not estimate thecamera position later because loop closing and relocalization were disabled in this experiment. Therefore, robustness
greatly contributed to the success rate.
Localization accuracy: The localization accuracy wascomputed by scaling the estimated trajectories so that the
RMSE from ground truth trajectories was minimized because scale is not available in monocular SLAM. Note that
we did not evaluate the accuracy when the success rate wasless than 90% because the RMSE of very short trajectories
tends to have an unfairly high accuracy.
Number of initialization retries: Initialization plays animportant role in monocular visual SLAM and significantly
affects the success rate. Because different methods have different initialization processes, the number of initialization
retries in each method is not directly comparable, but canbe a reference regarding whether the method has a weakness in initialization in certain cases.

5.3. Results and Discussion
Table 1 shows the experimental results. The results wereobtained by applying each method to the image sequencesin EuRoC five times, and Table 1 shows the average values and the standard deviations of the aforementioned criteria. Bold font is used to emphasize the highest accuracy
in each sequence. Regarding LSD-SLAM, the results of initialization retries are not included in the table because LSDSLAM did not have a re-initialization function and failed toinitialize in any sequences. We thus manually identified theframe for which the initialization worked well in each trial,so the number of retries of LSD-SLAM was excluded fromthe evaluation.
The EuRoC image sequences MH01, MH02, MH04,
MH05, V101, V201, and V202 are relatively easy cases
for visual SLAM because the camera motion is relatively
slow and the illumination does not change frequently. By
contrast, the camera moves fast in MH03, V102, V103, andV203, and additionally the lighting conditions dynamicallychange in V102, V103, and V203. Furthermore, in V203,
the exposure time is so long that we can see severe motion blur in the image sequence, particularly in an extremely
dark environment.
Even for the challenging environments, VITAMIN-E
never lost localization and outperformed other SLAM methods, DSO, ORB-SLAM, and LSD-SLAM, both in terms ofaccuracy and robustness, as shown in Table 1. Particularly,
in the sequences that contain fast camera motion, such as
V102 and V103, the proposed method was superior to theexisting methods. The reconstruction results are shown in
Figures 3 and 4. Despite the proposed VITAMIN-E successfully generating dense and accurate geometry comparedwith its competitors, it performed equally fast on a CPU, asTable 1. Experimental results : localization success or failure [X or ×], localization accuracy [cm], localization success rate [%], andnumber of initialization retries.
Sequence name(no. of images)

Our method

MH01 easy(3682)

X

MH02 easy(3040)X

MH03 medium(2700)X

MH04 difficult(2033)X

MH05 difficult(2273)

X

V101 easy(2911)

X

V102 medium(1710)

X

V103 difficult(2149)

X

V201 easy(2280)

X

V202 medium(2348)

X

V203 difficult(1922)

X

12.9 ± 0.5
100.0 ± 0.00±08.8 ± 0.5
100.0 ± 0.0
0±010.6 ± 1.3
100.0 ± 0.0
0±019.3 ± 1.6
100.0 ± 0.0
0±014.7 ± 1.1
100.0 ± 0.0
0±09.7 ± 0.2
100.0 ± 0.0
0±09.3 ± 0.6
100.0 ± 0.0
0±011.3 ± 0.5
100.0 ± 0.0
0±0
7.5 ± 0.4
100.0 ± 0.00±08.6 ± 0.7
100.0 ± 0.0
0±0140.0 ± 5.2
100.0 ± 0.0
0±0DSO[6]
cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

cm
%

X

6.0 ± 0.8
100.0 ± 0.0
0±04.2 ± 0.2
100.0 ± 0.0
0±021.1 ± 0.9
100.0 ± 0.0
0±020.3 ± 1.0
95.7 ± 0.0
5±010.2 ± 0.6
95.5 ± 0.02±013.4 ± 5.8
100.0 ± 0.0
0±053.0 ± 5.5
100.0 ± 0.0
0±085.0 ± 36.4
100.0 ± 0.0
0±07.6 ± 0.5
100.0 ± 0.0
0±011.8 ± 1.4
100.0 ± 0.0
0±0147.5 ± 6.6
100.0 ± 0.0
0±0

ORB-SLAM[21]w/o loop closure
cm
%

X

cm
%

X

cm
%

×

cm
%

X

cm
%

X

cm
%

X

cm
%

×

cm
%

×

cm
%

X

cm
%

X

cm
%

×

5.2 ± 1.1
97.7 ± 1.6
19 ± 114.1 ± 0.4
92.4 ± 1.1
56 ± 6(4.5 ± 0.4)
48.9 ± 0.8
0±033.6 ± 9.4
95.2 ± 0.8
6±114.9 ± 4.6
90.0 ± 4.0
18 ± 58.8 ± 0.1
96.6 ± 0.0
1±0(14.5 ± 11.7)
52.0 ± 3.3
17 ± 4(37.2 ± 20.7)
65.5 ± 8.8
56 ± 26
6.0 ± 0.1
95.2 ± 0.0
0±012.3 ± 2.7
99.5 ± 1.2
0±0(104.3 ± 64.0)
16.8 ± 15.9
233 ± 123

LSD-SLAM[5]w/o loop closure
cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

cm
%

×

(44.9 ± 7.2)
28.9 ± 23.6−
(58.3 ± 6.9)
73.0 ± 1.5−(266.2 ± 61.3)
28.4 ± 20.7−
(136.4 ± 114.3)
27.2 ± 7.0−(27.4 ± 16.4)
22.7 ± 0.5−(20.0 ± 22.8)
11.6 ± 11.2−
(67.0 ± 14.0)
15.2 ± 0.1−(29.3 ± 2.0)
11.0 ± 0.1−(131.3 ± 20.4)
74.1 ± 8.9−(42.1 ± 9.2)
11.3 ± 0.2−(17.7 ± 1.6)
11.9 ± 0.2
−

cm
%
cm
%
cm
%
cm
%
cm
%
cm
%
cm
%
cm
%
cm
%
cm
%
cm
%

Table 2. Average tracking time per frame in each method.
Our method

DSO

ORB-SLAM

LSD-SLAM

36 msec/frame

53 msec/frame

25 msec/frame

30 msec/frame

Table 3. Average computation time for each process of VITAMIN-E. Whereas the front-end processes ran in parallel for each frame, theback-end processes for generating the 3D mesh model were performed at a certain interval.
Feature tracking

Front-end
Localization & mapping

Meshing & denoising

36 msec/frame

25 msec/frame

45 msec/frame

Back-end
TSDF updating & marching cubes
TSDF updating & marching cubes(low-resolution; voxel size ' 15 cm) (high-resolution; voxel size ' 2.5 cm)

shown in Tables 2 and 3. Note that the smaller the size ofthe voxel in a TSDF for a detailed 3D model, the higher thecomputational cost.
The high accuracy and robustness of VITAMIN-E derives from tracking a large number of feature points andperforming bundle adjustment for every frame. Sharing reprojection errors among variables is important for accurate
monocular SLAM, and the proposed method efficiently diffuses errors to an enormous number of variables via fast
bundle adjustment. Simultaneously, it prevents localization
failure caused by losing sight of some feature points by han-

175 msec/time

4000 msec/timedling a large number of feature points.

6. Conclusion
In this paper, we proposed a novel visual SLAM methodthat reconstructs dense geometry with a monocular camera. To process a large number of feature points, we proposed curvature extrema tracking using the dominant flow
between consecutive frames. The subspace Gauss–Newton
method was also introduced to maintain an enormous number of variables by partially updating them to avoid a large

Figure 3. Reconstruction results : (a) dense extrema tracking in real time, (b) reconstructed 3D points, (c) mesh models, and (d) normal
maps generated with the proposed dense geometry reconstruction in different sized TSDF voxels, and reconstructed 3D points in the same
scenes with (e) ORB-SLAM and (f) DSO.

inverse matrix calculation in bundle adjustment. Moreover, supported by the accurate and dense point clouds,
we achieved highly dense geometry reconstruction withNLTGV minimization and TSDF integration.

should be improved when loop closing is introduced in
VITAMIN-E, and fusing IMU data would also be effective
to stably estimate the camera pose for challenging environments such as EuRoC V203.

VITAMIN-E is executable on a CPU in real time, and it
outperformed state-of-the-art SLAM methods, DSO, ORBSLAM, and LSD-SLAM, both in terms of accuracy and robustness on the EuRoC benchmark dataset. Performance

Acknowledgements: This work was supported in part by JSPS KAKENHI, Japan Grant Numbers 16K16084 and 18K18072, and a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO).

0.6

Our Method
DSO

0.5

Error[m]50 msec/frame

Absolute translation error on MH03.

0.7

ORB-SLAM
LSD-SLAM

0.41340 frame

1350 frame

1360 frame

Fast motion

0.3
0.2
0.1
0
0

250

500

750

1000

125013501500

1750

2250

2500

2750

2000 frame

0.6

Our Method
DSO

0.5

2010 frame2020 frame

Nearly pure rotation
50 msec/frame

Absolute translation error on V102.

0.7

Error[m]

2000

2010

Frame Index

ORB-SLAM
LSD-SLAM

0.4

90 frame

100 frame

110 frame

Drastic depth change

0.3
0.2
0.1
0
0

200

400

600

100

800

910 1000

1200

1400

1600

1800

900 frame

910 frame

Frame Index

Nearly pure rotation

Absolute translation error on V103.

50 msec/frame

920 frame

1.4
1.2

Our Method
DSO

Error[m]1

ORB-SLAM
LSD-SLAM
1327 frame

0.81337 frame

1347 frame

Fast motion & Rapid illumination change

0.6
0.4
0.2
0
0

250

500

750

1000

1250

Frame Index

1500

1337

1750

1880

2000

2250

1870 frame

1880 frame

1890 frame

Nearly pure rotation

Figure 4. Chronological RMSEs of estimated trajectories on EuRoC V102, V103, and MH03, and the reconstructed point clouds in eachscene: VITAMIN-E successfully estimated the camera trajectories despite a drastic depth change, nearly pure camera rotation, rapid camera
motion, and severe lighting conditions, whereas the competitors suffered from them and resulted in large trajectory errors or completely
getting lost.

References[1] Paul Bergmann, Rui Wang, and Daniel Cremers. Online photometric calibration of auto exposure video for realtime visual odometry and SLAM. IEEE Robotics and Automation
Letters, 3, 2018.
[2] Michael Blosch, Sammy Omari, Marco Hutter, and Roland
Siegwart. Robust visual inertial odometry using a direct ekfbased approach. In Proc. of International Conference on Intelligent Robots (IROS), 2015.
[3] Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas
Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart. The EuRoC micro aerial vehicle datasets. International Journal of Robotics Research,
35(10):1157–1163, 2016.
[4] Michael Calonder, Vincent Lepetit, Christoph Strecha, andPascal Fua. BRIEF: Binary robust independent elementaryfeatures. In Proc. of European Conference on Computer Vision (ECCV), 2010.
[5] Jakob Engel and Daniel Cremers. LSD-SLAM: Large-scaledirect monocular SLAM. In Proc. of European Conference
on Computer Vision (ECCV), 2014.
[6] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct
sparse odometry. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2018.
[7] Jakob Engel, Jörg Stückler, and Daniel Cremers. Large-scaledirect slam with stereo cameras. In Proc. of International
Conference on Intelligent Robots (IROS), 2015.
[8] Martin A. Fischler and Robert C. Bolles. Random sample
consensus: A paradigm for model fitting with applications toimage analysis and automated cartography. Communicationsof the ACM, 24(6):381–395, 1981.
[9] Christian Forster, Zichao Zhang, Michael Gassner, ManuelWerlberger, and Davide Scaramuzza. SVO: Semidirect
visual odometry for monocular and multicamera systems.
IEEE Transactions on Robotics, 33(2):249–265, 2017.
[10] Yasutaka Furukawa and Jean Ponce. Accurate, dense, androbust multi-view stereopsis. IEEE Transactions on PatternAnalysis and Machine Intelligence, 32(8):1362–1376, 2010.
[11] Wolfgang Hess, Damon Kohler, Holger Rapp, and Daniel
Andor. Real-time loop closure in 2D LIDAR SLAM. In
Proc. of International Conference on Robotics and Automation (ICRA), 2016.
[12] Albert S. Huang, Abraham Bachrach, Peter Henry, Michael
Krainin, Daniel Maturana, Dieter Fox, and Nicholas Roy.
Visual odometry and mapping for autonomous flight using
an RGB-D camera. In Proc. of International Symposium on
Robotics Research (ISRR), 2011.
[13] Tong Ke and Stergios Roumeliotis. An efficient algebraic
solution to the perspective-three-point problem. In Proc. ofInternational Conference on Computer Vision and Pattern
Recognition (CVPR), 2017.
[14] Christian Kerl, Jürgen Sturm, and Daniel Cremers. Robust
odometry estimation for RGB-D cameras. In Proc. of International Conference on Robotics and Automation (ICRA),
2013.

[15] Georg Klein and David Murray. Parallel tracking and mapping for smallar workspaces. In Proc. of International Symposium on Mixed and Augmented Reality (ISMAR), 2007.
[16] Matthew Klingensmith, Ivan Dryanovski, Siddhartha Srinivasa, and Jizhong Xiao. Chisel: Real time large scale 3D reconstruction onboard a mobile device using spatially hashedsigned distance fields. Robotics: Science and Systems, 2015.
[17] Laurent Kneip, Davide Scaramuzza, and Roland Siegwart. A
novel parametrization of the perspective-three-point problem
for a direct computation of absolute camera position and orientation. In Proc. of International Conference on Computer
Vision and Pattern Recognition (CVPR), 2011.
[18] Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland
Siegwart, and Paul Furgale. Keyframe-based visual-inertial
odometry using nonlinear optimization. International Journal of Robotics Research, 34(3):314–334, 2015.
[19] Simon Lynen, Markus Wilhelm Achtelik, Stephan Weiss,
and Margarita Chli. A robust and modular multi-sensor fusion approach applied to mav navigation. In Proc. of International Conference on Intelligent Robots (IROS), 2013.
[20] Anastasios I. Mourikis and Stergios I. Roumeliotis. A multistate constraint Kalman filter for vision-aided inertial navigation. In Proc. of International Conference on Robotics andAutomation (ICRA), 2007.
[21] Raul Mur-Artal and Juan D. Tardós. ORB-SLAM2: an
open-source SLAM system for monocular, stereo and RGBD cameras. IEEE Transactions on Robotics, 33(5):1255–
1262, 2017.
[22] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges,
David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew
Fitzgibbon. Kinectfusion: Real-time dense surface mapping
and tracking. In Proc. of International Symposium on Mixed
and Augmented Reality (ISMAR), 2011.
[23] W. Nicholas Greene and Nicholas Roy. Flame: Fast
lightweight mesh estimation using variational smoothing ondelaunay graphs. In Proc. of International Conference on
Computer Vision (ICCV), 2017.
[24] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland
Siegwart, and Juan Nieto. Voxblox: Incremental 3D euclidean signed distance fields for on-board mav planning.
In Proc. of International Conference on Intelligent Robots(IROS), 2017.
[25] Chanoh Park, Peyman Moghadam, Sooshin Kim, Alberto
Elfes, Clinton Fookes, and Sridha Sridharan. Elastic lidar
fusion: Dense map-centric continuous-time slam. In Proc.
of International Conference on Robotics and Automation(ICRA), 2018.
[26] Alex Poms, Chenglei Wu, Shoou-I Yu, and Yaser Sheikh.
Learning patch reconstructability for accelerating multi-view
stereo. In Proc. of International Conference on Computer
Vision and Pattern Recognition (CVPR), 2018.
[27] William H. Press, Saul A. Teukolsky, William T. Vetterling,
and Brian P. Flannery. Numerical Recipes in C. Cambridge
University Press, Cambridge, USA, second edition, 1992.
[28] Tong Qin, Peiliang Li, and Shaojie Shen. Vins-mono: A
robust and versatile monocular visual-inertial state estimator.
IEEE Transactions on Robotics, 34(4):1004–1020, 2018.

[29] Jianbo Shi and Carlo Tomasi. Good features to track. In
Proc. of International Conference on Computer Vision and
Pattern Recognition (CVPR), 1994.
[30] Frank Steinbrücker, Jürgen Sturm, and Daniel Cremers. Volumetric 3D mapping in real-time on a cpu. In Proc. of International Conference on Robotics and Automation (ICRA),
2014.
[31] Bill Triggs, Philip McLauchlan, Richard Hartley, and Andrew Fitzgibbon. Bundle adjustment - a modern synthesis.
In Proc. of the Int. WS on Vision Algorithms: Theory andPractice, 2000.
[32] R. Wang, M. Schwörer, and D. Cremers. Stereo DSO: Largescale direct sparse visual odometry with stereo cameras. In
International Conference on Computer Vision (ICCV), 2017.
[33] Ji Zhang, Michael Kaess, and Sanjiv Singh. Real-time depth
enhanced monocular odometry. In Proc. of International
Conference on Intelligent Robots (IROS), 2014.
[34] Ji Zhang and Sanjiv Singh. LOAM: Lidar odometry andmapping in real-time. Robotics: Science and Systems, 2014.
[35] Zichao Zhang, Christian Forster, and Davide Scaramuzza.
Active exposure control for robust visual odometry in HDR
environments. In Proc. of International Conference on
Robotics and Automation (ICRA), 2017.
[36] Qian-Yi Zhou and Vladlen Koltun. Color map optimizationfor 3d reconstruction with consumer depth cameras. ACM
Trans. Graph., 33(4):155:1–155:10, 2014.