This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2020

1

C*: Cross-modal Simultaneous Tracking And
Rendering for 6-DoF Monocular Camera
Localization Beyond Modalities
Shuji Oishi1 , Yasunori Kawamata2 , Masashi Yokozuka1 , Kenji Koide1 , Atsuhiko Banno1 , and Jun Miura2

Abstract—We present a monocular camera localization technique for a three-dimensional prior map. Visual localization has
been attracting considerable attention as a lightweight and widely
available localization technique for any mobilities; however, it still
suffers from appearance changes and a high computational cost.
With a view to achieving robust and real-time visual localization,
we first reduce the localization problem to alternate local tracking
and occasional keyframe rendering by following a simultaneous
tracking and rendering algorithm. At the same time, by using
an information-theoretic metric denoted normalized information
distance in the local tracking, we developed a 6-DoF localization
method robust to intensity variations between modalities andvarying sensor properties. We quantitatively evaluated the accuracy and robustness of our method using both synthetic and real
datasets and achieved reliable and practical localization even in
the case of extreme appearance changes.

Prior 3D map

Input image

with laser reflectivity

Rendered image
from estimated pose

Fig. 1. 6-DoF visual localization beyond modalities. See the attached video
or https://youtu.be/PW_DlMiH-_w for more information.

Index Terms—Localization, Multi-Modal Perception, Visual
Tracking

I. I NTRODUCTION
CCURATE and robust localization is a key technology
for any autonomous mobilities, and it has become increasingly important to achieve autonomous navigation. Givenan accurate pose, a mobile robot can plan a path toward thedestination and navigate itself using feedback control, thereby
resulting in an efficient and safe autonomous system.
Various approaches have been proposed toward achieving
reliable localization. Although using the global positioning
system would be one of the most common approaches, it
can be easily disrupted in several regions, such as urban andindoor environments, because of multi-path effects and skyview obstruction, while autonomous service robots are in highdemand in such regions.
Thus far, successful autonomous mobilities have adopted
three-dimensional (3D) map-based approaches that estimate

A

Manuscript received: February 21, 2020; Revised May 15, 2020; Accepted
June 23, 2020.
This paper was recommended for publication by Sven Behnke uponevaluation of the Associate Editor and Reviewers’ comments. This work was
supported by JSPS KAKENHI(Grant Number 18K18072), research grant by
Naito Science & Engineering Foundation, and a project commissioned by the
New Energy and Industrial Technology Development Organization (NEDO).
1 Mobile Robotics Research Team, National Institute of Advanced Industrial Science and Technology (AIST), Ibaraki, Japan (e-mail: {shuji.oishi,
yokotsuka-masashi, k.koide, atsuhiko.banno}@aist.go.jp)
2 Active Intelligent Systems Laboratory, Toyohashi University of Technology, Aichi, Japan (e-mail: kawamata@aisl.cs.tut.ac.jp, jun.miura@tut.jp)
Digital Object Identifier (DOI): see top of this page.

the robot pose by aligning the local observation to the previously constructed 3D map. Fortunately, performing the 3Dmodeling of a real site or building has become easy, as
3D reconstruction technologies, such as Simultaneous Localization And Mapping (SLAM) [1] [2] and Structure from
Motion [3], have evolved, and the reconstructed map can beused as a prior for localization. In the robotics community,
LiDAR-based approaches that utilize point cloud registration
techniques, such as iterative closest points (ICP) and normal
distributions transform, have achieved highly accurate androbust localization by estimating the pose at which the local
observation and prior map are geometrically consistent witheach other. However, LiDAR-based approaches significantly
increase the sensor cost.
As an alternative to them, visual localization has been attracting considerable attention to address the above-mentioned
issues. It infers the pose of an agile monocular camera in a
known 3D map and, therefore, substantially reduces the sensor
cost compared with that in LiDAR-based approaches, thereby
resulting in lightweight and widely available localization forany mobilities. It can be categorized as either indirect methods
that use image descriptor matching [4] or direct methods thatcompare the appearance (pixel intensities) in each view [5].
Although these approaches achieve high localization performances, they inherently perform in the limited cases whereinsensors with the same properties are used for mapping andlocalization to ensure that the local observation is consistent
with the appearance of the prior unless we overcome thedifference of sensor characteristics or modalities.

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
2IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2020

With a view to achieving easy and reliable pose tracking,we are interested in monocular camera localization for bothdifferent sensing modalities and appearance changes between
mapping and localization phases. The problem, however, iswhat is refer to as ”cross-modal / long-term visual localization,” which has been a widely studied topic. Inspired bysimultaneous tracking and rendering (STAR) [5], reducing the
localization problem to successive local tracking against synthetic keyframes, we progressed toward achieving accurate androbust visual localization, even for different sensor properties.
Specifically, by employing an information-theoretic metric in
the local tracking to estimate the intensity relationship between
the monocular camera image and appearance of the prior 3D
map, we developed cross-modal STAR, C ∗ , as depicted in
Fig.1.
II. RELATED WORK
Indirect visual localization first constructs a database ofimage descriptors with their 3D locations as a prior map
and, subsequently, estimates the camera pose via featurepoint matching [4] [6]. Specifically, it uses image descriptors
or pre-trained visual words to make the 3D feature points
discriminative and match feature points extracted in the current
frame against the database. According to the correspondences
it localizes the camera pose by minimizing 2D reprojection
errors. While descriptors can be repeatably matched evenwhen slight illumination changes occur, they become lessdiscriminative when viewpoint changes significantly. This isbecause the visual feature associated with each map point
is usually extracted from a local intensity pattern in a 2D
image obtained from a certain viewpoint, thereby resulting inunstable localization for large viewpoint changes.
Direct visual localization achieves stable tracking by performing a pixel-wise comparison between incoming camera
images and a set of colored 3D points [7] [5]. Compared withfeature-based methods, direct methods achieve more robust
pose estimation even for significant viewpoint changes. They
perform the pose estimation by relying only on map point colors (intensities), which results in low viewpoint dependency.
However, in direct methods, it is implicitly assumed that theappearance is captured using the same sensor and remains overtime; therefore, the performance can easily worsen becauseof illumination changes or different sensor property/modality,
thereby restricting their applications to only limited situations.
In both indirect and direct methods, the localization accuracy can be remarkably reduced or the localization canfail if the appearance excessively changes because of somereason. To develop visual localization that is robust to extreme
appearance changes, several approaches utilize geometric information. Caselitz et al. [8] employed ORB-SLAM [9] togenerate 3D points of the scene from image sequences andtracked the camera pose by aligning the points against the prior
map with 7-DoF ICP. Although this approach uses a monocular camera, it results in point cloud registration, which is
independent of appearance changes, thereby achieving robust
localization. However, 3D points reconstructed using indirect
visual SLAM are inherently sparse, and 7-DoF ICP withsparse point clouds may result in rough localization. Neubert etal. [10] developed an appearance-independent localization bydefining a likelihood function for a particle filter that strongly
assumed the co-occurrence of geometric and texture edges.
By successively updating particles according to the likelihood
function, it estimated the optimal pose from which the edges ina virtual depth image captured were highly correlated to thosein the current camera image. Similar methods that minimize
the distance between edges in input and reference images byassuming the co-occurrence between different modalities havebeen proposed [11] [12] [13]; however, the assumption should
be carefully designed to ensure the co-occurrence depending
on individual cases, otherwise the similarity evaluation may
fail.
Deep learning-based approaches to visual localization havealso come into fashion. For example, Kendall and Cipolla[14] developed an end-to-end robust pose estimation from amonocular camera image. They proposed novel loss functions
based on scene geometry, which allows simultaneous position
and rotation learning, and achieved highly robust localization
under different lighting conditions and appearance changes.
However, as discussed in [15], data-driven visual localization
tends to be less accurate than classical indirect approaches
due to the difficulty in training the pose regression layer
well. Focusing on this, [15] developed several approaches withhand-crafted or data-driven models for computing essential
matrices; however, they still lag behind classical approaches
in terms of accuracy.
To perform accurate localization even in case of illumination
changes or across different sensor modalities, some researchworks used information-theoretic metrics. Wolcott et al. [16]achieved localization on a road map captured using LiDAR bymaximizing the mutual information (MI) between in-vehicle
camera views and virtually rendered images; however, only
3-DoF pose estimation was performed. Pascoe et al. [17][18] also proposed a 6-DoF pose estimation by employing
normalized information distance (NID) [19] and demonstrated
a promising localization performance in aligning images fromdifferent modalities; however, the tracking rate was only 2 Hz.
Information-theoretic metrics can compare intensity distributions instead of intensities or intensity patterns, and do not
expect any specific types of coincidence or mapping. This allows robust similarity evaluation beyond modalities; however,simultaneously, the computation is relatively demanding.
Inspired by the above-mentioned works, we developed
a cross-modal monocular camera tracking algorithm towardachieving robust and practical localization. The main contribution of this paper is a new pipeline that combines STARand NID; The combination of keyframe-based localization androbust local tracking achieves efficient and reliable localizationeven under extreme appearance changes, as demonstrated in
Section IV.
III. PROPOSED METHOD
A. Overview
Figure 2 provides an overview of the proposed method.
Given an initial camera pose in a prior 3D map, we track the

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
OISHI et al.: C* : CROSS-MODAL SIMULTANEOUS TRACKING AND RENDERING FOR 6-DOF MONOCULAR CAMERA LOCALIZATION BEYOND MODALITIESColored 3D map

Map point i P

Current image

Current keyframekLocal tracking
against keyframewith metric 𝛿𝐼 𝑁𝐼𝐷

Sample i uiu Reprojection

Far from
current keyframe?

neighborsN ( i u)Keyframepose T kw

Y

Coordinatetransformation

Current frame I t

Keyframe I k

 w World coordinate

3tRelative pose Tk
to be estimated

New keyframe
rendering

Current pose

Global pose

Twt = Tkt  Twk

Fig. 2. Overview of the proposed method.

current camera pose TtW by comparing the appearances of the
3D prior map with a monocular image It . Using STAR [5], we
split the problem into the following two separate tasks: One
deals with relative pose estimation between the current frame
and a synthetic keyframe, while the other renders as necessary
the synthetic keyframe, which comprises a virtual monocular
image and its depth map, to generate a local view (map) of theprior 3D map from the given pose. The alternate tracking andrendering relieve us from the frequent re-rendering process ineach iteration, as in [18]. It also enables us to estimate the 6DoF camera pose even from a large-scale 3D map by reducing
the localization problem to successive local tracking, thereby
avoiding error accumulation in pose estimation over time.
To perform localization against a keyframe, we leverage an
information-theoretic metric for ensuring robustness. Because
prior 3D maps can be captured using different sensors, such asa digital camera with different photometric characteristics and
LiDAR, conventional photometric error minimization may failto estimate the accurate camera pose. Therefore, we use NID[19] to compare images with different properties or modalities.
NID is a true metric that is independent of the number ofsamples, unlike MI [20], and it is effective for aligning images
under different conditions, as employed in pose estimation in a
3D textured prior map [17] [18]. Benefitting from the abovementioned characteristics, we develop a STAR-based robust
and practical localization technique by leveraging NID.
B. Simultaneous Tracking and Rendering
1) Local tracking: Following prior work [5] [21], we
formulate the visual localization problem. Given a monocular
image It and a synthetic keyframe one Ik rendered at a posekTW
, the local tracking problem is defined as SE(3) optimization with regard to the relative motion Ttk , by comparing It
with Ik in terms of a certain similarity metric δI. Specifically,
because the keyframe image Ik has the corresponding depth
map Dk , each pixel i u = (i u,i v) ∈ Ik can be back-projected
to a 3D point i Pk = (i x,i y,i z) in the keyframe coordinate byusing the camera projection model π : R3 7→ R2 as follows:iPk= π −1 (i u, Dk (i u)) ,(1)

where π −1 : R2 7→ R3 denotes the inverse projection function.
The 3D point i Pk is mapped to the current frame coordinate′t= Ttk i Pk andi u by both relative transformation i P
subsequent camera projection π as follows:
( t)
′
(2)
iu = π iP .
Notably, the intrinsic camera parameters in the projection π
are obtained via a calibration process, and we use the same
camera model to render the synthetic keyframes in this paper.
Given the relative pose Ttk , the 2D–3D and 3D–2D projections enable us to find the pixel-wise correspondences between
the current image It and keyframe image Ik by observing thesame 3D point. Therefore, the most probable relative poseT̂tk ∈ SE(3) is obtained by minimizing the sum of per-pixel
differences as follows:
T̂tk = arg minδI (It , Ik )Ttk

= arg minTtk

= arg minTtk∑
i u∈Ωk∑δIi (It (i u′ ), Ik (i u))()
δIi It (π(Ttk · π −1 (i u, Dk (i u)))), Ik (i u) ,i u∈Ωk(3)
where Ωk ⊂ Ik denotes sample pixels that are to be projected
onto the current camera image It . In our method, all the pixels
in Ik with valid depth values are used as the samples Ωk for
performing dense localization.
For example, photometric registration solves the problem
by defining the metric δI as a photometric error δIphoto ;The difference in the intensity domain is directly measured
as follows:∑δI photo (It , Ik ) ≡σ (It (i u′ ) − Ik (i u))(4)
i u∈Ωkwhere σ denotes a robust kernel to suppress the effect ofoutliers, such as t-distribution and Huber kernel [5].
2) Keyframe selection: Because the accuracy of pose tracking against a keyframe deteriorates as the view overlap decreases, we render a new keyframe as necessary to sustain
the local tracking. The idea of keyframe selection is common
in not only visual localization but also visual SLAM context.

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
4IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2020

Inspired from [22], we create a new keyframe for the mostrecently tracked poses for ensuring conservative keyframe
update independent of co-visibility [5] or average scene depth[23]. Specifically, the following distance is defined for the exponential map ξ ∈ R6 of the Lie algebra se(3) corresponding
to the relative position and orientation Ttk :( )(5)dist Ttk ≡ ξ T Wξ,iy = (I k ( i u ))yC. Robust metric leveraging information theory(6)

where H(It , Ik ) and I(It ; Ik ), respectively, denote the joint
entropy and MI of samples Ωk , and they are calculated asfollows:
H (It ) = −H (It , Ik ) = −n∑pt (x) log (pt (x)) ,
x=1
n ∑
n
∑

(7)

pt,k (x, y) log (pt,k (x, y)) , (8)x=1 y=1

I(It ; Ik ) = H(It ) + H(Ik ) − H(It , Ik ),u

x = (I t ( j N ( i u)))Current frame bin

pt ,kFig. 3. Joint probability calculation via spatially weighted voting using 2D
cubic B-spline.

joint probability C 2 continuous for performing gradient-based
optimization as follows:∑
1
pt,k (x, y) =β (i u,j N (i u′ )) ,
(10)
∥Ωk ∥
i u∈ΩkThe similarity metric δI should be carefully chosen tocompare images with different sensor properties or modalities.
In this study, because we aim to localize a monocular camera
in a prior 3D map reconstructed using a variety of sensors
such as LiDAR and different digital cameras, the appearance
difference significantly matters. Because photometric error(see Eq.4), which directly evaluates the similarity of camera
images and a 3D prior map in the intensity domain, has anunderlying assumption that the appearances are consistent,
it is not robust to appearance differences/changes, thereby
resulting in tracking failure. Therefore, following [17] [18],we employ NID to stably localize a monocular camera fordifferent modalities.
1) Computation of joint probability and NID: The NID
of two discrete random variables, the current frame It andkeyframe Ik , is given as follows:
H(It , Ik ) − I(It ; Ik )
δI N ID (It , Ik ) ≡,H(It , Ik )Keyframe bin

p x , y + =  (i u, j N ( i u))xiwhere the first three elements of ξ represent a translation
whereas the latter three elements represent a rotation, andW denotes a diagonal weighting matrix to perform keyframe
selection that is sensitive to certain movements. When the distance exceeds the threshold τdist , a new keyframe is generatedusing the most recent camera pose.
The weights and threshold were empirically determined sothat each keyframe could sufficiently overlap with the incoming camera images; however, simultaneously, the rendering
process could not be performed frequently, as it incurs highcomputational cost.

u(9)where pt,k denotes the joint probability obtained using an n ×
n-dimensional histogram, and the marginal probabilities pt andpk are derived from pt,k .
To compute NID, we must obtain pt,k from samples Ωk . We
implemented a weighted voting approach based on [24] (seeFig.3), wherein a 2D cubic B-spline was employed to make thexy= Θ (It (j N (i u′ ))) ,= Θ (Ik (i u)) ,

(11)
(12)

where Θ returns a histogram bin corresponding to the input
intensity, and β (i u,j N (i u′ )) denotes the B-spline coefficient that weights the contribution of each sample i u overthe 4 × 4 local support region j N (i u′ ) centered at the′
reprojection
i u . Notably, the coefficients are normalized as∑′j β (i u,j N (i u )) = 1, ∀i u to ensure that each sample i ucontributes equally to the histogramming.
2) Computation of derivatives for optimization: Differentiating Eq.6 with respect to the relative transform Ttk yields
the Jacobian of NID, thereby allowing us to seek the optimaltransformation as Tˆtk , which minimizes the NID between thecurrent frame and keyframe:(dδIN ID
=
dTtk

dI
dHt,k
− dTt,kt
dTtk
k

)

dH

Ht,k − (Ht,k − It;k ) dTt,k
t
k

2
Ht,k

where

∑ dpt,k(1 + log (pt,k )) ,dTtk
i u∈Ωk())∑ dpt,k (pt,k1 + log
.
dTtk
pt

,
(13)

dHt,k
dTtk

= −

(14)

dIt;k
dTtk

=

(15)

i u∈ΩkThe primary computation involved in the differentiation is
finding the derivative of the joint distribution pt,k . Notably,
the joint distribution can be differentiated using the chain-rule
as follows:∑ dβ (i u,j N (i u′ ))1dpt,k (x, y)
=
t
dTk
∥Ωk ∥
dTtk
i u∈Ωk(16)∑ dβ di u′
1
=
,
∥Ωk ∥
di u′ di Ttk
i u∈ΩkCopyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
OISHI et al.: C* : CROSS-MODAL SIMULTANEOUS TRACKING AND RENDERING FOR 6-DOF MONOCULAR CAMERA LOCALIZATION BEYOND MODALITIESTABLE IPARAMETERS USED IN THE EXPERIMENTS .
Parameter
Camera-image size
NID-histogram bins n
Keyframe-selection threshold τdist
Keyframe-selection weights W

value
800 × 600
160.01
diag(0.1, 0.1, 0.1, 1.0, 1.0, 1.0)

TABLE IIQ UANTITATIVE - EVALUATION RESULTS : RMS TRANSLATION ERROR [ M ],ROTATION ERROR [ DEG ], AND LOCALIZATION - SUCCESS RATIO ( THE RATIOOF NUMBER OF FRAMES WITH ESTIMATION ERRORS WITHIN 1 M TO THAT
OF TOTAL FRAMES ) [%].
Index
R.1
R.2

′

where ∂∂iiTut denotes the differential of the reprojection i u′kwith respect to the relative pose Tkt [25] calculated using focal
lengths fx and fy as follows:


′ ′
′
−fx2 −i u′ 2−fx
i ui v
′
iu
′
v
0
di u
′
′
i
fx
fx
iz
.
=  iz′ ′′fy2 +i v ′ 2−fy
i ui v
iv
dTtk
0
−
−i u′
′
′
iz

iz

fy

R.3
R.4
R.5
R.6

fy

(17)
According to Eq.13, we perform a gradient-based optimization
to determine the most probable relative pose Tkt . Specifically,
the Broyden–Fletcher–Goldfarb–Shannon (BFGS) algorithm
implemented in Ceres Solver [26] was adopted for optimizing
the local tracking.
IV. EXPERIMENTS

R.7

A.1
A.2
A.3
A.4

A. Setup

A.5

In the following experiments, we used the parameters listed
in Table I. Each experiment was performed using a desktop
PC equipped with an Intel Core i7-6850K and a GeForce
GTX1080, and the NID-histogram calculation was parallelizedusing CUDA to reduce the computation time.
In addition, we implemented competitive localization methods that use the same STAR framework but different costfunctions. Specifically, we chose a method based on photometric errors [5] and one based on edge distances [11]as baselines to compare the localization performances under
the same conditions. Both of them use pixels with sufficient
gradient; thus, we extract high-gradient pixels with a Sobel
filter and image thresholding. In addition, to reproduce [11],a distance transform is applied to the edge image to evaluate
edge distances.

A.6

B. Quantitative evaluation in the Replica Dataset
To quantitatively evaluate the accuracy and robustness of our
method, we used a set of photo-realistic 3D models, namely
the Replica Dataset [27]. It provides the 3D models of various
indoor environments, and each of them has not only highquality texture but also semantic labels according to object
categories.
We selected “room0” and “apartment0” from the dataset
and generated camera-image sequences along predefined trajectories (see Fig.4(a)(i)), which included both translationaland rotational motions. Given the initial pose, we successively
estimated the camera poses from the sequences by using each
localization technique to evaluate the accuracy against the
ground truth. In addition, as depicted in the first row in Fig.
4, we used two types of map representations, texture andsemantics, and also deteriorated the trajectory image sequences

5

A.7

L.1
L.2
L.3
L.4

Map / Image

Photometric [5]Edge [11]I. room0 in the Replica dataset
0.0148 [m]0.0139 [m]Texture / Camera
0.241 [deg]0.276 [deg](Fig.4(a) / Fig.4(c))
100 [%]
100 [%]
0.170 [m]0.540 [m]Texture / Blurred
2.78 [deg]7.95 [deg](Fig.4(a) / Fig.4(d))
100 [%]
9.5 [%]
0.502 [m]0.736 [m]Texture / Overexposed
5.60 [deg]14.5 [deg](Fig.4(a) / Fig.4(e))
1.3 [%]
1.8 [%]
0.365 [m]- [m]Texture / Occlusion
4.63 [deg]- [deg](Fig.4(a) / Fig.4(f))
3.6 [%]
0 [%]0.567 [m]0.545 [m]Texture / Salt&Pepper
13.1 [deg]8.03 [deg](Fig.4(a) / Fig.4(g))
13.7 [%]
9.8 [%]- [m]- [m]Texture / Underexposed- [deg]- [deg](Fig.4(a) / Fig.4(h))
0 [%]
0 [%]
- [m]
0.0249 [m]Semantics / Camera- [deg]0.490 [deg](Fig.4(b) / Fig.4(c))
0 [%]100 [%]
II. apartment0 in the Replica dataset
0.0117[m]
0.0203[m]
Texture / Camera
0.233 [deg]0.418 [deg](Fig.4(i) / Fig.4(k))
100 [%]
100 [%]
0.193 [m]0.490 [m]Texture / Blurred
4.47 [deg]8.14 [deg](Fig.4(i) / Fig.4(l))
22.5 [%]
5.2 [%]
0.676 [m]0.402 [m]Texture / Overexposed
8.38 [deg]3.51 [deg](Fig.4(i) / Fig.4(m))
4.5 [%]
0.7 [%]- [m]- [m]Texture / Occlusion- [deg]- [deg](Fig.4(i) / Fig.4(n))
0 [%]
0 [%]
- [m]
0.514 [m]Texture / Salt&Pepper- [deg]8.56 [deg](Fig.4(i) / Fig.4(o))
0 [%]5.3 [%]- [m]- [m]Texture / Underexposed- [deg]- [deg](Fig.4(i) / Fig.4(p))
0 [%]
0 [%]
0.510 [m]0.171 [m]Semantics / Camera
5.82 [deg]2.68 [deg](Fig.4(j) / Fig.4(k))
0.5 [%]
32.2 [%]
III. An indoor environment scaned with a LiDAR
0.0696[m]
0.0487[m]
Texture / Camera
1.88 [deg]1.35 [deg](Fig.4(q) / Fig.4(s))
33.7 [%]
32.5 [%]
0.235 [m]0.387 [m]Texture / Dim light
8.43 [deg]7.52 [deg](Fig.4(q) / Fig.4(t))
3.3 [%]
18.7 [%]
0.0432 [m]0.0422 [m]Texture / Obstacle
0.896 [deg]0.915 [deg](Fig.4(q) / Fig.4(u))
100 [%]
55.6 [%]
0.786 [m]0.119 [m]Reflectance / Camera
19.0 [deg]3.12 [deg](Fig.4(r) / Fig.4(s))
6.6 [%]
7.6 [%]

Our method0.00490 [m]0.0763 [deg]100 [%]
0.0644 [m]1.06 [deg]100 [%]
0.0128 [m]0.232 [deg]100 [%]
0.00390 [m]0.0574 [deg]100 [%]
0.00550 [m]0.0677 [deg]100 [%]
0.00770 [m]0.1250 [deg]100 [%]
0.0710 [m]1.38 [deg]100 [%]
0.0147 [m]0.176 [deg]100 [%]
0.0537 [m]0.821 [deg]100 [%]
0.0141 [m]0.175 [deg]100 [%]
0.0123 [m]0.132 [deg]100 [%]
0.00410 [m]0.0544 [deg]100 [%]
0.0130 [m]0.185 [deg]100 [%]
0.195 [m]3.23 [deg]49.7 [%]
0.0196 [m]0.966 [deg]100 [%]
0.0359 [m]0.806 [deg]100 [%]
0.0191 [m]0.417 [deg]100 [%]
0.0589 [m]1.60 [deg]100 [%]

to examine robustness against extreme appearance changes orcross-modality.
First, we performed localization experiments in the textured
3D map (see Fig.4(a)) on the basis of the image sequences(see Fig.4(c)–(h)) by assuming different sensor properties
in both mapping and localization phases. The translationaland rotational RMS errors are depicted in Fig. 5(a)–(f) andTable II(R.1)–(R.6). Each method performed satisfactorily andachieved high accuracies upon performing localization using
the image sequence of the original texture. However, when
the appearance changed because of blur, over/under exposure, occlusion, or salt-and-pepper noise, tracking with theconventional methods [5] [11] immediately failed. However,
our method robustly tracked the agile monocular camera evenunder extreme appearance changes.
Next, similar experiments were performed using the 3D
map colored by semantics (see Fig.4(b)) and the texture
image sequence (see Fig.4(c)), assuming the case of different
modalities. We simply assigned to each map point a binnumber as a color corresponding to the object category because the joint entropy is independent of how the mapping
is performed. Figure5(g) and Table II(R.7) show the errors

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
6IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2020(a) Textured map(b) Semantic map(c) Camera(d) Blurred(e) Overexposed(f) Occlusion(g) Salt&Pepper(h) Underexposed(n) Occlusion(o) Salt&Pepper(p) Underexposed

I. room0 in the Replica dataset(i) Textured map(j) Semantic map(k) Camera(l) Blurred(m) Overexposed

II. apartment0 in the Replica dataset(q) Textured map(r) Reflectance map(s) Camera(t) Dim light(u) Dynamic obstacle

III. An indoor environment scanned using a LiDAR

6

8
7
6
5
4
3
2
1
0
0

Rotation Error [deg]

0.8
0.6
0.4
0.2
0.0
0

1

2

3
4
Time [sec]

5

1

2

3

4

5

6

1

2

(a) R.1

3
4Time [sec]

5

6

1

3.0
2.5
2.0
1.5
1.0
0.5
0.0
0

2

3

4

5

6

1

2

3

4

5

6

6

1.2
1.0
0.8
0.6
0.4
0.2
0.0
0

1

(b) R.2

2

3
4
Time [sec]

5

6

1

2

(c) R.3

3
4
Time [sec]

5

1

2

3

4

5

6

0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.00
0
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0

Ours
Photometric
Edge

1

2

3

4

5

Translation Error [m]

0.0
0

1.0

Ours
Photometric
Edge

6

1

2

(d) R.4

3
4Time [sec]

5

6

0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0
20

Rotation Error [deg]

0.1

0.10
0.08
0.06
0.04
0.02
0.00
0

Translation Error [m]

0.2

6

Ours
Photometric
Edge

Rotation Error [deg]

5

0.06
0.05
0.04
0.03
0.02
0.01
0.00
0
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0

Translation Error [m]4Ours
Photometric
Edge

Rotation Error [deg]

3

0.16
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
0

Translation Error [m]2

Ours
Photometric
Edge

0.3

Rotation Error [deg]1

0.4

Translation Error [m]Ours
Photometric
Edge

Rotation Error [deg]Translation Error [m]

0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.00
0

Rotation Error [deg]Translation Error [m]Fig. 4. Various 3D maps and deteriorated image sequences on the trajectories, which are illustrated in (a), (i), and (q). Notably, we captured the image
sequences using a hand-held camera in the indoor evaluation; therefore, multiple trajectories are depicted in (q) (blue for (s), orange for (t), green for (u), andblack for (r)).

12(e) R.5

3
4Time [sec]

5

6

Ours
Photometric
Edge

1

2

3

4

5

6

2

3
4
Time [sec]

5

6

15
10
5
0
0

1

(f) R.6(g) R.7

0.00
0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
0

2

4

6

8

10

12

1.0
0.5
2

4

6
8
Time [sec]

10

12

0.05
0.00
0

2

4

(h) A.1

6
8Time [sec]

10

12

2

1.2
1.0
0.8
0.6
0.4
0.2
0.0
0

4

6

8

10

12

2

(i) A.2

4

6
8
Time [sec]

10

12

2

4

6

8

10

12

1.5
1.0
0.5
0.0
0

2

4

6

8

10

12

0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
0

12

1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0

2

(j) A.3

4

6
8
Time [sec]

10

12

2

(k) A.4

4

6
8
Time [sec]

10

Ours
Photometric
Edge

2

4

2

4

(l) A.56

8

6
8
Time [sec](m) A.6

10

10Translation Error [m]

1.5

0.0
0

0.05

Ours
Photometric
Edge

16
14
12
10
8
6
4
2
0
0

2

Rotation Error [deg]

12

0.12
0.10
0.08
0.06
0.04
0.02
0.00
0
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0

Translation Error [m]

10

Ours
Photometric
Edge

Rotation Error [deg]

8

0.10

0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
0

Translation Error [m]6Ours
Photometric
Edge

0.15

Rotation Error [deg]

4

0.10

Translation Error [m]

2

0.15

Rotation Error [deg]

0.00
0

0.20

0.20

Rotation Error [deg]

0.02

Ours
Photometric
Edge

Translation Error [m]

0.04

0.25

Rotation Error [deg]0.06

Translation Error [m]Ours
Photometric
Edge

0.08

Rotation Error [deg]Translation Error [m]I. room0 in the Replica dataset
0.10

80
70
60
50
40
30
20
10
0
0

2

12

12

Ours
Photometric
Edge

4

6

8

10

12

4

6
8
Time [sec]1012(n) A.7

25

30

35

40

5

10

15

20

5

10

15 20 25
Time [sec](o) L.1

30

35

40

5

10
Time [sec]1520(p) L.2

Ours
Photometric
Edge

5

10

15

20

25

30

35

5
4
3
2
1
0
0

Translation Error [m]

20

0.12
0.10
0.08
0.06
0.04
0.02
0.00
0

0.30
0.25
0.20
0.15
0.10
0.05
0.00
0

Rotation Error [deg]

15

Ours
Photometric
Edge

Translation Error [m]10Translation Error [m]

5

0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
0
8
7
6
5
4
3
2
1
0
0

Rotation Error [deg]

7
6
5
4
3
2
1
0
0

Ours
Photometric
Edge

Rotation Error [deg]Rotation Error [deg]Translation Error [m]II. apartment0 in the Replica dataset
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
0

5

10

15
20
Time [sec]

25

30

35

(q) L.3

Ours
Photometric
Edge

5

10

15

20

25

10
15
Time [sec]

20

25

10
8
6
4
2
0
0

5

(r) L.4

III. An indoor environment scanned using a LiDAR
Fig. 5. Translational and rotational RMS errors: Overall, compared with the conventional methods, C ∗ provided more accurate and robust localization results.
Refer to Table II for the map and image sequence used in each experiment index.

of estimates against the ground truth. As in the previous
experiment, significant appearance changes made it difficult
for the photometric-based method [5] to evaluate the difference
between the input and rendered images, thereby resulting in
immediate tracking failure. However, edge-based method [11]and the proposed method successfully estimated the camera
poses by respectively determining the co-occurrence of edges

in both domain, and semantic labels denoted as brightness andtexture intensities by leveraging NID. Notably, the localization
accuracy of our method reduced at the point indicated by thebroken lines in Fig.5(g), and that will be discussed later in
Section V.
In addition, we performed similar experiments using another3D map and image sequences (see Fig.4(i)–(p)), and the

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
OISHI et al.: C* : CROSS-MODAL SIMULTANEOUS TRACKING AND RENDERING FOR 6-DOF MONOCULAR CAMERA LOCALIZATION BEYOND MODALITIES

7

input

renderedL.1: Textured map versus camera image
input

renderedL.2: Textured map versus camera image in dim light
input

renderedL.3: Textured map versus camera image with a dynamic obstacle
input

renderedL.4: Reflectance map versus camera image
Fig. 6. Images rendered from the poses estimated using C ∗ based on the corresponding input images in the indoor evaluation. C ∗ successfully localized themonocular camera even in cases with appearance changes due to sensor properties, lighting conditions, a dynamic obstacle, and different modalities.

corresponding results are shown in Fig.5(h)–(n) and Table
II(A.1)–(A.7).
The localization frequency of C ∗ was approximately 12.5
Hz, which achieved faster localization than that achieved by
2 Hz employed in the previous NID tracking [17] [18], owing
to the STAR algorithm, which enabled real-time and robust
visual localization. In our implementation, the average time of
keyframe update including rendering and the associated dataprocessing was, respectively, 9.36[ms] and 16.3[ms] in room0
and apartment0, and we updated keyframes only 34 and 56
times while the number of total frames on each trajectory was
170 and 289.
C. Evaluation in a LiDAR map
To evaluate our method in a real environment, we constructed a 3D map using a LiDAR Focus3D (FARO Technologies, Inc.), which can capture not only the RGB colors
but also near-infrared intensity (laser reflectance) of each 3D
point. In addition, we used a digital camera, Flea3 (FLIR
Systems, Inc.), to capture image sequences to be tested for
6-DoF localization (see Fig.4(q)–(u)). Notably, the spectral
sensitivity and wavelength are different for both the LiDAR

and digital cameras, thereby enabling us to examine thelocalization tolerance against different sensor properties ormodalities. Moreover, we estimated the ground truth posesof an agile camera by using OptiTrack (NaturalPoint, Inc.)
and, subsequently, compared the estimates from each methodwith the ground truth in the same manner as in the previous
evaluation.
Figure 5(o)–(r) and Table II(L.1)–(L.4) show the experimental results for the indoor environment. In addition, Fig. 6
depicts both the input camera images and the images renderedfrom the estimated poses in each trial. The results agree withthe previous evaluations performed using the Replica dataset.
V. CONCLUSIONSWe proposed a cross-modal monocular camera localization
method, C ∗ . Our method achieves highly reliable local tracking using NID while saving the cost of rendering and the associated data processing by occasionally updating keyframes. We
proved the validity of the proposed method via experimentswith a realistic synthesized dataset and LiDAR scan.
Future work includes improving the robustness of our
method. In conclusion, although C ∗ stably localized the

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/LRA.2020.3007120
8IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2020(a) Around the black broken line illustrated in Fig.5(g)(b) Around the black broken line illustrated in Fig.5(n)Fig. 7. Pairs of input and rendered images just before the unstable localizationof C ∗ in R.7 and A.7 : Several semantic colors have almost the same
RGB colors (white), thereby preventing the NID cost from shaping a ravine
appropriate for optimization.

camera poses in most of the experiments, it could not localize accurately or not accomplish to track a trajectorywith the semantic image sequences generated in the room0
and apartment0 datasets, as indicated by the broken lines in
Fig.5(g)(n). This implies the limitation of C ∗ : As depicted in
Fig.7, when intensity co-occurrence, here the co-occurrence of
texture and semantics, was not clearly observed, the NID could
not satisfactorily shape the converge basin, thereby resulting
in unstable local tracking. As Jeong et al. [28] suggested,
formulating a cost function as the sum of edge distance, NID,
and other available costs might be a solution to obtain abetter converge basin for global optimization. However, toapply our visual localization to autonomous navigation, wemust simultaneously consider the computational cost for realtime applications, and thus the future work also includes more
efficient cost evaluation and keyframe update [29].
R EFERENCES[1] J. Zhang and S. Singh, “LOAM: Lidar Odometry and Mapping in Realtime,” in Robotics: Science and Systems Conference, July 2014.
[2] M. Yokozuka, S. Oishi, S. Thompson, and A. Banno, “VITAMIN-E:
visual tracking and MappINg with extremely dense feature points,” in
Conference on Computer Vision and Pattern Recognition (CVPR), 2019,
pp. 9641–9650.
[3] J. L. Schönberger and J.-M. Frahm, “Structure-from-Motion Revisited,”in Conference on Computer Vision and Pattern Recognition (CVPR),
2016, pp. 4104–4113.
[4] L. Liu, H. Li, and Y. Dai, “Efficient Global 2D-3D Matching for Camera
Localization in a Large-Scale 3D Map,” in 2017 IEEE International
Conference on Computer Vision (ICCV), 2017, pp. 2391–2400.
[5] K. Ok, W. N. Greene, and N. Roy, “Simultaneous tracking and rendering: Real-time monocular localization for MAVs,” in 2016 IEEE
International Conference on Robotics and Automation (ICRA), 2016,
pp. 4522–4529.
[6] A. Sujiwo, E. Takeuchi, L. Y. Morales, N. Akai, Y. Ninomiya, and
M. Edahiro, “Localization based on multiple visual-metric maps,” in2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2017, pp. 212–219.
[7] T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, andJ. McDonald, “Real-time large scale dense rgb-d slam with volumetricfusion,” The International Journal of Robotics Research, vol. 34, pp.
598–626, April 2015.

[8] T. Caselitz, B. Steder, M. Ruhnke, and W. Burgard, “Monocular camera
localization in 3D LiDAR maps,” in 2016 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), 2016, pp. 1926–
1931.
[9] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “ORB-SLAM: A
Versatile and Accurate Monocular SLAM System,” IEEE Transactions
on Robotics, vol. 31, no. 5, pp. 1147–1163, 2015.
[10] P. Neubert, S. Schubert, and P. Protzel, “Sampling-based methods forvisual navigation in 3D maps by synthesizing depth images,” in 2017
IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), 2017, pp. 2492–2498.
[11] K. Qiu, T. Liu, and S. Shen, “Model-Based Global Localization for
Aerial Robots Using Edge Alignment,” IEEE Robotics and Automation
Letters, vol. 2, no. 3, pp. 1256–1263, 2017.
[12] D. Wong, Y. Kawanishi, D. Deguchi, I. Ide, and H. Murase, “Monocular
localization within sparse voxel maps,” in 2017 IEEE Intelligent Vehicles
Symposium (IV), 2017, pp. 499–504.
[13] Y. Lu, J. Huang, Y. Chen, and B. Heisele, “Monocular localizationin urban environments using road markings,” in 2017 IEEE Intelligent
Vehicles Symposium (IV), 2017, pp. 468–474.
[14] A. Kendall and R. Cipolla, “Geometric loss functions for camera pose
regression with deep learning,” in 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2017, pp. 5974–5983.
[15] Q. Zhou, T. Sattler, M. Pollefeys, and L. Leal-Taix, “To Learn or
Not to Learn: Visual Localization from Essential Matrices,” in IEEE
International Conference on Robotics and Automation (ICRA), 2020.
[16] R. W. Wolcott and R. M. Eustice, “Visual localization within lidar
maps for automated urban driving,” in 2014 IEEE/RSJ International
Conference on Intelligent Robots and Systems, 2014, pp. 176–183.
[17] G. Pascoe, W. Maddern, and P. Newman, “Robust Direct Visual Localisation using Normalised Information Distance,” in The British Machine
Vision Conference (BMVC), 2015, pp. 70.1–70.13.
[18] G. Pascoe, W. Maddern, A. D. Stewart, and P. Newman, “Farlap: Fastrobust localisation using appearance priors,” in 2015 IEEE International
Conference on Robotics and Automation (ICRA), 2015, pp. 6366–6373.
[19] M. Li, X. Chen, X. Li, B. Ma, and P. M. B. Vitanyi, “The similarity
metric,” IEEE Transactions on Information Theory, vol. 50, no. 12, pp.
3250–3264, Dec 2004.
[20] N. X. Vinh, J. Epps, and J. Bailey, “Information Theoretic Measures
for Clusterings Comparison: Variants, Properties, Normalization andCorrection for Chance,” The Journal of Machine Learning Research,
vol. 11, pp. 2837–2854, 2010.
[21] F. Steinbruecker, J. Sturm, and D. Cremers, “Real-Time Visual Odometry from Dense RGB-D Images,” in Workshop on Live Dense Reconstruction with Moving Cameras at the Intl. Conf. on Computer Vision(ICCV), 2011.
[22] J. Engel, T. Schöps, and D. Cremers, “LSD-SLAM: Large-Scale Direct
Monocular SLAM,” in European Conference on Computer Vision, 2014,
pp. 834–849.
[23] C. Forster, M. Pizzoli, and D. Scaramuzza, “SVO: Fast Semi-Direct
Monocular Visual Odometry,” in IEEE International Conference on
Robotics and Automation (ICRA), 2014.
[24] G. Pascoe, W. Maddern, M. Tanner, P. Pinies, and P. Newman, “NIDSLAM: Robust Monocular SLAM Using Normalised Information Distance,” in 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2017, pp. 1446–1455.
[25] S. Hutchinson, G. D. Hager, and P. I. Corke, “A tutorial on visual servo
control,” IEEE Transactions on Robotics and Automation, vol. 12, no. 5,
pp. 651–670, Oct 1996.
[26] S. Agarwal, K. Mierle, and Others, “Ceres solver,” http://ceres-solver.org(Accessed September, 15, 2019).
[27] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J.
Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge,
Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales,
T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M.
Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe,“The Replica Dataset: A Digital Replica of Indoor Spaces,” arXiv
preprint arXiv:1906.05797, 2019.
[28] J. Jeong, L. Y. Cho, and A. Kim, “Road is Enough! Extrinsic Calibrationof Non-overlapping Stereo Camera and LiDAR using Road Information,” arXiv preprint arXiv:1902.10586, 2019.
[29] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain,
J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat:
A Platform for Embodied AI Research,” in IEEE/CVF International
Conference on Computer Vision (ICCV), 2019.

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.