This paper has been accepted for publication at the IEEE International Conference on Robotics andAutomation (ICRA), Montreal, 2019. c IEEE

Event-based, Direct Camera Tracking from a
Photometric 3D Map using Nonlinear Optimization
Samuel Bryner, Guillermo Gallego, Henri Rebecq, Davide Scaramuzza
Abstract— Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes, called “events”,
instead of traditional video images. These asynchronous sensors
naturally respond to motion in the scene with very low latency(in the order of microseconds) and have a very high dynamic
range. These features, along with a very low power consumption, make event cameras an ideal sensor for fast robot localization and wearable applications, such as AR/VR and gaming.
Considering these applications, we present a method to trackthe 6-DOF pose of an event camera in a known environment,
which we contemplate to be described by a photometric 3D map(i.e., intensity plus depth information) built via classic dense 3D
reconstruction algorithms. Our approach uses the raw events,
directly, without intermediate features, within a maximumlikelihood framework to estimate the camera motion that best
explains the events via a generative model. We successfully
evaluate the method using both simulated and real data, andshow improved results over the state of the art. We release thedatasets to the public to foster reproducibility and research inthis topic.

Fig. 1. Considered scenario: the pose of the event camera is computedwith respect to a photometric 3D map of the environment, represented asa dense, colored point cloud or mesh. The event camera is represented asa green frustum whose vertex is at its optical center. The image plane of
the event camera (at the base of the frustum) displays the events, coloredaccording to polarity (red and blue). Events are caused by the apparent
motion of edges on the image plane.

SUPPLEMENTARY MATERIAL
Video: https://youtu.be/ISgXVgCR-lE
Code, Datasets: http://rpg.ifi.uzh.ch/direct event camera tracking
I. INTRODUCTION
In contrast to standard cameras, which output intensity
images at a constant rate, event cameras, such as the Dynamic Vision Sensor (DVS) [1], have independent pixels
that output only intensity changes (called “events”) at thetime they occur, with microsecond resolution. Hence, theoutput of an event camera is a stream of asynchronous
events. This bio-inspired way of sensing visual information
offers several advantages: high temporal resolution and low
latency (in the order of microseconds), a very high dynamic
range (140dB vs. 60dB of standard cameras), lack of motion
blur (since pixels are independent of each other), and low
power and bandwidth requirements. Hence, event cameras
have a large potential for robotics and wearable applications
in challenging scenarios for standard cameras, such as highspeed and high dynamic range. Recent plans for mass
production claimed by companies, such as Samsung [2] andProphesee, highlight that there is a big commercial interestin exploiting these new vision sensors for mobile robotic aswell as augmented and virtual reality (AR/VR) applications.
The authors are with the Robotics and Perception Group, Depts. ofInformatics and Neuroinformatics, University of Zurich and ETH Zurich,
Switzerland, http://rpg.ifi.uzh.ch. This work was supported by the Swiss
National Center of Competence Research Robotics, through the Swiss National Science Foundation, the SNSF-ERC Starting Grant and the Qualcomm
Innovation Fellowship Award 2018.

Motivated by these recent developments, in this work, wetackle the problem of tracking the six degree-of-freedom(6-DOF) pose of an event camera in a known environment,
as illustrated in Fig. 1. This is the typical scenario of robots
traversing previously-mapped spaces, and AR/VR applications (users wear head mounted displays and move freely
in the previously-mapped environment). Event cameras offer
great potential in this scenario, since they excel at sensing
motion, and they do so with very low latency and consuming
very little power. We envision that the mobile robot or user
(AR/VR) would first use a standard sensor to build a highresolution and high quality map of the environment, and thenwould take advantage of an event camera to achieve robustness to high-speed motion and low-power consumption.
Because the output of event cameras is fundamentallydifferent from that of standard cameras, traditional visionalgorithms cannot be applied, and so, new methods are
required. Previous work [3] presented a filter-based approachthat requires to maintain a history of past camera poses which
are continuously being refined. This led to an unnecessarily
complex scheme to update many unknowns: the full historyof camera poses. In contrast, we focus on designing a simple
yet principled scheme for camera tracking. To this end,
we develop a maximum likelihood approach to explain theobserved events based on a generative model of them given
the map of the environment. This results in a nonlinear optimization approach, as opposed to filtering [3], that allows toachieve accurate camera tracking without having to maintaina history of past poses.

Our method works by comparing the events acquiredby the camera to predictions of them, as they would be
generated according to the map of the scene and candidate
values for the camera pose and its velocity. The comparison
error, defined by means of an objective function over theentire image plane, drives the estimation of the unknowns of
the problem: the pose of the event camera. Our framework
allows to take advantage of the asynchronous, high dynamic
range, and low-latency nature of the events to produce lowlatency camera poses with high temporal resolution.
In summary, our contributions are the following:
A novel method for 6-DOF camera tracking with an
event camera given a photometric 3D map of the scene.
We (i) leverage a generative model to explain how
events are related to intensity patterns in the map, fully
exploiting the strength of the intensity gradients causing
the events, and (ii) simultaneously estimate the camera
pose as well as its linear and angular velocities.
• We thoroughly evaluate the proposed pose tracker on
a diversity of scenes and provide a comparison with astate-of-the-art method [3], showing that our approachprovides more accurate camera poses.
• We release the datasets used in the experiments, including the acquired images, ground truth camera trajectories, as well as the built photometric maps of the scenes,
to foster reproducibility and research in this topic.
•

Outline: The rest of the paper is organized as follows:
Section II reviews the related work on event-based posetracking, Section III presents our approach, consisting of a
maximum likelihood formulation of the observed events interms of the camera pose, its velocity and the photometric 3D
map of the scene. Experiments are carried out and discussed
in Section IV to assess the performance of our method, andfinally, conclusions are drawn in Section V.
II. RELATED WORK
Since the invention and commercialization of the Dynamic
Vision Sensor (DVS) in 2008 [1], the problem of ego-motion
estimation with event cameras has been addressed by multiple researchers. Event-based SLAM has been investigatedin [4]–[6], and extended to 6-DOF motions in [7]–[13]. Thesesystems, however, target a different scenario than the one weaddress, namely, they focus on simultaneous map building
and localization, which serves to constantly explore new
regions of the environment.
In this work, we focus on localization with respect toa given map. This has important applications in robotic
navigation (robot moving in a known environment, such as awarehouse) and augmented or virtual reality. Hence, the mapof the environment needs to be built only occasionally (e.g.,
if new objects appear in the scene). Our work is most related
to [3], where the authors propose a probabilistic filtering
approach to track the 6-DOF pose of the event camera innatural scenes. They use a photometric map of the sceneconsisting of a sparse set of keyframes (poses, intensity
images and depth maps), and consider a robust likelihood

function for the generation of the events in terms of theintensity images.
From a high-level point of view, both [3] and this worktarget the same scenario, and therefore, share the same
inputs (events, photometric map) and outputs (camera poses).
The differences between them lie in the solution procedure,
where [3] adopts a filtering approach, whereas we propose anonlinear least-squares optimization approach.
Specifically, the method in [3] operates on an event-byevent basis, updating the filter state on every incoming event,thus virtually eliminating latency. This requires to keep a history of past camera poses, which are refined (included in thefilter state) as newer events are processed. Past camera poses
are utilized to compute the predicted intensity change (i.e.,“contrast”) of the event, according to a per-event generation
model. However, having to update or interpolate poses ona per-event basis has a non-negligible computational cost,
regardless of the cost of later processing stages. In contrast,
our approach trades off some latency for a simpler event
generation model that does not require any reference to pastposes. This is sensible because localization does not need tohave the granularity of an event (1 microsecond); it can havethe granularity of a few events (within 1 millisecond or less).
Hence, we process events in small groups (or “windows”) to
produce a pose. This grouping is not a drawback since we
could still produce a pose for every incoming event by sliding
the window by a single event. We define the likelihood ofa group of events given a pose and seek its maximization,
leading to an equivalent, nonlinear least squares problem.
Additionally, in contrast to previous approaches, we jointly
estimate the camera pose as well as its velocity (linear andangular).
III. METHODOLOGYOur method is inspired by [14], where the principledidea of predicting intensity changes and comparing them tothose given by actual events was used for feature tracking.
Here, instead, we adapt the framework and apply it to
the estimation of quantities in 3D space: the 6-DOF poseof the event camera and its velocity. In a nutshell, wecast the pose estimation problem as that of registering two
intensity-change images that are related by a complex, depthdependent geometric transformation. In the following, we
present the two images being registered and the objectivefunction used to compare them. Our method is summarized
in Fig. 2, which illustrates how to compute intensity-change
images using the room scene of Fig. 8.
A. Event-Camera Working Principle
Each pixel of an event camera produces an event ek =(uk , tk , pk ) whenever it detects that the logarithmic luminance L at that pixel, uk = (xk , yk )⊤ , changes by a specified
amount C (called contrast sensitivity) [1]:.
∆L(uk , tk ) = L(uk , tk ) − L(uk , tk − ∆tk ) = pk C, (1)where the polarity pk = {+1, −1} is the sign of thebrightness change, and ∆tk is the time elapsed since the last

Input data
Operation

Legend:E Events

Intermediate results
Optimization parameters∆t

dt integration

C. Intensity-Change Image from the Map and Sensor Pose

M photometric 3D map
Rendering using GPU

R

where the Kronecker delta δ selects the pixel lattice. This is
illustrated on the left branch of the block diagram in Fig. 2.
Pixels of L(u) that do not change intensity are represented
in gray in ∆L, whereas pixels that increased or decreased
intensity are represented in bright and dark, respectively.

L(u) intensity

Z(u) depth

Gradient ∇Optic flow

∇L(u) gradientv(u) optic flow

T poseṪ velocity

Dot product∆L(u) measured
intensity change∆L̂(u) predicted
intensity change

Normalization and subtraction

Error image
Fig. 2. Block diagram of the proposed objective function, which includes
the generative event model. The left branch illustrates how the observed
intensity-change image is computed from the events (2). The right branchshows how the predicted intensity-change image (8) is computed fromthe photometric map and the camera motion (pose T and velocity Ṫ ).
The difference between both intensity-change images is used to define alikelihood function (or, equivalently, an objective function) to be optimized
with respect to the event camera motion parameters.

event at the same pixel. The event ek is a tuple consistingof the space-time coordinates (on the image plane) of theintensity change and its polarity. Event timestamps tk havemicrosecond resolution.
B. Intensity-Change Image obtained from the Events
As anticipated in Section II, we process events in groups.
e
E = {ek }N
k=1 , spanning a small time interval ∆t = tNe − t1 .
We pixel-wise collect the event polarities over ∆t on thewhole image plane, producing an image ∆L(u) with theamount of intensity change that occurred during the interval:
X
pk C δ(u − uk ),(2)∆L(u) =
tk ∈∆t

The photometric 3D map of the environment is used to
produce a prediction of the events that would be generated
if the camera moved with some velocity. First, let us define,based on (1), a linearized event generation model, andthen let us show how its elements can be computed using
the information available: photometric map and candidate
camera pose and velocity.
a) Linearized Event Generation Model: For small ∆t,such as in the above-mentioned example of Fig. 2, theintensity increments (2) are due to moving edges, as we show
next. Substituting the brightness constancy assumption (i.e.,
optical flow constraint)∂Lu(t), t + ∇L u(t), t · u̇(t) = 0,(3)∂twith image-point velocity v ≡ u̇, in Taylor’s approximation ∂L.
(u, t)∆t,∆L(u, t) = L(u, t) − L u, t − ∆t ≈∂tgives (see [15])∆L(u) ≈ −∇L(u) · v(u)∆t,

(4)

(5)

that is, increments (2) are caused by intensity gradients.
∂L ⊤∇L = ( ∂Lmoving with velocity v(u) over a dis∂x , ∂y .)placement ∆u = v∆t. As the dot product in (5) conveys, if
the motion is parallel to the edge (v ⊥ ∇L), the increment
vanishes, and, therefore, no events are generated. From now
on, let us denote the modeled intensity change (5) using ahat, ∆L̂. This is illustrated on the right branch of the diagramin Fig. 2.
b) Image ∆L̂ in terms of the Map and the Camera
Pose: The elements on the right-hand side of (5), namely theintensity gradient ∇L and the image-point velocity v (i.e.,the motion field, also called “optic flow” in spite of theirdifferent nature) can be computed from the photometric mapof the scene and the unknowns of the problem.
The image gradient ∇L (5) is obtained by differentiating
the intensity image produced by projecting the photometricmap M onto a candidate viewpoint, specified by pose T(see Fig. 2). The motion field v in (5) is purely geometric,
given in terms of the candidate camera pose T , its linear.
and angular velocities Ṫ = (V⊤ , ω ⊤ )⊤ , and the depth of
the scene Z ≡ Z(u) with respect to the camera, accordingto [15], [16]:v(u) = J(u) Ṫ ,(6)
with the 2 × 6 feature sensitivity matrix

−1/Z
0
x/Z
xy
−(1 + x2 )J(u) =
20−1/Z y/Z 1 + y
−xy


y
.
−x
(7)

Substituting (6) in (5) gives the intensity change caused by
the camera motion in 3D space,∆L̂(u) ≈ −∇L(u) · J(u)Ṫ ∆t.

(8)

Observe that the pose T and its velocity Ṫ are global
quantities shared by all pixels u of the image plane; these are
the unknowns of the problem. The rest, i.e., the intensity I
and depth Z(u) (given by the photometric map M) and thetime span ∆t (given by the event timestamps) are all known.
The depth Z(u) and the intensity gradient ∇L depend on the
projection of the map from the candidate pose T , although
this is omitted in the notation for readability.
D. Maximum Likelihood Optimization Framework
As anticipated at the beginning of the section and illustrated at the bottom of Fig. 2 (where both branches meet),
we propose to use the difference between two images: theobserved intensity changes ∆L from the events (2) and thepredicted ones ∆L̂ from the photometric map (8), to estimate
the camera motion (pose and velocity).
More specifically, assuming that the difference ∆L − ∆L̂follows a zero-mean additive Gaussian distribution withvariance σ 2 , the likelihood of a group of events E being
generated by a camera moving with respect to a given map
M is
Z
2
1
1
.
e− 2σ2 (∆L(u)−∆L̂(u;T ,Ṫ )) du.
p(E|T , Ṫ ; M) = √
2
2πσ Ω
(9)
Our goal is to find the camera pose T and velocity Ṫ thatmaximize the event likelihood (9), i.e., that best explain theevents with the proposed generative model.
Maximizing (9) with respect to the motion parameters T
and Ṫ (since M, i.e., ∇L, is known) is equivalent to theminimization of the L2 norm of the intensity-change residual,
min k∆L(u) − ∆L̂(u; T , Ṫ )k2L2 (Ω) ,

(10)

T ,Ṫ

. R
where kf (u)k2L2 (Ω) = Ω f 2 (u)du. However, (10) depends
on the contrast sensitivity C (via (2)), which is unknown inpractice. Inspired by [17], we instead propose to minimize
the difference between normalized (unit-norm) images:
min
T ,Ṫ

∆L̂(u; T , Ṫ )∆L(u)−k∆L(u)kL2 (Ω)
k∆L̂(u; T , Ṫ )kL2 (Ω)

2, (11)
L2 (Ω)

which cancels the terms in C and ∆t, and only depends onthe direction of the velocity vector Ṫ . The image registration
implied by the objective function (11) was implemented, as
is standard, using a multiresolution approach (i.e., imagepyramid) and was minimized using the nonlinear leastsquares solver in the Ceres library [18]. To gain resilience tooutliers, we replaced the simple L2 norm by robust kernels(Huber norm).
Fig. 3 shows an example of a 2D slice of the objectivefunction (11) for an experiment with real data, color codedfrom blue (small error) to red (large error). By parametrizing
the camera’s pose using local exponential coordinates (i.e.

+/- 50cm+/- 5cm+/- 0.5cm

Fig. 3. Objective function (11) (pseudo-colored from blue (low) to red(high)) for variations in the x and y translational components of the pose
T of the event camera, shown for ±50cm, ±5cm and ±5mm.

with respect to an offset rotation [19], [20]) and using thestraightforward representations for position and velocity, we
obtain a twelve-dimensional parameter space (of which weplot 2D slices). As it is observed, the landscape of theobjective function is complex, but it has a clear minimum,
reachable provided the solver is initialized in its basin of
attraction. As in [3], we assume the initial camera pose isgiven, and initialize the camera velocity randomly, with about
1 m/s magnitude. Upon convergence, the pose and velocities
are used to initialize the parameters of the optimization
corresponding to the next group of events, i.e., next intensitychange image.
E. Discussion of the Approach
As mentioned in Section II, our approach does not needto manage a history of past camera poses. This is prevented
by the linearization in the event generation model (4) (and(5)). However, for this to be valid, we require to computethe intensity change due to several events, as opposed to the
per-event generation model of previous approaches.
An interesting characteristic of our approach is that it isbased on the generative event model (8), which not only takes
into account that events are triggered at the intensity edges ofthe projected map, ∇L, but also that the triggering condition
depends on the appearance of such edges with respect tothe direction of the camera velocity (the appearance of ∆L̂significantly varies with Ṫ ). Our method not only estimatesthe event camera pose, but also its velocity, and consequently,
the optic flow. This dependency was not explicitly modeledin previous works, such as filtering approaches [3], [4], [12].
The number of events Ne used to build an intensity-change
images (2) plays an important role in the objective function.
A small Ne does not yield sufficient signal-to-noise ratioor evidence of existing edge motion to produce good pose
estimates. A large Ne increases latency, produces motion blurin the intensity-change images and breaks the assumption
about events being triggered by the camera at a “single”
location T (the poses for the first and last event become
further apart), which is also harmful for estimation. Hence,
there is a trade-off in the selection of Ne , as illustrated inFig. 4. (a reference value is 0.10 events/pixel). As shownin [14] for feature tracking, it is possible to dynamically set
Ne based on the amount of texture present in the scene. In
the experiments, we use Ne /Np = 0.20 events/pixel, which
gives about Ne ≈ 10 000 events per intensity-change image.

0.01 events/pixel

0.10 events/pixel

0.50 events/pixel

(a)

(b)

Fig. 5. Two of the synthetic scenes: (a) toy room depicts some textured
boxes on a carpet; (b) atrium consists of a virtual model of the Atrium
Sponza Palace in Dubrovnik.
TABLE IM EDIAN ACCURACY OF TRACKING OF ALL EXPERIMENTS .
Velocity Direction [◦ ]

Pose
Scene

Fig. 4. Effect of varying the integration time ∆t, or, equivalently, thenumber of accumulated events Ne . Columns are sorted according to theincreasing value of Ne /Np , where Np is the number of pixels of the image.
Top row: intensity image I(u) with overlaid events (in red and blue, coloredaccording to polarity). Middle row: intensity-change images ∆L(u) in (2),obtained by pixel-wise accumulation of event polarities. Bottom row: 2D
slice of the 12-D objective function across the X and Y translation axes(±3 cm). A yellow plus (+) indicates the ground truth pose at the centerof the event integration window ∆t; red crosses (×) indicate the groundtruth poses at the beginning and end of the integration window.

IV. EXPERIMENTS
To assess the accuracy of our method, we first evaluate
it on synthetic data, where we are able to control scene
parameters (depth, illumination, etc.) and we have perfectknowledge of the event camera trajectory. Then, we test our
method on real data acquired in an indoor scene. There,
the ground truth camera trajectory is provided by a motioncapture system with sub-millimeter accuracy. We releaseour datasets to the public at http://rpg.ifi.uzh.ch/direct event
camera tracking.
A. Evaluation Metrics
The pose estimation error is measured in both positionand orientation. The position error is given by the Euclidean
distance between the ground truth and the estimated event
camera trajectories (position of the optical center). We also
report the relative position error with respect to the mean
scene depth, as in [3]. The orientation error is measured
using the geodesic distance in the rotation group SO(3) [21],which is the angle of the residual rotation between theestimated pose and the ground truth. Since the velocity of
the camera Ṫ can only be estimated up to a non-zero scale
factor, to compare two velocity vectors Ṫ we use the angle
between them, using the dot product.
B. Synthetic Data. Assessing Tracking Accuracy
Using simulated data, we validate our method and assess
its tracking accuracy. We used the event camera simulator [22], [23] on several scenes with different types ofAve. Depth [m] Pos. [cm] Orient. [◦ ] Linear

Angularcarpetcarpet (D)toy room
toy room (D)atriumatrium (D)

2.11
2.11
1.99
1.99
10.60
10.60

0.73
1.12
0.45
0.89
6.53
6.90

0.16
0.21
0.20
0.28
0.43
0.52

23.56
20.24
17.54
17.53
9.58
11.17

17.76
16.17
62.69
61.17
5.47
6.04

room Traj. 1
room Traj. 2
room (D)

3.20
3.41
3.41

9.95
8.82
9.27

3.08
3.84
3.63

n/a
n/a
n/a

n/a
n/a
n/a

(D): downsampled intensity-change images.

textures and objects: carpet, toy room and atrium. The
carpet scene consists of a flat, textured surface. It is aninteresting scenario since pose estimation can be ambiguous
for planar scenes (rotational and translational motions have
similar motion fields), which is more severe if the fieldof-view (FOV) of the camera is limited. The toy room
scene consists of a couple of boxes with natural textures(e.g., plant leaves) on the same flat surface as carpet(see Fig. 5a). The non-flatness of the scene prevents theappearance of pose ambiguities. Finally, the atrium scene
is a famous computer graphics virtual model of the Atrium
Sponza Palace in Dubrovnik. It presents rock-like textures,with strong intensity edges mostly due to boundary of objects(columns, windows, doors, etc.), as shown in Fig. 5b.
In the synthetic scenes, we generated smooth camera
trajectories by fitting splines to randomly generated poses.
Events were triggered as the camera moved using the simulator [23], with a 240 × 180 pixel resolution, thus simulating
the event camera within the DAVIS240C device [24]. Then,we run our pose tracking algorithm, feeding as photometric
3D map the virtual model of the scene (i.e., a colored meshcomprising the surface of all objects in the scene). The
resulting pose tracking errors of our method are summarized
in the top half of Table I.
As can be observed in Table I, errors across all simulated scenes are remarkably small: less than 0.65 % in
relative position error (with respect to the mean depth of thescene) and less than 0.52◦ in orientation error. Moreover,tracking in the planar scene (carpet) does not produce
motion ambiguities, e.g., x-translation is properly estimated,disambiguated from a y-rotation, in spite of the limited FOV

position [m]

1.0
0.5

Z
X

0.0
Y
–0.5
trackedgroundtruth

–1.0
0.0

2.5

5.0

7.5

10.0
12.5
time [sec]

15.0

17.5

20.0

Fig. 6. Pose tracking results over time for a smooth random motion over aplanar texture (carpet scene). Tracking does not suffer from ambiguities.
The median position error is 0.73 cm, which corresponds to a relative errorof 0.34 % with respect to the mean scene depth. The median orientation
error is 0.16◦ (see Table I).

(a) Photometric information (color)

5

position [m]

Z
0

X

–5

–10

trackedgroundtruth

Y
0

2

4

6

8

(b) Geometric information (shape)

10time [sec]Fig. 7. Pose tracking results for a simulated flight trough the atrium
scene in Fig. 5b. The median position error is 6.53 cm, which amounts toa relative error of 0.62 % with respect to the mean scene depth (10.60 m).
The median orientation error is 0.43◦ .

Fig. 8. room dataset. Photometric map used, generated with ElasticFusion [25]. Some artifacts in texture-less areas are present due to autoexposure and white-balance of the consumer-grade RGB-D camera used.

C. Real Data◦◦of the camera (60 horizontally and 50 vertically). Fig. 6
displays the translation of the camera in this planar scene.
As it is observed, the estimated trajectory (solid lines) andground truth one (dashed lines) are almost indistinguishable
compared to the excursion of the motion.
The atrium experiments present the largest absolute
errors since the scene is considerably larger than the previous
two. However, in terms of relative error with respect to themean scene depth, errors are of the same order of magnitude
as those of the experiments in the carpet and toy room
scenes. Fig. 7 shows both the estimated trajectory of theevent camera moving through the scene (solid line) and theground truth trajectory (dashed line). The curves are almost
on top of each other since errors are small compared to theamount of motion.
Supplementary Material: We encourage the reader towatch the accompanying video, which shows the experiments
here presented in a better form than still images can convey.
Using Downsampled Intensity-Change Images: We also
tested the pose tracking algorithm using lower resolution
intensity-change images. As mentioned in Section III, themethod is implemented using a multiresolution approach.
In these experiments, we registered intensity-change images∆L and ∆L̂ at the second highest-resolution level of theimage pyramid (e.g., 120 × 90 pixels for the DAVIS240C).
Such experiments are marked as “(D)” in Table I. This
modification had a minimal effect on accuracy loss (Table I)
while it significantly sped up computation time, as reported
in Table II.

Experiments were also conducted on real scenes. The
event camera was moved hand-held in a room comprising
texture as well as large white walls (i.e., texture-less). For thisexperiment, we used the DAVIS346 (346 × 260 pixels) frominiVation. Ground truth poses were provided by a motioncapture system. The room was pre-mapped using a consumer
RGB-D camera (ASUS Xtion Pro) and running open-source
software ElasticFusion [25]. For improved accuracy of the
3D reconstructed map, we provided to ElasticFusion poses
from the motion capture system. The resulting color andshape of the 3D-reconstructed photometric map are shownin Figs. 8a and 8b, respectively. Auto-exposure and whitebalance were enabled during recording with the RGB-D
camera, leading to some unseemly intensity edges in the map
in what are actually uniform white walls (see Fig. 8a).
The results of our pose tracking method on a couple oftrajectories of the event camera under 6-DOF motion in thisroom environment are reported in the bottom half of Table I.
In this scenario tracking is more challenging than inprevious scenarios for a number of reasons: (i) the presenceof uniformly white walls and a floor with high-frequency
texture, (ii) the presence of noise in both inputs: the eventsand the photometric map, (iii) the presence of inaccuracies
in the calibration of the event camera and the hand-eye
calibration with respect to the motion-capture system. In
spite of these source of errors, our method is still able to track
gracefully, with relative position errors of 2.58 % to 3.10 %and orientation errors of 3.08◦ to 3.84◦ . A visualization of
the pose tracking results (events aligned with the reprojectedTABLE IIC OMPUTATION TIME REQUIRED FOR POSE TRACKING .
Processing

Fig. 9. Pose tracking visualization on one of the sequences of the room
dataset (Fig. 8). Events overlaid on the projected map from the viewpointof the estimated camera pose. Events are colored in red and blue, according
to polarity (positive in blue, negative in red). The alignment of the majorityof the events with respect to the intensity edges of the projected map is agood indicator of the accuracy of the estimated camera pose.

carpetcarpet (D)toy room
toy room (D)atriumatrium (D)

20
20
20
20
10
10

2857
12 789
3160
6433
4130
11 532

24.66
17.01
25.94
27.59
12.23
17.34

120.0638.52
132.2659.59
166.0741.01

3024.70675.703418.50
1679.30
2098.60
751.60room Traj. 1
room Traj. 2

8.32
43.24

995
1016

64.63
90.31

359.54315.0822 704.00
22 139.20(D): downsampled intensity-change images.
The upper half are simulated environments.
These results were obtained on a single Intel i7-870 core.
TABLE IIIC OMPARISON OF ROOT- MEAN - SQUARE (RMS) POSE ERRORSWITH RESPECT TO STATE - OF - THE - ART.

Gallego et al. [3]XThis work

Length Position Orientation Position Orientation[s][cm] [%][◦ ][cm] [%][◦ ]trackedgroundtruthboxes 1boxes 2boxes 3
pipe 1pipe 2

Y
Z
0

1

2

3

4
time [sec]

5

6

7

23.3
26.7
33.7
29.8
22.2

5.08
4.04
5.47
10.96
15.26

2.69
2.15
2.90
4.04
5.34

2.51
2.18
2.82
2.90
4.68

4.74
4.46
5.05
10.23
11.29

2.52
2.38
2.68
3.77
3.95

1.86
2.10
2.39
2.13
4.02

8

Fig. 10. Pose tracking results on the room scene (Traj. 1) using thephotometric map in Fig. 8. The median position error is 9.95 cm withrespect to a mean scene depth of 3.20 m, that is, a 3.10 % relative error.
The median orientation error is 3.08◦ .

photometric map) is given in Fig. 9. The estimated andground truth trajectories of one of the sequences are displayed in Fig. 10. Compared to noise-free synthetic data, weobserve the decrease in tracking accuracy in the presence of
the above-mentioned noise sources.
D. Computational Load
We also report the computational effort required by our
non-optimized C++ implementation of the proposed approach. Timing results are presented in Table II for both
simulated and real data. The current implementation, built
on top of Ceres [18], is one to three orders of magnitudes
slower than real-time, depending on the settings. Making
the approach real-time capable is of course essential forany practical application. As mentioned in Section IV-B,
using lower resolution intensity-change images is an option
to speed up the algorithm with a minor impact on accuracy
loss. To further reduce pixel count one could heuristicallyfilter pixels and only keep those that contribute significantly
to the error. On the implementation side there is room for
optimization using parallelization: the pixels are independent,
only the last step of solving (11) is inherently sequential. Our
method would thus be ideally suited for a more distributed
platform, such as a GPU (currently, all computation is doneon the CPU, except for the rendering of the map, where

4

Gallego et al.
This work

3
2
1
0

RMS

Mean

Std

4

Orientation error [deg]0

Position error [%]position [m]1–1

Full Tracking Step

Length [s] Events/sec Iter./step ms/iter. total [ms]Scene

Gallego et al.
This work

3
2
1
0

RMS

Mean

Std

Fig. 11. Comparison with [3, Fig. 13]. Error in position (relative to a mean
scene depth of 1.90 m) and orientation (in degrees) of the camera poses
recovered by the method in [3] (in red) and by our method (in blue) forthe boxes sequences from [3] (ground truth is given by a motion-capturesystem). We provide box plots of the root-mean-square (RMS) errors, themean errors and the standard deviation (Std) of the errors.

the GPU is mainly a convenience), which was used in otherworks [12], [26], [27].
E. Comparison with State of the Art
We compared our method against the state-of-the-art [3].
To this end, we run our pose tracking algorithm on the samedatasets used in [3], and compared the resulting tracking
errors. These are reported in Table III. As it can be observed,
our method offers improved accuracy over [3], specially inorientation estimation. Smallest errors per row, per error
type are highlighted in bold. Orientation error improves, on
average, by 15 %, and position error, by 7 %. Differences aremost notorious in the outdoor (pipe) sequences. Both errors

are related, as it is well known that errors in orientation
at some time t produce worse translation estimates at latertimes, and vice-versa. Additional comparison on more boxes
sequences of [3] is provided in Fig. 11. Our method presents
more concentrated error statistics than [3].
F. Discussion
The previous experiments demonstrate that the proposed
method accurately tracks the pose of the event camera inboth simulated and real-world environments. In simulation,
position errors are smaller than 0.65 %, and orientation errors
smaller than half a degree. These errors are very small
and validate our method with respect to noise-free event
data. The resulting pose tracking errors are imputable tomodeling errors (such as the linearization (4), the approximation of intensity gradients using finite differences, aswell as non-explicitly modeled effects, e.g., occlusions anddisocclusions). While poses are estimated very accurately,
camera velocities are not as accurately estimated, which isdue to velocities being very sensitive to error sources, such asmodeling errors, as can be seen with perfect data (noise-free
events and map).
The above tracking errors increase to about 3 % and 4◦ ,respectively, in real-world sequences. The accuracy gap
between simulation and real world experiments is due toadditional error sources: (i) noise in the photometric map
reconstructed from a noisy RGB-D camera, (ii) noise andnon-idealities of event cameras [1], (iii) small delays among
the sensors, and (iv) inaccuracies in internal and external
calibration of all sensors involved to produce the “groundtruth” data used for evaluation. In spite of all these factors,pose tracking still performs well in real-world conditions.
V. CONCLUSION
We have presented a method to track the 6-DOF pose of anevent camera with respect to a photometric 3D map, which isa paramount scenario in robotics and AR/VR scenarios. Our
method leverages a principled event generation model withina maximum-likelihood framework to jointly estimate thecamera poses and velocities, through nonlinear optimization,
fully exploiting the strength of the intensity gradients in thescene. A thorough evaluation on both synthetic and real data
proves that our method provides compelling accuracy, andit improves over the state-of-the-art filtering approach [3]while being simpler to formulate. A key characteristic of
our method is that it uses the events, directly, without intermediate hand-crafted features, thus exploiting as much of theinformation contained in the events as possible. Additionally,
we release all our datasets with ground truth poses, in aneffort to foster reproducibility and research in this topic.
R EFERENCES[1] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128×128 120 dB 15µs latency asynchronous temporal contrast vision sensor,” IEEE J.
Solid-State Circuits, vol. 43, no. 2, pp. 566–576, 2008.
[2] B. Son, Y. Suh, S. Kim, H. Jung, J.-S. Kim, C. Shin, K. Park, K. Lee,
J. Park, J. Woo, Y. Roh, H. Lee, Y. Wang, I. Ovsiannikov, and H. Ryu,
“A 640x480 dynamic vision sensor with a 9um pixel and 300Meps
address-event representation,” in IEEE Intl. Solid-State Circuits Conf.
(ISSCC), 2017.

[3] G. Gallego, J. E. A. Lund, E. Mueggler, H. Rebecq, T. Delbruck,and D. Scaramuzza, “Event-based, 6-DOF camera tracking fromphotometric depth maps,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 40, no. 10, pp. 2402–2412, Oct. 2018.
[4] H. Kim, A. Handa, R. Benosman, S.-H. Ieng, and A. J. Davison,“Simultaneous mosaicing and tracking with an event camera,” inBritish Mach. Vis. Conf. (BMVC), 2014.
[5] A. Censi and D. Scaramuzza, “Low-latency event-based visual odometry,” in IEEE Int. Conf. Robot. Autom. (ICRA), 2014, pp. 703–710.
[6] C. Reinbacher, G. Munda, and T. Pock, “Real-time panoramic tracking
for event cameras,” in IEEE Int. Conf. Comput. Photo. (ICCP), 2017.
[7] D. Weikersdorfer, D. B. Adrian, D. Cremers, and J. Conradt, “Eventbased 3D SLAM with a depth-augmented dynamic vision sensor,” inIEEE Int. Conf. Robot. Autom. (ICRA), 2014, pp. 359–364.
[8] B. Kueng, E. Mueggler, G. Gallego, and D. Scaramuzza, “Low-latency
visual odometry using event-based feature tracks,” in IEEE/RSJ Int.
Conf. Intell. Robot. Syst. (IROS), 2016, pp. 16–23.
[9] E. Mueggler, G. Gallego, and D. Scaramuzza, “Continuous-time
trajectory estimation for event-based vision sensors,” in Robotics:
Science and Systems (RSS), 2015.
[10] H. Rebecq, T. Horstschäfer, G. Gallego, and D. Scaramuzza, “EVO:
A geometric approach to event-based 6-DOF parallel tracking andmapping in real-time,” IEEE Robot. Autom. Lett., vol. 2, no. 2, pp.
593–600, 2017.
[11] H. Rebecq, G. Gallego, E. Mueggler, and D. Scaramuzza, “EMVS:
Event-based multi-view stereo—3D reconstruction with an event camera in real-time,” Int. J. Comput. Vis., vol. 126, no. 12, pp. 1394–1414,
Dec. 2018.
[12] H. Kim, S. Leutenegger, and A. J. Davison, “Real-time 3D reconstruction and 6-DoF tracking with an event camera,” in Eur. Conf. Comput.
Vis. (ECCV), 2016, pp. 349–364.
[13] G. Gallego, H. Rebecq, and D. Scaramuzza, “A unifying contrast
maximization framework for event cameras, with applications tomotion, depth, and optical flow estimation,” in IEEE Conf. Comput.
Vis. Pattern Recog. (CVPR), 2018, pp. 3867–3876.
[14] D. Gehrig, H. Rebecq, G. Gallego, and D. Scaramuzza, “Asynchronous, photometric feature tracking using events and frames,” inEur. Conf. Comput. Vis. (ECCV), 2018.
[15] G. Gallego, C. Forster, E. Mueggler, and D. Scaramuzza, “Eventbased camera pose tracking using a generative event model,” 2015,
arXiv:1510.01972.
[16] P. Corke, Robotics, Vision and Control: Fundamental Algorithms in
MATLAB, ser. Springer Tracts in Advanced Robotics. Springer, 2017.
[17] G. D. Evangelidis and E. Z. Psarakis, “Parametric image alignment
using enhanced correlation coefficient maximization,” IEEE Trans.
Pattern Anal. Mach. Intell., vol. 30, no. 10, pp. 1858–1865, Oct. 2008.
[18] S. Agarwal, K. Mierle, and Others, “Ceres solver,” http://ceres-solver.
org.
[19] Z. Zhang, G. Gallego, and D. Scaramuzza, “On the comparison ofgauge freedom handling in optimization-based visual-inertial state
estimation,” IEEE Robot. Autom. Lett., vol. 3, no. 3, pp. 2710–2717,
Jul. 2018.
[20] G. Gallego and A. Yezzi, “A compact formula for the derivative of a 3D rotation in exponential coordinates,” J. Math. Imaging Vis., vol. 51,
no. 3, pp. 378–384, Mar. 2015.
[21] D. Q. Huynh, “Metrics for 3D rotations: Comparison and analysis,”J. Math. Imaging Vis., vol. 35, no. 2, pp. 155–164, 2009.
[22] E. Mueggler, H. Rebecq, G. Gallego, T. Delbruck, and D. Scaramuzza,“The event-camera dataset and simulator: Event-based data for pose
estimation, visual odometry, and SLAM,” Int. J. Robot. Research,
vol. 36, no. 2, pp. 142–149, 2017.
[23] H. Rebecq, D. Gehrig, and D. Scaramuzza, “ESIM: an open event
camera simulator,” in Conf. on Robotics Learning (CoRL), 2018.
[24] C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck, “A
240x180 130dB 3us latency global shutter spatiotemporal vision
sensor,” IEEE J. Solid-State Circuits, vol. 49, no. 10, pp. 2333–2341,
2014.
[25] T. Whelan, S. Leutenegger, R. F. Salas-Moreno, B. Glocker, and A. J.
Davison, “ElasticFusion: Dense SLAM without a pose graph,” inRobotics: Science and Systems (RSS), 2015.
[26] P. Bardow, A. J. Davison, and S. Leutenegger, “Simultaneous opticalflow and intensity estimation from an event camera,” in IEEE Conf.
Comput. Vis. Pattern Recog. (CVPR), 2016, pp. 884–892.
[27] A. Z. Zhu, Y. Chen, and K. Daniilidis, “Realtime time synchronized
event-based stereo,” in Eur. Conf. Comput. Vis. (ECCV), 2018.