Event-IMU fusion strategies for faster-than-IMU estimation throughput
William Chamorro1,2
Joan Solà1
Juan Andrade-Cetto1
1
Institut de Robòtica i Informàtica Industrial CSIC-UPC, Barcelona, Spain
2
Universidad UTE, Quito, Ecuador{wchamorro, jsola, cetto}@iri.upc.edu, william.chamorro@ute.edu.ec

Abstract
This study presents new methods for integrating event
data and IMU readings to achieve ultra-fast camera pose
estimates. The conventional predict-with-IMU-correctwith-vision approach is no longer optimal because events
can be generated much more rapidly than IMU data. Therefore, two novel fusion schemes are proposed, which combine constant velocity and constant acceleration prediction
models with ultra-fast (10 kHz) event-based updates andslower 1 kHz IMU updates. The first scheme uses IMU
data as instantaneous measurements of acceleration andangular rate, while the second scheme considers these measurements as the average within the IMU sampling time.
To provide a basis for comparison, the traditional methodthat predicts motion using IMU data and updates the estimates with event-feature matching over a 1ms time windowis also implemented. All models are designed as Kalmanfilter variants, which act as the tracker module of a PTAM
system in a human-made indoor scenario and are subjected
to stress experiments to evaluate their capabilities. The
models are also compared against an event-only estimatorand a frame-based visual-inertial approach. The findings
demonstrate superior performance at a throughput that is
100 times faster than the state-of-the-art.

1. Introduction
Event cameras have emerged as novel sensors that canhandle high dynamic motion and challenging lighting conditions in computer vision since they were commerciallyavailable in 2008 [1, 16]. With the technological advancements in event cameras, there is a need to explore newtechniques to fuse event data with other sources of information [8]. For instance, the combination of events andframes produced accurate rotational motion estimations between frames [3, 9], and event cameras and pulsed line sensors can be used to capture high-frequency light rays andreconstruct three-dimensional objects [2, 15].
Several works focus on solving the problem of visual-inertial odometry estimation using event cameras [14,17,19,
21, 23] and face differently the challenge of asynchronous
event data arriving faster than the IMU and image frames.
For instance, [23] proposes a method that uses IMU predictions and feature tracking over event windows with a multistate constraint Kalman filter [13]. Despite achieving highthroughput rates of up to 100 Hz, this method suffers froma high computational cost. The work in [17] preintegrates
inertial measurements and corrects with visual keyframesof motion-compensated event images minimizing reprojection error. The event window size used is sufficiently large
to preintegrate several IMU readings. The Ultimate SLAM
approach, presented in [19], extends this idea by including
images in the process. The REVIO approach, presented
in [21], also follows feature tracking on event images andIMU preintegration, but includes a range distance sensor
and sliding window nonlinear optimization in the process.It
is worth mentioning that similar architectures have been
evaluated with a stereo rig of event cameras,like the workof [12], which is an extension of the event-only approachin [22]. It should be noted that all of these approaches
achieve a maximum throughput of 100 Hz.
Our approach is more related to IDOL [11], which isa line-based visual-inertial system that tracks clusters oflines across multiple event windows to refine IMU preintegrated predictions through non-linear optimization. However, IDOL is known to have a high computational cost that
limits its real-time performance. A more recent method,
PL-EVIO [10], follows a similar approach to [17] but uses
the Line Segment Detector (LSD) [20] and FAST corners[6] to detect features on motion-compensated images. Like
IDOL, PL-EVIO employs the IMU preintegration scheme
but achieves faster estimation rates of 30-100 Hz.
In this work, we propose novel strategies that do not follow the feature tracking and IMU preintegration schemes
from most state-of-the-art approaches. Our approach does
not create event images and relies only on small temporal windows of events. Our proposed camera tracking system uses the Lie Kalman filter architecture presented in [4],which uses only events and line features, and has a highthroughput rate of up to 10kHz, which is at least one hundred times faster than the state-of-the-art. We integrate
the proposed camera tracking approaches as part of a full
Parallel Tracking and Mapping (PTAM) system to assess
their performance in human-made environments and compare them against state-of-the-art methods. We use the mapping module presented in [5] to retrieve 3D lines from thescene using only event data, which works at a lower rate.
In addition, we experimentally assessed the proposed sensor fusion strategies to characterize the limits of the motion
dynamics estimation in terms of speed and acceleration.

2. Events & IMU fusion
Our tracking system utilizes the Lie Kalman filter formulation proposed in [4]. This filter is designed to workwith small event windows and can achieve a pose estimation rate that is up to 10 times faster than the DAVIS346
IMU sampling frequency. We describe here our strategy todeal with the fact that event-based poses are produced at amuch higher rate than IMU readings. It uses both events and
IMU data during the correction stage, each at its own sampling interval, producing estimates at the high rate of theevent-based tracker. We consider two variants corresponding to two different assumptions on the nature of the IMU
measurements, either instantaneous at the sampling time oraveraged over a sampling period. For comparison, we also
describe the classical strategy, which incorporates the IMU
readings in the prediction stage and the events in the correction stage, thus requiring a reduction of the throughput
to the lower rate of the IMU. In the following sections, wewill provide a detailed explanation of these two strategies
and their variants.

At central
window time: tc
Prediction

Event
Window[ tk ... tk+N ]Constant
velocityor

3D linesmap

Dataassociation

Constant
acceleration
Matched: lines & events

Correction

Instantaneous
model
or
Mean
model

Correction

For all matched events& IMU:

Projection

Event correction t k
IMUcorrection t c

For all matched events:

Yes

Event correction t k+N
t k+N

Exist
IMU

Pose

Event correction t k
No Event correction t c
Event correction t k+N
t k+N

Figure 1. Strategy 1: Events and IMU in the correction stage.

eration measurement model (IA):  \am &= {{\bf R}_k}^{\top }({\bf a}_k - {\bf g}) + {\bf b_a}_k + {\bm \sigma }_{\bf a} \\ \wm &= {\bm \omega }_k + {\bw }_k + { \bm \sigma }_{\bm \omega },(2)
where âmk and ω̂mk are the estimated IMU linear acceleration and angular velocity at the IMU reference frame B,
which are computed from the current estimates of the camera linear acceleration ak , its angular velocity ωk , and orientation Rk ∈ SO(3).
The Mean Acceleration measurement model (MA) deals
with IMU readings as mean values over a ∆T interval:
  \am &= {{\bf R}_k}^{\top }\left (\frac {{\bf v}_k-{\bf v}_{k-j}}{\Delta T} - {\bf g}\right ) + {\bf b_a}_k + {\bm \sigma }_{\bf a} \\ \wm &= \frac {{\bf R}_k \ominus {\bf R}_{k-j}}{\Delta T} + {\bw }_k + {\bm \sigma }_{\bm \omega },2.1. Events & IMU in correction stage
This strategy draws from the events-only correction
mechanism proposed in [4], which utilizes an ultra-fast
event-line data association algorithm over a small windowof events. Here, we combine it with IMU corrections, whichtake place at central window time. When there are no IMU
readings available, the correction is carried out solely with
event data. The event window has a typical size of 100 µs,
hence we incorporate one IMU reading every ten windows
to account for the IMU rate of 1 kHz. Figure 1 summarizes
the strategy.
2.1.1

Measurement models for IMU data

We consider two different assumptions on the nature of the
IMU readings: instantaneous and averaged. Each of thesemodels requires a particular parametrization of the state
space, which are described in Sec. 2.2.1.
The use of IMU readings as instantaneous values at the
sampling time yields what we call the Instantaneous Accel-(4)
where we compute the mean acceleration from the velocity
estimates at two consecutive IMU sampling times vk andvk−j . In addition, the mean angular velocity is computed
as an SO(3) right minus operation between the current orientation and the orientation at the previous IMU samplingtime Rk−j . Note that the right minus operator ⊖ is definedwith the SO(3) logarithmic map as Log(Rk−j −1 Rk ), and∆T is an IMU time step fixed at 1 ms for the DAVIS346
IMU sensor used in this work. Both measurement models⊤
include the gravity vector g = [0, 0, −9.81] m/s2 , the acceleration and angular velocity biases (bak , bω k ∈ R3 ),and the sensor noise (σa , σω ∈ R3 ).
The innovation related to IMU readings is computed as,

  \label {eq:z_IA} {\bf {z}} = \begin {bmatrix} {\bf a_m} - {\bf \hat a}_{\bf m}\\ {\bm \omega }_{\bf m} - {\bf \hat {\bm \omega }}_{\bf m} \end {bmatrix} \in \Rn {6}{1},

(5)

−1

EventsWindow

(a)

(b)

120

2.2. Measurement model for segment observations
The three-dimensional segments’ endpoints pj ∀ j ∈{1, 2} are projected to the image plane as follows,  \underline {\bf u}_j = {\bf K}{\Rab {C}{B}}^{\top } ({\bf R}^{\top }({\bf p}_j - {\bf r})-\tab {C}{B})~\in \mathbb {P}^2,(8)

where r and R are the system pose computed at IMU
reference frame, K is the intrinsic camera matrix, andB{rB
C , RC } are the IMU-to-camera extrinsic parameters.
In the absence of IMU data, the partial event innovations
belong to the signed event-line distance, which is computed⊤
considering the projected line l = u1 × u2 = [a, b, c] andthe event e associated to a line as follows,  z &= d({\bf e}, {\bf l}) = \frac {\underline {\bf e}^\top {\bf l}}{\sqrt {a^2 + b^2}} && \in \mathbb {R}~.\label {eq:z}(9)

The innovation covariance is Z = HPH⊤ + σd 2 ∈R, which results in scalar term. The measurement noise is

100

150

200

(c)

EventsThresholding

100

10

30

80

50

β

30
On events

70

90

Ambiguity
Zone

α

90

−1

where Jr and Jl stands for the SO(3) inverse right andleft Jacobians, which expressions are stated in the equations(144) and (146) in [18]. The Jacobians with respect to thebiases are trivial and equal to the identity, and the operator[·]× denotes a skew-symmetric matrix.
The Jacobians summarized in (6) and (7) are part of an
innovation Jacobian matrix H ∈ R6×m with m = 21 forthe IA model, and m = 24 for the MA model. The innovation covariance is computed as Z = HPH⊤ + N ∈ R6×6 ,where P stands for the state covariance of the transition
models of each filter variant detailed in Sec.2.2.1. In addition, N = diag(σa 2 I, σω 2 I) ∈ R6×6 is the measurement
noise matrix that contains the IMU sensor noise of linear
acceleration σa [m/s2 ], and angular velocity σω [rad/s].

50

80

(c)

(7)

80

60

(b)

  \text {IA model}& \left \{ \begin {array}{lll} \Jab {R}{am} &= & -{\bf R}^{\top } {\left [ {\bf a}-{\bf g}\right ]}_{\times } \\ \Jab {a}{am} &= & -{\bf R}^{\top } \end {array} \right . \label {eq:J_IA}\\ \text {MA model}& \left \{ \begin {array}{lll} \Jab {R}{am} &= & {\left [ {\bf R}^{\top }\left (\frac {{\bf v}_k-{\bf v}_{k-j}}{\Delta T} - {\bf g} \right )\right ]}_{\times } \\ \Jab {v_k}{am} &= & {\bf R}^{\top } {\Delta T}^{-1}\\ \Jab {v_{k-j}}{am} &= & -{\bf R}^{\top } {\Delta T}^{-1}\\ \Jab {R_k}{\omega m} &=& {{\bf J}_{r}}^{-1}({\bm \theta }){\Delta T}^{-1} \\ \Jab {R_{k-j}}{\omega m} &=& {{\bf J}_{l}}^{-1}({\bm \theta }){\Delta T}^{-1} \end {array} \right ., \label {eq:J_MA}

40

Image
Tesselation

which is the difference between the sensor readings of acceleration am , and angular velocity ωm , and the estimationfrom any of the measurement models.
The Jacobians of the innovation with respect to the state
terms are derived using the SO(3) Lie properties summarized in [18]. The Jacobians follow the notation Jab ≜∂a/∂b, and their computations are aided with the chain rule
resulting in the following expressions for each measurement
model:

40

50

60

70

Off events
Matched events
Line cells
Projected lines Tesellation guides

Figure 2. Data association algorithm [4]denoted with σd [pix], and the innovation Jacobians H ofthis measurement model are fully detailed in [4].
Recall that the data association algorithm is the sameused in [4] and [5] and is fully detailed in these works.
This algorithm allows for discarding or validating events asfast as possible. The fast operation is the result of the application of three simple but effective steps: the visible line
projection onto the image plane in an event window, Fig. 2a;
the image tessellation for a fast event searching, Fig. 2b;and the event thresholding considering a minimum eventline distance α [pix] and the distance with respect closer
lines β [pix], Fig. 2c.
2.2.1

State representations and prediction models

Depending on the measurement model chosen, a specificprediction model is to be used in the filter. Two variants
are proposed: the constant acceleration (CA) model combined with the instantaneous measurement model (CA +IA), and the constant velocity (CV) model corrected withthe mean measurement model (CV + MA). Note that bothvariants use event-only corrections in between IMU readings, as shown in Fig.1.
The state transitions for both variants are shown in Table 1, where in CV+MA, vk−j and Rk−j denote thedelayed-state IMU pose at the time when the previous IMU
data arrived. These terms are updated with current values attime k after the correction stage, and ∆t is the event win-

rk+1 =
vk+1 =
ak+1 =
vk−j =
Rk−j =
Rk+1 =
ωk+1 =
bak+1 =bω k+1 =CA + IA

CV + MA )rk + vk ∆t + 12 ak ∆t2 rk + vk ∆t
vk + ak ∆t
vk + vn
ak + an
−
−
vk−j
−
Rk−j
Rk ⊕ {ωk ∆t}ωk + ωnbak + banbω k + bω n

∈M
∈ R3
∈ R3
∈ R3
∈ R3
∈ SO(3)∈ SO(3)
∈ R3
∈ R3
∈ R3

IMU + erk + v∆T
vk + (R(am − bak ) + g + σa )∆T
Rk ⊕ {(ωm − bω k + σω )∆T }bak + banbω k + bω nPredictionDataassociation

IMU reading

Matched: lines & events

3D linesmap[ tk ... tk+N ]Projection
Event correction t k
Event correction t c
Event correction t k+N
Pose

Table 2. State transition for IMU + e variant
xk+1 = f (rk+1 =
vk+1 =
Rk+1 =
bak+1 =bω k+1 =At central window time: tc

Correction

xk+1 = f (EventWindow

For all matched events:

Table 1. State transitions for CA+IA and CV+MA variants

∈M
∈ R3
∈ R3
∈ SO(3)∈ R3∈ R3

dow time step, which is set to 100 µs in our experiments.
The perturbation in all models is modeled as a Gaussian
noise with mean zero and a covariance, which is computed
as a random impulse integrated over a timestep ∆t. The
following expressions detail the covariance name associate
the each noise term and the integration result as follows:  \begin {array}{lll} {\bf v_n}\sim \mathcal {N}(0, {\bf V_i}) & \text {with}& {\bf V_i}={{\sigma }_{\bf {vn}}}^2 \Delta t {\bf I} \\ {\bm \omega }_{\bf n}\sim \mathcal {N}(0, {\bm \Omega }_{\bf i}) & \text {with}& {\bm \Omega }_{\bf i}={{\sigma }_{{\bm \omega }{\bf n}}}^2 \Delta t {\bf I} \\ {\bf a_n}\sim \mathcal {N}(0, {\bf A_i}) & \text {with}& {\bf A_i}={{\sigma }_{\bf {an}}}^2 \Delta t {\bf I} \\ {\bf b}_{\bf a_n}\sim \mathcal {N}(0, {\bf Ba_i}) & \text {with}& {\bf Ba_i}={{\sigma }_{\bf {ban}}}^2 \Delta t {\bf I} \\ {\bf b}_{{\bm \omega }_{\bf n}}\sim \mathcal {N}(0, {{\bf B}{\bm \omega }}_{\bf i}) & \text {with}& {{\bf B}{\bm \omega }}_{\bf i}={{\sigma }_{\bf {b}{{\bm \omega }{\bf n}}}}^2 \Delta t {\bf I} \end {array}, \label {eq:impulses}

Figure 3. Strategy 2: Prediction with IMU and correction with
events.

The resulting variant of the Lie Kalman filter is denoted
IMU + e because it incorporates both IMU and event data.
The state transition follows the classic visual-inertial formulation detailed in Table 2, where the sensor bias and noise isincluded in the linear velocity and orientation estimations.
But in contrast to frame-based visual-inertial methods in thestate of the art, our method is 10 to 30 times faster, thanks
to the use of event windows of 1ms and centered at IMU
sampling times. The state Jacobians are computed using
the SO(3) Lie properties, where the nontrivial expressions
are listed as follows,  \begin {array}{lll} \Jab {R}{v} & = & -{\bf R} {[{\bf a_m} - {\bf b_a}]}_{\times } \\ \Jab {ba}{v} & = &-{\bf R} \Delta t \\ \Jab {R}{R} & = & {{\bf R}\{ ({\bm \omega }_{\bf m}-{\bf b}_{\bm \omega })\}}^{\top }\\ \Jab {b \omega }{R} & = &-{\bf J}_r({\bm \omega }_{\bf m} - {\bf b}_{\bm \omega })\Delta T \end {array}

(10)

(11)

√
√
√
where σvn√[m/s s], σωn [rad/s s], σan [m/s2 √s],σban [m/s2 s], are noise constants, and σbωn [rad/s s]
are bias drifts. These constants were tuned empirically
based on characteristics of the experiments and the IMU
sensor specifications.
The state covariance is propagated as Pk+1 =FPk F⊤ + Q, where F is Jacobian matrix with respect tothe state derived following SO(3) lie group properties andthe state transitions of Table 1. Q is the perturbation covariance conformed with the random impulses in (10), andbuilt according to the noise terms of each variant detailed inTable 1. The covariance matrices have a size of m × m with
m = 21 for the CA+IA variant and m = 24 for CV+MA.

The correction for the IMU+e variant follows the same
approach described in Sec. 2.2, which uses the signed eventto-line function stated in (9).

2.3. Prediction with IMU and correction with events
For sake of completeness, we also implement a second
strategy that incorporates the IMU readings in the prediction stage; as a result, each event window is now centered at
IMU time and has a fixed size of 1 ms, as detailed in Fig. 3.

2.4. Events & IMU in a PTAM system
Our proposed sensor fusion variants enable camera pose
estimation in conjunction with a mapping module within
a PTAM system. We leverage the mapping formulationfrom [5], which allows us to recover 3D lines from thescene. The mapping module builds a spatial grid at a
keyframe position, which is intersected by back-projectedrays from events in several event windows. The tracking
module determines the pose of each event window. The gridis divided into voxels, which are populated with votes fromeach ray that passes through them. Voxels with a greaternumber of votes are more likely to contain a 3D point. We
can retrieve a 3D line from a set of 3D points aligned instraight patterns located in straight edges.
Since the mapping and tracking modules operate independently, we use a local bundle adjustment module to op-

Table 3. RMSE for event-only and event-IMU PTAM configurations and for visual-inertial SVO.

DatasetOfficeL shape
Officelarge
Trihedron
Officefar
Pose throughput
StreamProcessing time
∗

X [m]
Y [m]
Z [m]

IMU+e

SVO

CA + IA

CA+IA
3.79
0.23
6.60
0.22
3.27
0.17
3.72
0.27
10
15.20

CV+MA
3.68
0.22
6.53
0.22
3.12
0.17
3.71
0.26
10
13.90

IMU+E
3.75
0.23
6.38
0.21
3.15
0.16
3.69
0.26
1
10.9

SVO
4.77
0.25
6.88
0.22
8.20
0.18
3.61
0.25
0.033∗
22.22(a)Seq. 1

0.4
0.2
0.0
0.0
-0.2

Reconstruction with 3D lines

Ground truth

X1

0.0
-0.2
-0.4

X1

-0.05
-0.10
-0.15
-0.20

Y1

(b)Seq. 0.40
2
0.35
0.30

Y1

-0.20

Z1

Mapping

-0.25

-0.4

Z1

1.0
0.5
0.0
1.0
0.5
0.0
-0.5
1.0

Roll
[rad]

0.50
0.37
0.50
(d)Seq.0.30
4

R1
P1

(c)Seq. 3

0.10
0.75

P1
Yw1

0.0

0.50

Yw1

10

15

20

25

30

35

0.25

40

45

50

55

30

Tracking

0.63

R1

Pitch
[rad]
Yaw
[rad]

CA
3.91
0.24
6.58
0.23
3.46
0.18
3.77
0.26
10
14.5

Throughput limited by the DAVIS346 camera

CV+MA

Time [s] 5

CV
3.87
0.23
6.62
0.23
3.31
0.20
3.84
0.27
10
11.6

Pos. [cm]Rot. [rad]Pos. [cm]Rot. [rad]Pos. [cm]Rot. [rad]Pos. [cm]Rot. [rad]
[kHz]
[ms]

31

32

On events

Off events

Tracked lines

Figure 4. Trajectory and 3D reconstruction with PTAM system using CV + MA variant in Office L shape scenario. The tracked lines arehighlighted in green in the tracking snapshots, and the line map was captured from Rviz at the end of the sequence.

timize 3D lines and camera poses using a cost function thatminimizes the distance between the event and line in the image plane. For more details on the mapping process, see [5].
In section 3.1, we present the results of a PTAM applicationthat uses our proposed sensor fusion variants.

3. Experiments and results
To evaluate the performance of all filter variants, we propose two experiments. Firstly, we integrate the tracking system into a PTAM (Parallel Tracking and Mapping) system
to assess the pose estimation accuracy while creating 3Dline-based maps. Secondly, we conduct a stress experiment

to assess the estimation limits in high dynamics. We use the
DAVIS346 event camera in all experiments, which includes
an MPU-6500 IMU.
The sensor noise and bias drifts are tuned to IMU
specifications.
The used values for all variants are2
σa = 0.93[m/s], σω = 0.017 [rad/s],
σban =√√
2
2.94 [m/s s], σbωn = 0.17 [rad/s s]. The reminding noise constants were set experimentally in order to geta response with the√lowest possible error,√these parametersare σvn = √3 [m/s s], σωn = 7 [rad/s s], and, σan =150 [m/s2 s]. The PTAM system runs in a standard multicore CPU with Ubuntu 18.04.6 and ROS Melodic.

2.2
1

2.5
2

6.6
13

6.3
12

5.9
11

2.9
3

14.3
33

12.1 12.5
27 28

14.8
34

16.5
39

37
35
33

Z [m]

Y [cm]

X [cm]

Frequency:
Step:

5

0
27
25

Time [s]

15

16

17

18

37

38

39

40

84

85

(a)

Error evolution

1.5
RMSE [cm]

19

86

92.5

93.3

94.2

99.5 100

Tracking captures

1.0
0.5
0

20
CV+MA

40

60
CA+IA

(b)

80

CV

100
IMU+E

(c)
Ground truth

Figure 5. Position estimation in stress test from 240 rpm to 990 rpm with CV+MA, CV+IA, and IMU+e variants: (a) position estimation,(b) error evolution, (c) four-bar mechanism with tracking snapshots until the system failure.

3.1. PTAM evaluation
The mapping node of [5] was used to recover 3D lines
for all tracking variants, and it runs at a lower asynchronous
rate of about 10 Hz. The PTAM’s tracking module was set
up for the event-only variants of CV and CA, as extensively
described in [4], and for the three proposed sensor fusion
variants: CV+MA, CA+IA, and IMU+e. Additionally, forcomparison purposes, the frame-based visual-inertial approach SVO [7] was also tested on our sequences. Table 3
summarizes the scenarios and the mean pose estimation errors, compared to Optitrack ground truth.
The tracking variants and the mapping node run in parallel threads. Our efficient implementation of CV+MA andCA+IA allows reaching a no-lag real-time throughput of up
to 10kHz, with event windows of 100µs. Event windows
smaller than 100µs diminished the mapping accuracy due tothe reduced amount of events per window. SVO works at aframe rate of 30 Hz for the DAVIS346 camera. Note that according to [7] SVO is capable of running at 300 fps showing
real-time performance. In our experiments the throughputof SVO is limited by the DAVIS camera.
A PTAM performance example in the office scenario isshown in Fig. 4. Notice that whereas SVO has large drifts insome short intervals, all the proposed methods better matchthe ground truth trajectory. The mean errors in Table 3 show
an overall accuracy improvement of about 5% in all sen-sor fusion variants with respect to event-only ones. The
best-achieving variants were CV+MA and IMU+e, which
present better results than the frame-based approach, probably because the scenarios are richer in straight edges than
texture. IMU+e was the most accurate approach but limited
to a throughput of 1kHz. Its accuracy is about 5% higher
than CV+MA or CA+IA. While our approach works well inhuman-made environments where straight lines are prevalent and provide stable estimations of camera pose, it may
face challenges in natural scenarios where straight lines areabsent. Retrieving small, noisy lines in such environments
can compromise the accuracy of our estimations. Therefore,
further research is needed to improve the robustness of theapproach in natural scenarios. Nevertheless, our approachremains a promising solution for fast and accurate camera
pose estimations in human-made environments.

The time required to process all event windows within a
33ms event stream is named stream processing time, and it
is displayed in the last row in Table 3. The sensor fusion
variants show an increase in computational time of about
6% with respect to the event-only CV and CA methods.
IMU+e was the cheapest option given its smaller state size.
The computational time of the mapping thread is the same
reported in [5] due to there were no modifications proposedin our approach.

In this experiment, the sensor fusion variants were tested
to determine their tracking limits under high dynamics.
The experiment involved an event camera placed on topof a four-bar mechanism that observes a statically known
marker, as shown in Fig.5c. The marker is used as a line
map to estimate the camera pose in high dynamics conditions The device’s kinematics and dimensions are wellknown, providing ground truth for the experiment.
The DC motor powering the device was discretely increased from 132 rpm (2.2Hz) until tracking failed. Fig. 5ashows that the IMU+e (dark yellow) failed first at 750 rpm(12.5 Hz), followed by CA+IA (cyan) at 858 rpm (14.3 Hz)
and CV+MA (gray) at 870 rpm (14.5 Hz), while the eventonly variant CV (dark blue) reached the maximum speed of
990 rpm (16 Hz).
Although the sensor fusion variants failed before theevent-only variant, they provided better estimations thanCV, as illustrated in the error evolution plot in Figure
5b. Specifically, Table 3 shows that IMU+e produced themost accurate pose estimations, followed by CV+MA andCA+IA, whose error ranges were similar.
However, the main cause of failure for the sensor fusion
variants was the acceleration exceeding the MPU-6500 acceleration limit of 16g at 750 rpm, leading to saturation.
This saturation issue was verified by comparing the estimated and measured linear acceleration signals. The estimated acceleration (light blue in Figure 6) exceeded 16g,
but the IMU sensor (dark blue in Figure 6) can not produce
measurements beyond that value, which introduced errors
in the estimations. Towards the end of the experiment, CV
experienced strong vibrations over 25g, which caused thetracking estimation to degrade. Due to the strong vibrationsin the mechanism, experiments with accelerations over 25g
were not conducted to avoid damaging the event camera.

4. Conclusions
This work presents a comprehensive evaluation of different event-camera tracking variants, including event-onlyand sensor fusion approaches, for ultra-fast response intracking and pose estimation.
The proposed variants, CV+MA and CA+IA, as well
as IMU+e, showed an accuracy improvement of approximately 6% over event-only approaches. IMU+e was foundto be the most accurate, albeit with limited throughput.
The proposed tracking architecture achieved a throughput
of about 10 kHz with CA+IA and CV+MA, and 1 kHz with
IMU+e, which is significantly faster than frame-based approaches like SVO (limited to 30Hz due to DAVIS model
design). Although the computational burden of sensor fusion variants is slightly higher than that of event-only methods, real-time performance is maintained.

Freq.[Hz]: 2.2 3.3
Step : 1 4

aZ [m/s2] aY [m/s2] aX [m/s2]

3.2. Stress evaluation

100
0
-100
-200
200
0
-200
-400
200
0
-200

4.8
8

5.9
11

7.0
14

7.7
16

8.1
17

9.6
20

9.9 11.0 12.5 14.7 16.5
21 24 28 34 39

-16g

16g
-16g

16g

-16g

Estimated
Acceleration

MeasuredAcceleration IMU

IMU
saturation limit

Figure 6. Linear acceleration measured and estimated during
the stress experiment. Note: IMU acceleration is saturated after
750 rpm (12.5 Hz) where the camera experiences an acceleration
higher than 16g

The stress experiment showed that the sensor fusion
variants failed before event-only ones due to the acceleration exceeding the sensor acceleration limit of 16g. Overall, the results demonstrate the potential of sensor fusion
approaches to achieve event-based pose estimation with athroughput of 1kHZ to 10kHz and accuracy superior toevent-only or image-based methods in human-made environments.

Acknowledgments
This work received support from projects
EBCON(PID2020-119244GB-I00)
and
AUDEL
(TED2021-131759A-I00) funded by MCIN/ AEI/
10.13039/501100011033 and by the ”European Union
NextGenerationEU/PRTR”, and the Consolidated ResearchGroup RAIG (2021 SGR 00510) of the Departament deRecerca i Universitats de la Generalitat de Catalunya.

References[1] Christian Brandli, Raphael Berner, Minhao Yang,
Shih-Chii Liu, and Tobi Delbruck. A 240× 180 130 db
3 µs latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circuits, 49(10):2333–2341,
2014. 1
[2] Christian Brandli, Thomas A. Mantel, Marco Hutter,
Markus A. Höpflinger, Raphael Berner, Roland Siegwart, and Tobi Delbruck. Adaptive pulsed laser line
extraction for terrain reconstruction using a dynamic
vision sensor. Frontiers in Neuroscience, 7(8):275,
2014. 1
[3] Andrea Censi and Davide Scaramuzza. Low-latency
event-based visual odometry. In IEEE Int. Conf.
Robotics and Automation, pages 703–710, 2014. 1[4] William Chamorro, Juan Andrade-Cetto, and Joan
Solà. High-speed event camera tracking. In British

Machine Vision Conf., pages 1–12, Jan 2020. 1, 2, 3,
6[5] William Chamorro, Joan Solà, and Juan AndradeCetto. Event-based line SLAM in real-time. IEEE
Robotics and Automation Lett., 7(3):8146 – 8153,
2022. 2, 3, 4, 5, 6
[6] Rosten Edward, Porter Reid, and Drummond Tom.
Faster and better: A machine learning approach to corner detection. IEEE Trans. Pattern Analysis and Machine Intelligence, 32(1):105–119, 2010. 1[7] Christian Forster, Zichao Zhang, Michael Gassner,
Manuel Werlberger, and Davide Scaramuzza. SVO:
Semidirect visual odometry for monocular and multicamera systems. IEEE Trans. Robotics, 33(2):249–
265, 2016. 6[8] Guillermo Gallego, Tobi Delbrück, Garrick Orchard,
Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt,
Kostas Daniilidis, et al. Event-based vision: A survey. IEEE Trans. Pattern Analysis and Machine Intelligence, 44(1):154–180, 2020. 1[9] Daniel Gehrig, Henri Rebecq, Guillermo Gallego, andDavide Scaramuzza. Asynchronous, photometric feature tracking using events and frames. In Eur. Conf.
Computer Vision, pages 750–765, 2018. 1[10] Weipeng Guan, Peiyu Chen, Yuhan Xie, and Peng Lu.
PL-EVIO: Robust Monocular Event-based Visual Inertial Odometry with Point and Line Features. arXiv
preprint arXiv:2209.12160, 2022. 1[11] Cedric Le Gentil, Florian Tschopp, Ignacio Alzugaray, Teresa Vidal-Calleja, Roland Siegwart, andJuan Nieto. IDOL: A framework for imu-dvs odometry using lines. In IEEE/RSJ Int. Conf. Intelligent
Robots and Systems, pages 5863–5870, 2020. 1[12] Zhe Liu, Dianxi Shi, Ruihao Li, and Shaowu Yang.
ESVIO: Event-Based Stereo Visual-Inertial Odometry. Sensors, 23(4):1998, 2023. 1[13] Anastasios I Mourikis, Stergios I Roumeliotis, et al.
A multi-state constraint Kalman filter for vision-aided
inertial navigation. In IEEE Int. Conf. Robotics andAutomation, volume 2, pages 3565–3572, 2007. 1[14] Elias Mueggler, Guillermo Gallego, Henri Rebecq,
and Davide Scaramuzza. Continuous-time visualinertial odometry for event cameras. IEEE Trans.
Robotics, 34(6):1425–1440, 2018. 1[15] Manasi Muglikar, Guillermo Gallego, and Davide
Scaramuzza. ESL: Event-based structured light. In
IEEE Int. Conf. 3D Vision, pages 1165–1174, 2021. 1[16] Lichtsteiner Patrick, Christoph Posch, and Tobi Delbruck. A 128 x 128 120 dB 15µs latency asynchronous
temporal contrast vision sensor. IEEE J. Solid-State
Circuits, 43:566–576, 2008. 1[17] Henri Rebecq, Timo Horstschaefer, and Davide Scaramuzza. Real-time visual-inertial odometry for event
cameras using keyframe-based nonlinear optimization. British Machine Vision Conf., 2017. 1[18] Joan Solà, Jeremie Deray, and Dinesh Atchuthan. A
micro lie theory for state estimation in robotics. arXiv
preprint arXiv:1812.01537, 2018. 3[19] Antoni Rosinol Vidal, Henri Rebecq, Timo
Horstschaefer, and Davide Scaramuzza.
Ultimate SLAM? combining events, images, and IMU
for robust visual SLAM in HDR and high-speed
scenarios. IEEE Robotics and Automation Lett.,
3(2):994–1001, 2018. 1[20] Rafael Grompone Von Gioi, Jérémie Jakubowicz,
Jean-Michel Morel, and Gregory Randall. LSD: a line
segment detector. J. Image Processing Online, 2:35–
55, 2012. 1[21] Yingxun Wang, Bo Shao, Chongchong Zhang, Jiang
Zhao, and Zhihao Cai. REVIO: Range-and EventBased Visual-Inertial Odometry for Bio-Inspired Sensors. Biomimetics, 7(4):169, 2022. 1[22] Yi Zhou, Guillermo Gallego, and Shaojie Shen.
Event-based stereo visual odometry. IEEE Trans.
Robotics, pages 1433–1450, 2021. 1[23] Alex Zihao Zhu, Nikolay Atanasov, and Kostas Daniilidis. Event-based visual inertial odometry. In
IEEE Conf. Computer Vision and Pattern Recognition,
pages 5391–5399, 2017. 1