Abstract

INTRODUCTION
11
Event Cameras, such as the DVS [[𝐄𝑗], are bio-inspired visual sensors that differ substantially from traditional frame-based cameras. The pixels of an event camera operate asynchronously and trigger an event whenever there is s sufficient change in the sensed logarithmic brightness level. More specifically, if the change of logarithmic brightness L(x,t) = logI(x,t) at pixel x = (x,y)  r on the image plane surpasses a threshold C , the event camera will output a four- tuple signal e = {x, y, t, s} where t is a timestamp and s is a binary polarity indicating whether the brightness has increased or decreased. Therefore, each pixel has its own sam- pling rate and outputs data proportionally to the amount of motion between camera and scene and in dependence of the gradient of the visual input. An event camera does not produce im- ages at a constant rate, but rather a stream of asynchronous, sparse events in a space-time volume with approximately microsecond time resolution. 2 Due to its exact nature, event cameras have several advantages over standard cameras including low latency ( Ī μ s), absence of motion blur, high dynamic range (140 dB vs 60 dB for traditional cameras [[ ] ), and low power consumption. These beneficial properties en- able an event camera to tackle vision tasks even in challenging conditions such as increased agility or low illumination conditions.  One of the critical applications of an event camera is ego-motion estimation given that existing pipelines based on standard camera easily fail under high-speed motion or challenging illumination [[ 3 , [ 3 ]]. However, the technical ob- stacle for event-based motion estimation is the fact that events are asynchronous and do not communicate absolute intensity information.  Traditional motion estimation algorithms are therefore not appropriate and novel algorithms are needed. Most of the state-of-the-art works in event-based motion estimation rely on learning- based approaches [[ . [ . ], filter-based methods [ . [ . [ ]] and optimization methods [ . [ ], [ . [ ]]. Learning-based approaches need a huge amount of data to train the network, and filter-based methods are computationally complex and often need an initial guess. Since most problems are nonlinear, the results of optimization methods highly depend on a good initial guess. [[9, [2] provide globally optimal solvers, which do not rely on good initial guess however they are computationally demanding and limited to homography environments.  However, the methods are computationally demanding and limited to homography scenarios.  Line features have already been used in event-based structure-from-motion frameworks. An ex- ample is given by Hough²Map [[ 8 ] which detects, tracks, and triangulates general lines. Other methods require highly artificial, black-and-white textures [ 2 ]]. Brändli et al. [ 0 ] pro- pose Event-Based Line Segment Detector (ELiSeD), which adopts the idea behind the LSD algorithm [ 28 ] to the event-based case.The method performs incremental event-based detec- tion and tracking of lines in arbitrary scenes, but is not yet validated in the context of a full structure-from-motion framework. DVS sensors are often equipped with an Inertial Mea- surement Unit (IMU) (i.e.  DAVIS 240 [ ]), which is why researchers have also considered event-based visual-inertial odometry [ [ ∞ , [ ∞ , [ ∞ ] In particular, Le Gentil et al. [ ∞ ]introduce a line-based event-inertial odometry framework. In their event-based line tracking front-end, they draw concepts from [[ ]] and [[ ], and detect line segments as locally spatio-temporal planar patches. A critical concern in inertial odometry frameworks is given by bootstrapping. Our method aims at linear velocity initialization, which can neither be obtained directly from IMU, nor an event camera. Most existing event-inertial odometry algorithms pay little atten- tion to the initialization question. Though using the assumption of known angular velocities, our work is the first to focus on linear velocity initialization and proposes a novel closed- form solver. It may be used to bootstrap other event-based inertial odometry frameworks, and in particular supports fusion at the level of velocities rather than positions. This has the advantage of requiring only single integration of inertial signals. The main contributions are listed as follows: • We make use of trifocal tensor geometry [[ ]] to formulate the relationship between events, lines and the ego-motion of the event camera, the so-called Continuous Event- Line Constraint (CELC). • To the best of our knowledge, this leads to the first closed-form translational veloc- ity solver. It relies on the assumptions of known angular velocity and constant speed resulting in a linear constraint, and furthermore enables nonlinear optimization to im- prove performance. • Important steps towards a DVS pendant of epipolar geometry and relative motion es- timation for normal cameras. The paper is organized as follows. Section 2 reviews the general idea of trifocal tensor geometry with lines and introduces the continuous event-line constraint. In Section 3, we employ CELC for linear velocity estimation with a closed-form solution, and provide all implementation details. In Section 4 , we analyze and evaluate the proposed algorithm via both simulated and real experiments. Section 5 gives final remarks.
Continuous Event-Line Constraint
2
We give a review of the trifocal tensor with standard cameras. We then derive the continuous event-line constraint (CELC) for an event camera which indicates the relationship between events, lines and camera dynamics.
Review of Trifocal Tensor
2.1
The trifocal tensor plays an analogous role in three views to that played by the fundamen- tal matrix in two [[…]]. It encapsulates all the (projective) geometric relations between three views including the incidence relationship of three corresponding points, three correspond- ing lines, and point with line incidence relations. The line-point-line incidence relation is the most relevant for our work. Lets denote a 3D line L, its tso corresponding line projections / 1 and / 3 in views 1 and 3, respectively, and an image point x 2 in view 2 which is the projection of a 3D point on L . The geometry is illustrated in Fig. 1(a). Let's define the second view as the reference view. We furthermore define [ R 12 [ t 12 ] as the Euclidean transformation parameters from view 2 to view 1, and [R 32 t 32] as the Euclidean transformation parameters from view 2 to view 3. In the calibrated case, the line-point–line incidence relation is given by
where f 2 is the bearing vector corresponding to pixel x 2 in view 2, and l i = K T l i with i = 1,3 are the normal vectors of the planes crossing L and the camera centers of views 1 and 3, respectively. [T 1 , T 2 , T 3 ] defines the trifocal tensor, and the formulation for the 3 × 3 matrices T i is given by T i = r i t 32 − t 12r i 2T , i = 1,2,3, (2) where r ¦ 2 denotes the i-th column of R 12 , and r ³ 2 the i-th column of R 32 . For further details on the trifocal tensor including its derivation, the reader is kindly referred to Chapter 15 of日.
2.2
Continuous Event-Line Constraint – CELC
Our task consists of event camera ego-motion estimation. Our assumption is that events are mostly triggered by the reprojection of sharp appearance and occlusion edges, which—for the sake of a simplified derivation—are furthermore assumed to be straight in 3D. Note that this assumption may not be limiting, as man-made environments often present themselves in a form where the majority of such edges are indeed straight. We detect continuous line projections from event streams and figure out the relation between those events, the underlying 3D lines, and dynamic motion parameters by using the aforementioned trifocal tensor relations.  The continuous set of events triggered by the projection of a straight 3D line L under motion forms a cluster of events E in a twisted manifold-like shape.  As denoted by the green and red lines in Figure 1(b), let the two lines l 1 and l 3 represent the reprojection of the 3D line L at timestamps t s and t e , respectively. As introduced in Sec. 2.1, for an event e k ∈ E , l 1 and l 3 must then satisfy the trifocal relation (1). Here, let's define [ T k , T k , T k ] as the trifocal tensor for the k -th event. The trifocal tensor is constructed by the transformation of the camera from t k to t s and the transformation from t k to t c . The trifocal tensor will be different for each individual event. Note that rather than introducing an individual rotation and translation for each event—which would introduce too many unknowns—, we make use of a locally constant velocity assumption and parameterize the relative translation and rotation as a continuous time function of the linear velocity v and angular velocity ω . Hence, the rotation R sk from time t k to time t s can be represented by the continuous time function
where ( a , θ ) are the axis-angle parameters of the rotation R sk . The translation t sk from time t k to time t s is given by
By replacing t s with t e in equations (3) and (4), we can obtain R ek and t ek . Based on (3) and (4), we finally obtain the continuous time formulation of the trifocal tensor
Using equation (5) and applying simple matrix multiplication, we obtain
The continuous event-line constraint (CELC) for the k -th event is obtained by combining (6) and (1), thus resulting in
where
The incidence relation expresses the intrinsic relationship between events generated by a 3D line and first order camera dynamics. Considering the transformation of lines and trifocal tensor geometry, Ì 2 = B k v represents the projected line in view k generated by reference lines l 1 and l 3 and motion dynamics. Any event e k triggered by the same line should lie on Ï 2 , i.e. f , Ī 2 = 0. As scale is unobservable in the monocular setting, the unknown motion parameters ω and v actually make up for only 5-DoF. However, equation (3) and (4) are nonlinear with respect to ω and v , which makes it hard to simultaneously figure out angular velocity and linear velocity. In the continuation, we therefore consider the case where angular velocities are given by an Inertial Measurement Unit (IMU).
Closed-form Velocity Initialization
3
Typically, a DVS sensor such as the DAVIS346, integrates an event camera and an IMU which provides angular velocity and acceleration. With the help of the prior known angular velocity, the nonlinear 5 DoF motion estimation problem is reduced to a 2 DoF problem: translational velocity estimation. The closed-form speed initialization algorithm proceeds in four steps.  The first step consists of event clustering.  Next, for each cluster we extract the lines l 1 and l 3 by using a small time interval of events at the beginning and the end of each cluster.  Finally, using (7) , we propose a linear closed-form speed solver for the linear velocity v. The fourth and final step consists of nonlinear optimization improving the estimation result.
3.1
Line Clustering and Extraction
We adopt a strategy similar to the one leveraged in [ 0 , [ 0 ], which considers events as a 3D point cloud in the space-time volume.  The coordinates are given by the pixel position of the event and the timestamp, i.e. e i = [ x i , y i , t i / c ]. To balance the magnitude of the image coordinates and the timestamp of an event, the latter is normalized by a constant c whose value is chosen according to the average level of texture in the scene. The time span over which event clusters are formed is dynamically defined by considering a fixed number of N events. Events generated by the same line will approximately form a local plane in the 3D space-time volume of the event stream.  Hence—ignoring the influence of rotational velocities, clustering events generated by the same line in 3D roughly amounts to plane clustering in a 3D point cloud. We employ the open-source C++ library Cilantro [ [ ]] to implement the clustering procedure, which operates in a region growing fashion inspired by connected component segmentation. For more details, please refer to [ [ 20 ]. For each event cluster ξ j in which events are sorted with increasing timestamps, we utilize the first and last 0.005s intervals of events to extract the lines l 1 j and l 3 j . We use cv::fitLine from OpenCV [ 8 ] to extract the lines, and the algorithm is based on an M- estimator that iteratively fits the line using a weighted least-squares algorithm. We choose the Huber norm strategy ( [ [ 3 ], page 43). Note that our algorithm uses variable timestamps around which l 1 ; and l 3 ; are fitted, which enables us to slide the 0.005s intervals towards the center of the entire cluster interval in case of insufficient events.
Linear Velocity Solver
3.2
With known angular velocity, the CELC ( 7 ) becomes linear in the translational velocity. It is furthermore easy to concatenate all linear constraints for all events of all clusters into one constraint. Given M event clusters E j where j = 1,2...,M and the corresponding extracted lines l 1 j and l 3 j , the constraints from each event cluster with N j events can be stacked into the single linear problem
A can be computed from the known angular velocity, the extracted lines l 1 j and l 3 j , and all measured events.  This linear problem could be solved using A T Av = 0 via SVD. However, given that least square estimation methods lack robustness [[ [ ][ [ ] , we choose to use another robust M-estimator with a Huber norm [ [ 3 ], and employ iteratively re-weighted least-squares fitting for the nullspace extraction.  As the number of events N is very large (about 100,000 in our real data experiments), and A is an N × 3 matrix, we improve efficiency by randomly selecting 1000 samples out of the N to perform the M-estimation.  Further details of our implementation can be found in the Definition part of Chapter 1.3 in [ 23 ].
3.3
Degenerated Case
Note that our linear solver cannot always determine a unique solution. It is obvious that if the motion of the camera is a pure rotation—meaning that v = 0 —solving our linear equation 9 via SVD will not be possible.  Another degenerate case exists if the camera moves along a straight line without rotation.  It is obvious that in this case any translational velocity component along the direction of the 3D line L = l 1 × l 3 will not contribute to any appearance changes in the image, and therefore also no events.  Hence, the 3D line direction needs to lie in the null space of the matrix A , which means A(l 1 × l 3 ) = 0.  Moreover, Av = A(V 1 + l 1 × l 3 ) = AV 1 . There exists an unobservable direction for the translational velocity given by the direction of the 3D line. In the case of linear motion, the unobservable direction also exists when there are multiple lines but all of them are parallel.
3.4
Nonlinear Optimization
In real cases, events are affected by both spatial and temporal noise as well as outliers in the form of salt and pepper noise.  This may lead to errors in the event clustering and the line extraction. The linear solver introduced in the previous section therefore only offers an initial guess, which may be further refined by a maximum likelihood estimation to make the estimation more accurate and robust. The objective is to minimize the geometric distance between the reprojected 3D line and the events. The cost function is given by
where d( · ) represents the distance function, and Î 2k j is the reprojected line in the image plane at timestamp t k j . One way to conveniently represent the 3D line during the nonlinear optimization is by its projections l 1 j and l 2 j in their relevant views. Given a candidate linear velocity, one can then again compute the trifocal tensor, and furthermore easily derive the location of the projected 3D line at the time of the corresponding event using the line transfer equation Î²k j = B k j V . The entire objective is minimized using a trust-region based method (e.g. Levenberg- Marquardt) and implemented with Ceres [闻] using a robust cost function (Huber norm).
Experiments
4
We analyze the proposed closed-form algorithm both in simulation and on real data. We use the Euclidean distance ε and the cosine distance φ between the estimated results and ground truth as a metric to evaluate the accuracy of the estimated results. They are given as follows: ε = || Vgt — Vest||2, φ = arccos( VgtVest),
where Vgt and Vest are the ground truth and estimated linear velocities, respectively.
Simulation
4.1
We start by evaluating the performance of the proposed approach over synthetic data.  To generate synthetic data, we randomly generate line segments in 3D space within a volume of x = [−2,2] m, y = [−2,2] m, and z = [3,6] m. With given angular and linear velocities, an event is generated by randomly choosing a 3D point on a line and projecting it into an image plane with the camera pose sampled by a random timestamp within a given time interval. In our experiments, we set the angular velocity of the camera as ω = [0, 0, 2] rad/s, and the linear velocity as v = [1, 2, 0] m/s. We generate events from 5 lines within a time interval of 0.5 s. We disturb the pixel location of each generated event by zero-mean Gaussian noise with a standard deviation of 2 pixels. We also add Gaussian noise of N (0,2) pixels to the ground-truth endpoints of each starting and ending line pair l 1 j and l 3 j . To evaluate the proposed solver, we adopt the single variable method to conduct various simulation experiments from two aspects. One is to evaluate the solver's robustness against noise including disturbance of events, errors of the extracted lines at the boundary of the interval and disturbances of angular velocities. The other one is to investigate the effect of certain factor such as the scale of the velocity, the length of the time interval, or the number of lines. Note that the errors for each level of each variable are averaged over 500 experiments. The detail configurations are as follows: • Robustness against event location noise: The disturbance of each event is varied with a standard deviation reaching from 0 to 5 pixels with a step size of 0.5 pixels. The average μ and standard deviation σ of ε and φ is presented in Figure 2(a). • Robustness against noise in the end points of l j and l 3j : We add pixel-level Gaus- sian noise to the two endpoints of the projected lines to test the robustness of the proposed linear solver. The noise is varied between 0 and 5 pixels with a step size of 0.5 pixels. Figure 2(b) indicates the simulated results. • Robustness against noise in angular velocity: We add Gaussian noise to the known angular velocity to test the robustness of the proposed linear solver. The noise is varied between 0 and 1 rad/s with a step size of 0.1 rad/s. Figure 2(c) indicates the respective results. • Effect of speed: We set the direction of the linear velocity as [0.447, 0.894, 0] and the scale of the velocity is varied between 0 and 10 m/s. The simulation results are shown Figure 2:  Noise analysis.  (a) shows the errors of the solver with increasing event location disturbance. (b) illustrates the error of our solver with different levels of noise added to the end-points of the fitted lines l 1 ; and l 3 ; . (c) displays the errors of the solver with increasing angular velocity disturbance. The error generally increases with noise. in Figure 3(a). As can be observed, errors are decreasing with an increasing norm of the speed. In other words, the higher speed, the more accurate our solver is operating. • Effect of the time interval size: We vary the time interval from 0.2 s to 2.2s with a step size of 0.2 s.  Results (Figure 3(b)) indicate that the errors are decreasing as the time interval is increasing. • Effect of the number of lines: The number of lines is varied from 2 to 10 with steps of 1. Figure 3(c) presents the results. The more lines are present in the scene, the higher the accuracy of the solver. Without loss of generality, the errors increase as noise level are increasing (cf. Figure 2). Note that the solver is rather sensitive to noise, which is analogous to the trifocal tensor- based approaches for standard cameras [ [ 1 , [1] . Furthermore, the more displacement the camera experiences during the chosen time interval, the higher the expected accuracy.
Real Data
4.2
To the best of our knowledge, we are the first to propose the CELC-based linear velocity solver for event cameras, and as such it is hard to compare against an existing SOTA algo- rithm.  We therefore design our own baseline algorithm to evaluate the proposed method, which is based on the line-line-line incidence relationship [ [ [ ]].
Using equation ( 7 ), the line-line constraint under continuous motion is given by l 2 B k v = 0,
Figure 4: (a)-(b) Example data captured by an AGV with a downward-facing event camera. (c)-(d) Example data captured by an UAV with a 45° downward-facing event camera. (a) and (c) denote grayscale images, whereas (b) and (d) the identified event clusters corresponding to real-world line segments in a spatio-temporal view. The extracted lines at the beginning and at the end of each interval are shown in black. The coordinate system in the upper left corner of (b) and (d) means uses the red and green axes to denote the x and y coordinates of each event, and the blue axis to indicate the temporal axis. where I f is the 3 × 3 screw symmetric matrix form of I 2 . The additional line l 2 is fitted at the center of the interval and by using the same strategy as introduced in Sec 3.1 . With line features l 1 , l 2 , l 3 in three views as well as known angular velocity, we can again stack the constraints for all clusters and figure out the linear velocity through a similar robust nullspace calculation method as before. We denote this baseline implementation the continuous event- based line-line constraint (CE3LC). We verify feasibility and practicality of our approach on two real-world datasets collected by an automated guided vehicle (AGV) and an unmanned aerial vehicle (UAV), respectively. The two datasets are collected by a DAVIS346, which has a resolution of 346x260 pixels.
(1) AGV with a downward-facing event camera
The AGV dataset is collected with a camera mounted on the front of an XQ-4 Pro robot and faces downwards (see Figure 4(a)-4(b)). We recorded a uniform circular motion se- quence on a chessboard. Ground truth is obtained via an Optitrack motion capture system. Our algorithm is working in normalized coordinates, which is why normalization and undis- tortion are computed in advance. Figure 4(b) shows the collected data and line clusterings. To alleviate the influence of noise on the events and resulting inaccuracies in the line clusters and extraction, we utilize a spatio-temporal window with 1,000,000 events (about 0.7s) to estimate the linear velocity.
(2) UAV with a 45° downward-facing event camera
We further evaluate our method on a sequence (Indoor45 9) from [ [ ] which is captured by an UAV equipped with a 45° downward-facing event camera with a resolution of 346×260 pixels.  The maximum velocity ( | v | max ) of the UAV is about 11.23 m/s , which means the intensity images are rather blurry.  Therefore, it is difficult to use the intensity images to extract lines for pose estimation. The event camera however works well in such a challenging scenario. As can be observed in Figure 4(d), our strategy maintains successfully extracted event clusters and starting and ending lines from the raw stream of events. Note that in order to better distinguish lines that are very close, we separate events into positive and negative sets before running the actual clustering algorithm.

4.2.1
Analysis of the Results
We select 5 sequences from each dataset and the results are listed in Table 1 and Table 2. CELC indicates the proposed solver without optimization, CELC+opt the proposed solver with nonlinear optimization, and CE3LC the proposed baseline algorithm without nonlinear optimization.  As can be observed, CELC+opt typically outputs better results than CELC, indicating the positive impact of nonlinear optimization. Furthermore, CELC outperforms CE3LC. The reason is given by the fact that CE3LC relies more heavily on the performance of 2D line fitting, while CELC utilizes all events measurements to constrain the problem. Note that the accuracy of the algorithm highly depends on the accuracy of the line de- tection and fitting, the resolution of the camera, the number of lines in the environment, and other factors analyzed in the simulation experiments. For example, the resolution we use is only 346×260 pixels. As demonstrated by the KITTI dataset [ [ [ ], a common resolution for normal cameras would be in the order of 1392×512, which is much higher. We furthermore believe that the number of studies on line detection and fitting in event streams is still rather limited, and better approaches would certainly benefit the method proposed in this paper.
CONCLUSION
5
Different from existing event-based motion estimation approaches, we are the first to exploit trifocal tensor geometry in order to constrain the dynamics of an event camera from an event stream generated by the continuous observation of arbitrary 3D lines. The closed- form velocity solver employs a novel constraint which we denote the Continuous Event Line Constraint (CELC). We believe that our algorithm is an important first step into the direction of velocity bootstrapping for DVS sensors, and our future work considers the embedding of this solver and the related constraints into a more complete, event-based visual-inertial framework for direct velocity estimation.