E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning
Authors: Xiuhong Lin
Changjie Qiu
zhipeng cai
Siqi Shen
Yu Zang
Weiquan Liu
Xuesheng Bian
Matthias Müller
Cheng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: event camera
2D-3D registration
representation learning
TL;DR: Our network extracts information from event camera data as spatiotemporal point clouds and transforms it into gridded feature tensors. It also pioneers the implementation of event-to-point cloud registration.

Abstract: Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration.
The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: https://github.com/Xmu-qcj/E2PNet.

E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning
Xiuhong Lin Changjie Qiu Zhipeng Cai', Siqi Shen th Yu Zang Weiquan Liu Xuesheng Bian" Matthias Müller Cheng Wangth
Z
Fujjan Key Lab of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University (XMU), China.
Keyy Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China. "Iteel Labs. "Apple Inc. "Yancheng Institute Of Technology, China. asiqishen, cwang xmu edu. en, zhipeng, cai @inteln com { [xhlinxm, qiuchang jie, xsbian @stu. xmu, edu. en
Abstract
Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning- barning-based method for event-to to- co-point cloud registration. The core of E2PNet is a novel feature representation network called Ponnt--onntstoo Tensor (EP2T), which encodes event data into 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameter and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-basee based methods. Compared RGB- based registration. E2PNet is more robust to extreme illumination or fast motion due the of event data. Beyond 2D-3D registration, potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: E2PNet.
also show the
Introduction
Computer vision aims to understand and reconstruct the world from 2D observations (2). Establishing the relationship (pose or correspondences) between 2D data and the 3D world is an important step toward this goal. Conventional 2D-3D registration mostly relies on multi-view geometry 3 4 5 Specifically, 3D (key) points are first triangulated 6 using multiple images with proper disparity. Subsequently, the corresponding 2D features 7 are attached to 3D points for subsequent 2D-3D registration. This framework is by design image-based, which limits the use of other sensors that obtain 3D point clouds directly (e.g., LiDAR, depth cameras).
Recently, learning-baste based methods for direct 2D-3D registration have been proposed However, these methods require high-quality RGB grayscale images which are not always feasible
Xiihhong Lin and Changjie Qiu contribute equally to this work. *Coresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Point cloud map
Points registered by E2PNet
GT points
Points registered by traditional camera-based method
Event Camera
Traditional Camera (Imogga for Registration)
(Vinalication of Raghtration)
Predicted Camera Pose
This work proposes the first learning-based event-to- ponnt-to-ponte point cloud registration method E2PNet. As shown in the figure, challenging illumination/fast motion may cause RGB-based methods (e.g., LCD (T) to fail, resulting in erroneous registration (red camera). Using event data as input, E2PNet is robust such illumination change and accurately recovers the camera pose (blue).
Figure I: Teaser.
in practice. Challenging environments with extreme illumination motion blur exist in many applications, which may cause the algorithm to fail completely (see Fig. for an example).
Unlike traditional cameras, the novel information acquisition principle of event cameras them less susceptible to challenging illumination and motion blur. This advantage motivates us to design robust 2D-3D registration method for event cameras.
I makes
High dynamic range (HDR, up 120 dB (13 and high temporal resolution bring excellent perfor- mance to event cameras but also create new challenges. The unstructured data stream of event cameras sparse in the spatial plane but dense (up to 300 million events per second [4] and non-ume oon-uniform in the temporal dimension. Existing methods mostly process events as image-like feature maps [15] [6] or voxels [8] [9]. which arguably loses spatio-temporal details. Although event cameras have been used for other vision tasks such as optical flow estimation [15 18 20], moving object detection
21 [2 and high dynamic image reconstruction
24 the task of event-to-point cloud registration
(E2P) has not yet been studied.
There are several challenges. First, event cameras only capture the surface appearance of the scene, whereas 3D point clouds encode geometric structures, which is inherently challenging [ Second, event cameras have lower resolution than traditional cameras and only capture the brightness variation rather than the brightness itself. Finally, event data and point clouds can both be considered as unstructured 3D data, which are harder to process than 2D images. Moreover, the lack of an E2P dataset with accurate annotations limits the research of such cross-modal registration methods.
To address these problems, we present E2PNet, the first E2P framework and the corresponding dataset for end-to-end learning. Specifically, we propose a modular event representation network called (EP2T), which applies local feature aggregation with different temporal and spatial distance weights and spatio-temporal separated attention mechanisms to effectively encode sparse and in-homogeneous event data. In the meantime, EP2T adaptively aggregates local neighborhood features with uneven event data in the temporal and spatial dimensions and outputs structured sgrid-shapedi feature tensor which can be directly plugged into image-based -aaed 2D-3D registration frameworks. E2PNet is capable of E2P registration for indoor scenes of different scales (single room to one floor) and produces the corresponding 6-DOF transformation between the event camera and 3D point clouds. As shown in Fig. I significantly improve the registration performance on challenging data. Our main contributions are:
replacing the RGB inputs with event data can
We propose EP2T, the first representation learning architecture transform event spatio- temporal point clouds into a grid-shaped feature tensor. EP2T can adaptively learn how extract critical spatio-temporal information according downstream tasks, and can be applied as plug-and-play module to existing event camera processing networks.
We propose E2PNet, the first approach that allows direct registration of 2D event cameras and 3D point clouds. To facilitate follow-up research, we also propose a framework to construct E2P datasets using existing SLAM datasets.
We conduct extensive experiments and analysis to demonstrate the effectiveness of E2PNet. for E2P, and the potential of the EP2T module benefit in other diverse tasks.

---

2 Related Works
2.1 Event Representation
The asynchronous unstructured event data poses great challenge for computer vision. While spiking neural networks (SNN) [25] can directly use the raw data of event cameras, they require special hardware and has low flexibility. A more common approach is to process batches of event data via an intermediate representation, e.g., an image of positive and negative events over certain horizon. The processing window is usually set to a fixed number of events (FEN) 226 20 or a fixed interval time (FIT) 115 20).
Hand-crafted representations usually convert raw event data into 2D grid-shaped feature maps with sensor resolution using the statistics of raw data, for example, event counting [27]. average/latest timestamp of events on each pixel [15] or temporal distance-weighted statistics [7.. While these strategies can approximate the distribution of event data with respect to space and time, some important details (i.e. spatio-temporal correlation) may be lost due to the aggregation. To address 30 31 discards most of the events and only keeps the latest few events for each pixel and analyzes them detail. However, these methods cannot encode distant apatio-temporal relationships. More recent work tries to incoporate these relationships by representing event data as graphs [32 33 34 35). Although more spatio-temport information can be extracted, building graph structure is complicated and computationally expensive. Each hand-crafted representation has different advantages and disadvantages. Hence, choosing the right representation for a given tasks is often not trivial. The proposed E2PNet also utilizes hand-crafted methods [15] 20 17 However, we use them in conjunction with our learned features instead of raw event data produce the final grid-shaped feature tensors.
this shortcoming, another set of methods 128 29
Learning-based representations train models to adaptively extract critical information based on the specific task. However, the inhomogeneity and lack of structure make event representation learning challenging. EST [16] and DDES (30] propose to learn tensor-bases representations. which regard the events of each pixel as pulse signals and optimize the impulse response function. ECSNet (36) uses 3D point cloud learning methods to extract point- features. However, these feature points are incompatible with the processing pipeline of downstream tasks, and generalize poorly to other tasks. In contrast, we aim to fully extract the spatio-tempora correlation of the event data and transform into a grid feature tensor representation for better compatibility.
2.2 2D-3D Registration
This work focuses on the direct 2D-3D registration problem, which has been studied for conventional images. Li et al. [37 use the multi-view projection image of 3D objects to extract the corresponding HOG features, and train a feature mapping network to push the features from the same 2D-3D) objects closer. 3DTNet 38 uses RGB-D point clouds and images to train distinctive 3D) descriptors. However, these methods can only be used to register images to colored point clouds, which is not always feasible. E2PNet does not generate features based on color information, hence it can be applied to conventional point clouds without colors. 2D3D-MatchNet [T] and LCD [I] learn consistent feature representations of 2D images and 3D point clouds, realizing 2D-3D registration through coosl feature retrieval. P2-Net 12] learns the detection and matching of potential feature points from images and RGB-D points. Deepl2P [II] transforms image- cloud registration into point cloud visibility classification and then maximizes the number of visible points by optimizing the camera pose. A common drawback of these image-based methods is that they require high-quality images to perform well, which cannot be guaranteed in applications with challenging illumination conditions or camera motion. As shown later in the experiments, introducing event data for 2D)31D registration makes E2PNet much more robust in challenging environments.
3 Methodology
E2PNet treats the input event data as spatio-temporal point clouds. Before explaining how this can be done, we briefly introduce the generation process of event data here. During data capture, each pixel of the event camera responds independently and asynchronously to illumination changes. Specifically, given image of size (H, W), denote 1 [tt,,) h, w) as the illumination at time t, for the pixel at (h, w).
3

---

Aggregation
Event Points
Feature Aggregation
Local neighborhood boundary
Spatio-temporal point cloud
Spatio-teppora
Aggregution center feature
point-wise feature
Heights
Height
Heighd
Width
Time
Widih
Time
Width
Time
Point-based Representation
Temporal patch aggregarion
Spatial patch
Center Fenture
Spatio-temporal
Tousor-based
augreguttion
Kernel
Global
Representation
Temporal Kernel
Feature
Input
Output
Event (bp:W2daPPP)
F
FSTA
Grided
Data
Spatial
Tensor
Fat
Figure 2: The proposed Prent-Points-to-Tensoo (EP2T) network. EP2T treats event data as spatio-temporal point clouds. Local Aggregation (LAA) first generates feature aggregation centers and aggregates local features from neighboring points. Subsequently, Spatio-temporal Separated Attention (STA) extracts global features using attention. To handle the spatio-temporal inhomogeneity problem, LA and STA separate spatial and temporal information during computation. Next, Feature Propagation (FP) propagates features from aggregation centers to each event point based on the distance. Finally, grid-shaped feature tensors are obtained by different yet complementary event tensorization methods.
When the inequality
[log(( h, w)) log([(t)-1,,,, w,)l
(I)
is satisfied. This pixel will generate an impulse response called an event, which is expressed as (h, w, p). The polarity p E {+1,-1} indicates the direction of the brightness change (increasing or decreasing) and 7 is the trigger threshold. f;-1 represents the time when the event was triggered last time pixel (h, w). The imaging plane and the time axis can be combined establish a three-dimensiones apatio-temporal coordinate system, where events characterized by coordinates (h, w, t4) collectively form a spatio-tempora point cloud.
In order to compute the pose of the camera w.r.t. a 3D point cloud, E2PNet first encodes the spatio-temporal point clouds into 2D grid-shaped feature tensor. This process relies EP2T, which is the core component E2PNet and will be introduced in detail Sec. Sec. 33 After obtaining the encoded feature tensor, we can formulate the event-to-point registration (E2P) problem as the registration of 2D feature maps and 3D point clouds, which can be handled by mature image-based frameworks mm Since EP2T is differentiable, the training of E2PNet is simply done by connecting EP2T with the registration network and performing back proopppopggttion in the same way as the original registration network. Empirically we found that directly adding EP2T to the registration network without changing any hyper- parameter works reasonably well, which is another advantage of E2PNet.
3.1 Spatio-Temporal Reprsentation Learning with wvent-Points-t-TTee Tensor (EP2T)
As illustrated in Fig. EP2T consists of three consecutive modules: Local Aggregation (LA), Spatio-temporal Separated Attention (STA), and Feature Propagation (FP).
Given a spatio-tempont point cloud derived from the input event data, LA evenly samples a set of feature aggregation centers in the spatial and temporal domains. For each sampled center, LA computes the locally aggregated features using the neighboring input points. To produce the global feature, STA performs attention over these aggregated features, with special design to handle the spatio-temporal inhomogeneity. To produce features on each input point (from aggregation centers), FP propagates each feature from STA to the neighboring points based on the distance. Finally, employ classical tensorization methods to process the point-wise features into grid-shaped tensors.
The novelty of LA (see Sec. 3.1.1 for more details) and STA (Sec. 3.1.2 lies in the separation of spatial and temporal domains, which is designed to handle the inhomogeneity of spatio-temporal point clouds. FP (Sec. 3.1.3) propagates aggregated features to neighboring spatio-temporal points so

---

that standard point-wise tensorization can be used to generate grid-shaped outputs, which enables the use of RGB-based 2D-3D registration frameworks for E2P.
3.1.1 Local Aggregation (LA)
Given an event camera of resolution (H, W), we extract N events from the input and convert them into a spatio-temporal point cloud with N points. Unlike conventional point clouds, the spatial and temporal domains of our inputs naturally have different metric scales, which makes existing point cloud representation networks [39] ineffective for feature encoding (see Sec.
To address this problem, the proposed LA module aggregates features across the spatial and temporal domains seperately. Specifically, for each input point e = (h, w, 1, L.P) with E {-1, representing the brightness change direction, we first normalize it via ê = H W tao PP), where /max and mii are the maximum and minimum time stamps of the input point cloud.
hh x
The feature aggregation process of LA is inspired by PointNet++ (39]. There are two steps during aggregation: (1) generate a set of aggregation centers; (2) calculate the aggregated feature for each center by employing neighboring points.
PointNet++ [39] generates aggregation centers using Farthest Point Sampling (FPS) 40), which is relatively slow in wvent-based applications that require a high response speed. Furthermore, FPS is not suitable for spatio-temporal point clouds with non-uniform density distributions (might generate redundant points in dense regions and fewer points in sparse regions). To address this issue, LA uses uniform sampling with fixed spatial and temporal interval generate uniformly distributed aggregation centers. Specifically, given the normalized point cloud E spanning over a spatio-temporal cube of size (H, W. T). we generate set of aggregation centers C' by sampling M (30*30*5 in our experiments) uniform grid locations within this cube.
To aggregate features from neighboring points, PointNet++ 39 first selects K nearest neighbors from É for each C E C' and then replaces neighbors whose Euclidean distance is too far away from с with the point closest to Cj. After selecting neighboring points, multi multi-layer convolution is performed to produce the aggregated feature. Using Euclidean distance omits the inhomogeneity of the spatial and temporal domains, which exists even with our initial normalization. Hence, we propose to separate the spatial and temporal domains during neighborhood selection. Specifically, we change the distance metric during the selection of nearest neighbors (K = 64 in LA) to axdspace (ê;, c,) + 9dTime (ê,, C,). We also two distances separately in the spatial and temporal domains to choose points for replacements, i.e., we replace a neighboring point é; with the closest point to c; if dsspcce (ê;, c,) 2 a or dimmm (é,, c,) 2 B. dl-)) is the mean squared distance of the corresponding domain.
Separating the spatial and temporal domains during feature aggregation addresses the inhomogeneity problem from two aspects. (1) We can provide constraints to the spatial and temporal domains separately to prevent choosing neighboring points with small distance in one domain but a far distance in another. (2) We can provide different a's and B3's to create multiple aggregated features that focuses on different domains. For the second aspect, we produce three sets of features in LA by setting = {0.8, 0.1, 0.5) and 3 = {0.1, 0.8.0.5) respectively, which allows the aggregated feature to focus more on spatial, temporal, and spatio-temporal domains.
3.1.2 Spatio-temporal Separated Attention (STA)
After generating the feature centers and the locally aggregated features, we use the STA module to extract global features and merge features from different domains into the uniform spatio-temporal domain. Similar to LA, STA also separates the spatial and temporal domains to effectively handle the inhomogencity problem.
F,, F.. and F.t in Fig. 3 denote the aggregated temporal, spatial and spatio-temporal features from LA respectively. We first create global features for the spatial and temporal domains separately using the temporal and spatial residual kernels. Each of these two kernels performs self- celf-attention over features from the same domain, and cross-attention between features from different domains. Subsequently, we merge them into a temporal and spatial residual using retention gates (sigmoid and tanh functions) inspired by 41 42 After producing the residuals for spatial and temporal domains separately, we merge them into the spatio-temporal domain using the spatio-temporal aggregated feature F to produce the global spatio-temporal feature fSTA (c,) for each aggregation center cj.
5

---

Cross- Aftention
Siguold
Output
Fs
Temporal
Center-wise
Residual
Temporal Residual Kernel
Fst
Self-
Atfeation
Tanh
Sigmold
Aftention
Og
Output
Tanh
Global
feature
Attention
Sputial Residual
Sigmoid
Spatie- Residual Kernel
STA
Siggaold
Output
Fz
Spatial Residual
Spatio-tempore Residual
Sigmoid
[a]Single Domain Residual Kernel
(bySpatio-temporal Residwal Kernel
(c]Spatio temporal Separated Attention Medulis
Figure 3: Detailed structure of the proposed STA module. (a) Encode features for the spatial and temporal domains separately using the spatial and temporal residual kernels. (b) Merge both residuals into the apatio-teppora domain. (c) Apply the apatio-tempont residuals the apatio-tempore features.
3.1.3 Feature Propagation (FP) and Tensorization
After producing the global features at the aggregation centers, we propagate them neighboring input event points using a weighted average to get feature for each input point. Specifically, given the global features FSTA {fsia (c,)) and the normalized event points E = {ē:}, we compute the propagated feature by
Eg [i.s E, Wss
frr(C;;
where Way max(]ee__c,1B..00 10 is used to prevent division by zero.
Finally, to convert point-based features into grid-shaped features, use three different yet comple- mentary event tensorization methods [5 20 I7 to convert the set of ffp(ê;) into three different types of 2D grid-shaped sparse feature tensors, and concatenate them to produce the final tensorized feature map.
3.2 E2P Dataset Construction
Since there is no dataset for E2P, we use multi-sensor SLAM datasets to construct the ground-truth (GT) labels for training. We select the widely used MVSEC 43 and VECtor 44 to build the MVSEC-E2P and VECtor-E2P datasets, which incorporate LiDAR, traditional cameras and event cameras at the same time. Furthermore, they have an accurate multi-sensor time synchronization and calibration. Through the GT pose provided in these two datasets, we establish the GT matching relationship between events (images) and point clouds for E2P tasks. A naive way to obtain such GT would be to project point clouds into the image plane, which is slow in practice. Instead, we construct a camera model in the world coordinate system and use this model to construct a quadrangular viewing frustum. Selecting the points in the quadrangular viewing frustum is practically > 2r faster than the naive solution.
Experiments
In this section, first demonstrate the effectiveness of the proposed E2PNet for event-to-point cloud registration (Sec. 4.I). which our primary goal. Then, we conduct ablation studies to verify the effectiveness of individual modules in EP2T (Sec. 4.2. Finally, analyze the generalization of EP2T on other event-based vision tasks (Sec.
4.1 Event to Point Cloud Registration
Datasets and Preprocessing. We use two representative datasets described in Sec.
3.2 MVSEC
4
uses a 16-beam LiDAR and an event camera with a resolution of (346,260); event camera simultancously generate grayscale images. Since only indoor sequences have high-precision pose

---

Table 1: Quantitative results of 2D-3D registration. Since the three hand-crafted event tensorization representation methods [15] 17 20 are complementary, we use them jointly to achieve the best performance and call this hybrid method E-Statistic. To fairly compare against point- based repre- sentations, we also remove the tensorization process of E2PNet, i.e., w/o TR. The best performing method is highlighted.
Deep12P
11 [Registration-aasee
LCD
1 (Retrioval-based)
Input Representation
Method
MVSEC-E2P
VECtor-E2P
MVSEC-E2P
VECtor-E2P
RE("()) TB(M)L) RE("() TE(m)) RECU()) TE(M)) RE()() TE(m))
Traditional Image
Grayscale Image
7.922
0.370
11.343
3.176
6.335
1.347
17.879
13.200
F-Statistic
6.748
0.250
10.654
3.524
4.968
1.297
11.034
9.416
Event(Tensoo (ensor-bbsed)
Tore 19
7,465
0.192
10.542
4.565
4.855
1,350
9.521
7.254
Ours
5.127
0,164
8.778
2,454
3.606
0.821
8.672
7,403
RCSNet
36
8.075
1.612
10.149
2,542
4.985
1.263
20.740
13.284
Event(Pointtbuse -ased)
Ours(w/o TR)
6.021
0.721
8.795
3.212
4.422
0.768
9.120
6.551
ground truth (GT), we use the indoor-a and indoor-y sequences for training and testing respectively, where E [1.3] and y 4. VECtor [44] uses 128-beam LiDAR and an event camera with a resolution of (640,480). Different from MVSEC, VECtor contains various large-scale indoor environments with long trajectories. The GT pose is derived via ICP(45) between LiDAR scans and dense environment point clouds captured by a high-precision laser scanner. We use the units- units-dolly, units-scooter, corridors-dolly and corridors-we wwalk sequences for training, and the school-dolly and school-scooter sequences for evaluation. For both datasets, we randomly acquire event data in an (256,256) area from event camera data input; simultaneously also acquire grayscale images with the same field of view and resolution for comparison.
Evaluation. Following the previous standard [46] we use the translation error TE- = ||Ter
to evaluate the accuracy of predicted camera poses in E2P. T and R. represents the translation vector and rotation matrix respectively, and tr(-) is the trace of matrix.
Teed la and the rotation error: RE arccos
Implementation Details. We follow the FEN 226 20 principle and acquire 20000 consecutive events at a time and sample N 8192 events from them. Since EP2T has encoded the spatio- temporal information as 2D grid tensors, we can regard E2P as the registration of 2D feature maps with 3D point clouds. We choose LCD [1 and DeepI2P (I]) as the representatives to test the performance of E2P using feature retrieval-based and point visibility-based registration. LCD maps the corresponding 2D tensor and 3D point cloud into a sigh--imensionall feature space, where training aims to make the feature distance as close as possible. During training, we fetch events (image) in the current pose and their corresponding point cloud patch according to the GT pose (see Sec. B.2). The principle of Deepl2P is to judge whether the input point cloud appears in the corresponding field of view of events (image). During training, we use the visible point clouds for each event view and its surrounding point clouds (enlarged by 30%) for supervision. All methods are implemented using Pytorch. Training is done following the setup of individual baselines on a 3090Ti GPU.
We compare E2tNet against conventional camera-based registration (i.e., the original version of LCD and Deep12P) and other event representation methods, including tensor-based [19
17 20 and point-based (36] methods. To compare against point-based methods, we do not perform the final tensorization (i.e., w/o TR) so that the encoded features are also point-based. Since there is standard technique to apply point-based event representations to image-based registration frameworks, for fair comparisons, we implement a simple encoder with max- max-pooling and linear layer (see appendix for details) to project the posnt-based features to global feature vector. Subsequently, we use this vector to replace the original global feature (computed by performing convolution on the tensor-based representation) in both L.D and Deep12P. For LCDD-based methods petrieval-bbsedd] we use the trained model to encode the feature on the target point cloud under different poses to build a point cloud feature library. During inference, we extract the corresponding pose in the feature library
15]
7

---

Point clouds projected through GT pose
Point clouds projected through predieted pose
Projected Point
Over
Exposure
Motion
Blur
Nou-
Challenging Situation
Traditional Camera
E-Statistic
TORE
Ours
Ground Truth
Figure 4: Qualitative comparison E2PNet with other methods. The green and cyan boxes represent the projected position of the 3D point clouds onto the view plane of the camera after rigidly transforming to the camera coordinate system using ground truth pose and predicted pose respectively, Compared to the image-based baseline (LCD), event-based methods are more robust to challenging illumination and motion. E2PNet (ours) performs best among all event-base prent-based methods.
Table 2: Ablation. Removing individual modules from E2PNet hurt the registration accuracy.
Ours w/o
Ours w/o
Ours w/o LA+FP
Ours w/o LA+STA+FP
Method
Ours
S-T separated
STA
RE(*))
3.606
3.710 0.824
3.922
4.878
4.968
TE(m)()
0.821
0.913
1.232
1.297
that is closest to the feature of the event data. For DeepI2P- based methods (point cloud visibility classification-bass basedd, we use the same optimization method as the original method [IT].
As shown in Tab. D replacing conventional image inputs with event data improves the registration accuracy significantly and consistently across datasets and buseline methods. E2PNet performs the best among both tensor-based and point-based E2P methods, outperforming both hand-crafted (E-Statistic) and learning-based methods. Qualitative results in Fig. Hig.-ffurti further show the robustness of E2PNet in different environments. Finally, due to better compatibility, tensor-based methods perform better than point-based methods, and sometimes by large margin. Note that although point-based representations can be applied to 2D-3D registration, since only global feature vector of the input (event) image required, they may not be compatible with tasks that require pixel-level feature maps.
4.2
Ablation studies
In this section, we analyze the effectiveness of individual components of E2PNet, i.e., the Local Aggregation module (LA), Spatio-temporal Separated Attention module (STA), Feature Propagation module (FP). and the mechanism for seperating the spatial and temporal domains in STA. We use LCD as our baseline model and conduct experiments on MVSEC-E2P dataset. As shown in Tab. 2 the registration accuracy reduces as components are removed, showing the importance of each are coupled modules (we need FP to propagate features from
component. Note that since LA aggregation centers individual event points), add and remove them together.

---

Table 3: Results of EP2T on other vision tasks. "Ours" means the result obtained after incorporating our EP2T module. For a fair comparison, The relevant training settings are the same as the original framework. The dataset used for evaluation is indicated after "@"
Flow Estimation
Event-to-Image
Reconstruction
Object Recognition
Method
AEB(J) Outlier(%)XL)
Method
MSE(J) SSIM(1) LPIPS(U)
Method
Accuracy(1)
cGNN 20 @
33
0.34
0.06
cGAN 20
43
0.479
0.431
0,450
[6 @ 31
0.746
cGAN(ours) @
13
0.30
0.005
couN( @
0.476
0.460
0.435
EST(ours) @ 31
0.771
EV-Flow 15 @
43
0.478
0.03
BTEB @ 48
0.20
0.32
0.46
[6 @ 49
0.949
(V-Flow(ouss)
0.462
0.03
(ors( 4S
0.16
0.32
0.42
EST(ours) @ : 0.957
4.3 Generalization Experiments
Though we focus on the task of E2P in this work, the architecture design of our EP2T module is also applicable to other vision tasks. In this section, we apply EP2T to three different tasks (see below) to show its potential. Specifically, we embed the EP2T module into existing baseline models as an additional information channel and compare the performance. For all baseline models, we use the released source code and the corresponding training setup during experiments.
Flow estimation aims to infer the instantancous speed and direction of the corresponding pixel motion of the object on the imaging plane. Compared to the traditional caume-differeenc-bbssee optical flow estimation methods, event camera optical flow estimation has a higher temporal resolution, enabling more accurate estimation of object motion in complex scenarios involving fast motion and high rotation rates. We employ the standard networks cGAN (00 and EV-FlowNet [15] as baselines in this experiment. Following standard practice, we utilize the average endpoint error (AEE) (50) and the outlier rate (the ratio of errors greater than three pixels) to evaluate different methods. We use the MVSEC dataset evaluate these different methods.
Event-to Event-to-image to-immge reconstruction is another canonical task for event cameras in which a high-contrast image with sharp edges is reconstructed from events, making full use of the advantages of the event camera's 's high contrast and fast response speed. In this experiment, we use BTEB 447 on the ECD dataset [51 and cGAN (20] on the MVSEC dataset [4] the baselines and use the mean squared error (MSE), structural similarity (SSIM) [52), perceptual similarity (LPIPS) 53 for evaluation.
Object recognition with conventional cameras remains challenging due to high latency and motion blur. Event cameras are naturally less sensitive to such problems. this experiment, we employ the widely used EST [54] network as the baseline. To assess the performance of our proposed methodology, we utilize the publicly available N-caltech10) dataset [49 and N-cars dataset [3] and use the recognition accuracy the evaluation metric.
As shown in Tab. 3 adding the EP2T module improved the performance of baselines on various datasets and tasks. This result clearly demonstrates the potential of EP2T for other vision tasks. Since our initial motivation to design EP2T driven by the 2D-3D registration task, we believe the current architecture design and the way to apply our method to other tasks both have further space to be optimized, which we leave as the future works.
4.4 Time and Memory Efficiency
We analyzed the time and memory efficiency of the event representation learning module EP2T, as well as the complete E2PNet consisting of EP2T and 2D-3D registration network. As shown in Tab. [ our E2PNet is slightly slower and uses more memory than baseline methods. However, the overhead introduced by EP2T is arguably small given the significantly improved accuracy.
We also analyzed of the impact of two hyperparameters (number of FEN and number of sampling points) from three perspectives: accuracy, time efficiency, and memory cost in Tab. [S] Selecting an appropriate number of events per batch (FEN) optimizes algorithm performance. A small FEN yields insufficient information, whereas an excessive FEN extends sampling time and limits the exploration of spatiotemporal details. Raising the number of EP2T sampling points enhances spatiotemporal detail capture but escalates computational costs. The major contributor to the higher memory usage is the EP2T network designed based on PointNet++ (39], where the woint-wise distance calculation

---

Table 4: Time and memory efficiency of different methods. We compared the event representation module and the Event-to-point cloud registration framework, which consists of the event represen- tation module and the 2D-3D registration network. All experiments were performed with batch size of on a 3090Ti GPU using the MVSEC-E2P dataset. We jointly use three cand-cratted event tensorization representation methods 13 7 20 to achieve the best performance and call this hybrid method E- S-atttistic. Compared to the baselines, E2PNet demonstrated significantly higher accuracy and marginally lower but still acceptable time and memory efficiency.
2D-3D registration using LCD
Event representation
Method
Grayscale Image
E2PNet
E-statistic
ECSNet
36
EP2T
E-statistic
Time(ms)(J
55.29
110.62
55.00
81.82
42.8
3,16
Space(MB)))
2218
7890
2216
2530
4650
0.68
RE(O()
6.336
3.606
4.968
4.985
x
TF(m)L)
1.347
0.821
1.297
1.263
x
x
Table 5: The influence of varying FEN and the number of EP2T sampling points on the accuracy and efficiency of E2PNet. FEN represents the event count in each batch, from which E-statistic employed to extract global spatial features. Our EP2T algorithm selects some sampling points from the events processed in each batch as the aggregation center of the spatiotemporal information. All experiments are conducted MVSEC-E2P dataset using LCD [ framework, with inference performed on 3090ti GPU and a batch size of 1.
Number FEN
EP2T sampling points
RE()())
TE(mX()
Time(ms)(L)
Space(MBX()
512
3.813
1.089
65.51
2992
4096
3.777
0.954
79.55
4394
20000 (Ours)
8192 (Ours)
3.606
0.821
110.62
7890
16384
3.247
0.789
144.09
12190
8192
3.479
1.074
90.34
7258
8192
60000
4.242
1.426
109.50
7908
operation leads to memory growth that scales with the number of EP2T sampling points. However, when reducing the number EP2T sampling points to 512, E2PNet runs a similar speed (65.5ms) and memory cost (2.99GB) as other baselines yet still has higher accuracy.
In summary, our EP2T module is both effective and adaptable, introducing a novel approach for extracting spatiotemporal correlations and detailed information from event camera data. According to different performance requirements and available computing resources, the number of EP2T sampling points and sampling strategies can be flexibly adjusted to strike an efficiency-accurag trade-off. The impact of different sampling strategies can also be viewed in the Appendix.
Conclusion and Limitation
In this work, we propose E2PNet, the first dearning-based event-to-point cloud registration (E2P) method. We design a novel Preet-Points-to-Teneeee
(EP2T) network, which encodes event data into feature tensors in the shape of 2D grid such that mature RGB-based 2D-3D registration frameworks can be used for the E2P task. EP2T treats event data as spatio-temporal point clouds and separates the temporal and spatial domains for processing to handle the inhomogeneity of the event point cloud, Our experiments show that E2PNet is superior compared to other learning- based or handcrafted event representation methods and the traditional RGB-based registration in extreme environments (overexposure, motion blur). In addition. we also show the potential of the EP2T module in vision tasks beyond 2D-3D registration. Since E2PNet inspired by 3D point-based based learning architectures [39]. similar limitations exist. For instance, an excessive number of sampling points will result large memory consumption and efficiency degradation of E2PNet. However, further efficiency improvements are possible by some optimization techniques like delayed aggregation (55) sparse convolution [56]. We believe that solving this problem an interesting and important future research direction.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China (61971363, 62301473); by the FuXiaQuan National Independent Innovation Demonstration Zone Collaborative Innovation Platform NN033002CQ0TT20210000 ties (No. 20720230033); by PDL (2022-PDL-12) by the China Postdoctoral Science Foundation (No.2021M690094)) We would like thank the anonymous reviewers for their valuable comments.
by the Fundamental Research Funds for the Central Universi-
References
[1] Quang- Hieu Pham, Mikacla Angelina Uy, Binh-Son Hua, Duc Thanh Nguyen, Gemma Roig, and Sai-Kit Yeung. Led: Learned cross-domain descriptors for 2d-3d matching. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 1 11856-1 1164, 2020.
[2] Richard Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
[3] Jogendra Nath Kundu, MV Rahul, Aditya Ganeshan, and Venkatesh Babu. Object pose estimation from monoeular image using multi-view keypoint correspondence. In Computer Visi 2018 Workshops: Munich, Germany, September 8-14, 2018, Proceedings, Part III 15, pages 993... Springer, 2019.
[4] Francese Moreno -oguer, Vincent Lepetit, and Pascal Fua. Accurate non-iterative o(n) solution to the pnp problem. 2007 IEEE I 11th International Conference on Computer Vision, pages 1-8. leee, 2007.
[5] Andrew Davison, lan Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real- time single camera slam. IEEE transactions pattern analysis and machine intelligence, 1956):10 2007.
[6] Johannes I. Schonberger and Jan-Michael Frahm. Structure- cowouur-ffoo-mooteee -moioo revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[7] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
[8] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions robotics, 31(5): 11(5):117-1166. 2015.
[9] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 467-483. Springer, 2016.
[10] Mengdan Feng. Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud. In 2019 International Conference Robotics and Automation (ICRA), pages 4790-4796. IEEE, 2019.
[11] Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-point cloud registration via deep classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15960-15969, 2021.
[12] Bing Wang, Changhao Chen, Zhaopeng Cui, Jie Qin, Chris Xiaoxuan Lu, Zhengdi Yu, Peijun Zhao, Zhen Dong, Fan Zhu, Niki Trigoni, et al. P2-net: Joint description and detection of local features for pixel and point matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16004-16013, 2021.
[13] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event- based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 4440):154-180, 2020.

---

[14] Bongki Son, Yunjae Suh, Sungho Kim, Heejae Jung, Jun-Seok Kim, Changwoo Shin, Keunju Park, Kyoobin Lee, Jinman Park, Jooyeon Woo, et al. 4.1 a x40x 480 dynamic vision sensor with a 9pm pixel and 300meps address-event representation. In 20/7 IEEE International Solid-State Circuits Conference (ISSCC), pages 66-67. IEFE, 2017.
[15] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Ev-flownet: Self- supervised optical flow estimation for event-based cameras. arXiv preprint a80i 2018.
[16] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End- to-end learning of representations for asynchronous event-based data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
[17] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danilidis. Unsupervised wvent-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 989-997, 2019.
[18] Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch. Learning an event sequence embedding for dense event-based deep stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1527-1537, 2019.
[19] R Wes Baldwin, Ruixu Liu, Mohammed Almatrafi, Vijayan Asari, and Keigo Hirakawa. Time- ordered recent event (tore) volumes for event cameras. IFEE Transactions on Puttern Analysis and Machine Intelligence, 45(2):2519-2533 2022.
[20] Mohammad Mostafavi, Lin Wang, and Kuk-Jin Yoon. Learning to reconstruct hdr images from events, with applications depth and flow prediction. International Journal of Computer Vision, 129:900-920, 2021.
[21] Anton Mitrokhin, Chengxi Ye, Cornelia Fermüller, Yiannis Aloimonos, and Tobi Delbruck. Ev- imo: Motion segmentation dataset and learning pipeline for event cameras. In 2019 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 6105-6112. IEEE, 2019.
[22] Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based motion segmentation with spatio-temporal graph cuts. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[23] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with event camera. IEEE transactions on pattern analysis and machine intelligence, (916))1964-199 2019.
[24] Xiang Zhang, Wei Liao. Lei Yu, Wen Yang, and Gui-Song Xia. Event-based synthetic aperture imaging with hybrid network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14235-14244 2021.
[25] Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural networks, 111:47-63, 2019.
[26] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence, 43(6) 2019.
[27] Ana Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scara- muzza. Event-based vision meets deep learning steering prediction for self-driving cars. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 2018
[28] Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi leng, and Chiara Bartolozzi. Event- based visual flow. IEEE transactions on neural networks and learning systems, 25(2):407-417, 2013.
12

---

[29] Mohammed Almatrafi, Raymond Baldwin, Kiyoharu Aizawa, and Keigo Hirakawa. Distance surface for event-based optical flow. IEEE transactions on pattern analysis and machine intelligence, 42(7):1547-15550 2020.
[30] Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch. Learning an event sequence embedding for dense event-based deep stereo. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1527-1537, 2019.
[31] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1731- 1740, 2018.
[32] Yijin Li, Han Zhou, Banghang Yang, Ye Zhang. Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Graph-based asynchronous event processing for rapid object recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 934 2021.
[33] Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based motion segmentation with spatio-temporal graph cuts. IEEE Transactions Neural Networks and Learning Systems, 2021.
[34] Anton Mitrokhin, Zhiyuan Hua, Comelia Fermuller, and Yiannis Aloimonos. Learning visual motion segmentation using event surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14414-14423, 2020.
[35] Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. Aegnn: Asynchronous event-based graph neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages
2022.
[36] Zhiwen Chen, Jinjian Wu, Junhui Hou, Leida Li, Weisheng Dong, and Guangming Shi. Ecsnet: Spatio-temporal feature learning for event camera. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
[37] Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas Guibas. Joint embeddings of shapes and images via cnn image purification. ACM transactions on graphics (TOG), 34(6):1-12, 2015.
[38] Xiaoxia Xing, Yinghao Cai, Tao Lu, Shaojun Cai, Yiping Yang, and Dayong Wen. 3dtnet: Learning local features using 2d and 3d cues. 2018 international conference on 3d vision (3dv), pages 435-443 IEEE, 2018.
[39] Charles Ruizhongtai Qi, Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[40] Carsten Moenning and Neil A Dodgson. Intrinsic point cloud simplification. Proc. 14th GrahiCon, 14:23, 2004.
[41] Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural networks: Lstm cells and network architectures. Neural computation, 31355:1255 2019.
[42] Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Strpm: A spatiotem- poral residual predictive model for high-resolution video prediction. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 13946-13955, 2022.
[43] Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters, 3(3):003--039, 2018.
[44] Ling Gao, Yuxuan Liang, Jiaqi Yang, Shaoxun Wu, Chenyu Wang, Jiaben Chen, and Laurent Kneip. Vector: versatile event-centric benchmark for multi-sensor slam. IEFE Robotics and Automation Letters, 7(3):227-8222 2022.
13

---

[45] Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[46] Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 1 11824-11889 11833, 2020.
[47] Federico Paredes- Vallés and Guido CHE de Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constaney. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3446-3455, 2021.
[48] Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research, 36(2):142- 149, 2017.
[49] Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
[50] Zhexiong Wan, Yuchao Dai, and Yuxin Mao. Learning dense and continuous optical flow from event camera. IEEE Transactions on Image Processing, 31:7237-72511 2022.
[51] Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research, 36(2):142-149, 2017.
[52] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, (34):000 612, 2004.
[53] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[54] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End- to-end learning of representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
[55] Yu Feng, Boyuan Tian, Tiancheng Xu, Paul Whatmough, and Yuhao Zhu. Mesorasi: Architec- ture support for point cloud analytics via International Symposium on Microarchirecttree (MICRO), pages 1037-1050. IEEE, 2020.
In 2020 53rd Anmual IEEE/ACM
[56) Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. Event-based asynchronous sparse convolutional networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 415-431. Springer, 2020.
14


Reviews:
[{"review_id": "cyjU4JJAhO", "reviewer": "Reviewer_mNvD", "summary": "In this paper, the authors proposed the first learning-based work that can handle event-to-point cloud registration (E2P). More specifically, a novel Event-Points-to-Tensor (EP2T) network is proposed to encode the data from the event camera into features tensors in the form of a 2D grid. The temporal patch aggregation and spatial patch aggregation combined with spatio-temporal kernel are applied to obtain the global feature. Then based on the output gridded tensor, a standard 2d-3d feature based algorithm is applied to obtain the structure and motion of the final 3D reconstruction scene.\n\nThe experimental results based on the two representative datasets demonstrate the performance, mainly in terms of accuracy, of the proposed E2PNet on the task of event to point cloud registration.", "questions": "See the \"Weakness\" section.", "limitations": "See the \"Weakness\" section. Furthermore, the final score assigned to the study is not solely determined by the discussion among peer reviewers. If the authors can solve my main concerns, I would like to raise the score.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation of this paper is well designed, that is introducing the first learning-based architecture to handle the sparse reconstruction based on event cameras. More specifically, the spatial and temporal attention mechanism together with the feature propagation module to obtain the final output gridded tensor. The design of the EP2T network is straightforward, elegant and easy to follow.\n\nIn addition to its innovative approach, the paper stands out for its well-structured writing style that includes a clear and concise method statement. The authors have taken great care in presenting their research in a manner that is easily understandable to the readers. Furthermore, the inclusion of visual representations helps to further elucidate the concepts and techniques discussed in the paper.\n\nMoreover, the performance shown in the statistical experiments especially in Table 1 demonstrate the superior performance of the proposed approach over the previous state-of-the-art approaches.", "weaknesses": "My major concerns lie in the following two aspects.\n\nFirst, the statistical experiments in Table 1-3 only demonstrate the accuracy in terms of camera position and direction, while the efficiency especially the time and memory efficiency is missed. \n\nSecond, the limitations or failure case of the proposed approach need to be discussed. I am wondering the performance of the proposed approach on more complex and even comparatively large-scale environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for your positive comments about the novelty, writing, and experiments of this work. Please see the following responses to your concerns. \n\n**Time and memory efficiency** We have uploaded a new PDF containing an analysis of the time and memory efficiency of E2PNet. As shown in Tab. 2 of the new PDF, E2PNet is slightly slower (110.6ms *VS.* 55ms) and uses more memory (7.89GB *VS.* 2.2GB) than baseline methods. However, the overhead introduced by EP2T is arguably small given the significantly improved accuracy (translation error decreased from 1.297m to 0.821m, rotation error decreased from 4.97 ° to 3.61°). Note that further efficiency improvements are possible by 1) combining EP2T with more efficient tensorized representation methods, 2) tuning the number of spatial-temporal sampling points for efficiency-accuracy trade-off (Tab. 1 and 2 of the newly added PDF, show that when reducing the number of sampling points to 512, E2PNet runs at a similar speed (65.5ms) and memory cost (2.99GB) as other baselines, yet still has higher accuracy), and 3) optimization techniques like delayed aggregation [1] (2.2x speed up on PointNet++ [2], using ModelNet40 [3] dataset), Sparse convolution [4] (20x acceleration on VGG13 [5], using N-Cars [6] dataset). We will add this analysis and discussion into the camera ready following your suggestion.\n\n**Limitations or failure cases** As discussed in lines 325 to 328 of the paper and the previous response, the limitations of the proposed method lie mainly in the efficiency, which is a common problem in 3D point-based learning architectures. We will state this limitation more clearly by adding the efficiency analysis results to the camera-ready paper and provide further discussions as mentioned in the previous response.\n\n**Complex or Large-scale Environment** Exploring datasets with a more complex structure or a larger scale is indeed a significant problem. Since the registration task requires accurate ground-truth poses for supervision, which (at the moment) is only available in existing indoor data, we evaluate algorithms on indoor scenes in this work. However, we tried our best to involve large-scale and challenging scenes in the experiments (Tab. 1 of the main paper). The VECtor [7] dataset contains indoor scenes with corridors that have long edges of more than $60$m. We use different scenes in the training and testing data to better evaluate the generalization capability of different algorithms. We will discuss this limitation in the paper but leave the construction of better benchmarks for future work.\n\n>[1] Feng, et al. \"Mesorasi: Architecture support for point cloud analytics via delayed-aggregation.\"\n\n>[2] Qi, et al. \"Pointnet++: Deep hierarchical feature learning on point sets in a metric space.\"\n\n>[3] Wuc \"3d shapenets: A deep representation for volumetric shapes.\"\n\n>[4] Messikommer, et al. \"Event-based asynchronous sparse convolutional networks.\"\n\n>[5]  Simonyan, et al. \"Very deep convolutional networks for large-scale image recognition.\"\n\n>[6] Sironi, et al. \"HATS: Histograms of averaged time surfaces for robust event-based object classification.\"\n\n>[7] Gao L, et al. \"Vector: A versatile event-centric benchmark for multi-sensor slam.\""}, {"review_id": "tYyeRRJJyP", "reviewer": "Reviewer_W8Di", "summary": "This paper proposed a Event-Points-to-Tensor (EP2T) network, which treats event data as spatio-temporal point clouds, to process event signals without losing the spatiotemporal information of event signals (especially temporal information, compared with other voxel grid-based methods). In terms of experiments, this work demonstrates the effectiveness of the EP2T network by using event-based point cloud registration as an example, resulting in the development of E2PNet. Furthermore, the authors have gone a step further and tested the generalization ability of the EP2T network in tasks such as optical flow estimation, image reconstruction, and object recognition, obtaining promising results.", "questions": "The authors are encouraged to respond to the discussion regarding the intuitive motivation behind EP2T and provide an explanation of whether this operation takes into account the mathematical/physical characteristics of events better or offers better feature extraction advantages compared to traditional dense CNN-based model structures, or if there are other considerations involved. Furthermore, the reviewers request additional targeted experiments to validate the proposed intuitive advantages.", "limitations": "The author points out that the problem of large memory consumption is an important future research direction, which is similar limitation to other point-based models. If this problem can be solved, it will indeed help to improve the practicability of the model. Furthermore, there is a potential limitation that the authors did not mention in the paper, namely, that E2PNet still relies on tensorization to convert event point clouds into dense grid representations, which are then fed into downstream models designed for traditional vision tasks. It would be intriguing if future work could bypass this step and directly output target results using discrete/sparse event signals.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed Event-Points-to-Tensor (EP2T) network in this paper takes a different approach compared to most existing event signal processing models. This method treats event signals as three-dimensional spatio-temporal point clouds and employs operations such as Local Aggregation (LA), Spatio-temporal Separated Attention (STA), and Feature Propagation (FP) for extracting and preprocessing spatio-temporal features of events. Finally, tensorization is performed to obtain event feature representations compatible with traditional visual models. Compared to previous models that always tensorize event signals before feeding them into the model, this method can fully utilize the discrete and sparse characteristics of events. It effectively extracts spatiotemporal features from events, especially rich temporal domain information. Additionally, the LA operation separates the extraction of features in the temporal and spatial domains, enabling feature extraction and aggregation on different dimensions/domains, considering the distinct physical meanings of the three dimensions in the event signal \"point cloud\".", "weaknesses": "Although the proposed EP2T network in this paper achieves sparse processing of event signals in the first half, it still requires tensorization of the events in the end. In practice, such an operation can significantly degrade the temporal precision of events. While the 2D-3D Registration task proposed in this paper may not aim for high temporal resolution, it is crucial for tasks that are sensitive to event precision, such as high-frame-rate video reconstruction and low-latency object tracking. Furthermore, although the idea of using point cloud networks for sparse processing of event signals in this paper is innovative, the intuitive motivation behind this approach is not clearly presented in the writing. For instance, compared to the operation of tensorization followed by feature extraction, it is not discussed in the paper which events EP2T can handle better and what advantages EP2T possesses. This discussion is lacking in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the value of this work and providing an in-depth review. We provide the responses to the questions/concerns below.  \n\n**The motivation, advantage, and limitation of EP2T+tensorization-based approaches** First of all, there is no existing event to point cloud registration framework. However, several 2D-3D (tensor-based, e.g., LCD [1] and DeepI2P [2] in our experiments) and 3D-3D (point-based, e.g., GeoTransformer [3]) frameworks exist for other modalities. Due to the direct use of spatial geometry to perform registration, 3D-3D frameworks are not easily applicable to event data since the event signal does not have 3D spatial information. 2D-3D registration frameworks can be directly applied to event data but most temporal information is lost. Our key idea is to design a feature representation network that can effectively extract the spatio-temporal feature before applied to different registration frameworks, whether tensor-based or point-based. In Tab. 1 of the main paper, we have compared the performance of E2PT when applied to both tensor and point-based methods. As described in line 260 of the main paper, we have modified LCD [1] and DeepI2P [2] so that we directly convert point-based features into a global feature without using any grid-based tensorization. We agree that designing a better fully point-based registration framework for event data is an interesting research direction. However, we do not see an obvious advantage of pure point-based methods over EP2T+tensor-based methods. Hence, based on the empirical accuracy, we choose to construct our E2PNet using E2PT+tensor-based methods. We will state this motivation more clearly in the camera-ready paper.\n\nIn terms of the advantage, focusing on the spatio-temporal feature learning makes it possible for our method to utilize different future registration frameworks, which can come from outside the field of event data processing. Since tensor-based methods are efficient, combining EP2T with tensor-based methods also provides reasonable speed and memory efficiency along with high registration accuracy. \n\n**Limitations on memory consumption** We appreciate your feedback on the limitation of memory consumption. To further analyze and address this limitation, we have uploaded a PDF that includes a detailed analysis of memory usage. As shown in Tab. 2 of the PDF, though E2PNet exhibits slightly lower speed (110.6ms *VS.* 55ms) and larger memory cost (7.89GB *VS.* 2.2GB), this overhead is arguably acceptable compared to the significantly improved accuracy (translation error decreased from 1.297m to 0.821m, rotation error decreased from 4.97 ° to 3.61°).\n\nReducing memory consumption is indeed an important direction for future work. The major contributor to the higher memory usage is the EP2T network designed based on PointNet++ [4], where the point-wise distance calculation operation leads to memory growth that scales with the number of points, resulting in a quadratic consumption increase. There are successful approaches that can significantly improve the space efficiency of point cloud frameworks, such as PAT [5] (2x memory reduction on PointNet++, using ModelNet40 [6] dataset) and PVCNN [7] (3x memory reduction on point-based models, using ModelNet40 dataset). Most of these methods can be readily integrated into EP2T, offering the potential to enhance (memory) efficiency without compromising performance. Meanwhile, for applications that require high inference speed, we can reduce the number of EP2T sampling points (Tab. 1 of the rebuttal PDF) to achieve a significant memory reduction (2.99GB with 512 sampling points) while still performing better than the baselines. We are dedicated to further optimize the memory consumption of EP2T, and your feedback has reinforced our commitment.\n\n\n>[1] Pham, et al. \"Lcd: Learned cross-domain descriptors for 2d-3d matching.\"\n\n>[2] Li, et al. \"DeepI2P: Image-to-point cloud registration via deep classification.\"\n\n>[3] Zheng, et al. \"GeoTransformer: Fast and Robust Point Cloud Registration With Geometric Transformer.\"\n\n>[4] Qi, et al. \"Pointnet++: Deep hierarchical feature learning on point sets in a metric space.\"\n\n>[5] Yang , et al. \"Modeling point clouds with self-attention and gumbel subset sampling.\"\n\n>[6] Wu, et al. \"3d shapenets: A deep representation for volumetric shapes.\"\n\n>[7] Liu, et al. \"Point-voxel cnn for efficient 3d deep learning.\""}, {"review_id": "VUcvxokUrC", "reviewer": "Reviewer_N3N6", "summary": "This paper proposes a learning-based event-to-point cloud registration method, which encodes event spatio-temporal data into a grid-shaped feature tensor, and propose a framework to construct E2P datasets using existing SLAM datasets. Experiments are conducted on MVSEC-E2P and VECtor-E2P datasets, and state-of-the-art results are achieved on these datasets.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The presentation is easy to understand. \n2. Experiments are well conducted and convincing.", "weaknesses": "1. Since event-based applications require a high response speed, the efficiency analysis should be given about the comparison of the runtime and memory usage between the proposed method and other state-of-the-art methods.\n2. Since the experimental datasets are generated by the authors, did the authors retrain the learning-based models on the datasets in order to compare with these learning-based methods?\n3. The comparisons are done to a few methods in the current manuscript, I suggest more state-of-the-art methods should be included for comparison.\n4. Minor Typos / Writing  \nFigure 3: what does O_{E} mean?  \nLine 211: the definition of f_{SP} is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive comments about our writing and experiments. In the following, we address the reviewer's concerns and back up our responses with additional experiments. We hope that with major concerns like efficiency analysis and training details resolved, the reviewer will consider improving the final ratings.\n\n**Comparison of the proposed method and other state-of-the-art methods w.r.t. runtime and memory usage.**\nThanks for the suggestion, we have uploaded a new PDF analyzing the time and space efficiency of E2PNet. Though slightly slower (110ms *VS.* 55ms inference time) and more memory-intensive (7.98GB *VS.* 2.2GB) compared to direct tensor-based approaches, given the significantly improved accuracy, this overhead is arguably acceptable for the registration task.  \n\nNote that the efficiency of the introduced E2PT network can be further improved by incorporating recent techniques, e.g., 2.2x potential speedup from the delayed aggregation [1]. We are committed to investigating this interesting direction to improve our method in the future. We also show in Tab. 1 of the rebuttal PDF that one can reduce the number of center points in EP2T to significantly reduce the speed and memory consumption (512 points result in 65ms inference speed and 2.9GB memory usage) while still achieving better performance than baselines.\n\nWe will add this discussion to the camera-ready paper.\n\n**Did the authors retrain the learning-based models?**\nYes. To fairly compare against different methods, we have retrained all learning-based baselines on the proposed datasets. We will clarify this in the camera-ready paper.\n\n**The comparisons are done to a few methods in the current manuscript. I suggest more state-of-the-art methods should be included for comparison.**\n\nThanks for the suggestion. This work focuses on the feature representation learning of event-to-point cloud registration. To demonstrate the effectiveness of our E2PT representation, we have compared it against hand-crafted features (discretized event volume [2], published at CVPR2019) and advanced methods with both tensor-based representations (Tore [3], published at T-PAMI 2023) and point-based representations (ECSNet [4], published at T-CSVT 2022). All of these methods are state-of-the-art in their respective domains and serve as solid benchmarks for evaluating the performance of our proposed event representation. \n\nTo verify the effectiveness of our method under different 2D-3D registration frameworks (used after the feature representation network), we also compared against different representations under representative methods based on both registration (DeepI2P [5], published at CVPR2021) and retrieval (LCD [6], published at AAAI2020). Though there are other potential candidate registration frameworks like P2Net [7] (ICCV2021) and Pump [8] (CVPR2022), they did not release the code and our main focus is designing a better feature representation learning network that can be plugged into different downstream registration frameworks.\n\n**Typo/writing** We appreciate your attention to the details of our work. In Fig. 3, $O_{E}$  refers to the attention feature obtained from the spatio-temporal feature $F_{st}$ at each feature aggregation center through a classic self-attention module ($O_{E}$ has the same dimension as $F_{st}$ ). Our approach involves not only assigning local neighborhood features to each aggregation center through the LA module but also considering the importance of global information. To achieve this, we introduce a self-attention module to enhance the long-distance correlation between the spatio-temporal features of each aggregation center, which allows for better information interaction among all centers. \n\nThe $f_{SP} $ in the paper is a typo, and it should be $f_{FP}$, which represents the point-wise spatio-temporal features obtained after feature propagation (FP module).\n\nWe will fix/clarify these points in the camera-ready paper and carefully proofread the manuscript once more.\n\n>[1] Feng, et al. \"Mesorasi: Architecture support for point cloud analytics via delayed-aggregation.\"\n\n>[2] Zhu, et al. \"Unsupervised event-based learning of optical flow, depth, and egomotion.\"\n\n>[3] Baldwin, et al. \"Time-ordered recent event (TORE) volumes for event cameras.\"\n\n>[4] Chen, et al. \"ECSNet: Spatio-Temporal Feature Learning for Event Camera.\"\n\n>[5] Li, et al. \"DeepI2P: Image-to-point cloud registration via deep classification.\"\n\n>[6] Pham, et al. \"Lcd: Learned cross-domain descriptors for 2d-3d matching.\"\n\n>[7] Wang, et al. \"P2-net: Joint description and detection of local features for pixel and point matching.\"\n\n>[8] Revaud, et al. \"Pump: Pyramidal and uniqueness matching priors for unsupervised learning of local descriptors.\""}, {"review_id": "OCqi0lIb6a", "reviewer": "Reviewer_mSRE", "summary": "The paper presents an method to perform point cloud registration using event camera data.\nThe method first learns a feature representation (E2PT) from a point cloud of events. This representation is used as input to standard registration networks.\nThe experimental section shows better accuracy compared to other event-based feature represenations as well as frame-based approaches.\nOther experiments also show that the E2PT representations is generic and cna be used for other event vision tasks. ", "questions": "Please describe more precisely the steps of the method as stated above.\n \nAlso, please clarify the event sampling strategy: \"We follow the FEN [25, 26] principle and acquire 20000 consecutive\nevents at a time and sample N = 8192 events from them.\" \nDoes it mean that out of 20k events, only N are used to build the tensor representation or that multiple consecutive tensor representations are used? why using such a specific value for N: 8192?", "limitations": "It would be interesting to understand the runtime of this approach, and how does it scale with respect to the event rate (number of events in the point cloud).\n\n", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper addresses a novel problem of using event camera data for 2d-3d registration. They adapt existing registration pipelines taking into account the specificity of the events.\nThey propose a new learned event-based representation which leads to good experimental results for the registration problem, but also for other event vision tasks (optical flow, classification, image reconstruction).", "weaknesses": "The clarity of the paper can be overall improved.\nIn particular the method section is not always clear and some details are missing.\nFor example it is not clear how the features Ft , Fs , and Fst are built (Sec. 3.1.1). the authors mention some \"multi-layer convolution\", but is not clear what is the input to these convolutions, what exactly is the architecture, if tehre is a separate network for  Ft , Fs , and Fst or if it is shared. What is the size of Ft , Fs , and Fst etc. Describing the algorithm with more equations and pseudo-code can make the method clearer, since text and figures alone can be ambigous.\n\nSimilarly there is no description in the method section of the E2Pnet architecture, which is presented as the second main contribution in the introduction. From the experimental details it seems that standard architecture from previous works [1,11] are used? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review and valuable suggestions. Due to space constraints, detailed explanations of the methodology are contained in Sec. A.1.1 of the appendix, including the technical details about the LA, STA, and FP modules. Our approach builds upon the 3D point-based architecture (PointNet++ [1]) and has been designed to leverage the unique characteristics of the event spatio-temporal data. For example, we introduce the spatio-temporal separation mechanism, which is explained in Sec. 3.1 of the paper. Below, we provide further details to better address your concern. We will carefully modify the camera-ready paper to make the method description more straightforward.\n\n**Weaknesses (How are the features $F_{t}$, $F_{s}$, and $F_{st}$ built? What exactly is the architecture? )**\n\n $F_{t}$, $F_{s}$, and $F_{st}$ are obtained by the LA module with different parameters. The LA module needs to find the corresponding K nearest neighbors for each feature aggregation center (sampling point) to perform local feature aggregation (LA). Traditional 3D point cloud approaches usually use Euclidean distance to determine nearest neighbors. In this paper, we consider the different physical meaning and distribution characteristics of event data in time and space. Hence, we use different weight parameters for time and space, and the neighborhood of each sampling point is determined according to the weighted distance. The features generated by LA with larger temporal and spatial weights are called $F_{t}$ and $F_{s}$, respectively. The features generated by LA with the same temporal and spatial weights are called $F_{st}$. Please refer to lines 180 to 190 of the main paper for a more details.\n\nThe structure of LA is similar to PointNet++ [1]. For each set of spatio-temporal distance weights, after determining the K nearest neighbors of each sampling point through the weighted spatio-temporal distance, we can obtain a feature matrix of size $S\\*K\\*3$; $S$ represents the number of aggregation centers, $K$ is the number of neighborhood points (64 in our experiments) and $3$ corresponds to the space-time coordinates ($h, w, t$) of the event.\nThe features are mapped to a high-dimensional space through a 3-layer MLP to obtain features of size $S\\*K\\*C$, where $C=3,16,32,64$ represents the number of feature channels. Each layer of the MLP is followed by a layer of batch normalization and ReLU activation function. After that, we perform maximum pooling on the $K$ dimension, retain the neighborhood point with the largest feature value in each feature channel, and finally obtain the $S\\*64$ dimensional feature.\n\n**Weaknesses (No description in the method section of the E2PNet architecture, standard architecture from previous works used?)**\n\nThe main focus of this work is the feature representation learning of the event data. While E2PNet can utilize different mature 2D-3D registration frameworks after obtaining the output from EP2T (line 152 of the paper), we do not propose a novel registration framework. In our experiments we tested the classical 2D-3D registration frameworks DeepI2P[2] and LCD[3]. We will make this more clear in the camera-ready paper and merge the contribution statement of E2PNet with EP2T to avoid ambiguity.\n\n**Questions (Does it mean that out of 20k events, only N are used to build the tensor representation or that multiple consecutive tensor representations are used? Why use such a specific value for N: 8192?)**\n\nAs correctly pointed out, we sample N=8192 events from the original consecutive 20k events. Empirically, this strikes a good balance between efficiency and high accuracy.\nTo further ablate our choice, we have conducted an analysis of the impact of these two hyperparameters (number of FEN and number of sampling points) from three perspectives: accuracy, time efficiency, and memory cost. The results can be found in the newly uploaded rebuttal PDF. Overall, the proposed hyperparameter settings makes E2PNet reasonably efficient (time and memory) yet significantly more accurate than the baselines. In practice, the value of N can be increased (see the result of 16384 in Tab. 1 of the rebuttal PDF) to achieve higher accuracy or decreased (512 in Tab. 1 of the rebuttal PDF) to achieve much higher efficiency at the cost of accuracy (still better than the baseline).\n\n**Limitations (Runtime of this approach and how it scales with respect to the event rate)** \n\nAccording to your suggestion, we conducted further analysis of runtime in the rebuttal PDF. Experiments demonstrate that our method achieves superior accuracy and acceptable overhead in terms of runtime and memory consumption. In terms of the scaling, the runtime and memory cost increase with the number of sampling points, with the accuracy also consistently improving. \n\nWe also identify some recently proposed techniques to optimize the efficiency of PointNet++, such as delayed aggregation [4] (2.2x speed up on PointNet++, using ModelNet40 [5] dataset), Sparse convolution [6] (20x acceleration on VGG13 [7], using N-Cars [8] dataset). Since our EP2T is inspired by PointNet++, a similar improvement can be expected by applying the same techniques, which will be an interesting direction for future work.\n\n>[1] Qi, et al. \"Pointnet++: Deep hierarchical feature learning on point sets in a metric space.\"\n\n>[2] Li, et al. \"DeepI2P: Image-to-point cloud registration via deep classification.\"\n\n>[3] Pham, et al. \"Lcd: Learned cross-domain descriptors for 2d-3d matching.\"\n\n>[4] Feng, et al. \"Mesorasi: Architecture support for point cloud analytics via delayed-aggregation.\"\n\n>[5] Wu, Zhirong, et al. \"3d shapenets: A deep representation for volumetric shapes.\"\n\n>[6] Messikommer, et al. \"Event-based asynchronous sparse convolutional networks.\"\n\n>[7] Simonyan, et al. \"Very deep convolutional networks for large-scale image recognition.\"\n\n>[8] Sironi, et al. \"HATS: Histograms of averaged time surfaces for robust event-based object classification.\""}]

Author rebuttal:
We thank all reviewers for their positive comments about the novelty (R1, R2, R4), significance (R1, R2, R4), writing quality (R1, R2, R3), and experiments (R1, R2, R3, R4) of this work.

A common question/concern was the efficiency of the proposed method. We have conducted several experiments and provide a detailed analysis in the uploaded rebuttal PDF. The results show that our method is reasonably efficient while achieving a much higher accuracy compared to the baselines. We also identify and discuss different approaches to further improve the efficiency. Please refer to the individual responses for further details and also responses to other specific concerns. We appreciate all reviewers' feedback and will incorporate it in the final manuscript.

Meta-review: This paper receives 4x positive ratings: 2x borderline accepts, 1x weak accept and 1x accept. The overall comments on this paper are positive. The reviewers think that this is the first learning-based architecture to handle the sparse reconstruction based on event cameras; The performance shown in the statistical experiments especially in Table 1 demonstrate the superior; This paper takes a different approach compared to most existing event signal processing models; It effectively extracts spatiotemporal features from events, especially rich temporal domain information; A new learned event-based representation which leads to good experimental results for the registration problem, but also for other event vision tasks (optical flow, classification, image reconstruction).
Decision: Accept (poster)