Title: E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: event camera

2D-3D registration

representation learning
Abstract: Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration.
The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: https://github.com/Xmu-qcj/E2PNet.

E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning

Abstract

Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-toTensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: E2PNet.

1 Introduction

Computer vision aims to understand and reconstruct the world from 2D observations [2]. Establishing the relationship (pose or correspondences) between 2D data and the 3D world is an important step toward this goal. Conventional 2D-3D registration mostly relies on multi-view geometry [3, 4, 5]. Specifically, 3D (key) points are first triangulated [6, 5] using multiple images with proper disparity. Subsequently, the corresponding 2D features [7, 8, 9] are attached to 3D points for subsequent 2D-3D registration. This framework is by design image-based, which limits the use of other sensors that obtain 3D point clouds directly (e.g., LiDAR, depth cameras).

Recently, learning-based methods for direct 2D-3D registration have been proposed [10, 1, 11, 12]. However, these methods require high-quality RGB or grayscale images which are not always feasible

(Visualization of Event Data for Registration)

（

Images for Registration

）

Point cloud map

GT points

Points registered by E2PNet

Points registered by traditional camera-based method

in practice. Challenging environments with extreme illumination or motion blur exist in many applications, which may cause the algorithm to fail completely (see Fig. 1 for an example).

Unlike traditional cameras, the novel information acquisition principle of event cameras [13] makes them less susceptible to challenging illumination and motion blur. This advantage motivates us to design a robust 2D-3D registration method for event cameras.

High dynamic range (HDR, up to 120 dB [13]) and high temporal resolution bring excellent performance to event cameras but also create new challenges. The unstructured data stream of event cameras is sparse in the spatial plane but dense (up to 300 million events per second [14]) and non-uniform in the temporal dimension. Existing methods mostly process events as image-like feature maps [15, 16] or voxels [17, 18, 19], which arguably loses spatio-temporal details. Although event cameras have been used for other vision tasks such as optical flow estimation [15, 18, 20], moving object detection [21, 22] and high dynamic image reconstruction [23, 24], the task of event-to-point cloud registration (E2P) has not yet been studied.

There are several challenges. First, event cameras only capture the surface appearance of the scene, whereas 3D point clouds encode geometric structures, which is inherently challenging [11]. Second, event cameras have lower resolution than traditional cameras and only capture the brightness variation rather than the brightness itself. Finally, event data and point clouds can both be considered as unstructured 3D data, which are harder to process than 2D images. Moreover, the lack of an E2P dataset with accurate annotations limits the research of such cross-modal registration methods.

To address these problems, we present E2PNet, the first E2P framework and the corresponding dataset for end-to-end learning. Specifically, we propose a modular event representation network called Event-Points-to-Tensor (EP2T), which applies local feature aggregation with different temporal and spatial distance weights and spatio-temporal separated attention mechanisms to effectively encode sparse and in-homogeneous event data. In the meantime, EP2T adaptively aggregates local neighborhood features with uneven event data in the temporal and spatial dimensions and outputs a structured (grid-shaped) feature tensor which can be directly plugged into image-based 2D-3D registration frameworks. E2PNet is capable of E2P registration for indoor scenes of different scales (single room to one floor) and produces the corresponding 6-DOF transformation between the event camera and 3D point clouds. As shown in Fig. 1, replacing the RGB inputs with event data can significantly improve the registration performance on challenging data. Our main contributions are:

We propose EP2T, the first representation learning architecture to transform event spatiotemporal point clouds into a grid-shaped feature tensor. EP2T can adaptively learn how to extract critical spatio-temporal information according to downstream tasks, and can be applied as a plug-and-play module to existing event camera processing networks.

We propose E2PNet, the first approach that allows direct registration of 2D event cameras and 3D point clouds. To facilitate follow-up research, we also propose a framework to construct E2P datasets using existing SLAM datasets.

We conduct extensive experiments and analysis to demonstrate the effectiveness of E2PNet for E2P, and the potential of the EP2T module to benefit in other diverse tasks.

2 Related Works

2.1 Event Representation

The asynchronous unstructured event data poses a great challenge for computer vision. While spiking neural networks (SNN) [25] can directly use the raw data of event cameras, they require special hardware and has low flexibility. A more common approach is to process batches of event data via an intermediate representation, e.g., an image of positive and negative events over a certain horizon. The processing window is usually set to a fixed number of events (FEN) [26, 20] or a fixed interval time (FIT) [15, 20].

Hand-crafted representations usually convert raw event data into 2D grid-shaped feature maps with sensor resolution using the statistics of raw data, for example, event counting [27], average/latest timestamp of events on each pixel [15] or temporal distance-weighted statistics [17]. While these strategies can approximate the distribution of event data with respect to space and time, some important details (i.e. spatio-temporal correlation) may be lost due to the aggregation. To address this shortcoming, another set of methods [28, 29, 30, 31] discards most of the events and only keeps the latest few events for each pixel and analyzes them in detail. However, these methods cannot encode distant spatio-temporal relationships. More recent work tries to incoporate these relationships by representing event data as graphs [32, 33, 34, 35]. Although more spatio-temporal information can be extracted, building a graph structure is complicated and computationally expensive. Each hand-crafted representation has different advantages and disadvantages. Hence, choosing the right representation for a given tasks is often not trivial. The proposed E2PNet also utilizes hand-crafted methods [15, 20, 17]. However, we use them in conjunction with our learned features instead of raw event data to produce the final grid-shaped feature tensors.

Learning-based representations train models to adaptively extract critical information based on the specific task. However, the inhomogeneity and lack of structure make event representation learning challenging. EST [16] and DDES [30] propose to learn tensor-based representations, which regard the events of each pixel as pulse signals and optimize the impulse response function. ECSNet [36] uses 3D point cloud learning methods to extract point-based features. However, these feature points are incompatible with the processing pipeline of downstream tasks, and generalize poorly to other tasks. In contrast, we aim to fully extract the spatio-temporal correlation of the event data and transform it into a grid feature tensor representation for better compatibility.

2.2 2D-3D Registration

This work focuses on the direct 2D-3D registration problem, which has been studied for conventional images. Li et al. [37] use the multi-view projection image of 3D objects to extract the corresponding HOG features, and train a feature mapping network to push the features from the same 2D-3D objects closer. 3DTNet [38] uses RGB-D point clouds and images to train distinctive 3D descriptors. However, these methods can only be used to register images to colored point clouds, which is not always feasible. E2PNet does not generate features based on color information, hence it can be applied to conventional point clouds without colors. 2D3D-MatchNet [10] and LCD [1] learn consistent feature representations of 2D images and 3D point clouds, realizing 2D-3D registration through cross-modal feature retrieval. P2-Net [12] learns the detection and matching of potential feature points from images and RGB-D points. DeepI2P [11] transforms image-point cloud registration into point cloud visibility classification and then maximizes the number of visible points by optimizing the camera pose. A common drawback of these image-based methods is that they require high-quality images to perform well, which cannot be guaranteed in applications with challenging illumination conditions or camera motion. As shown later in the experiments, introducing event data for 2D-3D registration makes E2PNet much more robust in challenging environments.

3 Methodology

E2PNet treats the input event data as spatio-temporal point clouds. Before explaining how this can be done, we briefly introduce the generation process of event data here. During data capture, each pixel of the event camera responds independently and asynchronously to illumination changes. Specifically, given an image of size ( H,W ) , denote I ( t i , h, w ) as the illumination at time t i for the pixel at ( h, w ) .

Feature Aggregation

LA

Height

Time

Spatio-temporal

point cloud

Height

Time

Height

Time

(h 1

,w 1

,t

1,

p 1

)

(h 2

,w 2

,t

2,

p 2

)

...

(h n

,w n ,t

n,

p n

)

Aggregation center feature

Spatio-temporal

point-wise feature

Input

Event

Data

Local neighborhood boundary

Point-based

Representation

Tensor-based

Representation

w

Temporal patch

aggregation

t

Spatial patch

aggregation

w

h

h

F t

F s

Center

Feature

Aggregation center

Output

Grided

Tensor

Global

Feature

Spatio-temporal

Kernel

Temporal

Kernel

Spatial

Kernel

F st

f

STA

FP

Event  Points

STA

When the inequality

is satisfied. This pixel will generate an impulse response called an event , which is expressed as e = ( h, w, t i , p ) . The polarity p ∈ { +1 , -1 } indicates the direction of the brightness change (increasing or decreasing) and τ is the trigger threshold. t i -1 represents the time when the event was triggered last time at pixel ( h, w ) . The imaging plane and the time axis can be combined to establish a three-dimensional spatio-temporal coordinate system, where events characterized by coordinates ( h, w, t i ) collectively form a spatio-temporal point cloud.

In order to compute the pose of the camera w.r.t. a 3D point cloud, E2PNet first encodes the spatio-temporal point clouds into a 2D grid-shaped feature tensor. This process relies on EP2T, which is the core component of E2PNet and will be introduced in detail in Sec. 3.1. After obtaining the encoded feature tensor, we can formulate the event-to-point registration (E2P) problem as the registration of 2D feature maps and 3D point clouds, which can be handled by mature image-based frameworks [1, 11]. Since EP2T is differentiable, the training of E2PNet is simply done by connecting EP2T with the registration network and performing back-propagation in the same way as the original registration network. Empirically we found that directly adding EP2T to the registration network without changing any hyper-parameter works reasonably well, which is another advantage of E2PNet.

3.1 Spatio-Temporal Reprsentation Learning with Event-Points-to-Tensor (EP2T)

As illustrated in Fig. 2, EP2T consists of three consecutive modules: Local Aggregation (LA), Spatio-temporal Separated Attention (STA), and Feature Propagation (FP).

Given a spatio-temporal point cloud derived from the input event data, LA evenly samples a set of feature aggregation centers in the spatial and temporal domains. For each sampled center, LA computes the locally aggregated features using the neighboring input points. To produce the global feature, STA performs attention over these aggregated features, with a special design to handle the spatio-temporal inhomogeneity. To produce features on each input point (from aggregation centers), FP propagates each feature from STA to the neighboring points based on the distance. Finally, we employ classical tensorization methods to process the point-wise features into grid-shaped tensors.

The novelty of LA (see Sec. 3.1.1 for more details) and STA (Sec. 3.1.2) lies in the separation of spatial and temporal domains, which is designed to handle the inhomogeneity of spatio-temporal point clouds. FP (Sec. 3.1.3) propagates aggregated features to neighboring spatio-temporal points so

that standard point-wise tensorization can be used to generate grid-shaped outputs, which enables the use of RGB-based 2D-3D registration frameworks for E2P.

3.1.1 Local Aggregation (LA)

Given an event camera of resolution (H, W), we extract N events from the input and convert them into a spatio-temporal point cloud with N points. Unlike conventional point clouds, the spatial and temporal domains of our inputs naturally have different metric scales, which makes existing point cloud representation networks [39] ineffective for feature encoding (see Sec. 4.2).

To address this problem, the proposed LA module aggregates features across the spatial and temporal domains seperately. Specifically, for each input point e = ( h, w, t, p ) with p ∈ {-1 , 1 } representing the brightness change direction, we first normalize it via ˆ e = ( h H , w W , t t max -t min , p ) , where t max and t min are the maximum and minimum time stamps of the input point cloud.

The feature aggregation process of LA is inspired by PointNet++ [39]. There are two steps during aggregation: (1) generate a set of aggregation centers; (2) calculate the aggregated feature for each center by employing neighboring points.

PointNet++ [39] generates aggregation centers using Farthest Point Sampling (FPS) [40], which is relatively slow in event-based applications that require a high response speed. Furthermore, FPS is not suitable for spatio-temporal point clouds with non-uniform density distributions (might generate redundant points in dense regions and fewer points in sparse regions). To address this issue, LA uses uniform sampling with a fixed spatial and temporal interval to generate uniformly distributed aggregation centers. Specifically, given the normalized point cloud ˆ E spanning over a spatio-temporal cube of size ( H,W,T ) , we generate a set of aggregation centers C by sampling M (30*30*5 in our experiments) uniform grid locations within this cube.

To aggregate features from neighboring points, PointNet++ [39] first selects K nearest neighbors from ˆ E for each c j ∈ C and then replaces neighbors whose Euclidean distance is too far away from c j with the point closest to c j . After selecting neighboring points, a multi-layer convolution is performed to produce the aggregated feature. Using Euclidean distance omits the inhomogeneity of the spatial and temporal domains, which exists even with our initial normalization. Hence, we propose to separate the spatial and temporal domains during neighborhood selection. Specifically, we change the distance metric during the selection of nearest neighbors ( K = 64 in LA) to αd Space ( ˆ e i , c j ) + βd Time ( ˆ e i , c j ) . We also use two distances separately in the spatial and temporal domains to choose points for replacements, i.e., we replace a neighboring point ˆ e i with the closest point to c j if d Space ( ˆ e i , c j ) ≥ α or d Time ( ˆ e i , c j ) ≥ β . d ( · ) is the mean squared distance of the corresponding domain.

Separating the spatial and temporal domains during feature aggregation addresses the inhomogeneity problem from two aspects. (1) We can provide constraints to the spatial and temporal domains separately to prevent choosing neighboring points with a small distance in one domain but a far distance in another. (2) We can provide different α 's and β 's to create multiple aggregated features that focuses on different domains. For the second aspect, we produce three sets of features in LA by setting α = { 0 . 8 , 0 . 1 , 0 . 5 } and β = { 0 . 1 , 0 . 8 . 0 . 5 } respectively, which allows the aggregated feature to focus more on spatial, temporal, and spatio-temporal domains.

3.1.2 Spatio-temporal Separated Attention (STA)

After generating the feature centers and the locally aggregated features, we use the STA module to extract global features and merge features from different domains into the uniform spatio-temporal domain. Similar to LA, STA also separates the spatial and temporal domains to effectively handle the inhomogeneity problem.

F t , F s , and F st in Fig. 3 denote the aggregated temporal, spatial and spatio-temporal features from LA respectively. We first create global features for the spatial and temporal domains separately using the temporal and spatial residual kernels. Each of these two kernels performs self-attention over features from the same domain, and cross-attention between features from different domains. Subsequently, we merge them into a temporal and spatial residual using retention gates (sigmoid and tanh functions) inspired by [41, 42]. After producing the residuals for spatial and temporal domains separately, we merge them into the spatio-temporal domain using the spatio-temporal aggregated feature F st to produce the global spatio-temporal feature f STA ( c j ) for each aggregation center c j .

(c)Spatio-temporal Separated Attention Module(STA)

(a)Single Domain Residual Kernel

F s

Sigmoid

Self-

Attention

Cross-

Attention

F t

Self-

Attention

Sigmoid

Cross-

Attention

Self-

Attention

Spatial

Residual

Kernel

Sigmoid

Temporal

Residual

Kernel

Sigmoid

+

OE

Output

Output

Spatial

Residual

F s

F t

F t

V s

F s

F t

Temporal

Residual

Center-wise Feature

F t

F s

O E

Spatio-temporal

Residual

Spatio-temporal

Residual Kernel

f

STA

Global

feature

F s

Output

F st

Sigmoid

3.1.3 Feature Propagation (FP) and Tensorization

After producing the global features at the aggregation centers, we propagate them to neighboring input event points using a weighted average to get a feature for each input point. Specifically, given the global features F STA = { f STA ( c j ) } and the normalized event points ˆ E = { ˆ e i } , we compute the propagated feature by

where w i,j = 1 max( || ˆ e i -c j || 2 2 , 10 -5 ) . 10 -5 is used to prevent division by zero.

Finally, to convert point-based features into grid-shaped features, we use three different yet complementary event tensorization methods [15, 20, 17] to convert the set of f FP ( ˆ e i ) into three different types of 2D grid-shaped sparse feature tensors, and concatenate them to produce the final tensorized feature map.

3.2 E2P Dataset Construction

Since there is no dataset for E2P, we use multi-sensor SLAM datasets to construct the ground-truth (GT) labels for training. We select the widely used MVSEC [43] and VECtor [44] to build the MVSEC-E2P and VECtor-E2P datasets, which incorporate LiDAR, traditional cameras and event cameras at the same time. Furthermore, they have an accurate multi-sensor time synchronization and calibration. Through the GT pose provided in these two datasets, we can establish the GT matching relationship between events (images) and point clouds for E2P tasks. A naive way to obtain such GT would be to project point clouds into the image plane, which is slow in practice. Instead, we construct a camera model in the world coordinate system and use this model to construct a quadrangular viewing frustum. Selecting the points in the quadrangular viewing frustum is practically > 2 x faster than the naive solution.

4 Experiments

In this section, we first demonstrate the effectiveness of the proposed E2PNet for event-to-point cloud registration (Sec. 4.1), which is our primary goal. Then, we conduct ablation studies to verify the effectiveness of individual modules in EP2T (Sec. 4.2). Finally, we analyze the generalization of EP2T on other event-based vision tasks (Sec. 4.3).

4.1 Event to Point Cloud Registration

Datasets and Preprocessing. We use two representative datasets described in Sec. 3.2. MVSEC [43] uses a 16-beam LiDAR and an event camera with a resolution of (346,260); the event camera can simultaneously generate grayscale images. Since only indoor sequences have high-precision pose

ground truth (GT), we use the indoorx and indoory sequences for training and testing respectively, where x ∈ [1 , 3] and y = 4 . VECtor [44] uses a 128-beam LiDAR and an event camera with a resolution of (640,480). Different from MVSEC, VECtor contains various large-scale indoor environments with long trajectories. The GT pose is derived via ICP[45] between LiDAR scans and dense environment point clouds captured by a high-precision laser scanner. We use the units-dolly, units-scooter, corridors-dolly and corridors-walk sequences for training, and the school-dolly and school-scooter sequences for evaluation. For both datasets, we randomly acquire event data in an (256,256) area from event camera data as input; simultaneously we also acquire grayscale images with the same field of view and resolution for comparison.

Evaluation. Following the previous standard [46], we use the translation error TE = || T GT -T pred || 2 and the rotation error: RE = arccos ( tr ( R -1 GT R pred ) -1 2 ) to evaluate the accuracy of predicted camera poses in E2P. T and R represents the translation vector and rotation matrix respectively, and tr ( · ) is the trace of a matrix.

Implementation Details. We follow the FEN [26, 20] principle and acquire 20000 consecutive events at a time and sample N = 8192 events from them. Since EP2T has encoded the spatiotemporal information as 2D grid tensors, we can regard E2P as the registration of 2D feature maps with 3D point clouds. We choose LCD [1] and DeepI2P [11] as the representatives to test the performance of E2P using feature retrieval-based and point visibility-based registration. LCD maps the corresponding 2D tensor and 3D point cloud into a high-dimensional feature space, where training aims to make the feature distance as close as possible. During training, we fetch events (image) in the current pose and their corresponding point cloud patch according to the GT pose (see Sec. 3.2). The principle of DeepI2P is to judge whether the input point cloud appears in the corresponding field of view of events (image). During training, we use the visible point clouds for each event view and its surrounding point clouds (enlarged by 30% ) for supervision. All methods are implemented using Pytorch. Training is done following the setup of individual baselines on a 3090Ti GPU.

We compare E2PNet against conventional camera-based registration (i.e., the original version of LCD and DeepI2P) and other event representation methods, including tensor-based [19, 15, 17, 20] and point-based [36] methods. To compare against point-based methods, we do not perform the final tensorization (i.e., w/o TR) so that the encoded features are also point-based. Since there is no standard technique to apply point-based event representations to image-based registration frameworks, for fair comparisons, we implement a simple encoder with max-pooling and a linear layer (see appendix for details) to project the point-based features to a global feature vector. Subsequently, we use this vector to replace the original global feature (computed by performing convolution on the tensor-based representation) in both LCD and DeepI2P. For LCD-based methods (retrieval-based), we use the trained model to encode the feature on the target point cloud under different poses to build a point cloud feature library. During inference, we extract the corresponding pose in the feature library

Traditional Camera

Ours

TORE

E-Statistic

Ground Truth

Projected Point

Motion

Blur

Point clouds projected through GT pose

Over

Exposure

Point clouds projected through predicted pose

Non-

Challenging

Situation

1

that is closest to the feature of the event data. For DeepI2P-based methods (point cloud visibility classification-based), we use the same optimization method as the original method [11].

As shown in Tab. 1, replacing conventional image inputs with event data improves the registration accuracy significantly and consistently across datasets and baseline methods . E2PNet performs the best among both tensor-based and point-based E2P methods, outperforming both hand-crafted (E-Statistic) and learning-based methods. Qualitative results in Fig. 4 further show the robustness of E2PNet in different environments. Finally, due to better compatibility, tensor-based methods perform better than point-based methods, and sometimes by a large margin. Note that although point-based representations can be applied to 2D-3D registration, since only a global feature vector of the input (event) image is required, they may not be compatible with tasks that require pixel-level feature maps.

4.2 Ablation studies

In this section, we analyze the effectiveness of individual components of E2PNet, i.e., the Local Aggregation module (LA), Spatio-temporal Separated Attention module (STA), Feature Propagation module (FP), and the mechanism for seperating the spatial and temporal domains in STA. We use LCD [1] as our baseline model and conduct experiments on the MVSEC-E2P dataset. As shown in Tab. 2, the registration accuracy reduces as components are removed, showing the importance of each component. Note that since LA and FP are coupled modules (we need FP to propagate features from aggregation centers to individual event points), we add and remove them together.

4.3 Generalization Experiments

Though we focus on the task of E2P in this work, the architecture design of our EP2T module is also applicable to other vision tasks. In this section, we apply EP2T to three different tasks (see below) to show its potential. Specifically, we embed the EP2T module into existing baseline models as an additional information channel and compare the performance. For all baseline models, we use the released source code and the corresponding training setup during experiments.

Flow estimation aims to infer the instantaneous speed and direction of the corresponding pixel motion of the object on the imaging plane. Compared to the traditional frame-difference-based optical flow estimation methods, event camera optical flow estimation has a higher temporal resolution, enabling more accurate estimation of object motion in complex scenarios involving fast motion and high rotation rates. We employ the standard networks cGAN [20] and EV-FlowNet [15] as baselines in this experiment. Following standard practice, we utilize the average endpoint error (AEE) [50] and the outlier rate (the ratio of errors greater than three pixels) to evaluate different methods. We use the MVSEC dataset to evaluate these different methods.

Event-to-image reconstruction is another canonical task for event cameras in which a high-contrast image with sharp edges is reconstructed from events, making full use of the advantages of the event camera's high contrast and fast response speed. In this experiment, we use BTEB [47] on the ECD dataset [51] and cGAN [20] on the MVSEC dataset [43] as the baselines and use the mean squared error (MSE), structural similarity (SSIM) [52], perceptual similarity (LPIPS) [53] for evaluation.

Object recognition with conventional cameras remains challenging due to high latency and motion blur. Event cameras are naturally less sensitive to such problems. In this experiment, we employ the widely used EST [54] network as the baseline. To assess the performance of our proposed methodology, we utilize the publicly available N-caltech101 dataset [49] and N-cars dataset [31] and use the recognition accuracy as the evaluation metric.

As shown in Tab. 3, adding the EP2T module improved the performance of baselines on various datasets and tasks. This result clearly demonstrates the potential of EP2T for other vision tasks. Since our initial motivation to design EP2T is driven by the 2D-3D registration task, we believe the current architecture design and the way to apply our method to other tasks both have further space to be optimized, which we leave as the future works.

4.4 Time and Memory Efficiency

We analyzed the time and memory efficiency of the event representation learning module EP2T, as well as the complete E2PNet consisting of EP2T and 2D-3D registration network. As shown in Tab. 4, our E2PNet is slightly slower and uses more memory than baseline methods. However, the overhead introduced by EP2T is arguably small given the significantly improved accuracy.

We also analyzed of the impact of two hyperparameters (number of FEN and number of sampling points) from three perspectives: accuracy, time efficiency, and memory cost in Tab. 5. Selecting an appropriate number of events per batch (FEN) optimizes algorithm performance. A small FEN yields insufficient information, whereas an excessive FEN extends sampling time and limits the exploration of spatiotemporal details. Raising the number of EP2T sampling points enhances spatiotemporal detail capture but escalates computational costs. The major contributor to the higher memory usage is the EP2T network designed based on PointNet++ [39], where the point-wise distance calculation

operation leads to memory growth that scales with the number of EP2T sampling points. However, when reducing the number of EP2T sampling points to 512, E2PNet runs at a similar speed (65.5ms) and memory cost (2.99GB) as other baselines yet still has higher accuracy.

In summary, our EP2T module is both effective and adaptable, introducing a novel approach for extracting spatiotemporal correlations and detailed information from event camera data. According to different performance requirements and available computing resources, the number of EP2T sampling points and sampling strategies can be flexibly adjusted to strike an efficiency-accuracy trade-off. The impact of different sampling strategies can also be viewed in the Appendix.

5 Conclusion and Limitation

In this work, we propose E2PNet, the first learning-based event-to-point cloud registration (E2P) method. We design a novel Event-Points-to-Tensor (EP2T) network, which encodes event data into feature tensors in the shape of a 2D grid such that mature RGB-based 2D-3D registration frameworks can be used for the E2P task. EP2T treats event data as spatio-temporal point clouds and separates the temporal and spatial domains for processing to handle the inhomogeneity of the event point cloud. Our experiments show that E2PNet is superior compared to other learning-based or handcrafted event representation methods and the traditional RGB-based registration in extreme environments (overexposure, motion blur). In addition, we also show the potential of the EP2T module in vision tasks beyond 2D-3D registration. Since E2PNet is inspired by 3D point-based learning architectures [39], similar limitations exist. For instance, an excessive number of sampling points will result in large memory consumption and efficiency degradation of E2PNet. However, further efficiency improvements are possible by some optimization techniques like delayed aggregation [55], sparse convolution [56]. We believe that solving this problem is an interesting and important future research direction.

References

Quang-Hieu Pham, Mikaela Angelina Uy, Binh-Son Hua, Duc Thanh Nguyen, Gemma Roig, and Sai-Kit Yeung. Lcd: Learned cross-domain descriptors for 2d-3d matching. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 11856-11864, 2020.

Richard Szeliski. Computer vision: algorithms and applications . Springer Nature, 2022.

Jogendra Nath Kundu, MV Rahul, Aditya Ganeshan, and R Venkatesh Babu. Object pose estimation from monocular image using multi-view keypoint correspondence. In Computer Vision-ECCV 2018 Workshops: Munich, Germany, September 8-14, 2018, Proceedings, Part III 15 , pages 298-313. Springer, 2019.

Francesc Moreno-Noguer, Vincent Lepetit, and Pascal Fua. Accurate non-iterative o(n) solution to the pnp problem. In 2007 IEEE 11th International Conference on Computer Vision , pages 1-8. Ieee, 2007.

Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Realtime single camera slam. IEEE transactions on pattern analysis and machine intelligence , 29(6):1052-1067, 2007.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4104-4113, 2016.

David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision , 60:91-110, 2004.

Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147-1163, 2015.

Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14 , pages 467-483. Springer, 2016.

Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud. In 2019 International Conference on Robotics and Automation (ICRA) , pages 4790-4796. IEEE, 2019.

Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-point cloud registration via deep classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15960-15969, 2021.

Bing Wang, Changhao Chen, Zhaopeng Cui, Jie Qin, Chris Xiaoxuan Lu, Zhengdi Yu, Peijun Zhao, Zhen Dong, Fan Zhu, Niki Trigoni, et al. P2-net: Joint description and detection of local features for pixel and point matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16004-16013, 2021.

Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Eventbased vision: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(1):154-180, 2020.

Bongki Son, Yunjae Suh, Sungho Kim, Heejae Jung, Jun-Seok Kim, Changwoo Shin, Keunju Park, Kyoobin Lee, Jinman Park, Jooyeon Woo, et al. 4.1 a 640 × 480 dynamic vision sensor with a 9 µ m pixel and 300meps address-event representation. In 2017 IEEE International Solid-State Circuits Conference (ISSCC) , pages 66-67. IEEE, 2017.

Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Ev-flownet: Selfsupervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898 , 2018.

Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. Endto-end learning of representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5633-5643, 2019.

Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 989-997, 2019.

Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch. Learning an event sequence embedding for dense event-based deep stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1527-1537, 2019.

R Wes Baldwin, Ruixu Liu, Mohammed Almatrafi, Vijayan Asari, and Keigo Hirakawa. Timeordered recent event (tore) volumes for event cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(2):2519-2532, 2022.

Mohammad Mostafavi, Lin Wang, and Kuk-Jin Yoon. Learning to reconstruct hdr images from events, with applications to depth and flow prediction. International Journal of Computer Vision , 129:900-920, 2021.

Anton Mitrokhin, Chengxi Ye, Cornelia Fermüller, Yiannis Aloimonos, and Tobi Delbruck. Evimo: Motion segmentation dataset and learning pipeline for event cameras. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6105-6112. IEEE, 2019.

Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based motion segmentation with spatio-temporal graph cuts. IEEE Transactions on Neural Networks and Learning Systems , 2021.

Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence , 43(6):1964-1980, 2019.

Xiang Zhang, Wei Liao, Lei Yu, Wen Yang, and Gui-Song Xia. Event-based synthetic aperture imaging with a hybrid network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14235-14244, 2021.

Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural networks , 111:47-63, 2019.

Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence , 43(6):1964-1980, 2019.

Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5419-5427, 2018.

Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi Ieng, and Chiara Bartolozzi. Eventbased visual flow. IEEE transactions on neural networks and learning systems , 25(2):407-417, 2013.

Mohammed Almatrafi, Raymond Baldwin, Kiyoharu Aizawa, and Keigo Hirakawa. Distance surface for event-based optical flow. IEEE transactions on pattern analysis and machine intelligence , 42(7):1547-1556, 2020.

Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch. Learning an event sequence embedding for dense event-based deep stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1527-1537, 2019.

Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 17311740, 2018.

Yijin Li, Han Zhou, Bangbang Yang, Ye Zhang, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Graph-based asynchronous event processing for rapid object recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 934-943, 2021.

Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based motion segmentation with spatio-temporal graph cuts. IEEE Transactions on Neural Networks and Learning Systems , 2021.

Anton Mitrokhin, Zhiyuan Hua, Cornelia Fermuller, and Yiannis Aloimonos. Learning visual motion segmentation using event surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14414-14423, 2020.

Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. Aegnn: Asynchronous event-based graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12371-12381, 2022.

Zhiwen Chen, Jinjian Wu, Junhui Hou, Leida Li, Weisheng Dong, and Guangming Shi. Ecsnet: Spatio-temporal feature learning for event camera. IEEE Transactions on Circuits and Systems for Video Technology , 2022.

Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J Guibas. Joint embeddings of shapes and images via cnn image purification. ACM transactions on graphics (TOG) , 34(6):1-12, 2015.

Xiaoxia Xing, Yinghao Cai, Tao Lu, Shaojun Cai, Yiping Yang, and Dayong Wen. 3dtnet: Learning local features using 2d and 3d cues. In 2018 international conference on 3d vision (3dv) , pages 435-443. IEEE, 2018.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Carsten Moenning and Neil A Dodgson. Intrinsic point cloud simplification. Proc. 14th GrahiCon , 14:23, 2004.

Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural networks: Lstm cells and network architectures. Neural computation , 31(7):1235-1270, 2019.

Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Strpm: A spatiotemporal residual predictive model for high-resolution video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13946-13955, 2022.

Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters , 3(3):2032-2039, 2018.

Ling Gao, Yuxuan Liang, Jiaqi Yang, Shaoxun Wu, Chenyu Wang, Jiaben Chen, and Laurent Kneip. Vector: A versatile event-centric benchmark for multi-sensor slam. IEEE Robotics and Automation Letters , 7(3):8217-8224, 2022.

Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures , volume 1611, pages 586-606. Spie, 1992.

Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11824-11833, 2020.

Federico Paredes-Vallés and Guido CHE de Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3446-3455, 2021.

Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research , 36(2):142-149, 2017.

Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience , 9:437, 2015.

Zhexiong Wan, Yuchao Dai, and Yuxin Mao. Learning dense and continuous optical flow from an event camera. IEEE Transactions on Image Processing , 31:7237-7251, 2022.

Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research , 36(2):142-149, 2017.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. Endto-end learning of representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5633-5643, 2019.

Yu Feng, Boyuan Tian, Tiancheng Xu, Paul Whatmough, and Yuhao Zhu. Mesorasi: Architecture support for point cloud analytics via delayed-aggregation. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) , pages 1037-1050. IEEE, 2020.

Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. Event-based asynchronous sparse convolutional networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16 , pages 415-431. Springer, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an method to perform point cloud registration using event camera data.\nThe method first learns a feature representation (E2PT) from a point cloud of events. This representation is used as input to standard registration networks.\nThe experimental section shows better accuracy compared to other event-based feature represenations as well as frame-based approaches.\nOther experiments also show that the E2PT representations is generic and cna be used for other event vision tasks.\n\nSTRENGTHS:\nThe paper addresses a novel problem of using event camera data for 2d-3d registration. They adapt existing registration pipelines taking into account the specificity of the events.\nThey propose a new learned event-based representation which leads to good experimental results for the registration problem, but also for other event vision tasks (optical flow, classification, image reconstruction).\n\nWEAKNESSES:\nThe clarity of the paper can be overall improved.\nIn particular the method section is not always clear and some details are missing.\nFor example it is not clear how the features Ft , Fs , and Fst are built (Sec. 3.1.1). the authors mention some \"multi-layer convolution\", but is not clear what is the input to these convolutions, what exactly is the architecture, if tehre is a separate network for  Ft , Fs , and Fst or if it is shared. What is the size of Ft , Fs , and Fst etc. Describing the algorithm with more equations and pseudo-code can make the method clearer, since text and figures alone can be ambigous.\n\nSimilarly there is no description in the method section of the E2Pnet architecture, which is presented as the second main contribution in the introduction. From the experimental details it seems that standard architecture from previous works [1,11] are used?\n\nQUESTIONS:\nPlease describe more precisely the steps of the method as stated above.\n \nAlso, please clarify the event sampling strategy: \"We follow the FEN [25, 26] principle and acquire 20000 consecutive\nevents at a time and sample N = 8192 events from them.\" \nDoes it mean that out of 20k events, only N are used to build the tensor representation or that multiple consecutive tensor representations are used? why using such a specific value for N: 8192?\n\nLIMITATIONS:\nIt would be interesting to understand the runtime of this approach, and how does it scale with respect to the event rate (number of events in the point cloud).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proposes a learning-based event-to-point cloud registration method, which encodes event spatio-temporal data into a grid-shaped feature tensor, and propose a framework to construct E2P datasets using existing SLAM datasets. Experiments are conducted on MVSEC-E2P and VECtor-E2P datasets, and state-of-the-art results are achieved on these datasets.

STRENGTHS:
1. The presentation is easy to understand. 
2. Experiments are well conducted and convincing.

WEAKNESSES:
1. Since event-based applications require a high response speed, the efficiency analysis should be given about the comparison of the runtime and memory usage between the proposed method and other state-of-the-art methods.
2. Since the experimental datasets are generated by the authors, did the authors retrain the learning-based models on the datasets in order to compare with these learning-based methods?
3. The comparisons are done to a few methods in the current manuscript, I suggest more state-of-the-art methods should be included for comparison.
4. Minor Typos / Writing  
Figure 3: what does O_{E} mean?  
Line 211: the definition of f_{SP} is missing.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a Event-Points-to-Tensor (EP2T) network, which treats event data as spatio-temporal point clouds, to process event signals without losing the spatiotemporal information of event signals (especially temporal information, compared with other voxel grid-based methods). In terms of experiments, this work demonstrates the effectiveness of the EP2T network by using event-based point cloud registration as an example, resulting in the development of E2PNet. Furthermore, the authors have gone a step further and tested the generalization ability of the EP2T network in tasks such as optical flow estimation, image reconstruction, and object recognition, obtaining promising results.\n\nSTRENGTHS:\nThe proposed Event-Points-to-Tensor (EP2T) network in this paper takes a different approach compared to most existing event signal processing models. This method treats event signals as three-dimensional spatio-temporal point clouds and employs operations such as Local Aggregation (LA), Spatio-temporal Separated Attention (STA), and Feature Propagation (FP) for extracting and preprocessing spatio-temporal features of events. Finally, tensorization is performed to obtain event feature representations compatible with traditional visual models. Compared to previous models that always tensorize event signals before feeding them into the model, this method can fully utilize the discrete and sparse characteristics of events. It effectively extracts spatiotemporal features from events, especially rich temporal domain information. Additionally, the LA operation separates the extraction of features in the temporal and spatial domains, enabling feature extraction and aggregation on different dimensions/domains, considering the distinct physical meanings of the three dimensions in the event signal \"point cloud\".\n\nWEAKNESSES:\nAlthough the proposed EP2T network in this paper achieves sparse processing of event signals in the first half, it still requires tensorization of the events in the end. In practice, such an operation can significantly degrade the temporal precision of events. While the 2D-3D Registration task proposed in this paper may not aim for high temporal resolution, it is crucial for tasks that are sensitive to event precision, such as high-frame-rate video reconstruction and low-latency object tracking. Furthermore, although the idea of using point cloud networks for sparse processing of event signals in this paper is innovative, the intuitive motivation behind this approach is not clearly presented in the writing. For instance, compared to the operation of tensorization followed by feature extraction, it is not discussed in the paper which events EP2T can handle better and what advantages EP2T possesses. This discussion is lacking in the paper.\n\nQUESTIONS:\nThe authors are encouraged to respond to the discussion regarding the intuitive motivation behind EP2T and provide an explanation of whether this operation takes into account the mathematical/physical characteristics of events better or offers better feature extraction advantages compared to traditional dense CNN-based model structures, or if there are other considerations involved. Furthermore, the reviewers request additional targeted experiments to validate the proposed intuitive advantages.\n\nLIMITATIONS:\nThe author points out that the problem of large memory consumption is an important future research direction, which is similar limitation to other point-based models. If this problem can be solved, it will indeed help to improve the practicability of the model. Furthermore, there is a potential limitation that the authors did not mention in the paper, namely, that E2PNet still relies on tensorization to convert event point clouds into dense grid representations, which are then fed into downstream models designed for traditional vision tasks. It would be intriguing if future work could bypass this step and directly output target results using discrete/sparse event signals.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed the first learning-based work that can handle event-to-point cloud registration (E2P). More specifically, a novel Event-Points-to-Tensor (EP2T) network is proposed to encode the data from the event camera into features tensors in the form of a 2D grid. The temporal patch aggregation and spatial patch aggregation combined with spatio-temporal kernel are applied to obtain the global feature. Then based on the output gridded tensor, a standard 2d-3d feature based algorithm is applied to obtain the structure and motion of the final 3D reconstruction scene.\n\nThe experimental results based on the two representative datasets demonstrate the performance, mainly in terms of accuracy, of the proposed E2PNet on the task of event to point cloud registration.\n\nSTRENGTHS:\nThe motivation of this paper is well designed, that is introducing the first learning-based architecture to handle the sparse reconstruction based on event cameras. More specifically, the spatial and temporal attention mechanism together with the feature propagation module to obtain the final output gridded tensor. The design of the EP2T network is straightforward, elegant and easy to follow.\n\nIn addition to its innovative approach, the paper stands out for its well-structured writing style that includes a clear and concise method statement. The authors have taken great care in presenting their research in a manner that is easily understandable to the readers. Furthermore, the inclusion of visual representations helps to further elucidate the concepts and techniques discussed in the paper.\n\nMoreover, the performance shown in the statistical experiments especially in Table 1 demonstrate the superior performance of the proposed approach over the previous state-of-the-art approaches.\n\nWEAKNESSES:\nMy major concerns lie in the following two aspects.\n\nFirst, the statistical experiments in Table 1-3 only demonstrate the accuracy in terms of camera position and direction, while the efficiency especially the time and memory efficiency is missed. \n\nSecond, the limitations or failure case of the proposed approach need to be discussed. I am wondering the performance of the proposed approach on more complex and even comparatively large-scale environment.\n\nQUESTIONS:\nSee the \"Weakness\" section.\n\nLIMITATIONS:\nSee the \"Weakness\" section. Furthermore, the final score assigned to the study is not solely determined by the discussion among peer reviewers. If the authors can solve my main concerns, I would like to raise the score.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}