SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Quantitative results of 2D-3D registration. Since the three hand-crafted event tensorization representation methods [15, 17, 20] are complementary, we use them jointly to achieve the best performance and call this hybrid method E-Statistic. To fairly compare against point-based representations, we also remove the tensorization process of E2PNet, i.e., w/o TR. The best performing method is highlighted.
Input Representation	Method	DeepI2P [11](Registration-based)	LCD [1] (Retrieval-based)
MVSEC-E2P	VECtor-E2P	MVSEC-E2P	VECtor-E2P
RE(°)(↓)	TE(m)(↓)	RE(°)(↓)	TE(m)(↓)	RE(°)(↓)	TE(m)(↓)	RE(°)(↓)	TE(m)(↓)
Traditional Image	Grayscale Image	7.922	0.370	11.343	3.176	6.335	1.347	17.879	13.200
Event(Tensor-based)	E-Statistic	6.748	0.250	10.654	3.524	4.968	1.297	11.034	9.416
Tore [19]	7.465	0.192	10.542	4.565	4.855	1.350	9.521	7.254
Ours	5.127	0.164	8.778	2.454	3.606	0.821	8.672	7.403
Event(Point-based)	ECSNet [36]	8.075	1.612	10.149	2.542	4.985	1.263	20.740	13.284
Ours(w/o TR)	6.021	0.721	8.795	3.212	4.422	0.768	9.120	6.551
ground truth (GT), we use the indoor-  $x$  and indoor-  $y$  sequences for training and testing respectively, where  $x \in [1,3]$  and  $y = 4$ . VECTor [44] uses a 128-beam LiDAR and an event camera with a resolution of (640,480). Different from MVSEC, VECTor contains various large-scale indoor environments with long trajectories. The GT pose is derived via ICP[45] between LiDAR scans and dense environment point clouds captured by a high-precision laser scanner. We use the units-dolly, units-scooter, corridors-dolly and corridors-walk sequences for training, and the school-dolly and school-scooter sequences for evaluation. For both datasets, we randomly acquire event data in an (256,256) area from event camera data as input; simultaneously we also acquire grayscale images with the same field of view and resolution for comparison.
Evaluation. Following the previous standard [46], we use the translation error  $\mathrm{TE} = ||\mathbf{T}_{\mathrm{GT}} - \mathbf{T}_{\mathrm{pred}}||_2$  and the rotation error:  $\mathrm{RE} = \arccos \left(\frac{tr\left(\mathbf{R}_{\mathrm{GT}}^{-1}\mathbf{R}_{\mathrm{pred}}\right) - 1}{2}\right)$  to evaluate the accuracy of predicted camera poses in E2P.  $\mathbf{T}$  and  $\mathbf{R}$  represents the translation vector and rotation matrix respectively, and  $tr(\cdot)$  is the trace of a matrix.
Implementation Details. We follow the FEN [26, 20] principle and acquire 20000 consecutive events at a time and sample  $N = 8192$  events from them. Since EP2T has encoded the spatiotemporal information as 2D grid tensors, we can regard E2P as the registration of 2D feature maps with 3D point clouds. We choose LCD [1] and DeepI2P [11] as the representatives to test the performance of E2P using feature retrieval-based and point visibility-based registration. LCD maps the corresponding 2D tensor and 3D point cloud into a high-dimensional feature space, where training aims to make the feature distance as close as possible. During training, we fetch events (image) in the current pose and their corresponding point cloud patch according to the GT pose (see Sec. 3.2). The principle of DeepI2P is to judge whether the input point cloud appears in the corresponding field of view of events (image). During training, we use the visible point clouds for each event view and its surrounding point clouds (enlarged by  $30\%$ ) for supervision. All methods are implemented using Pytorch. Training is done following the setup of individual baselines on a 3090Ti GPU.
We compare E2PNet against conventional camera-based registration (i.e., the original version of LCD and DeepI2P) and other event representation methods, including tensor-based [19, 15, 17, 20] and point-based [36] methods. To compare against point-based methods, we do not perform the final tensorization (i.e., w/o TR) so that the encoded features are also point-based. Since there is no standard technique to apply point-based event representations to image-based registration frameworks, for fair comparisons, we implement a simple encoder with max-pooling and a linear layer (see appendix for details) to project the point-based features to a global feature vector. Subsequently, we use this vector to replace the original global feature (computed by performing convolution on the tensor-based representation) in both LCD and DeepI2P. For LCD-based methods (retrieval-based), we use the trained model to encode the feature on the target point cloud under different poses to build a point cloud feature library. During inference, we extract the corresponding pose in the feature library
Figure 4: Qualitative comparison of E2PNet with other methods. The green and cyan boxes represent the projected position of the 3D point clouds onto the view plane of the camera after rigidly transforming to the camera coordinate system using ground truth pose and predicted pose respectively. Compared to the image-based baseline (LCD), event-based methods are more robust to challenging illumination and motion. E2PNet (ours) performs best among all event-based methods.
Table 2: Ablation. Removing individual modules from E2PNet hurt the registration accuracy.
Method	Ours	Ours w/o S-T separated in STA	Ours w/o STA	Ours w/o LA+FP	Ours w/o LA+STA+FP
RE(°)(↓)	3.606	3.710	3.922	4.878	4.968
TE(m)(↓)	0.821	0.824	0.913	1.232	1.297
that is closest to the feature of the event data. For DeepI2P-based methods (point cloud visibility classification-based), we use the same optimization method as the original method [11].
As shown in Tab. 1, replacing conventional image inputs with event data improves the registration accuracy significantly and consistently across datasets and baseline methods. E2PNet performs the best among both tensor-based and point-based E2P methods, outperforming both hand-crafted (E-Statistic) and learning-based methods. Qualitative results in Fig. 4 further show the robustness of E2PNet in different environments. Finally, due to better compatibility, tensor-based methods perform better than point-based methods, and sometimes by a large margin. Note that although point-based representations can be applied to 2D-3D registration, since only a global feature vector of the input (event) image is required, they may not be compatible with tasks that require pixel-level feature maps.
In this section, we analyze the effectiveness of individual components of E2PNet, i.e., the Local Aggregation module (LA), Spatio-temporal Separated Attention module (STA), Feature Propagation module (FP), and the mechanism for seperating the spatial and temporal domains in STA. We use LCD [1] as our baseline model and conduct experiments on the MVSEC-E2P dataset. As shown in Tab. 2, the registration accuracy reduces as components are removed, showing the importance of each component. Note that since LA and FP are coupled modules (we need FP to propagate features from aggregation centers to individual event points), we add and remove them together.
Table 3: Results of EP2T on other vision tasks. "Ours" means the result obtained after incorporating our EP2T module. For a fair comparison, The relevant training settings are the same as the original framework. The dataset used for evaluation is indicated after "@".
Flow Estimation	Event-to-Image Reconstruction	Object Recognition
Method	AEE(↓)	Outlier(%) (↓)	Method	MSE(↓)	SSIM(↑)	LPIPS(↓)	Method	Accuracy(↑)
cGAN[20] @ [43]	0.34	0.06	cGAN[20] @ [43]	0.479	0.431	0.450	EST[16] @ [31]	0.746
cGAN(ours) @ [43]	0.30	0.005	cGAN(ours) @ [43]	0.476	0.460	0.435	EST(ours) @ [31]	0.771
EV-Flow[15] @ [43]	0.478	0.03	BTEB[47] @ [48]	0.20	0.32	0.46	EST[16] @ [49]	0.949
EV-Flow(ours) @ [43]	0.462	0.03	BTEB(ours) @ [48]	0.16	0.32	0.42	EST(ours) @ [49]	0.957
4.3 Generalization Experiments
Though we focus on the task of E2P in this work, the architecture design of our EP2T module is also applicable to other vision tasks. In this section, we apply EP2T to three different tasks (see below) to show its potential. Specifically, we embed the EP2T module into existing baseline models as an additional information channel and compare the performance. For all baseline models, we use the released source code and the corresponding training setup during experiments.
Flow estimation aims to infer the instantaneous speed and direction of the corresponding pixel motion of the object on the imaging plane. Compared to the traditional frame-difference-based optical flow estimation methods, event camera optical flow estimation has a higher temporal resolution, enabling more accurate estimation of object motion in complex scenarios involving fast motion and high rotation rates. We employ the standard networks cGAN [20] and EV-FlowNet [15] as baselines in this experiment. Following standard practice, we utilize the average endpoint error (AEE) [50] and the outlier rate (the ratio of errors greater than three pixels) to evaluate different methods. We use the MVSEC dataset to evaluate these different methods.
Event-to-image reconstruction is another canonical task for event cameras in which a high-contrast image with sharp edges is reconstructed from events, making full use of the advantages of the event camera's high contrast and fast response speed. In this experiment, we use BTEB [47] on the ECD dataset [51] and cGAN [20] on the MVSEC dataset [43] as the baselines and use the mean squared error (MSE), structural similarity (SSIM) [52], perceptual similarity (LPIPS) [53] for evaluation.
Object recognition with conventional cameras remains challenging due to high latency and motion blur. Event cameras are naturally less sensitive to such problems. In this experiment, we employ the widely used EST [54] network as the baseline. To assess the performance of our proposed methodology, we utilize the publicly available N-caltech101 dataset [49] and N-cars dataset [31] and use the recognition accuracy as the evaluation metric.
As shown in Tab. 3, adding the EP2T module improved the performance of baselines on various datasets and tasks. This result clearly demonstrates the potential of EP2T for other vision tasks. Since our initial motivation to design EP2T is driven by the 2D-3D registration task, we believe the current architecture design and the way to apply our method to other tasks both have further space to be optimized, which we leave as the future works.
4.4 Time and Memory Efficiency
We analyzed the time and memory efficiency of the event representation learning module EP2T, as well as the complete E2PNet consisting of EP2T and 2D-3D registration network. As shown in Tab. 4, our E2PNet is slightly slower and uses more memory than baseline methods. However, the overhead introduced by EP2T is arguably small given the significantly improved accuracy.
We also analyzed the impact of two hyperparameters (number of FEN and number of sampling points) from three perspectives: accuracy, time efficiency, and memory cost in Tab. 5. Selecting an appropriate number of events per batch (FEN) optimizes algorithm performance. A small FEN yields insufficient information, whereas an excessive FEN extends sampling time and limits the exploration of spatiotemporal details. Raising the number of EP2T sampling points enhances spatiotemporal detail capture but escalates computational costs. The major contributor to the higher memory usage is the EP2T network designed based on PointNet++ [39], where the point-wise distance calculation
Table 4: Time and memory efficiency of different methods. We compared the event representation module and the Event-to-point cloud registration framework, which consists of the event representation module and the 2D-3D registration network. All experiments were performed with a batch size of 1 on a 3090Ti GPU using the MVSEC-E2P dataset. We jointly use three hand-crafted event tensorization representation methods [15, 17, 20] to achieve the best performance and call this hybrid method E-Statistic. Compared to the baselines, E2PNet demonstrated significantly higher accuracy and marginally lower but still acceptable time and memory efficiency.
Method	2D-3D registration using LCD [1]	Event representation
Grayscale Image	E2PNet	E-statistic	ECSNet [36]	EP2T	E-statistic
Time(ms)(↓)	55.29	110.62	55.00	81.82	42.8	3.16
Space(MB)(↓)	2218	7890	2216	2530	4650	0.68
RE(°)(↓)	6.336	3.606	4.968	4.985	×	×
TE(m)(↓)	1.347	0.821	1.297	1.263	×	×
Table 5: The influence of varying FEN and the number of EP2T sampling points on the accuracy and efficiency of E2PNet. FEN represents the event count in each batch, from which E-statistic is employed to extract global spatial features. Our EP2T algorithm selects some sampling points from the events processed in each batch as the aggregation center of the spatiotemporal information. All experiments are conducted on MVSEC-E2P dataset using LCD [1] framework, with inference performed on a 3090ti GPU and a batch size of 1.
Number of FEN	EP2T sampling points	RE(°)(↓)	TE(m)(↓)	Time(ms)(↓)	Space(MB)(↓)
20000 (Ours)	512	3.813	1.089	65.51	2992
4096	3.777	0.954	79.55	4394
8192 (Ours)	3.606	0.821	110.62	7890
16384	3.247	0.789	144.09	12190
8192	8192	3.479	1.074	90.34	7258
60000	4.242	1.426	109.50	7908
operation leads to memory growth that scales with the number of EP2T sampling points. However, when reducing the number of EP2T sampling points to 512, E2PNet runs at a similar speed (65.5ms) and memory cost (2.99GB) as other baselines yet still has higher accuracy.
In summary, our EP2T module is both effective and adaptable, introducing a novel approach for extracting spatiotemporal correlations and detailed information from event camera data. According to different performance requirements and available computing resources, the number of EP2T sampling points and sampling strategies can be flexibly adjusted to strike an efficiency-accuracy trade-off. The impact of different sampling strategies can also be viewed in the Appendix.
In this work, we propose E2PNet, the first learning-based event-to-point cloud registration (E2P) method. We design a novel Event-Points-to-Tensor (EP2T) network, which encodes event data into feature tensors in the shape of a 2D grid such that mature RGB-based 2D-3D registration frameworks can be used for the E2P task. EP2T treats event data as spatio-temporal point clouds and separates the temporal and spatial domains for processing to handle the inhomogeneity of the event point cloud. Our experiments show that E2PNet is superior compared to other learning-based or handcrafted event representation methods and the traditional RGB-based registration in extreme environments (overexposure, motion blur). In addition, we also show the potential of the EP2T module in vision tasks beyond 2D-3D registration. Since E2PNet is inspired by 3D point-based learning architectures [39], similar limitations exist. For instance, an excessive number of sampling points will result in large memory consumption and efficiency degradation of E2PNet. However, further efficiency improvements are possible by some optimization techniques like delayed aggregation [55], sparse convolution [56]. We believe that solving this problem is an interesting and important future research direction.
This work was supported by the National Natural Science Foundation of China (61971363, 62301473); by the FuXiaQuan National Independent Innovation Demonstration Zone Collaborative Innovation Platform (No.3502ZCQXT2021003); by the Fundamental Research Funds for the Central Universities (No. 20720230033); by PDL (2022-PDL-12); by the China Postdoctoral Science Foundation (No.2021M690094). We would like thank the anonymous reviewers for their valuable comments.
[1] Quang-Hieu Pham, Mikaela Angelina Uy, Binh-Son Hua, Duc Thanh Nguyen, Gemma Roig, and Sai-Kit Yeung. Lcd: Learned cross-domain descriptors for 2d-3d matching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11856–11864, 2020.
[2] Richard Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
[3] Jogendra Nath Kundu, MV Rahul, Aditya Ganeshan, and R Venkatesh Babu. Object pose estimation from monocular image using multi-view keypoint correspondence. In Computer Vision-ECCV 2018 Workshops: Munich, Germany, September 8-14, 2018, Proceedings, Part III 15, pages 298-313. Springer, 2019.
[4] Francesc Moreno-Noguer, Vincent Lepetit, and Pascal Fua. Accurate non-iterative o(n) solution to the pnp problem. In 2007 IEEE 11th International Conference on Computer Vision, pages 1-8. IEEE, 2007.
[5] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Realtime single camera slam. IEEE transactions on pattern analysis and machine intelligence, 29(6):1052-1067, 2007.
[6] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[7] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
[8] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147-1163, 2015.
[9] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 467-483. Springer, 2016.
[10] Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud. In 2019 International Conference on Robotics and Automation (ICRA), pages 4790-4796. IEEE, 2019.
[11] Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-point cloud registration via deep classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15960-15969, 2021.
[12] Bing Wang, Changhao Chen, Zhaopeng Cui, Jie Qin, Chris Xiaoxuan Lu, Zhengdi Yu, Peijun Zhao, Zhen Dong, Fan Zhu, Niki Trigoni, et al. P2-net: Joint description and detection of local features for pixel and point matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16004-16013, 2021.
[13] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-180, 2020.
[45] Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[46] Zi Jian Yew and Gim Hee Lee. RPM-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11824-11833, 2020.
[47] Federico Paredes-Vallés and Guido CHE de Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3446-3455, 2021.
[48] Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research, 36(2):142-149, 2017.
[49] Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
[50] Zhexiong Wan, Yuchao Dai, and Yuxin Mao. Learning dense and continuous optical flow from an event camera. IEEE Transactions on Image Processing, 31:7237-7251, 2022.
[51] Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research, 36(2):142-149, 2017.
[52] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[53] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[54] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
[55] Yu Feng, Boyuan Tian, Tiancheng Xu, Paul Whatmough, and Yuhao Zhu. Mesorasi: Architecture support for point cloud analytics via delayed-aggregation. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1037-1050. IEEE, 2020.
[56] Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. Event-based asynchronous sparse convolutional networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 415-431. Springer, 2020.