SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiuhong Lin<sup>ab</sup>, Changjie Qiu<sup>ab*</sup>, Zhipeng Cai<sup>c</sup>, Siqi Shen<sup>ab†</sup>, Yu Zang<sup>ab</sup>, Weiquan Liu<sup>ab</sup>, Xuesheng Bian<sup>a</sup>, Matthias Müller<sup>d</sup>, Cheng Wang<sup>ab</sup>  
<sup>a</sup>Fujian Key Lab of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University (XMU), China.
<sup>b</sup>Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China. <sup>c</sup>Intel Labs. <sup>d</sup>Apple Inc. <sup>e</sup>Yancheng Institute Of Technology, China. {siqishen, cwang}@xmu.edu.cn, zhipeng.cai@intel.com {xhlinxm, qiuchangjie, xsbian}@stu.xmu.edu.cn
Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECTor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: E2PNet.
Computer vision aims to understand and reconstruct the world from 2D observations [2]. Establishing the relationship (pose or correspondences) between 2D data and the 3D world is an important step toward this goal. Conventional 2D-3D registration mostly relies on multi-view geometry [3, 4, 5]. Specifically, 3D (key) points are first triangulated [6, 5] using multiple images with proper disparity. Subsequently, the corresponding 2D features [7, 8, 9] are attached to 3D points for subsequent 2D-3D registration. This framework is by design image-based, which limits the use of other sensors that obtain 3D point clouds directly (e.g., LiDAR, depth cameras).
Recently, learning-based methods for direct 2D-3D registration have been proposed [10, 1, 11, 12]. However, these methods require high-quality RGB or grayscale images which are not always feasible
*Xiuhong Lin and Changjie Qiu contribute equally to this work.
Figure 1: Teaser. This work proposes the first learning-based event-to-point cloud registration method E2PNet. As shown in the figure, challenging illumination/fast motion may cause RGB-based methods (e.g., LCD [1]) to fail, resulting in erroneous registration (red camera). Using event data as input, E2PNet is robust to such illumination change and accurately recovers the camera pose (blue).
in practice. Challenging environments with extreme illumination or motion blur exist in many applications, which may cause the algorithm to fail completely (see Fig. 1 for an example).
Unlike traditional cameras, the novel information acquisition principle of event cameras [13] makes them less susceptible to challenging illumination and motion blur. This advantage motivates us to design a robust 2D-3D registration method for event cameras.
High dynamic range (HDR, up to 120 dB [13]) and high temporal resolution bring excellent performance to event cameras but also create new challenges. The unstructured data stream of event cameras is sparse in the spatial plane but dense (up to 300 million events per second [14]) and non-uniform in the temporal dimension. Existing methods mostly process events as image-like feature maps [15, 16] or voxels [17, 18, 19], which arguably loses spatio-temporal details. Although event cameras have been used for other vision tasks such as optical flow estimation [15, 18, 20], moving object detection [21, 22] and high dynamic image reconstruction [23, 24], the task of event-to-point cloud registration  $(E2P)$  has not yet been studied.
There are several challenges. First, event cameras only capture the surface appearance of the scene, whereas 3D point clouds encode geometric structures, which is inherently challenging [11]. Second, event cameras have lower resolution than traditional cameras and only capture the brightness variation rather than the brightness itself. Finally, event data and point clouds can both be considered as unstructured 3D data, which are harder to process than 2D images. Moreover, the lack of an E2P dataset with accurate annotations limits the research of such cross-modal registration methods.
To address these problems, we present E2PNet, the first E2P framework and the corresponding dataset for end-to-end learning. Specifically, we propose a modular event representation network called Event-Points-to-Tensor (EP2T), which applies local feature aggregation with different temporal and spatial distance weights and spatio-temporal separated attention mechanisms to effectively encode sparse and in-homogeneous event data. In the meantime, EP2T adaptively aggregates local neighborhood features with uneven event data in the temporal and spatial dimensions and outputs a structured (grid-shaped) feature tensor which can be directly plugged into image-based 2D-3D registration frameworks. E2PNet is capable of E2P registration for indoor scenes of different scales (single room to one floor) and produces the corresponding 6-DOF transformation between the event camera and 3D point clouds. As shown in Fig. 1, replacing the RGB inputs with event data can significantly improve the registration performance on challenging data. Our main contributions are:
- We propose EP2T, the first representation learning architecture to transform event spatiotemporal point clouds into a grid-shaped feature tensor. EP2T can adaptively learn how to extract critical spatio-temporal information according to downstream tasks, and can be applied as a plug-and-play module to existing event camera processing networks.
- We propose E2PNet, the first approach that allows direct registration of 2D event cameras and 3D point clouds. To facilitate follow-up research, we also propose a framework to construct E2P datasets using existing SLAM datasets.
- We conduct extensive experiments and analysis to demonstrate the effectiveness of E2PNet for E2P, and the potential of the EP2T module to benefit in other diverse tasks.
2.1 Event Representation
The asynchronous unstructured event data poses a great challenge for computer vision. While spiking neural networks (SNN) [25] can directly use the raw data of event cameras, they require special hardware and has low flexibility. A more common approach is to process batches of event data via an intermediate representation, e.g., an image of positive and negative events over a certain horizon. The processing window is usually set to a fixed number of events (FEN) [26, 20] or a fixed interval time (FIT) [15, 20].
Hand-crafted representations usually convert raw event data into 2D grid-shaped feature maps with sensor resolution using the statistics of raw data, for example, event counting [27], average/latest timestamp of events on each pixel [15] or temporal distance-weighted statistics [17]. While these strategies can approximate the distribution of event data with respect to space and time, some important details (i.e. spatio-temporal correlation) may be lost due to the aggregation. To address this shortcoming, another set of methods [28, 29, 30, 31] discards most of the events and only keeps the latest few events for each pixel and analyzes them in detail. However, these methods cannot encode distant spatio-temporal relationships. More recent work tries to incorporate these relationships by representing event data as graphs [32, 33, 34, 35]. Although more spatio-temporal information can be extracted, building a graph structure is complicated and computationally expensive. Each hand-crafted representation has different advantages and disadvantages. Hence, choosing the right representation for a given tasks is often not trivial. The proposed E2PNet also utilizes hand-crafted methods [15, 20, 17]. However, we use them in conjunction with our learned features instead of raw event data to produce the final grid-shaped feature tensors.
Learning-based representations train models to adaptively extract critical information based on the specific task. However, the inhomogeneity and lack of structure make event representation learning challenging. EST [16] and DDES [30] propose to learn tensor-based representations, which regard the events of each pixel as pulse signals and optimize the impulse response function. ECSNet [36] uses 3D point cloud learning methods to extract point-based features. However, these feature points are incompatible with the processing pipeline of downstream tasks, and generalize poorly to other tasks. In contrast, we aim to fully extract the spatio-temporal correlation of the event data and transform it into a grid feature tensor representation for better compatibility.
2.2 2D-3D Registration
This work focuses on the direct 2D-3D registration problem, which has been studied for conventional images. Li et al. [37] use the multi-view projection image of 3D objects to extract the corresponding HOG features, and train a feature mapping network to push the features from the same 2D-3D objects closer. 3DTNet [38] uses RGB-D point clouds and images to train distinctive 3D descriptors. However, these methods can only be used to register images to colored point clouds, which is not always feasible. E2PNet does not generate features based on color information, hence it can be applied to conventional point clouds without colors. 2D3D-MatchNet [10] and LCD [1] learn consistent feature representations of 2D images and 3D point clouds, realizing 2D-3D registration through cross-modal feature retrieval. P2-Net [12] learns the detection and matching of potential feature points from images and RGB-D points. DeepI2P [11] transforms image-point cloud registration into point cloud visibility classification and then maximizes the number of visible points by optimizing the camera pose. A common drawback of these image-based methods is that they require high-quality images to perform well, which cannot be guaranteed in applications with challenging illumination conditions or camera motion. As shown later in the experiments, introducing event data for 2D-3D registration makes E2PNet much more robust in challenging environments.
E2PNet treats the input event data as spatio-temporal point clouds. Before explaining how this can be done, we briefly introduce the generation process of event data here. During data capture, each pixel of the event camera responds independently and asynchronously to illumination changes. Specifically, given an image of size  $(H,W)$ , denote  $I(t_{i},h,w)$  as the illumination at time  $t_i$  for the pixel at  $(h,w)$ .
Figure 2: The proposed Event-Points-to-Tensor (EP2T) network. EP2T treats event data as spatio-temporal point clouds. Local Aggregation (LA) first generates feature aggregation centers and aggregates local features from neighboring points. Subsequently, Spatio-temporal Separated Attention (STA) extracts global features using attention. To handle the spatio-temporal inhomogeneity problem, LA and STA separate spatial and temporal information during computation. Next, Feature Propagation (FP) propagates features from aggregation centers to each event point based on the distance. Finally, grid-shaped feature tensors are obtained by different yet complementary event tensorization methods.
When the inequality
$$
\left\| \log \left(I \left(t _ {i}, h, w\right)\right) - \log \left(I \left(t _ {i - 1}, h, w\right)\right) \right\| > \tau \tag {1}
$$
is satisfied. This pixel will generate an impulse response called an event, which is expressed as  $\mathbf{e} = (h,w,t_i,p)$ . The polarity  $p\in \{+1, - 1\}$  indicates the direction of the brightness change (increasing or decreasing) and  $\tau$  is the trigger threshold.  $t_{i - 1}$  represents the time when the event was triggered last time at pixel  $(h,w)$ . The imaging plane and the time axis can be combined to establish a three-dimensional spatio-temporal coordinate system, where events characterized by coordinates  $(h,w,t_i)$  collectively form a spatio-temporal point cloud.
In order to compute the pose of the camera w.r.t. a 3D point cloud, E2PNet first encodes the spatio-temporal point clouds into a 2D grid-shaped feature tensor. This process relies on EP2T, which is the core component of E2PNet and will be introduced in detail in Sec. 3.1. After obtaining the encoded feature tensor, we can formulate the event-to-point registration (E2P) problem as the registration of 2D feature maps and 3D point clouds, which can be handled by mature image-based frameworks [1, 11]. Since EP2T is differentiable, the training of E2PNet is simply done by connecting EP2T with the registration network and performing back-propagation in the same way as the original registration network. Empirically we found that directly adding EP2T to the registration network without changing any hyper-parameter works reasonably well, which is another advantage of E2PNet.
3.1 Spatio-Temporal Representation Learning with Event-Points-to-Tensor (EP2T)
As illustrated in Fig. 2, EP2T consists of three consecutive modules: Local Aggregation (LA), Spatio-temporal Separated Attention (STA), and Feature Propagation (FP).
Given a spatio-temporal point cloud derived from the input event data, LA evenly samples a set of feature aggregation centers in the spatial and temporal domains. For each sampled center, LA computes the locally aggregated features using the neighboring input points. To produce the global feature, STA performs attention over these aggregated features, with a special design to handle the spatio-temporal inhomogeneity. To produce features on each input point (from aggregation centers), FP propagates each feature from STA to the neighboring points based on the distance. Finally, we employ classical tensorization methods to process the point-wise features into grid-shaped tensors.
The novelty of LA (see Sec. 3.1.1 for more details) and STA (Sec. 3.1.2) lies in the separation of spatial and temporal domains, which is designed to handle the inhomogeneity of spatio-temporal point clouds. FP (Sec. 3.1.3) propagates aggregated features to neighboring spatio-temporal points so
that standard point-wise tensorization can be used to generate grid-shaped outputs, which enables the use of RGB-based 2D-3D registration frameworks for E2P.
3.1.1 Local Aggregation (LA)
Given an event camera of resolution (H, W), we extract N events from the input and convert them into a spatio-temporal point cloud with N points. Unlike conventional point clouds, the spatial and temporal domains of our inputs naturally have different metric scales, which makes existing point cloud representation networks [39] ineffective for feature encoding (see Sec. 4.2).
To address this problem, the proposed LA module aggregates features across the spatial and temporal domains separately. Specifically, for each input point  $\mathbf{e} = (h,w,t,p)$  with  $p\in \{-1,1\}$  representing the brightness change direction, we first normalize it via  $\hat{\mathbf{e}} = \left(\frac{h}{H},\frac{w}{W},\frac{t}{t_{\mathrm{max}} - t_{\mathrm{min}}},p\right)$ , where  $t_\mathrm{max}$  and  $t_\mathrm{min}$  are the maximum and minimum time stamps of the input point cloud.
The feature aggregation process of LA is inspired by PointNet++ [39]. There are two steps during aggregation: (1) generate a set of aggregation centers; (2) calculate the aggregated feature for each center by employing neighboring points.
PointNet++ [39] generates aggregation centers using Farthest Point Sampling (FPS) [40], which is relatively slow in event-based applications that require a high response speed. Furthermore, FPS is not suitable for spatio-temporal point clouds with non-uniform density distributions (might generate redundant points in dense regions and fewer points in sparse regions). To address this issue, LA uses uniform sampling with a fixed spatial and temporal interval to generate uniformly distributed aggregation centers. Specifically, given the normalized point cloud  $\hat{E}$  spanning over a spatio-temporal cube of size  $(H,W,T)$ , we generate a set of aggregation centers  $C$  by sampling  $M$  ( $30^{*}30^{*}5$  in our experiments) uniform grid locations within this cube.
To aggregate features from neighboring points, PointNet++ [39] first selects  $K$  nearest neighbors from  $\hat{E}$  for each  $\mathbf{c}_j\in C$  and then replaces neighbors whose Euclidean distance is too far away from  $\mathbf{c}_j$  with the point closest to  $\mathbf{c}_j$ . After selecting neighboring points, a multi-layer convolution is performed to produce the aggregated feature. Using Euclidean distance omits the inhomogeneity of the spatial and temporal domains, which exists even with our initial normalization. Hence, we propose to separate the spatial and temporal domains during neighborhood selection. Specifically, we change the distance metric during the selection of nearest neighbors ( $K = 64$  in LA) to  $\alpha d_{\mathrm{Space}}(\hat{\mathbf{e}}_i,\mathbf{c}_j) + \beta d_{\mathrm{Time}}(\hat{\mathbf{e}}_i,\mathbf{c}_j)$ . We also use two distances separately in the spatial and temporal domains to choose points for replacements, i.e., we replace a neighboring point  $\hat{\mathbf{e}}_i$  with the closest point to  $\mathbf{c}_j$  if  $d_{\mathrm{Space}}(\hat{\mathbf{e}}_i,\mathbf{c}_j)\geq \alpha$  or  $d_{\mathrm{Time}}(\hat{\mathbf{e}}_i,\mathbf{c}_j)\geq \beta$ .  $d(\cdot)$  is the mean squared distance of the corresponding domain.
Separating the spatial and temporal domains during feature aggregation addresses the inhomogeneity problem from two aspects. (1) We can provide constraints to the spatial and temporal domains separately to prevent choosing neighboring points with a small distance in one domain but a far distance in another. (2) We can provide different  $\alpha$ 's and  $\beta$ 's to create multiple aggregated features that focuses on different domains. For the second aspect, we produce three sets of features in LA by setting  $\alpha = \{0.8, 0.1, 0.5\}$  and  $\beta = \{0.1, 0.8, 0.5\}$  respectively, which allows the aggregated feature to focus more on spatial, temporal, and spatio-temporal domains.
3.1.2 Spatio-temporal Separated Attention (STA)
After generating the feature centers and the locally aggregated features, we use the STA module to extract global features and merge features from different domains into the uniform spatio-temporal domain. Similar to LA, STA also separates the spatial and temporal domains to effectively handle the inhomogeneity problem.
$\mathbf{F}_t, \mathbf{F}_s,$  and  $\mathbf{F}_{st}$  in Fig. 3 denote the aggregated temporal, spatial and spatio-temporal features from LA respectively. We first create global features for the spatial and temporal domains separately using the temporal and spatial residual kernels. Each of these two kernels performs self-attention over features from the same domain, and cross-attention between features from different domains. Subsequently, we merge them into a temporal and spatial residual using retention gates (sigmoid and tanh functions) inspired by [41, 42]. After producing the residuals for spatial and temporal domains separately, we merge them into the spatio-temporal domain using the spatio-temporal aggregated feature  $\mathbf{F}_{st}$  to produce the global spatio-temporal feature  $f_{\mathrm{STA}}(\mathbf{c}_j)$  for each aggregation center  $\mathbf{c}_j$ .
(a)Single Domain Residual Kernel
(b)Spatio-temporal Residual Kernel
(c)Spatio-temporal Separated Attention Module(STA)
Figure 3: Detailed structure of the proposed STA module. (a) Encode features for the spatial and temporal domains separately using the spatial and temporal residual kernels. (b) Merge both residuals into the spatio-temporal domain. (c) Apply the spatio-temporal residuals to the spatio-temporal features.
3.1.3 Feature Propagation (FP) and Tensorization
After producing the global features at the aggregation centers, we propagate them to neighboring input event points using a weighted average to get a feature for each input point. Specifically, given the global features  $F_{\mathrm{STA}} = \{f_{\mathrm{STA}}(\mathbf{c}_j)\}$  and the normalized event points  $\hat{E} = \{\hat{\mathbf{e}}_i\}$ , we compute the propagated feature by
$$
f _ {\mathrm {F P}} \left(\hat {\mathbf {e}} _ {i}\right) = \frac {\sum_ {j} w _ {i , j} f _ {\mathrm {S T A}} \left(\mathbf {c} _ {j}\right)}{\sum_ {j} w _ {i , j}}, \tag {2}
$$
where  $w_{i,j} = \frac{1}{\max(|\hat{\mathbf{e}}_i - \mathbf{c}_j|_2^2, 10^{-5})}$ .  $10^{-5}$  is used to prevent division by zero.
Finally, to convert point-based features into grid-shaped features, we use three different yet complementary event tensorization methods [15, 20, 17] to convert the set of  $f_{\mathrm{FP}}(\hat{\mathbf{e}}_i)$  into three different types of 2D grid-shaped sparse feature tensors, and concatenate them to produce the final tensorized feature map.
3.2 E2P Dataset Construction
Since there is no dataset for E2P, we use multi-sensor SLAM datasets to construct the ground-truth (GT) labels for training. We select the widely used MVSEC [43] and VECTor [44] to build the MVSEC-E2P and VECTor-E2P datasets, which incorporate LiDAR, traditional cameras and event cameras at the same time. Furthermore, they have an accurate multi-sensor time synchronization and calibration. Through the GT pose provided in these two datasets, we can establish the GT matching relationship between events (images) and point clouds for E2P tasks. A naive way to obtain such GT would be to project point clouds into the image plane, which is slow in practice. Instead, we construct a camera model in the world coordinate system and use this model to construct a quadrangular viewing frustum. Selecting the points in the quadrangular viewing frustum is practically  $>2x$  faster than the naive solution.
In this section, we first demonstrate the effectiveness of the proposed E2PNet for event-to-point cloud registration (Sec. 4.1), which is our primary goal. Then, we conduct ablation studies to verify the effectiveness of individual modules in EP2T (Sec. 4.2). Finally, we analyze the generalization of EP2T on other event-based vision tasks (Sec. 4.3).
4.1 Event to Point Cloud Registration
Datasets and Preprocessing. We use two representative datasets described in Sec. 3.2. MVSEC [43] uses a 16-beam LiDAR and an event camera with a resolution of (346,260); the event camera can simultaneously generate grayscale images. Since only indoor sequences have high-precision pose