SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junsheng Zhou $^{1*}$  Baorui Ma $^{1,2*}$  Wenyuan Zhang $^{1}$  Yi Fang $^{3}$  Yu-Shen Liu $^{1\dagger}$  Zhizhong Han $^{4}$
School of Software, Tsinghua University, Beijing, China<sup>1</sup>  
Beijing Academy of Artificial Intelligence<sup>2</sup> New York University Abu Dhabi, Abu Dhabi, UAE<sup>3</sup>  
Department of Computer Science, Wayne State University, Detroit, USA<sup>4</sup>  
zhoujs21@mails.tsinghua.edu.cn brma@baai.ac.cn zhangwen21@mails.tsinghua.edu.cn 3mmyfang@nyu.edu liuyushen@tsinghua.edu.cn h312h@wayne.edu
Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at https://github.com/junshengzhou/VP2P-Match.
Image-to-Point Cloud registration finds the rigid transformation (e.g. translation and rotation) that aligns the projection of a LiDAR frame represented as a point cloud to the reference image captured by cameras. The key here is to determine the extrinsic parameters of the camera with respect to the reference frame of LiDAR. It plays an important role in autonomous driving, 3D computer vision, augmented/virtual reality, etc. Despite a plethora of approaches have explored the same-modality registration tasks (e.g. Image-to-Image [50, 13] and Point Cloud-to-Point Cloud registration [25, 1]) and achieve promising results, few researches have shown convincing performances on cross-modality registration between images and point clouds due to its inherent difficulty.
*Equal contribution.† The corresponding author is Yu-Shen Liu.
Previous methods use MLPs [34, 35] on point clouds and CNNs [19, 23] on images separately with contrastive losses to learn distinctive features, and try to establish 2D-3D correspondences by matching the learned features. However, they fail to learn a structured latent space shared by 2D and 3D data due the calculation differences in MLPs or CNNs, which leads to different feature domains.
To solve these issues, we propose a novel framework to learn a structured cross-modality latent space for robust 2D-3D feature matching, where 2D elements are represented as pixels and the 3D elements are represented as the combination of voxels and points. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching for cross-modality registration, where both the voxel and pixel branch are designed based on CNNs to operate spatially-local convolutions on voxels/pixels represented in grids, and the point branch is integrated to regain the information lost during voxelization.
To learn a distinctive latent space where the correct 2D-3D correspondences can be guaranteed with cross-modality feature matching, we propose a novel loss with adaptive-weighted optimization which allows to learn distinctive 2D-3D correspondences by optimizing positive and negative matches in a robust self-paced manner. Since the LiDAR point cloud is captured around the car with a  $360^{\circ}$  perception, and the image is captured eyes forward, there is a large range of outliers on both modalities (e.g. points/voxels behind the car and sky pixels), between which there is no explicit correspondences. To handle the outliers, we design a detection strategy to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences.
Furthermore, previous methods [14, 36, 26] adopt Perspective-n-Point (PnP) [24, 16] solver as a post-processing to estimate poses from matching results. They merely use the pseudo supervision conducted from 2D-3D correspondences as the optimizing target during training. The insufficient supervision leads to large errors since the network has no ability to handle incorrect matching pairs which have a highly negative effect on the results. Inspired by EPro-PnP [8], we propose to train our framework in an end-to-end manner. With a differentiable probabilistic PnP solver, we can impose supervision directly on the predicted pose by minimizing the Kullback-Leibler (KL) divergence between the predicted and target pose distribution. Our main contributions can be summarized as:
- We propose a novel framework to learn Image-to-Point Cloud registration by learning a structured cross-modality latent space with adaptive-weighted optimization, through an end-to-end training schema with a differentiable PnP solver.
- We propose to represent the 3D elements as the combination of voxels and points to overcome the pattern gap between points and pixels, where a triplet network is designed to learn VoxelPoint-to-Pixel matching.
- We demonstrate our superior performance over the state-of-the-arts by conducting extensive experiments on KITTI and nuScenes datasets.
2.1 Same-Modality Registration
Image Registration. Image-to-Image Registration is the key of structure-from-motion (SfM) [41] and SLAM [32]. Classic methods [39, 40] extract feature descriptors from image pairs with SIFT [28] or ORB [38], and establish correspondences based on the descriptor matching, where the Perspective-n-Point (PnP) [24] or Bundle Adjustment algorithm [45] can be further applied to estimate rotations and translations from the correspondences. Recently, learning-based approaches [50, 33, 13] have shown promising results on image registration. LIFT [50] replaces the key steps of SIFT to neural network layers.
Point Cloud registration. Traditional methods [2, 9] on point cloud registration requires a proper initialization to estimate the rigid transformation. With the rapid development of deep learning, the neural networks have shown great potential in 3D applications [34, 55, 54, 56, 48, 27, 31, 22, 53]. Recently, learning-based approaches like 3DMatch [51], PerfectMatch [18], PPFNet [12], USIP [25] have advanced a lot on point cloud registration. 3DFeat-Net [49] attempts to joint description and detection on 3D keypoints. However, both the image and point cloud registration methods depend
Figure 1: Overview of our method. Given a pair of mis-registered image  $I$  and point cloud  $P$  as input, (a) we first operate sparse voxelization to generate the sparse voxel  $V$ , and the triplet network is then applied to extract patterns from the three modalities. We represent the 2D patterns as the pixel features and the 3D patterns as the combination of voxel and point features, respectively. The adaptive-weighted loss is then used to learn distinctive 2D-3D cross-modality patterns. (b) We detect the intersection regions on 2D/3D space with cross-modality feature fusion. (c) We remove the outlier regions based on the results of intersection detection and build 2D-3D correspondences with 2D-3D feature matching, where the probabilistic PnP is then applied to predict the distribution of the extra poses to impose end-to-end supervisions with ground truth poses.
on the matching of feature descriptors designed for a specific modal, which fail dramatically on cross-modality registration.
2.2 Cross-Modality Registration
Visual Localization. A well-searched task on cross-modality registration is visual localization, which aims to estimate the 6DOF camera pose of a query image in a 3D scene model presented in point clouds. Traditional methods [44, 47] build 3D maps using SfM [41] to store 3D visual descriptors. To determine the pose of a query image, those methods match visual descriptors obtained from the query image with the ones stored in the point cloud, and apply PnP solver [24] to estimate the camera pose from 2D-3D matches. Recently, Go-Match [57] proposes a learning-based schema to solve localization with only geometry information. Some other methods [3, 7, 5] further explore the differentiability of PnP solver. But those visual localization methods only focus on locating query images on a pre-built environment, and fail to generalize to dynamic scenes captured in real-time.
Another series of methods [6, 20, 30] explore the task of LiDAR and camera self-calibration. LccNet [29] projects mis-calibrated LiDAR point clouds onto depth images, together with the images captured by cameras to regress the relative rigid transformation. However, those methods can only handle a very small mis-calibration range since a relatively large range will lead to failures on depth projection. Moreover, the self-calibration task assumes that the relative transformation between camera and LiDAR is constant and is only predicted once for each sequence, while our method focuses on the registration for each frame of the sequence which is much more difficult.
Image-to-Point Cloud Registration. 2D3D-MatchNet [14] detects keypoints from images and point clouds using SIFT and ISS, and feed the local patches around those 2D/3D keypoints to CNNs and PointNet. The networks are trained with triplet loss to learn cross-modality correspondences. However, the hand-crafted keypoint detectors for different modalities don't guarantee the correct matching of keypoints, which leads to a poor registration accuracy. DeepI2P [26] converts the registration problem into a classification and inverse camera projection optimization problem, but the performance is limited since the frustum classification only indicates coarse 2D-3D correspondence. Recently, CorrI2P [36] proposes to learn dense 2D-3D correspondences by matching point-wise and pixel-wise features for fine registration. However, it fails to learn a distinctive latent space for 2D and 3D data, which leads to a large ratio of incorrect correspondences. Furthermore, all the above methods can not predict the rigid transformation directly, which require a post processing (e.g. Perspective-n-Point) to estimate rigid transformation from the 2D-3D correspondences.
Problem statement. Given a pair of 2D image  $I \in \mathbb{R}^{H \times W \times 3}$  and 3D point cloud  $P \in \mathbb{R}^{N \times 3}$ , the cross-modality Image-to-Point Cloud registration is to predict the rigid transformation  $\mathcal{T} = [R|t]$  in 3D space that can be used to align the projection of point cloud  $P$  to image  $I$ , where  $t \in \mathbb{R}^3$  is the translation vector and  $R \in SO(3)$  is the rotation matrix.
In this section, we first introduce the detailed framework of our proposed VoxelPoint-to-Pixel Matching for learning a structured cross-modality latent space. Next, a novel loss with adaptive-weighted optimization is proposed for learning distinctive cross-modality patterns. Finally, we present the differentiable probabilistic PnP solver which drives our end-to-end learning schema. The overview of our framework is shown in Figure 1.
3.1 VoxelPoint-to-Pixel Matching
The first step of our framework is to obtain element-wise 2D and 3D features. We represent 2D elements as pixels and 3D elements as the combination of voxels and points. To achieve this, we design a triplet network consisting of Voxel/Point/Pixel branches as shown in Figure 1.(a).
Triplet Network. To design a voxel branch to obtain voxel-wise features  $f_{\text{voxel}}$ , a naive implementation is to generate dense voxels from  $P$  and operate volumetric convolutions [58]. However, suffering from the cubic complexity of voxels, the resolution for voxel data is usually limited (e.g.  $128^3$ ), which will lead to a large information loss, especially for large scale LiDAR data. To address this issue, we leverage sparse convolution [10, 43] on high-resolution sparse voxels where the empty voxels are skipped.
Although the voxel branch can represent powerful 3D spatial patterns with convolutions, it still suffers from the information loss during voxelization. Therefore, we integrate the point branch to regain the lost 3D detailed patterns by learning point features. The point branch is designed based on PointNet++ [35]. It operates hierarchical set abstraction with self-attention modules [52] to get global feature  $g_{3D} \in \mathbb{R}^{C_{3D} \times 1}$ , and the feature propagation module [35] is further applied to learn point-wise feature  $f_{point} \in \mathbb{R}^{N \times C}$ . The pixel branch is a convolutional U-Net [37], where the ResNet [19] is used as the basic layers to get global 2D image feature  $g_{2D} \in \mathbb{R}^{C_{2D} \times 1}$ , and the upsampling module with skip-connection is further applied to get pixel-wise feature  $f_{pixel} \in \mathbb{R}^{H \times W \times C}$ .
2D-3D Feature Matching. We represent 3D elements as the combination of voxels and points. To achieve this, we first transform the voxel feature to the point-wise voxel representation  $f_{voxel}^{\prime} \in \mathbb{R}^{N \times C}$  by interpolating each point-location with its 8 neighbor voxel grids using trilinear interpolation. Thus, the 3D features  $f_{3D}$  can be donated as:
$$
f _ {3 D} = f _ {\text {p o i n t}} + f _ {\text {v o x e l}} ^ {\prime}, \in \mathbb {R} ^ {N \times C}, \tag {1}
$$
and the 2D features  $f_{2D} \in T \times C$  can be achieved by reshaping pixel-wise features  $f_{pixel}$  where  $T = H \times W$ . After being processed with the embedding heads, the 2D and 3D features can be mapped into the same latent space, where we establish correspondences according to cosine similarities between  $f_{2D}$  and  $f_{3D}$ .
Comparison to Point-to-Pixel Matching. Due to the different characteristics between 3D point cloud and 2D images, it is extremely difficult to operate the same feature extractors on both modals. Existing approaches [14, 36] learn point-to-pixel matching with MLP-based point network and CNN-based image network, which leads to different feature domains.
We find that although the huge domain gap between points and pixels, while the voxels and pixels share great similarities. Therefore, we introduce the voxel branch with spare convolution to capture spatially-local patterns as 2D convo
lution operated on pixel branch. Since both the 2D and 3D features are processed by convolutions,
Figure 2: The t-SNE visualization of learned latent space with Point-to-Pixel (P2P) and VoxelPoint-to-Pixel (VP2P) Matching.
they share similar characteristics in feature space, leading to a structured shared latent space for both 2D and 3D features.
We visualize the learned latent space of Point-to-Pixel Matching and our proposed VoxelPoint-to-Pixel Matching with t-SNE [46] to convert features to 2D space through cosine similarities in Figure 2. The latent space learned by P2P Matching is extremely irregular with large gaps between several clusters and some space only contains the features of a single modality. In contrast, VP2P Matching leads to a structured cross-modality latent space where the features are evenly distributed throughout the space. See appendix for more analysis.
Intersection Detection. Since the images and LiDAR point clouds are captured in quite different ways, there is a large range of outliers on both modalities where no correspondences can be founded. We define the intersection region as the overlap between the 2D projection of a LiDAR point cloud using ground truth camera parameters and the reference image. To handle the outlier regions, we design a detection strategy as shown in Figure 1(b) to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences.
For detecting the outlier region in 3D space, we first repeat the 2D global feature  $g_{2D}$  and concatenate it with the 3D element-wise feature  $f_{3D}$  as  $[g_{2D}:f_{3D}]$ , where “:” donates the feature concatenation. Then several MLPs  $\phi$  are used to learn the probability  $d_{3D}^{i}$  of a point  $p^i$  to lie in the 2D-3D intersection region as:  $d_{3D}^{i} = \phi (f_{3D}:g_{2D}), i\in [1,N]$ , vice versa for detecting 2D space outliers. We define a threshold  $\sigma$  where 2D/3D elements with probabilities smaller than  $\sigma$  will be considered as outliers.
To learn the detection, we sample  $Z_{in}$  pairs of pixels and points with probability  $d$  from the intersection region and  $Z_{out}$  pairs of pixels and points with probability  $\hat{d}$  from the outlier regions. The detection loss is then formulated as:
$$
\mathcal {L} _ {\text {d e t}} = \frac {1}{Z _ {\text {o u t}}} \sum_ {i = 0} ^ {Z _ {\text {o u t}}} \left(\hat {d} _ {2 D} ^ {i} + \hat {d} _ {3 D} ^ {i}\right) - \frac {1}{Z _ {\text {i n}}} \sum_ {j = 0} ^ {Z _ {\text {i n}}} \left(d _ {2 D} ^ {j} + d _ {3 D} ^ {j}\right) \tag {2}
$$
3.2 Adaptive-Weighted Optimization
To explore distinctive patterns, various optimization strategies like contrastive loss and triplet loss are widely used in 2D or 3D tasks. However, these formulations treat each pair of samples equally, which leads to ambiguous convergence, especially in the difficult situation of 2D-3D feature matching. Inspired by the Circle Loss [42], we design a flexible optimization strategy with adaptive weighting for learning a distinctive cross-modality latent space where we can establish 2D-3D correspondences more accurately.
Given a set of 2D-3D pairs  $K = \{\alpha^i,\beta^i\}$ $i\in [1,v]$ , which are sampled from the intersection region. We separate them into positive pairs  $k_{p}$  and negative pairs  $k_{n}$  determined by whether the distance between the location of  $\alpha^i$  and the projection of  $\beta^i$  on the image is larger than a radius  $r$  or not. We define the positive similarity for a positive pair  $k_{p}^{i}$  as:
$$
s _ {p} ^ {i} = f _ {2 D} ^ {i} \cdot f _ {3 D} ^ {i} = \sum_ {c} f _ {2 D} ^ {i c} f _ {3 D} ^ {i c}, \tag {3}
$$
where  $c$  is the channel number of features. And vice versa for negative pairs  $s_n$ .
To avoid ambiguous convergence and allow flexible optimization, we design adaptive weighting
factors for positive and negative pairs as  $\rho_{p} = \varPsi(\gamma(1 - s_{p}^{j} + m))$  and  $\rho_{n} = \varPsi(\gamma(s_{n}^{k} - m))$ , in which  $\gamma$  is a scale factor,  $m$  is a margin for better similarity separation and  $\varPsi$  is the function to detach  $\rho_{p}$  and  $\rho_{n}$  from gradients to serve as weights for optimization. The adaptive-weighted loss is then derived as:
$$
\mathcal {L} _ {A W} = \log [ 1 + \sum_ {j} \exp^ {\rho_ {p} (1 - s _ {p} ^ {j} + m)} \sum_ {k} \exp^ {\rho_ {n} \left(s _ {n} ^ {k} - m\right)} ]. \tag {4}
$$
Figure 3: The illustration of Adaptive-weighted optimization.
We illustrate the adaptive-weighted optimization in Figure 3. The 3D space is achieved by projecting points into image space. Given a 3D point  $\beta$ , we define two negative pairs  $\{\alpha_n^1, \beta\}$  and  $\{\alpha_n^2, \beta\}$  where  $\alpha_n^1$  and  $\alpha_n^2$  lie outside the safe radius  $r$ . It is obviously that the negative pair  $\{\alpha_n^2, \beta\}$  is much harder to learn since  $\alpha_n^2$  lies near the positive region and some samples around  $\alpha_n^2$  is from positive pairs, while  $\{\alpha_n^1, \beta\}$  is easier since all the samples around  $\alpha_n^1$  is also negative. And vice versa for the positive pairs. Optimizing all the samples with the same weight will lead to ambiguous convergence at the regions containing both negative and positive samples, i.e., it will be difficult to distinct the negative sample  $\alpha_n^2$  with positive samples. Driven by this analysis, we optimize the patterns with adaptive weighting to force the network focus more on the harder samples (indicated by the color shading above).
(a) Contrastive Loss
(b) Adaptive-Weighted Loss
Figure 4: The cosine similarity visualization with different losses. We select one point  $p$  at the rear of the car from the point cloud and compute the cosine similarity between  $p$  and each pixel in the reference image. The colors from blue to red indicate the increase of similarities.
We visually compare the feature matching results of our proposed adaptive-weighted optimization with the contrastive loss as used in CorrI2P [36] in Figure 4, where our proposed loss shows significantly superior performance in learning a distinctive cross-modality latent space.
3.3 Differentiable PnP Solver
Correspondences Establishing. To establish correspondences, we first remove the outlier regions on both modalities with intersection detection and leverage the nearest neighbor principle on the cross-modality latent space for 2D-3D feature matching. Due to the different density of points and pixels, they do not have completely one-to-one correspondences. To avoid ambiguous matching, we establish the 2D-3D correspondence  $X^i$  by searching for the point coordinate  $P^z$  with the largest similarity in cross-modality latent space for each 2D pixel coordinate  $I^i$ ,  $i \in [1, T_{in}]$ ,  $T_{in}$  is the number of pixels which lie in the intersection region. The operation achieved by arg max to find  $P^z$  is inherently non-differentiable, to train our framework end-to-end, we leverage Straight-Through Gumbel Estimator [21] to estimate the gradient for backward optimization.
Probabilistic PnP. Given a set of correspondences  $X = \{I^i, P^i\}$ ,  $i \in [1, T_{in}]$ , the PnP problem [24] is to search for an optimal pose  $\mathcal{T} = [R|t]$  that minimizes the reprojection error:
$$
\underset {\mathcal {T}} {\arg \max } \frac {1}{2} | | \underbrace {\pi (R P ^ {i} + t) - I ^ {i}} _ {w _ {i} (\mathcal {T})} | | ^ {2}, \tag {5}
$$
where  $\pi$  is the projection function and  $w^{i}(\mathcal{T})$  is the reprojection error at  $X^i$ . Inspired by Epro-PnP [8], we solve the non-differentiable PnP problem by interpreting the output as a probabilistic distribution:
$$
p (\mathcal {T} | X) = \frac {\exp - \frac {1}{2} \sum_ {i = 1} ^ {T _ {i n}} \left\| w ^ {i} (\mathcal {T}) \right\| ^ {2}}{\int \exp - \frac {1}{2} \sum_ {i = 1} ^ {T _ {i n}} \left\| w ^ {i} (\mathcal {T}) \right\| ^ {2} \mathrm {d} \mathcal {T}}. \tag {6}
$$
The KL divergence loss is then computed to minimize the distance between the predicted pose distribution and ground truth pose distribution:
$$
\mathcal {L} _ {K L} = \frac {1}{2} \sum_ {i = 1} ^ {T _ {i n}} \left\| w ^ {i} \left(\mathcal {T} _ {g t}\right) \right\| ^ {2} + \log \int \exp - \frac {1}{2} \sum_ {i = 1} ^ {T _ {i n}} \left\| w ^ {i} (\mathcal {T}) \right\| ^ {2} d \mathcal {T}, \tag {7}
$$
where the first term is the the reprojection error at ground truth pose, and the second term is the integral of reprojection error at predicted pose distribution. In practice, we leverage Monto Carlo strategy to collect samples to approximate the integration with Adaptive Multiple Importance Sampling algorithm [11].
Except for conducting supervisions on the probabilistic distribution, we further estimate the exact pose  $\mathcal{T}' = [R'|t']$  by solving Eq. (5) with an iterative PnP solver based on Gauss-Newton (GN) algorithm, and compute the pose loss as: