SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\text {p o s e}} = 2 - 2 \left(R ^ {\prime T} R _ {g t}\right) ^ {2} + \left| \left| t ^ {\prime} - t _ {g t} \right| \right| _ {2} ^ {2}, \tag {8}
$$
which can also participate on the optimization since the iterative part in GN algorithm is differentiable.
Table 1: Registration accuracy on the KITTI and nuScenes datasets. Lower is better for RTE and RRE, higher is better for Acc.
Method	KITTI	nuScenes
RTE(m)↓	RRE(°)↓	Acc.↑	RTE(m)↓	RRE(°)↓	Acc.↑
Grid Cls. + PnP [26]	3.64 ± 3.46	19.19 ± 28.96	11.22	3.02 ± 2.40	12.66 ± 21.01	2.45
DeepI2P (3D) [26]	4.06 ± 3.54	24.73 ± 31.69	3.77	2.88 ± 2.12	20.65 ± 12.24	2.26
DeepI2P(2D) [26]	3.59 ± 3.21	11.66 ± 18.16	25.95	2.78 ± 1.99	4.80 ± 6.21	38.10
CorrI2P [36]	3.78 ± 65.16	5.89 ± 20.34	72.42	3.04 ± 60.76	3.73 ± 9.03	49.00
Ours	0.75 ± 1.13	3.29 ± 7.99	83.04	0.89 ± 1.44	2.15 ± 7.03	88.33
We set the channel dimension  $C$  of 2D/3D feature to 64, and set  $C_{2D}$  and  $C_{3D}$  for the 2D/3D global feature both to 512. We set the margin  $m$  to 0.25, the scale factor  $\gamma$  to 32 and the safe radius  $r$  to 1 pixel. To enhance the representation ability of the voxel branch, we keep the point transformation pipe used in SPVNAS [43]. And the probability threshold  $\sigma$  in intersection detection is set to 0.95. Although the GN-based PnP solver used in Sec. 3.3 can solve exact pose while remain differentiable, its iterative solution is very time-consuming. For efficient registration, we leverage the EPnP [24] with  $O(n)$  time complexity during inference since we do not focus on the differentiability at inference time. The RANSAC [15] is applied with EPnP for more robust registrations.
We evaluate our performance on Image-to-Point Cloud registration task on two wildly used benchmarks KITTI and nuScenes. On both dataset, the images and point clouds are captured simultaneously with 2D cameras and 3D LiDARs.
KITTI Odometry [17]. We generate the image-point cloud pairs from the same data frame of 2D/3D sensors. We follow previous works [26] to utilize the 0-8 sequences for training, and 9-10 for testing. The mis-registration transformation is conducted with a 2D translation on the ground within  $\pm 10$  and a rotation around the up-axis with no limited range. We downsample the image resolution to  $160\times 512$  and the point cloud size to 40960 for training and testing.
nuScenes [4]. The image-point cloud pairs are generated by official SDK where the point cloud is accumulated from the nearby frames and the image is from the current frame. We follow the official data spilt of nuScenes to utilize 850 scenes for training, and 150 scenes for testing. The mis-registration transformation is conducted in a similar way as the one in KITTI dataset. We downsample the image resolution to  $160 \times 320$  and the point cloud size to 40960, respectively.
We compare our method with the state-of-the-art methods DeepI2P [26] and CorrI2P [36]:
1) Grid Cls. + PnP. The grid classification setting is a baseline approach proposed in DeepI2P to divide the image into  $32 \times 32$  grids and learn to classify each 3D point to a unique 2D grid with a neural network. The EPnP with RANSAC is then applied to predict the rigid transformation.
2) Frus.Cls. + Inv.Proj. DeepI2P proposes to perform the frustum classification with the inverse camera projection to obtain final rigid transformation. We report their results with 2D and 3D inverse camera projection as DeepI2P(2D) and DeepI2P(3D).
3) CorrI2P. CorrI2P is the latest work on Image-to-Point Cloud registration which learns dense correspondences for image-point cloud pairs, and the EPnP with RANSAC is applied to predict the rigid transformation.
Metrics. We follow DeepI2P to evaluate the registration performance with average Relative Translational Error (RTE) and average Relative Rotation Error (RRE). We do not follow CorrI2P to remove the image-point cloud samples with large errors before averaging, which in our opinion, is an unsuit-
9.04 m/178.34°
Input
DeepI2P
$3.29\mathrm{m} / 10.58^{\circ}$
Corr12P
$2.51\mathrm{m} / 7.84^{\circ}$
Ours
0.22 m/0.94°
GT
RTE(m) / RRE (°)
5.30 m/141.30°
$1.78\mathrm{m} / 17.63^{\circ}$
$1.56\mathrm{m} / 4.43^{\circ}$
0.34 m / 0.57°
RTE(m)/RRE  $(^{\circ})$
$3.66\mathrm{m} / 51.20^{\circ}$
Figure 5: Visual comparison of Image-to-Point Cloud registration results under KITTI dataset.
$2.42\mathrm{m} / 20.80^{\circ}$
$2.49\mathrm{m} / 4.33^{\circ}$
0.28 m / 0.68°
able way to report real performances. We further report the registration accuracy (Acc.) which is the proportion of fine registrations with  $\mathrm{RTE} < 2\mathrm{m}$  and  $\mathrm{RRE} < 5^o$ .
4.3 Registration Accuracy
Quantitative Comparison. The results of cross-modality registration are shown in Table 1, in which our proposed method achieves the superior performance over the compared methods on both KITTI and nuScenes dataset. Especially, the latest work CorrI2P [36] adopted a similar way as ours to establish 2D-3D correspondences with feature matching, which is the most relevant method to ours. However, our method is about 4 times better than CorrI2P in terms of RTE. The main reason is that the previous methods fails to conduct end-to-end supervisions and can not learn robust 2D-3D correspondences due to the huge domain gap. In contrast, our proposed method can learn a structured cross-modality latent space for robust 2D-3D feature matching, together with an end-to-end training schema driven by a probabilistic PnP solver. As a result, our method is able to predict highly accuracy 2D-3D correspondences and achieve better performance than its counterparts.
Visual Comparison. The visual comparison is shown in Figure 5. For intuitive visualization, the point cloud is projected into image space with the predicted extrinsic parameters of different approaches and the known camera internal parameters. The color indicates the distance between a point and the camera. For DeepI2P, we adopt the setting with highest accuracy, i.e. DeepI2P(2D) for visual comparison. Compared with the other methods, our method achieves better registration accuracy in different road situations. For example, on the difficult tuning situation (e.g. the  $1^{st}$  and  $2^{nd}$  row), both DeepI2P and CorrI2P fail to solve correct registration, where the projections of trees and cars are largely dis-matched with the corresponding pixels in the image.
4.4 Feature Matching Accuracy
We further provide the feature matching visualization in Figure 6. We visualize the double-side error maps by computing the matching distance on both modalities. Specifically, for 2D-to-3D matching, we search for a point with the greatest similarity in cross-modality latent space for each 2D pixel in the intersection region, and compute the error by first projecting the matching point into image space and compute the Euler distance
Figure 6: Feature matching error visualization.
between the projected matching point and the 2D pixel. And vice versa for 3D-to-2D matching. As shown in Figure 6, our method significantly outperforms CorrI2P [36] in both 2D-to-3D and 3D-to-2D matchings. Most of the pixels in 2D-to-3D matching or points in 3D-to-2D matching of our method can reach slight errors within 2 pixels, which demonstrates that our learned shared latent space can distinct cross-modality patterns apart and lead to an accurate feature matching for each single 2D/3D element. We also observe that some elements on the 2D/3D edges will lead to relative large errors, since it is often difficult for intersection detection to perform perfectly in the edges.
We further compare the efficiency of our method with other counterparts, where all the methods are evaluated with an NVIDIA RTX 3090 GPU and Intel(R) Xeon(R) E5-2699 CPU. The
Table 2: The efficiency comparisons with other methods.
DeepI2P (2D)	DeepI2P (3D)	CorrI2P	Ours
Model size (MB)	100.12	100.12	141.07	30.73
Pose Inference (s)	23.47	35.61	8.96	0.19
results are shown in Table 2, where our method requires much fewer parameters but gets significantly better performances. Furthermore, our method only requires  $0.19s$  for network inference and pose estimation for one frame, which is about  $50\times$  (or more) faster than the previous works. The reason is that DeepI2P solves the time-consuming inverse camera projection problem for pose estimation, even with a 60-fold pose initialization for avoiding crashing, and CorrI2P spends a lot time to eliminate the wrong correspondences through iteration. Therefore, all the previous works fail to be applied to real automatic driving scenarios where the low latency is required, while our method can solve poses much more efficiently. Note that our result is achieved by replacing the GN-based PnP solver with the O(n) EPnP [24] at inference time, while the time will increase to  $2.38s$  with the GN-based PnP inference.
We conduct ablation studies to justify the effectiveness of each design in our method and the effect of some important parameters. We report the performance in terms of RTE/RRE/Acc. under the KITTI dataset.
Framework designs. We first justify the effectiveness of each design of our framework. Specifically, we develop four different variations for comparison: (1)  $w/o$  voxel branch is the variation removing the voxel branch from the triplet network; (2)  $w/o$  point branch is the variation removing
Table 3: The effect of each design in our framework.
RTE(m)↓	RRE(°)↓	Acc.↑
w/o voxel branch	1.25 ± 4.54	7.03 ± 10.19	73.53
w/o point branch	1.08 ± 3.09	6.91 ± 13.26	80.80
w/o A-W optimization	1.04 ± 2.88	4.81 ± 8.96	77.01
w/o Diff. PnP	0.83 ± 1.60	3.44 ± 10.02	82.18
Full	0.75 ± 1.13	3.29 ± 7.99	83.04
the point branch from the triplet network; (3)  $w / o$  A-W optimization is the variation replace our adaptive-weighted optimization loss with the contrastive loss as used in CorrI2P [36]; (4)  $w / o$  Diff.  $PnP$  is the variation removing the end-to-end supervisions driven by the differentiable PnP solver.
The results are shown in Table 3, from which we can find that our Full model achieves the best performances over all variations. Such results prove the effectiveness of each design in our framework. Moreover, by comparing w/o voxel branch and w/o point branch to the Full model, we can find that the voxel branch plays a more important role in our framework, which demonstrates that the voxel modality with similar characteristics (represented in grids) and pattern extractors (CNNs) as pixels is more suitable for learning Image-to-Point Cloud registration.
The effect of input resolutions. We further explore the effect of input image resolutions and point densities. Table 4 shows the performance of different settings, from which we can find that higher resolutions on both modalities lead to better results since the low resolution images will omit some visual information while the low den
Table 4: Ablations on image resolutions and point densities.
Image	Point	RTE(m)↓	RRE(°)↓	Acc.↑
40 × 128	40960	0.92 ± 1.52	3.54 ± 12.08	82.50
80 × 256	40960	0.84 ± 1.07	3.65 ± 8.83	82.67
160 × 512	40960	0.75 ± 1.13	3.29 ± 7.99	83.04
160 × 512	20480	0.93 ± 1.33	3.59 ± 8.62	82.75
160 × 512	61440	0.72 ± 1.17	2.82 ± 5.37	83.70
sity point clouds will lose the detailed geometries. We select the proper setting with a balance between the performance and efficiency.
In this work, we propose a novel framework to learn Image-to-Point Cloud registration with VoxelPoint-to-Pixel Matching, where we learn a structured cross-modality latent space with a novel adaptive-weighted loss. We represent the 3D elements as the combination of voxels and points to overcome the domain gap between points and pixels. Moreover, we train our framework end-to-end
by imposing supervision directly on the predicted pose distributions with a differentiable PnP solver. The extensive experiments on KITTI and nuScenes datasets demonstrate our superior performances.
This work was supported by National Key R&D Program of China (2022YFC3800600), the National Natural Science Foundation of China (62272263, 62072268), and in part by Tsinghua-Kuaishou Institute of Future Media Data.