Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching
Authors: Junsheng Zhou
Baorui Ma
Wenyuan Zhang
Yi Fang
Yu-Shen Liu
Zhizhong Han
Venue: NeurIPS 2023 spotlight (2023)
Keywords: LiDAR Point Clouds
2D images
Cross-modality registration
Matching

Abstract: Cross-modality registration between 2D images captured by cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods.

Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching
Junsheng Zhou!* 1+ Yu-Shen Liu" Zhizhong Han
Baorui Ma Ma 1.2 Wenyuan Zhang'
Yi Fang
School of Software, Tsinghua University, Beijing, China'
Beijing Academy of Artificial Intelligence" New York University Abu Dhabi, Abu Dhabi, UAE
Department of Computer Science, Wayne State University, Detroit, USA!
aeinghua.edd..o zhoujs2 s21 @mails tsinghua. edu. cn brma@baai ac. en zhangwen21@mails ttt.ei00e0aalsss.snttta.ee edu. cn 3mmy angenyu. edu Liuyushen@tsinghua. cn h312h@wayne. edu
Abstract
Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n -nrictssvee--oontss (PnP) to estimate rigid transforma- tion during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very differ- ent characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxclPoint-to-Piig matching, where represent 3D elements using both voxels and points learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented grids, and integrate an additional point branch to regain the information lost dur- ing voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinetive patterns cross-modality features, we design novel loss with adaptive- weighhed optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improve- ments over the state-of-the thee----tthrtttt art methods. The code and models are available at bttps::/githae com/ hwt.///jnnnggghoo///P--......
Introduction
Image-to- Point Cloud registration finds the rigid transformation (c.g. translation and rotation) that aligns the projection of LiDAR frame represented point cloud to the reference image captured by cameras. The key here to determine the extrinsic parameters the camera with respect to the reference frame of LiDAR. It plays important role in autonomous driving, 3D computer vision, augmented/virtual reality, etc. Despite plethora of approaches have explored the same-modality registration tasks (e.g. Image-to-Image [50, 13] and Point Cloud-to- fo-Point Cloud registration [25, 1]) and achieve promising results, few researches have shown convincing performances on cross-modality registration between images and point clouds due to its inherent difficulty.
"qqual contribution. 'hhe corresponding author is Yu-Shen Liu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Previous methods use MLPs [34, 35] on point clouds and CNNs [19, 23] on images separately with contrastive losses to learn distinctive features, and try establish 2D-3D correspondences by matching the learned features. However, they fail to learn structured latent space shared by 2D and 3D data due the calculation differences in MI.s or CNNs, which leads to different feature domains.
To solve these issues, we propose a novel framework to learn a structured cross-modality latent space for robust 2D-3D feature matching, where 2D elements are represented as pixels and the 3D elements are represented as the combination of voxels and points, Specifically, we design triplet network to learn VoxelPoint-too to-oiett--Pixel matching for cross-modality registration, where both the voxel and pixel branch are designed based on CNNs to operate spatially- convolutions on voxels/pixels represented in grids, and the point branch integrated regain the information lost during voxelization.
To learn distinctive latent space where the correct 2D-3D correspondences can be guaranteed with cross-modality feature matching, we propose a novel loss with adaptive-weighted optimization which allows learn distinctive 2D-3D correspondences by optimizing positive and negative matches in a robust self-paced manner. Since the LiDAR point cloud is captured around the car with 360 perception, and the image is captured eyes forward, there large range of outliers on both modalities (e.g. points/voxels behind the car and sky pixels), between which there is no explicit correspondences. To handle the outliers, we design a detection strategy to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences.
Furthermore, previous methods [14, 36, 26] adopt Perspective-n-Point (PnP) [24, 16] solver as a post-processing to estimate poses from matching results. They merely the pseudo supervision conducted from 2D-3D correspondences as the optimizing target during training. The insufficient supervision leads to large errors since the network has no ability handle incorrect matching pairs which have a highly negative effect on the results. Inspired by EPro-PnP [8]. we propose to train our framework in an end-to-end manner. With differentiable probabilistic PnP solver, we can impose supervision directly on the predicted pose by minimizing the Kullback-Leible (KL) divergence between the predicted and target pose distribution. Our main contributions can be summarized as:
We propose novel framework to learn Image-to-Point Cloud registration by learning structured cross- coss-moddllty latent space with adaptive-weigg optimization, through an end-to-end training schema with differentiable PnP solver.
We propose to represent the 3D elements as the combination of voxels and points to overcome the pattern gap between points and pixels, where triplet network is designed to learn VoxelPoint-to-Pixe matching.
We demonstrate our superior performance over the state-of-the-are by conducting extensive experiments on KITTI and nuScenes datasets.
2 Related Work
2.1 Same-Modality Registration
Image Registration.
Image-to-lmage Registration is the key of frommmrr-frommmotiion (SfM) [41]
and SLAM [32]. Classic methods [39, 40] extract feature descriptors from image pairs with SIFT [28] or ORB [38], and establish correspondences based on the descriptor matching, where the Perspective- n-Point (PnP) [24] or Bundle Adjustment algorithm [45] can be further applied to estimate rotations
and translations from the correspondences. Recently, learning- based approaches [50, 33, 13] have shown promising results on image registration. LIFT [50] replaces the key steps of SIFT to neural network layers.
Point Cloud registration. Traditional methods [2, 9] on point cloud registration requires proper initialization to estimate the rigid transformation. With the rapid development of deep learning, the neural networks have shown great potential in 3D applications [34, 55, 54, 56, 48, 27, 31, 22, 53]. Recently, learning-based approaches like 3DMatch [51], PerfectMatch [18], PPFNet [12], USIP [25] have advanced a lot on point cloud registration. 3DFeat-Net [49] attempts to joint description and detection on 3D keypoints. However, both the image and point cloud registration methods depend

---

Crozs- Caeo-Moddlllyy Learning
b
Internaction
CoRP
Coor
det
KLP
Map
Come
Coros
6 Subve Puse End-ko End
doo
Hodiue
dist
MLP
MLP
das
Curres
2.
Law Aitiiooooogo
Uec Detectio
Figure I: Overview of our method. Given a pair of mis-registered image / and point cloud P as input, (a) we first operate sparse voxelization generate the sparse voxel V, and the triplet network is then applied to extract patterns from the three modalities. We represent the 2D patterns as the pixel features and the 3D patterns as the combination of voxel and point features, respectively. The adaptive-weighted loss is then used to learn distinctive 2D-3D cross-modality patterns. (b) We detect the intersection regions on 2D/3D space with eross cross-modality feature fusion. (c) We remove the outlier regions based on the results of intersection detection and build 2D-3D correspondences with 2D-3D feature matching, where the probabilistic PnP then applied to predict the distribution of the extra poses to impose end-to-end supervisions with ground truth poses.
on the matching of feature descriptors designed for specific modal, which fail dramatically on cross-modality registration.
2.2 Cross-Modality Registration
Visual Localization. A well-searched task on cross-modality registration visual localization, which aims to estimate the 6DOF camera pose of a query image in a 3D scene model presented in point clouds. Traditional methods [44, 47] build 3D maps using SfM [41] to store 3D visual descriptors. To determine the pose of query image, those methods match visual descriptors obtained from the query image with the ones stored in the point cloud, and apply PnP solver [24] estimate the camera pose from 2D-3D matches. Recently, Go-Match [57] proposes learning-baa -basd schema to solve localization with only geometry information. Some other methods [3, 5J further explore the differentiability of PnP solver. But those visual localization methods only focus on locating query images on a pre-built environment, and fail to generalize to dynamic scenes captured real-time.
Another series of methods [6, 20, 30] explore the task of LiDAR and camera self-calibration. LeeNet [29] projects mis-calibrated LiDAR point clouds onto depth images, together with the images captured by cameras to regress the relative rigid transformation. However, those methods can only handle a very small caii-caibrrtion range since a relatively large range will lead failures on depth projection. Moreover, the self-calibb calibratiio task assumes that the relative transformation between camera and LiDAR is constant and is only predicted once for each sequence, while our method focuses on the registration for each frame of the sequence which is much more difficult.
Metch-ttt [14] detects keypoints from images and point clouds using SIFT and ISS, and feed the local patches around those 2D/3D keypoints to CNNs and PointNet. The networks are trained with triplet loss learn cross-modality correspondences. However, the hand-crafted keypoint detectors for different modalities don't 't guarantee the correct matching of keypoints, which leads a poor registration accuracy. Deep12P [26] converts the registration problem into classification and inverse camera projection optimization problem, but the performance limited since the frustum classification only indicates coarse 2D-3D correspondence. Recently, Corr12 [36] proposes to learn dense 2D-3D correspondences by matching wisnt and pixel-wise features for fine registration. However, it fails to learn distinctive latent space for 2D and 3D data, which leads to large ratio of incorrect correspondences. Furthermore, all the above methods can not predict the rigid transformation directly, which require a post processing (e.g. Perspective-n (-Pint))) to estimate rigid transformation from the 2D-3D correspondences.
Image-to-Point Cloud Registration.
3

---

3 Method
Problem statement. Given a pair of 2D image IR x and 3D point cloud P E R RNxx3, the cross-modality Image-to- to-gett Cloud registration is to predict the rigid transformation T [R]t] in 3D space that can be used to align the projection of point cloud to image I, whhere E IR the translation vector and R (O(3) is the rotation matrix.
In this section, we first introduce the detailed framework of our proposed VoxelPoint-to- Pixel Matching for learning a structured cross-modality latent space. Next, a novel loss with adaptive- weighted optimization is proposed for learning distinctive cross- modsllty patterns. Finally, we present the differentiable probabilistic PnP solver which drives our end-to-end learning schema. The overview of our framework is shown in Figure 1.
3.1 VoxelPoint-to-Pixel Matching
The first step of our framework is to obtain element-wise 2D and 3D features. We represent 2D elements pixels and 3D elements as the combination of voxels and points. To achieve this, we design a triplet network consisting of Voxel/Point/Pixel branches as shown Figure I.(a).
Triplet Network. To design a voxel branch to obtain voxel-wise features feorel. a naive implementa- tion is to generate dense voxels from P and operate volumetric convolutions [58]. However, suffering from the cubic complexity of voxels, the resolution for voxel data is usually limited (e.g. 128"), which will lead to large information loss, especially for large scale LiDAR data. address this issue, we leverage sparse convolution [10, 43] on high-resolution sparse voxels where the empty voxels are skipped.
Although the voxel branch can represent powerful 3D spatial patterns with convolutions, it still suffers from the information loss during voxelization. Therefore, integrate the point branch to regain the lost 3D detailed patterns by learning point features. The point branch is designed based on +ointNet++ [35]. It operates hierarchical set abstraction with self-attention modules [52] to get global feature 9&D E IR CaD x1 and the feature propagation module [35] is further applied to learn point-wise feature fpoine E NxC The pixel branch is convolutional U-Net [37], where the ResNet [19] is used the basic layers get global 2D image feature 92D RC and the upsampling module with skip-connection is further applied to get pixel-wise feature fpize E R XW xC
2D-3D Feature Matching. We represent 3D elements as the combination of voxels and points. To achieve this, we first transform the voxel feature the point-wise voxel representation frozet R by interpolating each point-location with its 8 neighbor voxel grids using trilinear interpolation. Thus, the 3D features faD can be donated as:
faD = Spoint + fronet: R.NCC
and the 2D features f2D E T x C' can be achieved by reshaping pixel-wise features fpizel where T' H x W. After being processed with the embedding heads, the 2D and 3D features can be mapped into the same latent space, where we establish correspondences according to cosine similarities between f2D and faD.
Comparison Point-to-Pixel Matching. Due to the different characteristics between 3D point cloud and 2D images, is extremely difficult to operate the same feature extractors on both modals. Existing approaches [14, 36] learn point-to-pixel matching with MLP-based point network and CNN-based image network, which leads to different feature domains.
69 P.PMacciing
da Maachin
We find that although the huge domain gap be- tween points and pixels, while the voxels and pixels share great similarities. Therefore, we in- troduce the voxel branch with spare convolution to capture spatially-Iocal patterns as 2D convo- lution operated pixel branch. Since both the 2D and 3D features are processed by convolutions,
3E Festure
Figure The t-SNE visualization of learned latent space with Point-to-Pixel (P2P) and VoxelPoint-to- Pixel (VP2P) Matching.

---

they share similar characteristics in feature space, leading to structured shared latent space for both 2D and 3D features.
We visualize the learned latent space of Point-to-Piie Pixel Matching and our proposed VoxelPoint-to-PPiee Matching with t-SNE [46] to convert features 2D space through cosine similarities in Figure 2. The latent space learned by P2P Matching is extremely irregular with large gaps between several clusters and some space only contains the features of single modality. In contrast, VP2P Matching leads to a structured cross-modality latent space where the features are evenly distributed throughout the space. See appendix for more analysis.
Intersection Detection. Since the images and LiDAR point clouds are captured in quite different ways, there is large range of outliers on both modalities where correspondences can be founded. We define the intersection region as the overlap between the 2D projection of a LiDAR point cloud using ground truth camera parameters and the reference image. To handle the outlier regions, design a detection strategy as shown in Figure ((b) to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences.
For detecting the outlier region in 3D space, we first repeat the 2D global feature 92D and concatenate with the 3D element-wing feature fsD as [92D fapl, where "." donates the feature concatenation. Then several MLPs o are used to learn the probability d&D of a point p' to lie in the 2D-3D intersection region as: d0 @aaD 92D), i E [1, N]. vice versa for detecting 2D space outliers. We define threshold a where 2D/3D elements with probabilities smaller than o will be considered as outliers.
To learn the detection, we sample Zin pairs of pixels and points with probability d from the intersection region and Zoue pairs of pixels and points with probability d from the outlier regions. The detection loss then formulated as:
Zout
Zin Σ
Cdet
(dio + dgD)
1
(dio d2D)
(2)
Zout
Zin j=0
3.2 Adaptive-Weise Optimization
To explore distinctive patterns, various optimization strategies like contrastive loss and triplet loss are widely used 2D or 3D tasks. However, these formulations treat each pair of samples equally, which leads to ambiguous convergence, especially in the difficult situation of 2D-3D feature matching. Inspired by the Circle Loss [42], we design flexible optimization strategy with adaptive weighting for learning distinctive cross-modality latent space where we can establish 2D-3D correspondences more accurately.
Given set of 2D-3D pairs K = 8" E [1, v). which are sampled from the intersection region. We separate them into positive pairs kip and negative pairs kn determined by whether the distance between the location of a and the pro- jection of 3 on the image larger than a radius not. We define the positive similarity for a positive pair k., as:
3D-to22 pars
pairs
2D space
B
ogaan
5 = f2n fāD = ESa
pic fanján
where c is the channel number of features. And vice versa for negative pairs sn.
Figure 3: The illustration of Adaptive-weighted optimization.
To avoid ambiguous convergence and allow flex- ible optimization, we design adaptive weighting
factors for positive and negative pairs Pr = (1 - + mm) and p., ( - mm). ), in which
y is scale factor, m a margin for better similarity separation and v is the function to detach Ap and On. from gradients to serve as weights for optimization. The adaptive- loss is then derived as: ex" (m]-
L log[1 Σ exp=
5

---

We illustrate the Shuptive-weighted optimization in Figure 3. The 3D space is achieved by projecting points into image space. Given 3D point B, we define two negative pairs {a 88 and {a B} where az and oz lies outside the safe radius r. It is obviously that the negative pair {a 9 is much harder to learn since af lies near the positive region and some samples around a is from positive pairs, while {a [8} is easier since all the samples around o, is also negative. And vice versa for the positive pairs. Optimizing all the samples with the same weight will lead to ambiguous convergence at the regions containing both negative and positive samples, i.e., it will be difficult to distinct the negative sample az, with positive samples. Driven by this analysis, we optimize the patterns with adaptive weighting to force the network focus more the harder samples (indicated by the color shading above).
(a) Contrastive Loss
b adapttve-Welghter Loss
We visually compare the fea- ture matching results of our pro- posed adaptive-weighter opti- mization with the contrastive loss used in Corr12P [36] in Figure 4, where our proposed loss shows significantly supe- rior performance in learning a distinctive cross-modality latent space.
Figure 4: The cosine similarity visualization with different losses. We select one point p at the rear of the car from the point cloud and compute the cosine similarity between and each pixel in the reference image. The colors from blue to red indicate the increase of similarities.
3.3
Differentiable PnP Solver
Correspondences Establishing, To establish correspondences, we first remove the outlier regions on both modalities with intersection detection and leverage the nearest neighbor principle on the cross-modality latent space for 2D-3D feature matching. Due to the different density of points and pixels, they do not have completely one-to-one correspondences. To avoid ambiguous matching, we establish the 2D-3D correspondence X by searching for the point coordinate p with the largest similarity in cross- moos--y latent space for each 2D pixel coordinate I', E [1, [i]] Ton is the number of pixels which lie in the intersection region. The operation achieved by arg max to find P inherently non-differentiable, to train our framework end-to-end, we leverage Straight-Throuu Gumbel Estimator [21] to estimate the gradient for backward optimization.
Probabilistic PnP. Given set of correspondences X P*), E [i.]. the PnP problem [24] is to search for an optimal pose T= - -[R][] that minimizes the reprojection error:
arg max 2 π (RP +) - 12
(5)
where is the projection function and w (7) is the reprojection error at X Inspired by Epro-PnP [8]. we solve the non-differentiable PnP problem by interpreting the output as a probabilistic distribution:
exp 2/2 Emn
Tin I|w (T)I2
p (T]))=
f exp 1/2 Ee 1/ (T)
The KL divergence loss is then computed to minimize the distance between the predicted pose distribution and ground truth pose distribution:
Tin
Tin exp ΣΣ | (T))I
LKL
Ilww (Txi)I + log
where the first term is the the reprojection error at ground truth pose, and the second term is the integral of reprojection error at predicted pose distribution. In practice, we leverage Monto Carlo strategy to collect samples to approximate the integration with Adaptive Multiple Importance Sampling algorithm [1]].
Except for conducting supervisions on the probabilistic distribution, we further estimate the exact pose T [['((') by solving Eq. (5) with an iterative PnP solver based on Gauss-Newton (GN) algorithm, and compute the pose loss as:
fpose = 2 2( R RRTTRge
2² + |||' - 1ge 12,
(8)
which can also participate on the optimization since the iterative part in GN algorithm is differentiable.

---

Table 1: Registration accuracy on the KITTI and nuScenes datasets. Lower is better for RTE and RRE, higher is better for Ace.
KITTI
nuScenes
Method
RTE(m)
Acc.1
RTE(m),
RRE(") )J
Acc.
Grid Cls. + PnP [26]
3.64 + 3.46
19.19 + 28.96
11.22
3.02 = 2.40
12.66 + 21.01
2.45
Deep12P (3D) [26]
4.06 + 3.54
24.73 = 31.69
3.77
2.88 = 2.12
20.65 + 12.24
2.26
Deep12P(2D) [26]
3.59 3.21
11.66 = 18.16
25.95
2.78 = 1.99
4.80 + 6.21
38,10
Cor112 [36]
3.78 65.16
5.89 + 20.34
72.42
3.04
60.76
3.73 9.03
49.00
Ours
0.75 + 1.13
3.29 + 7.99
83.04
0.89
1.44
2.15 + 7.03
88.33
3,4 Implementation Details
We set the channel dimension of 2D/3D feature to 64, and set C2D and CAD for the 2D/3D global feature both to 512. We set the margin to 0.25, the scale factor 7 to 32 and the safe radius r to 1 pixel. To enhance the representation ability the voxel branch, we keep the point transformation pipe used in SPVNAS [43]. And the probability threshold in intersection detection is set to 0.95. Although the GN-based PnP solver used in Sec. 3.3 can solve exact pose while remain differentiable, its iterative solution is very time-consuming. For efficient registration,
leverage the EPnP [24] with O(n) time complexity during inference since we do not focus on the differentiability at inference time. The RANSAČ [15] is applied with EPnP for more robust registrations.
Experiments
4.1
Dataset
We evaluate our performance on Image-to-Point Cloud registration task on two wildly used bench- marks KITTI and nuScenes. On both dataset, the images and point clouds are captured simultaneously with 2D cameras and 3D LiDARs.
[17]. We generate the image-point cloud pairs from the same data frame of 2D/3D sensors. We follow previous works [26] to utilize the 0-8 sequences for training, and 9-10 for testing. The mis-registration transformation is conducted with 2D translation the ground within +10 and a rotation around the up-axis with no limited range. We downsample the image resolution to 160x512 and the point cloud size 40960 for training and testing.
KITTI Odometry
nuScenes [4]. The image-point cloud pairs are generated by official SDK where the point cloud accumulated from the nearby frames and the image is from the current frame. We follow the official data spilt of nuScenes to utilize 850 scenes for training, and 150 scenes for testing. The mis risirriiittation transformation is conducted in similar way the one in KITTI dataset. We downsample the image resolution to 160x320 and the point cloud size to 40960, respectively.
4.2 Baselines and Metrics
We compare our method with the state-of of-e---ftheeeee methods Deep12P [26] and Corr12P [36]:
1) Grid Cls. + PnP. The grid classification setting is a baseline approach proposed in Deep12P to divide the image into 32x32 grids and learn to classify each 3D) point unique 2D grid with a neural network. The EPnP with RANSAC is then applied to predict the rigid transformation. 2) Frus.Cls. + Inv.Proj. DeepI2P proposes to perform the frustum classification with the inverse camera projection to obtain final rigid transformation. We report their results with 2D and 3D inverse camera projection DeepI2P(2D) and DeeD12P(3D).
3) CorrI2P. Corr12P is the latest work on Image-to-Pon -Pint Cloud registration which learns dense correspondences for image-point cloud pairs, and the EPaP with RANSAC is applied to predict the rigid transformation.
Metrics. We follow Deepl2P to evaluate the registration performance with average Relative Transla- tional Error (RTE) and average Relative Rotation Error (RRE). We do not follow Corr12P to remove the image-point cloud samples with large errors before averaging, which in our opinion, is unsuit-

---

Input
Deep12P
Com122
Ours
GT
178.34
3.29 10.58
2.51 7.84
m 0..4
RTT ARE (*)
5.30 m/ 140..0
1.78 m/ 17.63
1.56 m 4.43
0.34 m/ 00577
RTB(m) / RRE (*)
3.66 51.20 2
2.42 20.80
2449 4.33
0.28 m 0.68
RTEim/ RRE (*)
Figure 5: Visual comparison of maae--to-Point Cloud registration results under KITTI dataset.
able way to report real performances. We further report the registration accuracy (Acc.) which is the proportion of fine registrations with RTE < 2m and RRE < 5%.
4.3 Registration Accuracy
Quantitative Comparison. The results of cross-modality registration are shown Table which our proposed method achieves the superior performance over the compared methods on both KITTI and nuScenes dataset. Especially, the latest work Corr12P [36] adopted a similar way as ours to establish 2D-3D correspondences with feature matching, which is the most relevant method to ours. However, our method is about times better than Cor112P in terms of RTE. The main reason is that the previous methods fails to conduct end-to-end supervisions and can not learn robust 2D-3D correspondences due to the huge domain gap. In contrast, our proposed method can learn a structured cross-modality latent space for robust 2D-3D feature matching, together with end-to-end training schema driven by a probabilistic PnP solver. As a result, our method is able to predict highly accuracy 2D-3D correspondences and achieve better performance than its counterparts.
Visual Comparison. The visual comparison shown in Figure 5. For intuitive visualization, the point cloud is projected into image space with the predicted extrinsic parameters of different approaches and the known camera internal parameters. The color indicates the distance between a point and the camera. For DeepI2P, we adopt the setting with highest accuracy, i.e. Deep12P(2D) for visual comparison. Compared with the other methods, our method achieves better registration accuracy in different road situations. For example, on the difficult tuning situation (e.g. the 1s and 2nd row), both Deepl2P and Corr12 fail to solve correct registration, where the projections of trees and cars are largely dis dis-matched with the corresponding pixels in the image.
4.4 Feature Matching Accuracy
We further provide the feature matching visual- ization in Figure 6. We visualize the double-side error maps by computing the matching distance on both modalities. Specifically, for 2D-to-31) matching, search for a point with the greatest similarity in cross-modality latent space for each 2D pixel in the intersection region, and compute the error by first projecting the matching point into image space and compute the Euler distance between the projected matching point and the 2D pixel. And vice versa for 3D-to-2D matching. As shown in Figure 6, our method significantly outperforms Corr12P [36] in both 2D-to-3D and 3D-to-2D matchings. Most of the pixels in 2D-to-3D matching or points in 3D-to-2D matching of our method can reach slight errors within pixels, which demonstrates that our learned shared latent space can distinct cross-modality patterns apart and lead to an accurate feature matching for each single 2D/3D element. We also observe that some elements on the 2D/3D edges will lead to relative large errors, since it is often difficult for intersection detection to perform perfectly in the edges.
Figure 6: Feature matching error visualization.
8

---

4.5 Efficiency Comparison
We further compare the efficiency of our method with other counterparts, where all the methods are evaluated
Table 2: The efficiency comparisons with other methods.
Deep12P (2D)
Deepl2P (3D)
Cor12P
Ours
Model size (MB)
100.12
100.12
141.07
30.73
with an NVIDIA RTX 3090 GPU and Intel(R) Xe)n(R) E5-2699 CPU. The results are shown in Table 2, where our method requires much fewer parameters but gets significantly better performances. Furthermore, our method only requires 0.199 for network inference and pose estimation for one frame, which is about 50x (or more) faster than the previous works. The reason is that Deepl2P solves the consummnnn inverse camera projection problem for pose estimation, even with a 60-fold pose initialization for avoiding crashing, and Corrl2P spends lot time to eliminate the wrong correspondences through iteration. Therefore, all the previous works fail to be applied to real automatic driving scenarios where the low latency is required, while our method can solve poses much more efficiently. Note that our result is achieved by replacing the GN-based PnP solver with the O(n) EPnP [24] at inference time, while the time will increase to 2.38s with the GN-ba based PnP inference.
Pose Inference
23.47
35.61
8.96
0.19
4.6 Ablation Study
We conduct ablation studies to justify the effectiveness of each design in our method and the effect of some important parameters. We report the performance in terms of RTE/RRE/Acc. under the KITTI dataset.
Framework designs. We first justify the effectiveness of each design of our frame- work. Specifically, we develop four dif- ferent variations for comparison: (1) w/o voxel branch is the variation removing the voxel branch from the triplet network; (2)
Table 3: The effect of each design in our framework.
RTE(m))
RRE(")
Acc.t
w/o voxel branch w/o point branch
1,25 + 4.54
7.03 + 10.19
73.53
1.08 1 3.09
6.91 1 13.26
80.80
w/o A-W optimization
1.04 = 2.88
4.81 + 8.96
77.01
Diff. PnP
0.83 1.60
3.44
10.02
82.18
w/o point branch is the variation removing Full
0.75 1.13
3.29 7.99
83.04
the point branch from the triplet network; (3) w/o A-W optimization is the variation replace our
adaptive- Seetttve-weigghedd optimization loss with the contrastive loss used in Corr12P [36]; (4) w/o Diff: PnP is the variation removing the end-to-end supervisions driven by the differentiable PnP solver.
The results are shown in Table 3, from which we can find that our Full model achieves the best performances over all variations. Such results prove the effectiveness of each design our framework. Moreover, by comparing w/o voxel branch and w/o point branch to the Full model, we can find that the voxel branch plays more important role in our framework, which demonstrates that the voxel modality with similar characteristics (represented in grids) and pattern extractors (CNNs) as pixels is more suitable for learning Image-to-Pel to-Ponnt Cloud registration.
The effect of input resolutions. We fur-
Table 4: Ablations on image resolutions and point den- sities.
ther explore the effect of input image res- olutions and point densities. Table 4 shows
Image
Point
RTF(m)
RRF(")
Acc.
the performance different settings. from which we can find that higher resolutions
40 128
40960
0,92 1: 1.52
3.54 1: 12.08
82.50
80 x 256
40960
0.84 + 1.07
3.65 = 8.83
82.67
on both modalities lead to better results
160 x 512
40960
0.75 + 1.13
3.29 + 7.99
83.04
since the low resolution images will omit
160 x 512
20480
0.93 - 1.33
3.59 = 8.62
82.75
some visual information while the low den-
160 x 512
61440
0.72 = 1.17
2.82 = 5.37
83.70
sity point clouds will lose the detailed geometries. We select the proper setting with a balance between the performance and efficiency.
5 Conclusions
In this work, we propose novel framework to learn Image-to-Point Cloud registration with MoxelPoint-to-Pixe Matching, where we learn structured cross-modality latent space with novel adaptive-weige weighted loss. We represent the 3D elements as the combination of voxels and points to overcome the domain gap between points and pixels. Moreover, train our framework end-to-end

---

by imposing supervision directly on the predicted pose distributions with a differentiable PnP solver. The extensive experiments on KITTI and nuScenes datasets demonstrate our superior performances.
Acknowledgement
This work supported by National Key R&D Program of China (2022YFC3800600; the National Natural Science Foundation of China (62272263, 62072268), and in part by Tsinghua- Institute of Future Media Data.
Lsinghua-Kuaishou Kuaishou
References
[1] Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEF/CVF
Conference on Computer Vision and Pattern Recognition, pages 6359-63677 2020.
[2] Paul J Besl and Neil McKay. Method for registration of 3-d shapes. In censorfusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[3] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dbaacdiifferentaable ransac for camera localization. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6684-6692, 2017.
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishman, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11621-11633 2020.
[5] Dylan Campbell, Liu Liu, and Stephen Gould. Solving the blind perspective-n-point problem end-to-end with robust differentiable geometric optimization. In European Conference on Computer Vision, page preprint. Springer, Aug 2020.
[6] Daniele Cattaneo, Matteo Vaghi, Augusto Luis Ballardini, Simone Fontana, Domenico G Sorrenti, and Wolfram Burgard. Cmrnet: Camera to lidar-map registration. In 2019 IEEE
intelligent Iransportation systems conference (ITSC), pages 1283-1289. IEEE, 2019. [7] Bo Chen, Alvaro Parra, Jiewei Cao, Nan Li, and Tat-Jun Chin. End-to-end learnable geometric vision by backpropagating pop optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8100-8109, 2020.
[8) Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end probabilistic perspective-n-point for monocular object pose estimation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 2781-2790. 2022.
[9] Yang Chen and Gérard Medioni. Object modelling by registration of multiple range images. Image and Vision Computing, 1455):455155. 1992.
[10] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[11] JEAN-MARIE CORNUET, JEAN-MICHEL MARIN, Antonietta Mira, and Christian P Robert. Adaptive multiple importance sampling. Scandinavian Journal of Statistics, 39(4):798-8 -12, 2012.
[12] Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 195-205, 2018.
[13] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable cnn for joint description and detection local features. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8092-8101, 2019.
[14] Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud. In 2019 International Conference on Robotics and Automation (ICRA), pages 4790-4796. IEEE, 2019.
[15] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381--8) 1981.
10

---

[16] Xiao- Stao-Shan Gao, Xiao-Rong Hou, Jianliang Tang, and Hang-Fei Cheng. Complete solution problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8):930 -9300-43, 2003.
classification for the perspective-three
[17] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1 1):1111 2013.
[18] Zan Gojcic. Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 5545-5554, 2019.
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[20] Ganesh Iyer, R Karnik Ram, J Krishna Murthy, and K Madhava Krishna. Calibnet: Geometri- cally supervised extrinsic calibration using 3d spatial transformer networks. In 2018 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 1110-1117 IEEE,
2018.
[21] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrizationn with sumble-softmax. In International Conference on Learning Representations (ICLR 2017). OpenReview. net, 2017.
[22] Chuan Jin, Tieru Wu, and Junsheng Zhou. Multi- grid representation with field regularization
for self-supervised surface reconstruction from point clouds. Computers & Graphics, 2023.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, (0(6):84-90) 2017.
[24] Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Epnp: An accurate o (n) solution to the pup problem. International journal of computer vision, 81 81(2):155- 166, 2009.
[25] Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 361-370. 2019.
[26] Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-point cloud registration via deep classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19990915969... 2021.
[27] Shujuan Li. Junsheng Zhou, Baorui Ma, Yu-Shen Liu, and Zhizhong Han. NeAF: Learning neural angle fields for point normal estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[28] David G Lowe. Object recogmition from local invaraan features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, volume 2, pages 1150-1157. Ieee. 1999.
[29] Xudong Lv, Boya Wang. Ziwen Dou, Dong Ye, and Shuo Wang. Leenet: Lidar and camera self-calibration using cost volume network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2894-2901, 2021. [30] Xudong Lv, Shuo Wang. and Dong Ye. Cinet: Lidar-camera registration using calibration flow network. Sensors, 21(23):8112 2021.
[31] Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient con- sistency for neural signed distance functions via level set aligmment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17724-17734 2023, [32] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5): 144--1161.., 2015.
[33] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. Advances in Neural Information Processing Systems, 31, 2018
[34] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[35] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical
feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 30, 2017
[36] Siyu Ren. Yiming Zeng, Junhui Hou, and Xiaodong Chen. Corri2p: Deep image-to-point cloud registration via dense correspondence. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
[37] Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for
biomedical image segmentation. In International Conference on Medical image computing and computer-assiste assisted intervention, pages 234-241. Springer, 2015.
[38] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative
to sift or surf. In Proceedings of the IEEE/CVE International Conference on Computer Vision,
11

---

pages 2564-2571. Icee. 2011.
[39]
Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Improving image-based localization by active correspondence search. In European Conference on Computer Vision, pages 752-765. Springer, 2012.
[40] Torsten Sattler, Will Maddemn, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg. Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8601-8610. 2018.
[41]
Johannes L Schonberger and Jan-Michacl Frahm. Structure- foo---froommmmee motion revisited. In Pro- ceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages
1104-4113, 2016.
[42] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang. and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6398 -407,, 2020.
[43] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang. and Song Han. Searching efficient 3d architectures with sparse point-voxed convolution. In European Confer- ence on Computer Vision, pages 685-702. Springer, 2020.
[44] Carl Toft, Will Maddem, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, Tomas Pajdla, et al. Long-term visual localization revisited. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2020. [45] Bill Triggs, Philip F McLauchlan, Richard Hartley, and Andrew W Fitzgibbon. Bundle adjustment modern synthesis. In International Workshop on Vision Algorithmss, pages
298-372. Springer. 1999.
[46] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine
Learning Research, 9(11) 1), 2008
[47] Qianqian Wang. Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. In European Conference on Computer Vision, pages
757-774. Springer, 2020.
[48] Xin Wen, Junsheng Zhou, Yu-Shen Liu, Hua Su, Zhen Dong, and Zhizhong Han. 3D shape
reconstruction from 2D images with disentangled attribute flow. In Proceedings of the IEEF/CV
Conference on Computer Vision and Pattern Recognition, pages 3803-381, 13, 2022.
[49] Zi Jian Yew and Gim Hee Lee. 3dfcat-net: Weakly supervised local 3d features for point cloud registration. In European Conference on Computer Vision, pages 607-623, 2018
[50] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In European Conference on Computer Vision, pages 467-483. Springer, 2016.
[51] Andy Zeng. Shuran Song. Matthias Nießnnrr Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1802 1811, 2017.
[52] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16259- 16268. 2021.
[53] Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEE/CVF international conference on computer vision, 2023.
[54] Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Learning consistency- aware unsigned distance functions progressively from raw point clouds. In Advances Neural Information Processing Systems (NeurIPS). 2022.
[55] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.0677 2023. [56] Junsheng Zhou, Xin Wen, Baorui Ma. Yu-Shen Liu, Yue Gao, Yi Fang, and Zhizhong Han. 3d-oae: Occlusion auto-encoders for self-supervised learning on point clouds. arXiv preprint UrXiv:2203.1400 2022.
[57] Qunjie Zhou, Sergio Agostinho, Aljosa Osep, and Laura Leal-Taixe. Is geometry enough for matching in visual localization? In European Conference on Computer Vision, 2022. [58] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4490-4499, 2018.
12


Reviews:
[{"review_id": "ei3fD4itYa", "reviewer": "Reviewer_CqXY", "summary": "The paper proposes to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver, which designs a triplet network to learn VoxelPoint-to-Pixel matching. The proposed method is trained in end-to-end manner by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. The experiments seem good.", "questions": "Please see weaknesses.", "limitations": "Limitations and failure cases should be discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1). A framework to learn Image-to-Point Cloud registration by learning a\nstructured cross-modality latent space with adaptive-weighted optimization, together with\nan end-to-end training schema driven by a differentiable PnP solver.\n\n(2) The paper represents the 3D elements as the combination of voxels and points to\novercome the pattern gap between points and pixels, where a triplet network is designed to learn VoxelPoint-to-Pixel matching.", "weaknesses": "\n1. The motivation that use voxel information is unclear, as described in the paper, one of the bottleneck of previous methods is points and pixels are with different characteristics with patterns learned in different manners (MLP and CNN), however, domain gap is also exited between voxeled information and pixels, why does the paper use voxel information should be discussed.\n\n2. The ablation studies are not enough, \n(1). in 3.2 Adaptive-Weighted Optimization, a hyper-parameter radius r is used, which is should be discussed, because it affects the positive and negative pairs, experiments should be provided to verify the robustness.\n\n(2). In probabilistic PnP, the effectiveness of L_{pose} and L_{kl} should be verified separately.\n\n(3). I agree that the settings in CorrI2P is unsuitable that exclude RTE larger than 5m and RRE larger than 10◦, however, it is better to provide these results same with CorrI2P, since it would be better for readers to understand where does main improvement come from.\n\n(4). Limitatations and failure cases should be discussed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Motivation of introducing voxel branch.**\n\nOur motivation is that irregular points are merely suitable to be processed by MLP to learn representations, while pixels are regular and processed by CNNs. The large differences between points and pixels and the one between calculations in MLPs and CNNs lead to different features domains and make it hard for previous works to learn a structured shared latent space for 2D and 3D data. We observe that voxels share much greater similarities with pixels than points since both voxels and pixels are regular and represented in grids, which are suitable to be processed by CNNs to operate spatially-local convolutions. We analyzed the effectiveness of introducing voxel information in “Comparison to Point-to-Pixel Matching” of Sec.3.1 and “The Analysis of VP2P Matching” of Sec.3.1 in the supplementary. We visualized the learned latent space of our VoxelPoint-to-Pixel Matching and quantitatively and qualitatively compared with Point-to-Pixel Matching to demonstrate that using regular voxels can reduce the domain gap between 2D and 3D data.\n\n\n**Q2: Ablations on hyper-parameter radius $r$.**\n\nWe have provided the ablation studies on the safe radius $r$ in Sec.3.3 in the supplementary. We set the safe radius $r$ to 0.5, 1, 2 and 4 pixels, and report the performances in Table 2 of the supplementary.\n\n\n**Q3: The effectiveness of $L_{pose}$ and $L_{KL}$.**\n\nWe conducted the ablations to verify $ L_{KL}$ and $ L_{pose}$ separately in Table B in the rebuttal PDF. We observe that when removing the probabilistic PnP loss $ L_{KL}$ in Eq. (7) or the pose loss $L_{pose}$ in Eq. (8) separately, the RTE (lower is better) rise from 0.65 to 0.72 / 0.68. And when further removing both of them, the RTE further degenerates to 0.75. These results demonstrate that the probabilistic PnP loss in Eq. (7) brings major improvement to the registration accuracy and the pose loss in Eq. (8) provides additional enhancements.\n\n\n**Q4: Comprehensive comparison with baseline methods.**\n\nWe provide the comprehensive comparison with baseline methods (e.g. CorrI2P and DeepI2P) under the same setting of CorrI2P to exclude RTE larger than 5m and RRE larger than 10$^o$ in Table A of the rebuttal PDF. \nTo further provide a more real and convincing comparison with SoTA method CorrI2P in the performance without extreme cases, we remove the same number of bad samples as CorrI2P and report the performance of our method as “Ours *”. By evaluating on the same number of test samples without extreme cases, we believe it is a relative fair comparison with CorrI2P. As shown in Table A of the rebuttal PDF, we achieve the best performance under all metrics. Especially, our method is about 3 times better than the SoTA baseline CorrI2P under the difficult benchmark nuScenes, even after removing the extreme samples. The result demonstrates that our method not only produces more stable registrations with much fewer failure cases but also produces much precise registrations for the success samples.\n\n\n**Q5: Limitations and failure cases.**\n\nOne of our limitations is that the feature matching errors at some noisy points of LiDAR point clouds may be very large, which have very negative influence on cross-modality registrations. As shown in Figure 3 of the supplementary, although the feature matchings at most of pixels/points are accurate, some feature matching results at noisy points (e.g. scans of bushes) of the scene are not stable. The reason is that the network is limited to handle noisy points without any special designs, leading to unstable 3D features at noisy points and further affect the registration accuracy at some complex scenes. We will add more discussion on the limitations and failure cases of our method in the revision.\n"}, {"review_id": "67F9PA8J4V", "reviewer": "Reviewer_QyPU", "summary": "This paper proposes a method to register an image with its nearby Lidar scan, and it comes with three modules:\n\n1) A sparse 3D conv-net and point-net to extract 3D features; A 2D conv-net to extract 2D features;\n\n2)  An intersection detection module to discard non-matchable 2D and 3D points;\n\n3) Modified Circle, Probabilistic PnP, and pose losses are used to train the proposed triple network.", "questions": "Please refer to the weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) This paper is well-written and easy to follow;\n\n2) This paper makes good practice to combine multiple existing modules to address the image2Lidar registration problem;\n\n3) According to Table 1, the proposed method outperforms previous works addressing the image2Lidar registration problem.\n", "weaknesses": "Though this paper successfully combines multiple existing modules, some design choices are yet to be investigated. Specifically,\n\n1) Generalization ability. \n\nAccording to the paper, the proposed method focuses on autonomous driving datasets and conducts experiments on 3-DoF registration problem (Line 267). I have a concern that the network overfits on this specific configuration. To address my concern, please conduct the following experiments:\n\na) Cross-validation. Using a network trained on the KITTI dataset to test on the nuScenes dataset;\n\nb) Conducting 6-DoF registration, rather than 3-DoF registration. Please at least add rotations around the x-axis and y-axis;\n\n\n2) Pipelines.\n\na) The effectiveness of the modified Circle Loss. Please add the comparison with respect to the original Circle Loss in Table 3;\n\nb) Since metric-learning plays an important role in this paper, I would expect authors to conduct comparisons with off-the-shelf metric-learning losses other than the Circle Loss. Please refer to https://github.com/KevinMusgrave/pytorch-metric-learning;\n\nc) The effectiveness of the Probabilistic PnP and pose loss. I would expect a two-stage training strategy here by first training the network by only using the adaptive-weighted loss, and then adding the Probabilistic PnP and pose loss. This reminds me of the work [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization]. Please separate the impact of KL divergence loss and pose loss, and check their effectiveness independently in Table 3. It would be nice to further compare the proposed Probabilistic PnP with respect to the differentiable PnP module in [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization].\n\n\n3) Minor.\n\nI think processing 2D and 3D data with convolutions may not justify the claim of \"Sharing similar characteristics in feature space\" (Line 162), as 3D convolution is invariant to shift, viewpoint, distance, etc. In contrast, 2D convolution is vulnerable to viewpoint and scale differences.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Cross dataset validation.**\n\nWe refer the reviewer to ”Global-Q1: Cross dataset validation.“ of the global response for justifying cross dataset validations.\n\n**Q2: Conducting 6-DoF registration, rather than 3-DoF registration.**\n\nWe follow previous methods (e.g. CorrI2P and DeepI2P) to conduct 3-DoF registration for a fair comparison. However, we justify that our proposed method is designed to match cross-modality features for image-to-point cloud registration and is not limited to the certain 3-DoF problem. We conduct 6-DoF registration under KITTI dataset to demonstrate the ability of our method in handling more difficult situations. Specifically, we conduct 6-DoF mis-registration transformation with 3D translations on x-axis, y-axis, and z-axis within $\\pm$ 5 m, and rotations around the x-axis, y-axis and z-axis within $\\pm$ 120$^o$. We report our performance and make a comparison with previous SoTA method CorrI2P in Table D in the rebuttal PDF. The results demonstrate that our method can also work in 6-DoF image-to-point cloud registration task with large mis-registrations where we achieve accurate registration performance of 0.96/3.73/76.7 in terms of RTE/RRE/Acc., while previous SoTA method CorrI2P fails at the more difficult task and produces much worse results (4.29/12.28/42.53) than ours. \n\n**Q3: Comparison with original Circle loss.**\n\nWe provide the comparison with original Circle loss in “Original CircleLoss” in Table E in the rebuttal PDF, where the performance drops from 0.65/2.10/91.14 to 0.96/2.94/88.10 in terms of RTE/RTE/Acc without our designs on the adaptive weighted optimization.\n\n**Q4: Comparisons with off-the-shelf metric-learning losses.**\n\nWe make a comparison with some off-the-shelf metric-learning losses in Table E in the rebuttal PDF. The implementation of these losses are from the GitHub repo as you suggested. The results demonstrate the effectiveness of our proposed adaptive-weighted optimization where we achieve the best performance among all the other losses (e.g. ContrastiveLoss, LiftedStructureLoss, GeneralizedLSLoss). The reason is that other metric-learning losses treat each pair of samples equally and are unable to distinguish hard and easy pairs, which leads to ambiguous convergence, especially in the difficult cross-modality matching. While we design a flexible optimization strategy with adaptive weighting to force the network to focus more on the harder samples, leading to a distinctive cross-modality latent space where we can establish 2D-3D correspondences more accurately. We further justify that our adaptive-weighted strategy is a general term that can be leveraged in improving other metric-learning losses, where we integrate the strategy to GeneralizedLSLoss and report the performance as “GeneralizedLSLoss-AW”. \n\n\n**Q5: The effectiveness of the Probabilistic PnP and pose loss.**\n\nWe provide the ablation studies on Probabilistic PnP and pose loss in Table F of the rebuttal PDF, following your suggestions. We first report the performance of a two-stage training strategy as “TwoStage (w/ Ours PnP)” by first training the network only using the adaptive-weighted loss for half epochs and then adding the Probabilistic PnP losses for the rest epochs. The two-stage strategy outperforms the result of using only adaptive-weighted optimization as “w/o Diff. PnP” in Table F (0.69 vs. 0.75) and is slightly worse than the result of our one-stage training with Probabilistic PnP shown as “w/ Ours PnP” in Table F (0.69 vs. 0.65). We further provide the ablation studies to separately explore the impact of KL divergence loss and pose loss as shown in Table B of the rebuttal PDF, where both losses can improve the performance.\n\n**Q6: Compare Probabilistic PnP with BlindPnP.**\n\nWe provide the ablations of replacing our Probabilistic PnP with BlindPnP [1] in Table F of the rebuttal PDF, where BlindPnP performs worse than our Probabilistic PnP (0.73 vs. 0.65). By introducing supervisions on the predicted pose distribution, the Probabilistic PnP brings more robust guidance for optimizations than BlindPnP which only leverages the L2 loss to guide the pose learning.\n\n[1] Campbell D,  et.al. Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization. ECCV2020\n\n**Q7: Can processing 2D and 3D data with convolutions reduce the domain gap?**\n\nOur motivation is that irregular points are merely suitable to be processed by MLP to learn representations, while pixels are regular and processed by CNNs. The large differences between points and pixels and the one between calculations in MLPs and CNNs lead to different features domains and make it hard for previous works to learn a structured shared latent space for 2D and 3D data.\nWe observe that voxels share much greater similarities with pixels than points since both voxels and pixels are regular and represented in grids, which are suitable to be processed by CNNs. We agree that 2D and 3D CNNs also have some differences, but they share similar operation to perform convolutions on regular data (pixels or voxels) represented in grids to explore spatially-local patterns, which is quite different to the feature patterns obtained by performing MLPs in irregular point sets. We analyzed the effectiveness of introducing voxel information in “Comparison to Point-to-Pixel Matching” of Sec.3.1 and “The Analysis of VP2P Matching” of Sec.3.1 in the supplementary. We visualized the learned latent space of our VoxelPoint-to-Pixel Matching and quantitatively and qualitatively compared with Point-to-Pixel Matching to demonstrate that using regular voxels can reduce the domain gap between 2D and 3D data.\n"}, {"review_id": "ubaHMBbAYl", "reviewer": "Reviewer_yFFc", "summary": "The paper proposes a 2D to 3D registration pipeline using a differentiable PnP method (Epro-PnP) and integrates 3D information using both voxel and point-based features. These designs target on previous problems such as the domain differences when fusing MLP-based point features and CNN-based pixel features, and the non-differentiable post-processing when computing the transformation that cannot train the entire model end-to-end. Experiments on KITTI and nuScenes, along with ablation studies show the effect of the proposed pipeline while achieving great efficiency. ", "questions": "Please address my questions/concerns above. Specifically, my main concerns are some unclear/inconsistent arguments and experimental designs.", "limitations": "The limitation of the method is not discussed. Some of the limitations that can be discussed are as follows: \n\n1. The method uses a non-differentiable fast PnP solver during the inference, which is different from the optimization process. How will it affect the performance? And will any fast PnP solver be suitable here? What is the limitation during inference?\n\n2. What is the limitation of the method when applied to different datasets/scenes?\n\n3. How to choose model parameters in order to balance accuracy and efficiency? The authors have included some ablation studies in both the main paper and the supplementary materials. However, the tradeoff between accuracy and efficiency could be further discussed.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method improves the current state-of-the-art methods by a large margin in terms of the accuracy and efficiency on KITTI and nuScenes datasets.\n\n- Although most techniques used in the paper are not new, combining these techniques yield effective results.\n\n- The idea of combining both the voxel-based features and the point cloud-based features helps with the domain difference between MLP-based and CNN-based features.\n\n- The writing of the paper is okay and is easy to follow.\n", "weaknesses": "Major:\n\n- In the ablation study Table 3, the authors showed that the differentiable PnP solver improves the accuracy of the method. Could the authors further explain the reasons?\n\n- In Line 287-288, the authors mentioned that they did not exclude extreme cases when reporting the performance. However, the authors could report both the performance with/without extreme cases for a clear and fair comparison to other methods.\n\n- The paper uses non-differentiable PnP to solve pose during inference. What about other methods applying a fast PnP during inference? Will they achieve similar speedups?\n\n- Ablation study (Table 3): how about using 1 or 2 of the components to show a more comprehensive study of the effectiveness of the combination of the module?\n\n- Table 4 better adds efficiency comparison.\n\n- The limitations of the method should be discussed in the main paper.\n\n\n\nMinor:\n\n- The attached code does not show many intuitions. A pseudo code is preferred in the text to aid the understanding of the method.\n\n- There are some typos in the paper that need to attend to.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Why differentiable PnP can improve the accuracy?**\n\nThe main insight that we introduce the end-to-end PnP is to impose supervisions directly on the predicted transformations. Previous works learn 2D-3D feature matching with non-differentiable PnP as a post-processing procedure to estimate the transformations The optimization target during training is only the pseudo supervision conducted from 2D-3D correspondences. The insufficient supervision leads to large errors since the network has no ability to handle incorrect matching pairs which have a highly negative effect on the results. By implementing an end-to-end framework, we are able to bring direct supervisions to the final transformation, which is the most accurate supervision and leads to a more stable and more accurate cross-modality registrations.\n\n**Q2: Comprehensive comparison with baseline methods.**\n\nWe refer the reviewer to “Global-Q2: Comprehensive comparison with baseline methods. ” of the global response for the results and analyses.\n\n**Q3: Can fast PnP speedup other methods.**\n\nActually, we already use the same setting as ours to apply EPnP with RANSAC to predict the rigid transformation at inference time for the SoTA baseline CorrI2P as described in L.283-L.285 of Sec. 4.2. The reason why CorrI2P fails to achieve fast pose inference is that CorrI2P produces lots of wrong matchings which results in wrong 2D-3D correspondences, and requires a large number of iterations for RANSAC to eliminate the wrong correspondences which is very time-consuming. On the contrary, our method produces much better cross-modality matchings and leads to robust 2D-3D correspondences as shown in Figure 6, thus only a few iterations are required for RANSAC to filter wrong correspondences. The other baseline DeepI2P performs the frustum classification where no 2D-3D correspondences are conducted and is not suitable for EPnP to solve the final pose estimation. Specifically, DeepI2P solves the time-consuming inverse camera projection problem for pose estimation, even with a 60-fold pose initialization for avoiding crashing, leading to long pose inference time. We also justify that our method requires much fewer parameters than the previous SoTA methods (e.g. CorrI2P and DeepI2P) as shown in Table 2, leading to much shorter time for network inference.\n\n**Q4: Comprehensive ablations on the designs.**\n\nWe provide the comprehensive ablations on our framework designs in Table H of the rebuttal PDF. We also report the performance using one, two or three of our designs at the same in Table H of the rebuttal PDF, where the results comprehensively demonstrate the effectiveness of each of our design.\n\n**Q5: Add efficiency comparison to Table 4.**\n\nWe add the efficiency comparison in Table 4 and provide the full results in Table G of in the rebuttal PDF.\n\n**Q6: Pseudo code for proposed method.**\n\nWe provide the pseudo code for the training and testing progress of our propose method in Algorithm 1 and 2 in the rebuttal PDF.\n\n**Q7: What is the limitation during inference? How will leveraging EPnP for inference affect the performance? Will other fast PnP suitable here?**\n\nThe main limitation during inference is the time spent, which determines the potential of our method to be applied in automatic driving scenarios where the low latency is required. We replace the Epro-PnP with EPnP during inference to reduce the pose inference time. We further test the performance of directly leveraging Epro-PnP to predict poses, which achieves similar performance (RTE = 0.74) with EPnP (RTE = 0.75) on the KITTI dataset. However, as reported in L.340-L.341, the inference time will increase from 0.19 s to 2.38 s if we use Epro-PnP. The reason is that Epro-PnP adopt a Gaussian-Newton algorithm-based iterative PnP solver with a time complexity of O($N^2$), while the EPnP is much faster with a time complexity of O($N$). Therefore, to enable fast registration, we leverage EPnP as the solver at the inference time. EPnP is the most widely-used and well-explored method for efficient pose estimation from correspondences with a time complexity of O(N), therefore we choose EPnP instead of other PnP solvers for fast and robust registration at inference time.\n\n**Q8: Limitations when applied to different datasets/scenes.**\n\nOur method shows great robustness and generality when getting applied in different datasets (e.g. KITTI and nuScenes), where we achieve more accurate registration results on both KITTI and nuScenes datasets as shown in Table 3 of the main paper. However, previous SoTA methods CorrI2P and DeepI2P show a performance decline in nuScenes. The result demonstrates that our method is more general than previous methods and achieves accurate performance in different dataset. We also made further discussions in the “KITTI vs. nuScenes” in Sec.3.2 of the supplementary.\nWe further justify that in our experiments setting, each dataset is split among sequences which are collected in different scenes, and the test sequences are unseen scenes for the trained model. \nTherefore, our performance on KITTI or nuScenes dataset can already demonstrate the ability of our methods to generalize to unseen scenes. \n\nFor more limitations of our method, please refer to “Global-Q3” and “Global-Q1” of the global respond. \n\n**Q9: How to choose model parameters for balancing accuracy and efficiency?**\n\nWe mainly consider the accuracy when choosing model parameters and also take into account efficiency for the model lightweight and faster convergence. As shown in Table G of the rebuttal PDF, we observe that increasing image resolutions can greatly improve the performance (e.g. from 0.84/2.90 to 0.65/2.10 in terms of RTE/RRE), therefore we choose to use higher image resolution with acceptable computational cost. When increasing the point density, only a marginal improvement is achieved (e.g. from 0.71/2.01 to 0.60/2.09), therefore we choose a proper density with fine performance and efficiency.\n"}, {"review_id": "hLfSvt1D6L", "reviewer": "Reviewer_ec8L", "summary": "This work aims to address the image-to-point cloud registration task. The authors propose a VoxelPoint-to-Pixel matching framework, which consists of three network branches dedicated to extracting features from voxel, point, and pixel representations, respectively, for 2D-3D matching. The network is trained with four different losses: an overlap prediction loss, a circle loss for 2D-3D feature matching, a KL divergence loss for probabilistic PnP, and a pose loss supervised by ground-truth. To assess the effectiveness of their approach, the authors perform experiments on two existing LiDAR datasets, showing improved 2D-3D matching performance compared to existing methods.", "questions": "Please see the Weaknesses section.", "limitations": "The authors did not discuss the limitations of their method. It would benefit the reader to include a failure case analysis, which would provide a more comprehensive understanding of the proposed approach.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed 2D-3D matching framework combines several well-established techniques. To bridge the domain gap between points and pixels, the authors suggest employing a triplet network to extract element-wise 2D and 3D features. To enhance the robustness of the 2D-3D matching process, the authors incorporate an overlap prediction branch. The whole pipeline is supervised by a 2D-3D matching (circle) loss and pose estimation (PnP) losses. \n\n- The authors assess the matching performance on the KITTI and nuScenes benchmarks, demonstrating that their method has better performance when compared with DeepI2P and CorrI2P.\n\n- The paper is generally easy to comprehend. I appreciate the presence of suitable illustrations accompanying the explanations in Sec. 3.", "weaknesses": "- One main concern for this work is its limited technical innovations: the adaptive-weighted loss is basically the circle loss [39], while the KL divergence loss for probabilistic PnP is borrowed from Epro-PnP [8]. Additionally, there is a lack of ablation studies for those four losses - it is unclear how the KL divergence loss $L\\_{KL}$ and the pose loss $L\\_{pose}$ contribute to the matching performance and whether both of them are necessary.\n\n- For the experiments on the KITTI and nuScenes datasets, the comparisons are limited to DeepI2P and CorrI2P. However, several other works related to 2D-3D matching are absent from the comparisons, such as the ones below. Additionally, it would be interesting to assess the generalization of the proposed method across datasets, for instance, from KITTI to nuScenes.\n  - Hierarchical Scene Coordinate Classification and Regression for Visual Localization. CVPR 2020.\n  - P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching. ICCV 2021.\n\n- For the ablation study in Sec. 4.6, it is unclear why only sequences 0-1 are used. It would be more beneficial to include all available data in order to thoroughly validate the design choices.\n\n- Typos:\n  - L267, “mis-registration”?\n  - L309 “dis-matched”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Technical innovations.**\n\nWe did get inspiration from previous methods on some loss designs. However, to best of our knowledge, almost no work explored the cross-modality contrastive learning between image features and voxel-point features, where we design a triplet network to learn VoxelPoint-to-Pixel matching to reduce modality domain gap and lead to robust 2D-3D correspondences and registrations. Epro-PnP cannot be directly leveraged in the Image-to-Point Cloud task since the images and LiDAR point clouds are captured in quite different ways. There is a large range of outliers on both modalities where no correspondences can be founded. To handle the outlier regions, we design a detection strategy to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences to solve Probabilistic PnP. Moreover, using the original contrastive equation, the network fails to produce a structured cross-modality latent space to represent both 2D and 3D features as shown in Figure 4. To address this issue, we introduce a loss with adaptive-weighted optimization inspired by Circle loss to explore a distinctive cross-modality latent space and also design a radius-based pair generation strategy to conduct negative and positive pairs, as discussed in Sec. 3.2. By directly introducing the original circle loss without our designed radius-based pair generation strategy, the network also fails to produce a structured cross-modality latent space to represent both 2D and 3D features, and the performance drops from 0.65/2.10/91.14 to 0.96/0.94/88.10 in terms of RTE/RRE/Acc.\n\nIn all, we proposed the first end-to-end framework for learning Image-to-Point Cloud registration, which enables a fast inference. Compared to previous state-of-the-art methods, our framework reduces the registration error from 3.59 to 0.75 in terms of RTE. Additionally, it significantly reduces the inference time from 8.96s to 0.19s, demonstrating its promising capabilities in autonomous driving systems.\n\n**Q2: More ablations for designed losses.**\n\nWe provide comprehensive ablation studies on each of the designed losses following your suggestions. We first demonstrate the effectiveness of our adaptive-weighted loss by comparing it with other metric learning losses (e.g. ContrastiveLoss, LiftedStructureLoss, GeneralizedLSLoss) in Table E of the rebuttal PDF, where we achieve the best performance among all the other losses. Then, we provide the ablations of replacing our Probabilistic PnP with another differential PnP BlindPnP [1] in Table F of the rebuttal PDF. We further provide the ablation studies to separately explore the impact of KL divergence loss and the pose loss in Table B of the rebuttal PDF, where both losses can improve the performance.\n\n[1] Campbell D, et.al. Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization. ECCV2020\n\n**Q3: Related works HSCNet and P2-Net.**\n\nWe focus on difference tasks with the mentioned HSCNet and P2-Net. HSCNet focuses on the visual location task which aims to estimate the camera pose of a query image with respect to a known environment, as described in the first sentence of the introduction in HSCNet. We have distinguished our task with the visual location task in L.87-L.92 in the Related Work, where we focus on the image-to-point cloud registration task in dynamic scenes captured at different time steps instead of first pre-building the whole environment and learning to locate a query image. P2-Net learns pixel-point matching for establishing 2D-3D correspondences, and also focuses on the visual location task, which is a different task of ours as described above. We agree that the P2-Net which also learns 2D-3D feature matchings can be leveraged in the Image-to-Point Cloud registration task as ours with some modifications. However, P2-Net does not open source, where no implementations can be found for a fair comparison. Note that the GitHub link provided in the abstract of P2-Net is empty. \n\n\n**Q4: Generalize the proposed method across datasets.**\n\nWe refer the reviewer to ”Global-Q1: Cross dataset validation.“ of the global response for justifying cross dataset validations.\n\n\n**Q5: The data used for ablation studies.**\n\nWe conduct ablation studies to explore each design in our frameworks and some important hyper-parameters like image resolutions, point densities, safe radius, feature dimensions, etc. Using a subset of the large-scale dataset for ablation studies is an efficient way and can justify the effectiveness of the designs.\nWe leverage the first two sequence of KITTI dataset, which contains about 30% Image-to-Point Cloud pairs of the KITTI dataset, as a subset to conduct comprehensive ablations for efficiently verifying more hyper-parameters and designs. \nWe agree that conducting ablation studies with the same setting as the main experiment to use the whole KITTI dataset can validate the design choices more thoroughly than only using a subset dataset. We will conduct the ablation studies under the whole KITTI dataset in the revision, due to the limited time in rebuttal period. \n\n\n**Q6: Limitations and failure cases.**\n\nOne of our limitations is that the feature matching errors at some noisy points of LiDAR point clouds may be very large, which have very negative influence on cross-modality registrations. As shown in Figure 3 of the supplementary, although the feature matchings at most of pixels/points are accurate, some feature matching results at noisy points (e.g. scans of bushes) of the scene are not stable. The reason is that the network is limited to handle noisy points without any special designs, leading to unstable 3D features at noisy points and further affect the registration accuracy at some complex scenes. We will add more discussion on the limitations and failure cases of our method in the revision.\n"}, {"review_id": "V8lbbir201", "reviewer": "Reviewer_Ehvn", "summary": "This paper introduces a framework for registering 2D images and 3D point clouds. The framework consists of three branches, for processing 2D images, 3D point clouds and 3D voxel constructed from the original 3D point clouds. The fused features of 3D point clouds and 3D voxel are used to establish correspondences with the features of the 2D images, and a differentiable PnP solver is used to calculate the transformation for end-to-end training. Experiments are conducted on two public datasets, KITTI and nuScenes, and the proposed method outperforms several existing baseline methods.", "questions": "Please respond to questions raised in weakness.", "limitations": "limitation is not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The three-branch structure is novel for this task and seems effective in improving the registration performance.\n\n2.The proposed method achieved much higher performance than existing methods in the experimental setting.\n\n3.The paper is generally well written and easy to follow. ", "weaknesses": "Major\n\n1.My major concern is the practical meaning of the proposed work. I wonder if there is a real scenario where this kind of registration is essential. Why not establish the correspondences between 2D images and 3D point clouds by device calibration? If we calibrate a LiDAR and an image camera, which should not be difficult, we can register the data captured with it (including both images and point clouds) to either an image or a point cloud by mono-modal registration. This two kinds of mono-modal registrations are both well studied with much higher accuracy. \n\n2.The ablation study has some issues. First, I notice that ablation is conducted on a different setting from the main experiment. Why not use the same setting as in Table 1? In the ablation study, only sequence 0-1 is used for training and only sequence 7 is used for testing, which are much less than the main experiment and makes the ablation results unstable. Second, from the ablation study we can see that removing any one component results in a light decrease of performance, but all ablation study performance are still much higher than baseline methods in Table 1. Existing ablation results can not show where the performance gain of the proposed method comes from. Further analysis is needed, such as removing two or three components at the same.\n\n3.The data are split into training and testing sets, but the author did not mention if a validation set is used and how they chose the best model for inference.\n\n4.RANSAC is used for more robust registration of the proposed method. We know that RANSAC plays a very important role in correspondence-based registration. Can RANSAC also be used in the comparing methods to improve their performance?\n\nMinor\n\n1.The references [31,32] and [19,22] in the first sentence of the second paragraph of Introduction are not proper, since these papers does not study the problem of establishing 2D-3D correspondences.\n\n2.In the introduction of the KITTI datasets, “2D translation on the ground within ±10”, what’s the unit of the translation? \n\n3.“registration accuracy (Acc.)” is not a good name for its current definition. Registration recall may be better.\n\n4.The inference time is 0.19s, which can not be said as real-time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Practical meaning of the proposed work.**\n\nThere are two types of calibration between vehicle-mounted camera and LiDAR, which is pre-calibration and online-calibration, both of them play important roles in the autonomous driving systems. We focus on the online-calibration which is much harder but crucial for autonomous driving scenarios.\nThe pre-calibration requires manual setup and adjustment, which brings significant labor costs. For example, the KITTI [1] dataset points out that the pre-calibration is not accurate where manual calibration is required to obtain the real transformation. As described in Sec. 2.2 of the KITTI [1] paper, the accurate transformation is achieved by first using the pre-calibration methods and then selecting a few manually correspondences between the LiDAR point clouds and pixels on the images to adjust the transformations, which is very labor-intensive. On the contrary, we aim to calibrate online automatically at any time during driving, which do not require any manual operation.\n\nAnother main difficulty in calibration is the fact that the transformation between the vehicle-mounted camera and LiDAR is not constant during driving due to factors such as vehicle shake and rough road conditions. As a result, mis-calibration occurs in almost every frame, and the mis-calibration errors will continue to accumulate over time as the vehicle travels. The naïve pre-calibration methods that only calibrate once are unable to handle these mis-calibrations during driving and require manual adjustment every once in a while.\nOur method enables online calibration automatically with a fast inference of only 0.19s per frame, which can handle the mis-calibration errors at any time during driving and provide an intelligent and effective way for improving the robustness of autonomous driving systems.\n\n[1] Geiger, Andreas, et.al. Are we ready for autonomous driving? the kitti vision benchmark suite. CVPR2012\n\n**Q2: The data used for ablation studies.**\n\nWe conduct ablation studies to explore each design in our frameworks and some important hyper-parameters like image resolutions, point densities, safe radius, feature dimensions, etc. Using a subset of the large-scale dataset for ablation studies is an efficient way and can justify the effectiveness of the designs.\nWe leverage the first two sequence of KITTI dataset, which contains about 30% Image-to-Point Cloud pairs of the KITTI dataset, as a subset to conduct comprehensive ablations for efficiently verifying more hyper-parameters and designs. \nWe agree that conducting ablation studies with the same setting as the main experiment to use the whole KITTI dataset can validate the design choices more thoroughly than only using a subset dataset. We will conduct the ablation studies under the whole KITTI dataset in the revision, due to the limited time in rebuttal period. \n\n\n**Q3: Comprehensive ablations on the designs.**\n\nWe provide the comprehensive ablations on our framework designs in Table H of the rebuttal PDF. By removing all our designs on the Voxel branch, Adaptive-Weighted Optimization and Differential PnP, the performance degenerates from 0.65/2.10/91.14 to 1.98/4.73/61.50 in terms of RTE/RRE/Acc., which is a total failure. We also report the performance using one, two or three of our designs at the same in Table H of the rebuttal PDF, where the results comprehensively demonstrate the effectiveness of each of our design.\n\n**Q4: How to choose the best model?**\n\nWe follow previous works (e.g. CorrI2P and DeepI2P) to split the data into training and testing sets for a fair comparison, where no validation set is used. We train our method for 50 epochs and use the final model for inference.\n\n**Q5: Can RANSAC improve other methods.**\n\nActually, we already use the same setting as ours to apply EPnP with RANSAC to predict the rigid transformation at inference time for the SoTA baseline CorrI2P as described in L.283-L.285 of Sec. 4.2. The reason why CorrI2P fails to achieve accurate registrations is that CorrI2P produces lots of wrong matchings which results in wrong 2D-3D correspondences, and after lots of iterations，RANSAC can hardly eliminate the wrong correspondences. On the contrary, our method produces much better cross-modality matchings and leads to robust 2D-3D correspondences, thus can achieve accurate registrations where only a few iterations are required for RANSAC. The other baseline DeepI2P performs the frustum classification where no 2D-3D correspondences are conducted and is not suitable for EPnP with RANSAC to solve the final pose estimation. However, DeepI2P solves the inverse camera projection problem for pose estimation, with a 60-fold pose initialization for avoiding crashing, which plays a similar role as RANSAC in EPnP. \n\n**Q6: Minor problems.**\n\n1. We will change the references [31, 32] and [19, 22] in the introduction to proper papers studying 2D-3D correspondences. \n\n2. The unit of the translation in L.267 which introduces KITTI dataset is m (meter), we will correct the statement as “2D translation on the ground within ±10 m”.\n\n3. We will change the name “Registration accuracy” to “Registration recall” in the full text following your suggestions.\n\n4. We will correct the “real-time” statement into “fast registration with 5 fps”.\n\n\n"}]

Author rebuttal:
We upload a rebuttal PDF with some experimental results requested by the reviewers. For the following rebuttals, we use “rebuttal PDF” to point to the provided PDF like “in Table A of the rebuttal PDF”.

We respond to some common questions in reviews as follows.

**Global-Q1: Cross dataset validation.**

Almost all the widely-used methods (e.g. Cylinder3D[1], CenterPoint[2])in 3D perception (e.g. segmentation / detection) train a single model for a dataset, i.e. one model for KITTI and one model for nuScenes, and do not perform cross dataset validation. The reason is that cross dataset validation is extremely difficult for 3D perception tasks of autopilot, since the devices (e.g. LiDARs and cameras) and the ways to collect data are quite different from dataset to dataset (e.g. KITTI and nuScenes). This leads to large differences of predicted feature distributions when apply a model trained on one dataset to directly leverage it to evaluate on another dataset. However, we justify that the experiments under a single dataset can already demonstrate the ability of our methods to generalize to unseen sequences or scenes. Specifically, each dataset (e.g. KITTI, nuScenes) is split among sequences and different sequences are collected in different scenes, therefore the test sequences are unseen scenes for the trained model. 

Moreover, we justify that our trained model in a single dataset can learn some underlying patterns that can be shared across different datasets. Specifically, we leverage our pretrained model of KITTI dataset to conduct few-shot registration experiments under nuScenes dataset. The model is finetuned using only 10 % samples in nuScenes training set, and is evaluated on the full test set. The result is shown in “$Ours_{Scratch}$” and “$Ours_{KITTIPretrain}$” in Table C of the rebuttal PDF, where fine-tuning the pretrained model of KITTI dataset on the small subset of nuScenes dataset significantly outperforms the result of training a randomly initialized model from scratch ,i.e. 1.79 vs. 2.72 in terms of RTE.

[1] Zhu X, et al. Cylindrical and asymmetrical 3d convolution networks for LiDAR segmentation. CVPR2021

[2] Yin T, et.al. Center-based 3d object detection and tracking. CVPR2021

**Global-Q2: Comprehensive comparison with baseline methods.**

We provide the comprehensive comparison with baseline methods (e.g. CorrI2P and DeepI2P) under the same setting of CorrI2P to exclude RTE larger than 5m and RRE larger than 10$^o$ in Table A of the rebuttal PDF. 
To further provide a more real and convincing comparison with SoTA method CorrI2P in the performance without extreme cases, we remove the same number of bad samples as CorrI2P and report the performance of our method as “Ours *”. By evaluating on the same number of test samples without extreme cases, we believe it is a relative fair comparison with CorrI2P. As shown in Table A of the rebuttal PDF, we achieve the best performance under all metrics. Especially, our method is about 3 times better than the SoTA baseline CorrI2P under the difficult benchmark nuScenes, even after removing the extreme samples. The result demonstrates that our method not only produces more stable registrations with much fewer failure cases but also produces much precise registrations for the success samples.

**Global-Q3: Limitations and failure cases.**

One of our limitations is that the feature matching errors at some noisy points of LiDAR point clouds may be very large, which have very negative influence on cross-modality registrations. As shown in Figure 3 of the supplementary, although the feature matchings at most of pixels/points are accurate, some feature matching results at noisy points (e.g. scans of bushes) of the scene are not stable. The reason is that the network is limited to handle noisy points without any special designs, leading to unstable 3D features at noisy points and further affect the registration accuracy at some complex scenes. We will add more discussion on the limitations and failure cases of our method in the revision.


Meta-review: This paper received unanimously positive reviews, including three weak accepts and two borderline accepts. All the reviewers recognized the effective design of the proposed approach and the state-of-the-art performance on standard benchmarks.

The AC thus recommends to accept it as a Spotlight.
Decision: Accept (spotlight)