Title: Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching
Venue: NeurIPS 2023
Accepted: yes
Keywords: LiDAR Point Clouds

2D images

Cross-modality registration

Matching
Abstract: Cross-modality registration between 2D images captured by cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods.

Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

Abstract

Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at https://github.com/junshengzhou/VP2P-Match .

1 Introduction

Image-to-Point Cloud registration finds the rigid transformation (e.g. translation and rotation) that aligns the projection of a LiDAR frame represented as a point cloud to the reference image captured by cameras. The key here is to determine the extrinsic parameters of the camera with respect to the reference frame of LiDAR. It plays an important role in autonomous driving, 3D computer vision, augmented/virtual reality, etc. Despite a plethora of approaches have explored the same-modality registration tasks (e.g. Image-to-Image [50, 13] and Point Cloud-to-Point Cloud registration [25, 1]) and achieve promising results, few researches have shown convincing performances on cross-modality registration between images and point clouds due to its inherent difficulty.

Previous methods use MLPs [34, 35] on point clouds and CNNs [19, 23] on images separately with contrastive losses to learn distinctive features, and try to establish 2D-3D correspondences by matching the learned features. However, they fail to learn a structured latent space shared by 2D and 3D data due the calculation differences in MLPs or CNNs, which leads to different feature domains.

To solve these issues, we propose a novel framework to learn a structured cross-modality latent space for robust 2D-3D feature matching, where 2D elements are represented as pixels and the 3D elements are represented as the combination of voxels and points. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching for cross-modality registration, where both the voxel and pixel branch are designed based on CNNs to operate spatially-local convolutions on voxels/pixels represented in grids, and the point branch is integrated to regain the information lost during voxelization.

To learn a distinctive latent space where the correct 2D-3D correspondences can be guaranteed with cross-modality feature matching, we propose a novel loss with adaptive-weighted optimization which allows to learn distinctive 2D-3D correspondences by optimizing positive and negative matches in a robust self-paced manner. Since the LiDAR point cloud is captured around the car with a 360 o perception, and the image is captured eyes forward, there is a large range of outliers on both modalities (e.g. points/voxels behind the car and sky pixels), between which there is no explicit correspondences. To handle the outliers, we design a detection strategy to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences.

Furthermore, previous methods [14, 36, 26] adopt Perspective-n-Point (PnP) [24, 16] solver as a post-processing to estimate poses from matching results. They merely use the pseudo supervision conducted from 2D-3D correspondences as the optimizing target during training. The insufficient supervision leads to large errors since the network has no ability to handle incorrect matching pairs which have a highly negative effect on the results. Inspired by EPro-PnP [8], we propose to train our framework in an end-to-end manner. With a differentiable probabilistic PnP solver, we can impose supervision directly on the predicted pose by minimizing the Kullback-Leibler (KL) divergence between the predicted and target pose distribution. Our main contributions can be summarized as:

We propose a novel framework to learn Image-to-Point Cloud registration by learning a structured cross-modality latent space with adaptive-weighted optimization, through an end-to-end training schema with a differentiable PnP solver.

We propose to represent the 3D elements as the combination of voxels and points to overcome the pattern gap between points and pixels, where a triplet network is designed to learn VoxelPoint-to-Pixel matching.

We demonstrate our superior performance over the state-of-the-arts by conducting extensive experiments on KITTI and nuScenes datasets.

2 Related Work

2.1 Same-Modality Registration

Image Registration. Image-to-Image Registration is the key of structure-from-motion (SfM) [41] and SLAM [32]. Classic methods [39, 40] extract feature descriptors from image pairs with SIFT [28] or ORB [38], and establish correspondences based on the descriptor matching, where the Perspectiven-Point (PnP) [24] or Bundle Adjustment algorithm [45] can be further applied to estimate rotations and translations from the correspondences. Recently, learning-based approaches [50, 33, 13] have shown promising results on image registration. LIFT [50] replaces the key steps of SIFT to neural network layers.

Point Cloud registration. Traditional methods [2, 9] on point cloud registration requires a proper initialization to estimate the rigid transformation. With the rapid development of deep learning, the neural networks have shown great potential in 3D applications [34, 55, 54, 56, 48, 27, 31, 22, 53]. Recently, learning-based approaches like 3DMatch [51], PerfectMatch [18], PPFNet [12], USIP [25] have advanced a lot on point cloud registration. 3DFeat-Net [49] attempts to joint description and detection on 3D keypoints. However, both the image and point cloud registration methods depend

Voxelize

𝑀𝐿𝑃

𝑀𝐿𝑃

𝐶𝑜𝑛𝑣

𝐶𝑜𝑛𝑣

𝐶𝑜𝑛𝑣

𝐶𝑜𝑛𝑣

+

Devoxelize

Repeat

&cat

Repeat

&cat

𝐶𝑜𝑛𝑣

𝑀𝐿𝑃

𝑑2𝐷

𝑑3𝐷

ℒ 𝑑𝑒𝑡

𝑑3𝐷

𝑑2𝐷

Reshape

&map

Feature Matching

𝐶𝑜𝑟𝑟𝑒𝑠

Probabilistic PnP

ℒ 𝑝𝑜𝑠𝑒

ℒ 𝐾𝐿

𝑑𝑖𝑠𝑡 𝑔𝑡

𝑑𝑖𝑠𝑡

(a) Cross-Modality Learning

(b) Intersection Det.

(c) Solve Pose End-to-End

𝑔𝑡

𝐼

𝑉

𝑃

ℒ𝐴𝑊

ℒ𝐴𝑊

Adaptive-weighted Loss

ℒ 𝑑𝑒𝑡

Detection Loss

+

Element-wise Add

on the matching of feature descriptors designed for a specific modal, which fail dramatically on cross-modality registration.

2.2 Cross-Modality Registration

Visual Localization. Awell-searched task on cross-modality registration is visual localization, which aims to estimate the 6DOF camera pose of a query image in a 3D scene model presented in point clouds. Traditional methods [44, 47] build 3D maps using SfM [41] to store 3D visual descriptors. To determine the pose of a query image, those methods match visual descriptors obtained from the query image with the ones stored in the point cloud, and apply PnP solver [24] to estimate the camera pose from 2D-3D matches. Recently, Go-Match [57] proposes a learning-based schema to solve localization with only geometry information. Some other methods [3, 7, 5] further explore the differentiability of PnP solver. But those visual localization methods only focus on locating query images on a pre-built environment, and fail to generalize to dynamic scenes captured in real-time.

Another series of methods [6, 20, 30] explore the task of LiDAR and camera self-calibration. LccNet [29] projects mis-calibrated LiDAR point clouds onto depth images, together with the images captured by cameras to regress the relative rigid transformation. However, those methods can only handle a very small mis-calibration range since a relatively large range will lead to failures on depth projection. Moreover, the self-calibration task assumes that the relative transformation between camera and LiDAR is constant and is only predicted once for each sequence, while our method focuses on the registration for each frame of the sequence which is much more difficult.

Image-to-Point Cloud Registration. 2D3D-MatchNet [14] detects keypoints from images and point clouds using SIFT and ISS, and feed the local patches around those 2D/3D keypoints to CNNs and PointNet. The networks are trained with triplet loss to learn cross-modality correspondences. However, the hand-crafted keypoint detectors for different modalities don't guarantee the correct matching of keypoints, which leads to a poor registration accuracy. DeepI2P [26] converts the registration problem into a classification and inverse camera projection optimization problem, but the performance is limited since the frustum classification only indicates coarse 2D-3D correspondence. Recently, CorrI2P [36] proposes to learn dense 2D-3D correspondences by matching point-wise and pixel-wise features for fine registration. However, it fails to learn a distinctive latent space for 2D and 3D data, which leads to a large ratio of incorrect correspondences. Furthermore, all the above methods can not predict the rigid transformation directly, which require a post processing (e.g. Perspective-n-Point) to estimate rigid transformation from the 2D-3D correspondences.

3 Method

Problem statement. Given a pair of 2D image I ∈ R H × W × 3 and 3D point cloud P ∈ R N × 3 , the cross-modality Image-to-Point Cloud registration is to predict the rigid transformation T = [ R | t ] in 3D space that can be used to align the projection of point cloud P to image I , where t ∈ R 3 is the translation vector and R ∈ SO (3) is the rotation matrix.

In this section, we first introduce the detailed framework of our proposed VoxelPoint-to-Pixel Matching for learning a structured cross-modality latent space. Next, a novel loss with adaptiveweighted optimization is proposed for learning distinctive cross-modality patterns. Finally, we present the differentiable probabilistic PnP solver which drives our end-to-end learning schema. The overview of our framework is shown in Figure 1.

3.1 VoxelPoint-to-Pixel Matching

The first step of our framework is to obtain element-wise 2D and 3D features. We represent 2D elements as pixels and 3D elements as the combination of voxels and points. To achieve this, we design a triplet network consisting of Voxel/Point/Pixel branches as shown in Figure 1.(a).

Triplet Network. To design a voxel branch to obtain voxel-wise features f voxel , a naive implementation is to generate dense voxels from P and operate volumetric convolutions [58]. However, suffering from the cubic complexity of voxels, the resolution for voxel data is usually limited (e.g. 128 3 ), which will lead to a large information loss, especially for large scale LiDAR data. To address this issue, we leverage sparse convolution [10, 43] on high-resolution sparse voxels where the empty voxels are skipped.

Although the voxel branch can represent powerful 3D spatial patterns with convolutions, it still suffers from the information loss during voxelization. Therefore, we integrate the point branch to regain the lost 3D detailed patterns by learning point features. The point branch is designed based on PointNet++ [35]. It operates hierarchical set abstraction with self-attention modules [52] to get global feature g 3 D ∈ R C 3 D × 1 , and the feature propagation module [35] is further applied to learn point-wise feature f point ∈ R N × C . The pixel branch is a convolutional U-Net [37], where the ResNet [19] is used as the basic layers to get global 2D image feature g 2 D ∈ R C 2 D × 1 , and the upsampling module with skip-connection is further applied to get pixel-wise feature f pixel ∈ R H × W × C .

2D-3D Feature Matching . We represent 3D elements as the combination of voxels and points. To achieve this, we first transform the voxel feature to the point-wise voxel representation f ′ voxel ∈ R N × C by interpolating each point-location with its 8 neighbor voxel grids using trilinear interpolation. Thus, the 3D features f 3 D can be donated as:

and the 2D features f 2 D ∈ T × C can be achieved by reshaping pixel-wise features f pixel where T = H × W . After being processed with the embedding heads, the 2D and 3D features can be mapped into the same latent space, where we establish correspondences according to cosine similarities between f 2 D and f 3 D .

Comparison to Point-to-Pixel Matching. Due to the different characteristics between 3D point cloud and 2D images, it is extremely difficult to operate the same feature extractors on both modals. Existing approaches [14, 36] learn point-to-pixel matching with MLP-based point network and CNN-based image network, which leads to different feature domains.

We find that although the huge domain gap between points and pixels, while the voxels and pixels share great similarities. Therefore, we introduce the voxel branch with spare convolution to capture spatially-local patterns as 2D convo-

(a) P2P Matching

(b) VP2P Matching

2D Feature

3D Feature

lution operated on pixel branch. Since both the 2D and 3D features are processed by convolutions,

they share similar characteristics in feature space, leading to a structured shared latent space for both 2D and 3D features.

Wevisualize the learned latent space of Point-to-Pixel Matching and our proposed VoxelPoint-to-Pixel Matching with t-SNE [46] to convert features to 2D space through cosine similarities in Figure 2. The latent space learned by P2P Matching is extremely irregular with large gaps between several clusters and some space only contains the features of a single modality. In contrast, VP2P Matching leads to a structured cross-modality latent space where the features are evenly distributed throughout the space. See appendix for more analysis.

Intersection Detection . Since the images and LiDAR point clouds are captured in quite different ways, there is a large range of outliers on both modalities where no correspondences can be founded. We define the intersection region as the overlap between the 2D projection of a LiDAR point cloud using ground truth camera parameters and the reference image. To handle the outlier regions, we design a detection strategy as shown in Figure 1(b) to predict the probability of lying in the intersection region for each 2D/3D elements, and remove the outlier regions on both modalities before inferring 2D-3D correspondences.

For detecting the outlier region in 3D space, we first repeat the 2D global feature g 2 D and concatenate it with the 3D element-wise feature f 3 D as [ g 2 D : f 3 D ] , where ':" donates the feature concatenation. Then several MLPs ϕ are used to learn the probability d i 3 D of a point p i to lie in the 2D-3D intersection region as: d i 3 D = ϕ ( f 3 D : g 2 D ) , i ∈ [1 , N ] , vice versa for detecting 2D space outliers. We define a threshold σ where 2D/3D elements with probabilities smaller than σ will be considered as outliers.

To learn the detection, we sample Z in pairs of pixels and points with probability d from the intersection region and Z out pairs of pixels and points with probability ˆ d from the outlier regions. The detection loss is then formulated as:

3.2 Adaptive-Weighted Optimization

To explore distinctive patterns, various optimization strategies like contrastive loss and triplet loss are widely used in 2D or 3D tasks. However, these formulations treat each pair of samples equally, which leads to ambiguous convergence, especially in the difficult situation of 2D-3D feature matching. Inspired by the Circle Loss [42], we design a flexible optimization strategy with adaptive weighting for learning a distinctive cross-modality latent space where we can establish 2D-3D correspondences more accurately.

Given a set of 2D-3D pairs K = { α i , β i } , i ∈ [1 , v ] , which are sampled from the intersection region. We separate them into positive pairs k p and negative pairs k n determined by whether the distance between the location of α i and the projection of β i on the image is larger than a radius r or not. We define the positive similarity for a positive pair k i p as:

where c is the channel number of features. And vice versa for negative pairs s n .

To avoid ambiguous convergence and allow flexible optimization, we design adaptive weighting

3D-to-2D pairs

2D-to-3D pairs

2D space

3D space

𝛽

𝛼𝑛

1

𝛼𝑛

2

𝛼𝑝

3

𝛼𝑝

4

𝑟

factors for positive and negative pairs as ρ p = Ψ ( γ (1 -s j p + m )) and ρ n = Ψ ( γ ( s k n -m )) , in which γ is a scale factor, m is a margin for better similarity separation and Ψ is the function to detach ρ p and ρ n from gradients to serve as weights for optimization. The adaptive-weighted loss is then derived as:

We illustrate the adaptive-weighted optimization in Figure 3. The 3D space is achieved by projecting points into image space. Given a 3D point β , we define two negative pairs { α 1 n , β } and { α 2 n , β } where α 1 n and α 2 n lies outside the safe radius r . It is obviously that the negative pair { α 2 n , β } is much harder to learn since α 2 n lies near the positive region and some samples around α 2 n is from positive pairs, while { α 1 n , β } is easier since all the samples around α 1 n is also negative. And vice versa for the positive pairs. Optimizing all the samples with the same weight will lead to ambiguous convergence at the regions containing both negative and positive samples, i.e., it will be difficult to distinct the negative sample α 2 n with positive samples. Driven by this analysis, we optimize the patterns with adaptive weighting to force the network focus more on the harder samples (indicated by the color shading above).

(𝒂)

Contrastive Loss

(𝒃)

Adaptive-Weighted Loss

3.3 Differentiable PnP Solver

Correspondences Establishing. To establish correspondences, we first remove the outlier regions on both modalities with intersection detection and leverage the nearest neighbor principle on the cross-modality latent space for 2D-3D feature matching. Due to the different density of points and pixels, they do not have completely one-to-one correspondences. To avoid ambiguous matching, we establish the 2D-3D correspondence X i by searching for the point coordinate P z with the largest similarity in cross-modality latent space for each 2D pixel coordinate I i , i ∈ [1 , T in ] , T in is the number of pixels which lie in the intersection region. The operation achieved by arg max to find P z is inherently non-differentiable, to train our framework end-to-end, we leverage Straight-Through Gumbel Estimator [21] to estimate the gradient for backward optimization.

Probabilistic PnP. Given a set of correspondences X = { I i , P i } , i ∈ [1 , T in ] , the PnP problem [24] is to search for an optimal pose T = [ R | t ] that minimizes the reprojection error:

where π is the projection function and w i ( T ) is the reprojection error at X i . Inspired by Epro-PnP [8], we solve the non-differentiable PnP problem by interpreting the output as a probabilistic distribution:

The KL divergence loss is then computed to minimize the distance between the predicted pose distribution and ground truth pose distribution:

where the first term is the the reprojection error at ground truth pose, and the second term is the integral of reprojection error at predicted pose distribution. In practice, we leverage Monto Carlo strategy to collect samples to approximate the integration with Adaptive Multiple Importance Sampling algorithm [11].

Except for conducting supervisions on the probabilistic distribution, we further estimate the exact pose T ′ = [ R ′ | t ′ ] by solving Eq. (5) with an iterative PnP solver based on Gauss-Newton (GN) algorithm, and compute the pose loss as:

which can also participate on the optimization since the iterative part in GN algorithm is differentiable.

We visually compare the feature matching results of our proposed adaptive-weighted optimization with the contrastive loss as used in CorrI2P [36] in Figure 4, where our proposed loss shows significantly superior performance in learning a distinctive cross-modality latent space.

3.4 Implementation Details

We set the channel dimension C of 2D/3D feature to 64, and set C 2 D and C 3 D for the 2D/3D global feature both to 512. We set the margin m to 0.25, the scale factor γ to 32 and the safe radius r to 1 pixel. To enhance the representation ability of the voxel branch, we keep the point transformation pipe used in SPVNAS [43]. And the probability threshold σ in intersection detection is set to 0.95. Although the GN-based PnP solver used in Sec. 3.3 can solve exact pose while remain differentiable, its iterative solution is very time-consuming. For efficient registration, we leverage the EPnP [24] with O (n) time complexity during inference since we do not focus on the differentiability at inference time. The RANSAC [15] is applied with EPnP for more robust registrations.

4 Experiments

4.1 Dataset

We evaluate our performance on Image-to-Point Cloud registration task on two wildly used benchmarks KITTI and nuScenes. On both dataset, the images and point clouds are captured simultaneously with 2D cameras and 3D LiDARs.

KITTI Odometry [17]. We generate the image-point cloud pairs from the same data frame of 2D/3D sensors. We follow previous works [26] to utilize the 0-8 sequences for training, and 9-10 for testing. The mis-registration transformation is conducted with a 2D translation on the ground within ± 10 and a rotation around the up-axis with no limited range. We downsample the image resolution to 160 × 512 and the point cloud size to 40960 for training and testing.

nuScenes [4]. The image-point cloud pairs are generated by official SDK where the point cloud is accumulated from the nearby frames and the image is from the current frame. We follow the official data spilt of nuScenes to utilize 850 scenes for training, and 150 scenes for testing. The mis-registration transformation is conducted in a similar way as the one in KITTI dataset. We downsample the image resolution to 160 × 320 and the point cloud size to 40960, respectively.

4.2 Baselines and Metrics

We compare our method with the state-of-the-art methods DeepI2P [26] and CorrI2P [36]:

1) Grid Cls. + PnP. The grid classification setting is a baseline approach proposed in DeepI2P to divide the image into 32 × 32 grids and learn to classify each 3D point to a unique 2D grid with a neural network. The EPnP with RANSAC is then applied to predict the rigid transformation. 2) Frus.Cls. + Inv.Proj. DeepI2P proposes to perform the frustum classification with the inverse camera projection to obtain final rigid transformation. We report their results with 2D and 3D inverse camera projection as DeepI2P(2D) and DeepI2P(3D).

3) CorrI2P. CorrI2P is the latest work on Image-to-Point Cloud registration which learns dense correspondences for image-point cloud pairs, and the EPnP with RANSAC is applied to predict the rigid transformation.

Metrics. We follow DeepI2P to evaluate the registration performance with average Relative Translational Error (RTE) and average Relative Rotation Error (RRE). We do not follow CorrI2P to remove the image-point cloud samples with large errors before averaging, which in our opinion, is an unsuit-

Input

DeepI2P

CorrI2P

Ours

GT

0.22 m / 0.94

°

0.34 m / 0.57

°

0.28 m / 0.68

°

2.51 m / 7.84

°

1.56 m / 4.43

°

2.49 m / 4.33

°

3.29 m / 10.58

°

1.78 m / 17.63

°

2.42 m / 20.80

°

9.04 m / 178.34

°

5.30 m / 141.30

°

3.66 m / 51.20

°

RTE(m) / RRE (

°

)

RTE(m) / RRE (

°

)

RTE(m) / RRE (

°

)

able way to report real performances. We further report the registration accuracy (Acc.) which is the proportion of fine registrations with RTE < 2 mand RRE < 5 o .

4.3 Registration Accuracy

Quantitative Comparison. The results of cross-modality registration are shown in Table 1, in which our proposed method achieves the superior performance over the compared methods on both KITTI and nuScenes dataset. Especially, the latest work CorrI2P [36] adopted a similar way as ours to establish 2D-3D correspondences with feature matching, which is the most relevant method to ours. However, our method is about 4 times better than CorrI2P in terms of RTE. The main reason is that the previous methods fails to conduct end-to-end supervisions and can not learn robust 2D-3D correspondences due to the huge domain gap. In contrast, our proposed method can learn a structured cross-modality latent space for robust 2D-3D feature matching, together with an end-to-end training schema driven by a probabilistic PnP solver. As a result, our method is able to predict highly accuracy 2D-3D correspondences and achieve better performance than its counterparts.

Visual Comparison. The visual comparison is shown in Figure 5. For intuitive visualization, the point cloud is projected into image space with the predicted extrinsic parameters of different approaches and the known camera internal parameters. The color indicates the distance between a point and the camera. For DeepI2P, we adopt the setting with highest accuracy, i.e. DeepI2P(2D) for visual comparison. Compared with the other methods, our method achieves better registration accuracy in different road situations. For example, on the difficult tuning situation (e.g. the 1 st and 2 nd row), both DeepI2P and CorrI2P fail to solve correct registration, where the projections of trees and cars are largely dis-matched with the corresponding pixels in the image.

4.4 Feature Matching Accuracy

We further provide the feature matching visualization in Figure 6. We visualize the double-side error maps by computing the matching distance on both modalities. Specifically, for 2D-to-3D matching, we search for a point with the greatest similarity in cross-modality latent space for each 2D pixel in the intersection region, and compute the error by first projecting the matching point into image space and compute the Euler distance

20

0

10

CorrI2P

Ours

2D-to-3D

3D-to-2D

between the projected matching point and the 2D pixel. And vice versa for 3D-to-2D matching. As shown in Figure 6, our method significantly outperforms CorrI2P [36] in both 2D-to-3D and 3D-to-2D matchings. Most of the pixels in 2D-to-3D matching or points in 3D-to-2D matching of our method can reach slight errors within 2 pixels, which demonstrates that our learned shared latent space can distinct cross-modality patterns apart and lead to an accurate feature matching for each single 2D/3D element. We also observe that some elements on the 2D/3D edges will lead to relative large errors, since it is often difficult for intersection detection to perform perfectly in the edges.

4.5 Efficiency Comparison

We further compare the efficiency of our method with other counterparts, where all the methods are evaluated with an NVIDIA RTX 3090 GPU and Intel(R) Xeon(R) E5-2699 CPU. The

results are shown in Table 2, where our method requires much fewer parameters but gets significantly better performances. Furthermore, our method only requires 0.19 s for network inference and pose estimation for one frame, which is about 50 × (or more) faster than the previous works. The reason is that DeepI2P solves the time-consuming inverse camera projection problem for pose estimation, even with a 60-fold pose initialization for avoiding crashing, and CorrI2P spends a lot time to eliminate the wrong correspondences through iteration. Therefore, all the previous works fail to be applied to real automatic driving scenarios where the low latency is required, while our method can solve poses much more efficiently. Note that our result is achieved by replacing the GN-based PnP solver with the O(n) EPnP [24] at inference time, while the time will increase to 2.38 s with the GN-based PnP inference.

4.6 Ablation Study

We conduct ablation studies to justify the effectiveness of each design in our method and the effect of some important parameters. We report the performance in terms of RTE/RRE/Acc. under the KITTI dataset.

Framework designs. We first justify the effectiveness of each design of our framework. Specifically, we develop four different variations for comparison: (1) w/o voxel branch is the variation removing the voxel branch from the triplet network; (2) w/o point branch is the variation removing

the point branch from the triplet network; (3) w/o A-W optimization is the variation replace our adaptive-weighted optimization loss with the contrastive loss as used in CorrI2P [36]; (4) w/o Diff. PnP is the variation removing the end-to-end supervisions driven by the differentiable PnP solver.

The results are shown in Table 3, from which we can find that our Full model achieves the best performances over all variations. Such results prove the effectiveness of each design in our framework. Moreover, by comparing w/o voxel branch and w/o point branch to the Full model, we can find that the voxel branch plays a more important role in our framework, which demonstrates that the voxel modality with similar characteristics (represented in grids) and pattern extractors (CNNs) as pixels is more suitable for learning Image-to-Point Cloud registration.

The effect of input resolutions. We further explore the effect of input image resolutions and point densities. Table 4 shows the performance of different settings, from which we can find that higher resolutions on both modalities lead to better results since the low resolution images will omit some visual information while the low den-

sity point clouds will lose the detailed geometries. We select the proper setting with a balance between the performance and efficiency.

5 Conclusions

In this work, we propose a novel framework to learn Image-to-Point Cloud registration with VoxelPoint-to-Pixel Matching, where we learn a structured cross-modality latent space with a novel adaptive-weighted loss. We represent the 3D elements as the combination of voxels and points to overcome the domain gap between points and pixels. Moreover, we train our framework end-to-end

by imposing supervision directly on the predicted pose distributions with a differentiable PnP solver. The extensive experiments on KITTI and nuScenes datasets demonstrate our superior performances.

References

Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6359-6367, 2020.

Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures , volume 1611, pages 586-606. Spie, 1992.

Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dsac-differentiable ransac for camera localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6684-6692, 2017.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11621-11631, 2020.

Dylan Campbell, Liu Liu, and Stephen Gould. Solving the blind perspective-n-point problem end-to-end with robust differentiable geometric optimization. In European Conference on Computer Vision , page preprint. Springer, Aug 2020.

Daniele Cattaneo, Matteo Vaghi, Augusto Luis Ballardini, Simone Fontana, Domenico G Sorrenti, and Wolfram Burgard. Cmrnet: Camera to lidar-map registration. In 2019 IEEE intelligent transportation systems conference (ITSC) , pages 1283-1289. IEEE, 2019.

Bo Chen, Alvaro Parra, Jiewei Cao, Nan Li, and Tat-Jun Chin. End-to-end learnable geometric vision by backpropagating pnp optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8100-8109, 2020.

Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2781-2790, 2022.

Yang Chen and Gérard Medioni. Object modelling by registration of multiple range images. Image and Vision Computing , 10(3):145-155, 1992.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019.

JEAN-MARIE CORNUET, JEAN-MICHEL MARIN, Antonietta Mira, and Christian P Robert. Adaptive multiple importance sampling. Scandinavian Journal of Statistics , 39(4):798-812, 2012.

Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 195-205, 2018.

Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable cnn for joint description and detection of local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8092-8101, 2019.

Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud. In 2019 International Conference on Robotics and Automation (ICRA) , pages 4790-4796. IEEE, 2019.

Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM , 24(6):381-395, 1981.

Xiao-Shan Gao, Xiao-Rong Hou, Jianliang Tang, and Hang-Fei Cheng. Complete solution classification for the perspective-three-point problem. IEEE Transactions on Pattern Analysis and Machine Intelligence , 25(8):930-943, 2003.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Zan Gojcic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5545-5554, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Ganesh Iyer, R Karnik Ram, J Krishna Murthy, and K Madhava Krishna. Calibnet: Geometrically supervised extrinsic calibration using 3d spatial transformer networks. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1110-1117. IEEE, 2018.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In International Conference on Learning Representations (ICLR 2017) . OpenReview. net, 2017.

Chuan Jin, Tieru Wu, and Junsheng Zhou. Multi-grid representation with field regularization for self-supervised surface reconstruction from point clouds. Computers & Graphics , 2023.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Epnp: An accurate o (n) solution to the pnp problem. International journal of computer vision , 81(2):155-166, 2009.

Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 361-370, 2019.

Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-point cloud registration via deep classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15960-15969, 2021.

Shujuan Li, Junsheng Zhou, Baorui Ma, Yu-Shen Liu, and Zhizhong Han. NeAF: Learning neural angle fields for point normal estimation. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , volume 2, pages 1150-1157. Ieee, 1999.

Xudong Lv, Boya Wang, Ziwen Dou, Dong Ye, and Shuo Wang. Lccnet: Lidar and camera self-calibration using cost volume network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2894-2901, 2021.

Xudong Lv, Shuo Wang, and Dong Ye. Cfnet: Lidar-camera registration using calibration flow network. Sensors , 21(23):8112, 2021.

Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient consistency for neural signed distance functions via level set alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17724-17734, 2023.

Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147-1163, 2015.

Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. Advances in Neural Information Processing Systems , 31, 2018.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems , 30, 2017.

Siyu Ren, Yiming Zeng, Junhui Hou, and Xiaodong Chen. Corri2p: Deep image-to-point cloud registration via dense correspondence. IEEE Transactions on Circuits and Systems for Video Technology , 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pages 2564-2571. Ieee, 2011.

Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Improving image-based localization by active correspondence search. In European Conference on Computer Vision , pages 752-765. Springer, 2012.

Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8601-8610, 2018.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4104-4113, 2016.

Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6398-6407, 2020.

Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision , pages 685-702. Springer, 2020.

Carl Toft, Will Maddern, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, et al. Long-term visual localization revisited. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020.

Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle adjustment-a modern synthesis. In International Workshop on Vision Algorithms , pages 298-372. Springer, 1999.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. In European Conference on Computer Vision , pages 757-774. Springer, 2020.

Xin Wen, Junsheng Zhou, Yu-Shen Liu, Hua Su, Zhen Dong, and Zhizhong Han. 3D shape reconstruction from 2D images with disentangled attribute flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3803-3813, 2022.

Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In European Conference on Computer Vision , pages 607-623, 2018.

Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In European Conference on Computer Vision , pages 467-483. Springer, 2016.

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1802-1811, 2017.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1625916268, 2021.

Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEE/CVF international conference on computer vision , 2023.

Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Learning consistencyaware unsigned distance functions progressively from raw point clouds. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773 , 2023.

Junsheng Zhou, Xin Wen, Baorui Ma, Yu-Shen Liu, Yue Gao, Yi Fang, and Zhizhong Han. 3d-oae: Occlusion auto-encoders for self-supervised learning on point clouds. arXiv preprint arXiv:2203.14084 , 2022.

Qunjie Zhou, Sergio Agostinho, Aljosa Osep, and Laura Leal-Taixe. Is geometry enough for matching in visual localization? In European Conference on Computer Vision , 2022.

Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4490-4499, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for registering 2D images and 3D point clouds. The framework consists of three branches, for processing 2D images, 3D point clouds and 3D voxel constructed from the original 3D point clouds. The fused features of 3D point clouds and 3D voxel are used to establish correspondences with the features of the 2D images, and a differentiable PnP solver is used to calculate the transformation for end-to-end training. Experiments are conducted on two public datasets, KITTI and nuScenes, and the proposed method outperforms several existing baseline methods.\n\nSTRENGTHS:\n1.The three-branch structure is novel for this task and seems effective in improving the registration performance.\n\n2.The proposed method achieved much higher performance than existing methods in the experimental setting.\n\n3.The paper is generally well written and easy to follow.\n\nWEAKNESSES:\nMajor\n\n1.My major concern is the practical meaning of the proposed work. I wonder if there is a real scenario where this kind of registration is essential. Why not establish the correspondences between 2D images and 3D point clouds by device calibration? If we calibrate a LiDAR and an image camera, which should not be difficult, we can register the data captured with it (including both images and point clouds) to either an image or a point cloud by mono-modal registration. This two kinds of mono-modal registrations are both well studied with much higher accuracy. \n\n2.The ablation study has some issues. First, I notice that ablation is conducted on a different setting from the main experiment. Why not use the same setting as in Table 1? In the ablation study, only sequence 0-1 is used for training and only sequence 7 is used for testing, which are much less than the main experiment and makes the ablation results unstable. Second, from the ablation study we can see that removing any one component results in a light decrease of performance, but all ablation study performance are still much higher than baseline methods in Table 1. Existing ablation results can not show where the performance gain of the proposed method comes from. Further analysis is needed, such as removing two or three components at the same.\n\n3.The data are split into training and testing sets, but the author did not mention if a validation set is used and how they chose the best model for inference.\n\n4.RANSAC is used for more robust registration of the proposed method. We know that RANSAC plays a very important role in correspondence-based registration. Can RANSAC also be used in the comparing methods to improve their performance?\n\nMinor\n\n1.The references [31,32] and [19,22] in the first sentence of the second paragraph of Introduction are not proper, since these papers does not study the problem of establishing 2D-3D correspondences.\n\n2.In the introduction of the KITTI datasets, “2D translation on the ground within ±10”, what’s the unit of the translation? \n\n3.“registration accuracy (Acc.)” is not a good name for its current definition. Registration recall may be better.\n\n4.The inference time is 0.19s, which can not be said as real-time.\n\nQUESTIONS:\nPlease respond to questions raised in weakness.\n\nLIMITATIONS:\nlimitation is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to address the image-to-point cloud registration task. The authors propose a VoxelPoint-to-Pixel matching framework, which consists of three network branches dedicated to extracting features from voxel, point, and pixel representations, respectively, for 2D-3D matching. The network is trained with four different losses: an overlap prediction loss, a circle loss for 2D-3D feature matching, a KL divergence loss for probabilistic PnP, and a pose loss supervised by ground-truth. To assess the effectiveness of their approach, the authors perform experiments on two existing LiDAR datasets, showing improved 2D-3D matching performance compared to existing methods.\n\nSTRENGTHS:\n- The proposed 2D-3D matching framework combines several well-established techniques. To bridge the domain gap between points and pixels, the authors suggest employing a triplet network to extract element-wise 2D and 3D features. To enhance the robustness of the 2D-3D matching process, the authors incorporate an overlap prediction branch. The whole pipeline is supervised by a 2D-3D matching (circle) loss and pose estimation (PnP) losses. \n\n- The authors assess the matching performance on the KITTI and nuScenes benchmarks, demonstrating that their method has better performance when compared with DeepI2P and CorrI2P.\n\n- The paper is generally easy to comprehend. I appreciate the presence of suitable illustrations accompanying the explanations in Sec. 3.\n\nWEAKNESSES:\n- One main concern for this work is its limited technical innovations: the adaptive-weighted loss is basically the circle loss [39], while the KL divergence loss for probabilistic PnP is borrowed from Epro-PnP [8]. Additionally, there is a lack of ablation studies for those four losses - it is unclear how the KL divergence loss $L\\_{KL}$ and the pose loss $L\\_{pose}$ contribute to the matching performance and whether both of them are necessary.\n\n- For the experiments on the KITTI and nuScenes datasets, the comparisons are limited to DeepI2P and CorrI2P. However, several other works related to 2D-3D matching are absent from the comparisons, such as the ones below. Additionally, it would be interesting to assess the generalization of the proposed method across datasets, for instance, from KITTI to nuScenes.\n  - Hierarchical Scene Coordinate Classification and Regression for Visual Localization. CVPR 2020.\n  - P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching. ICCV 2021.\n\n- For the ablation study in Sec. 4.6, it is unclear why only sequences 0-1 are used. It would be more beneficial to include all available data in order to thoroughly validate the design choices.\n\n- Typos:\n  - L267, “mis-registration”?\n  - L309 “dis-matched”?\n\nQUESTIONS:\nPlease see the Weaknesses section.\n\nLIMITATIONS:\nThe authors did not discuss the limitations of their method. It would benefit the reader to include a failure case analysis, which would provide a more comprehensive understanding of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a 2D to 3D registration pipeline using a differentiable PnP method (Epro-PnP) and integrates 3D information using both voxel and point-based features. These designs target on previous problems such as the domain differences when fusing MLP-based point features and CNN-based pixel features, and the non-differentiable post-processing when computing the transformation that cannot train the entire model end-to-end. Experiments on KITTI and nuScenes, along with ablation studies show the effect of the proposed pipeline while achieving great efficiency.\n\nSTRENGTHS:\n- The proposed method improves the current state-of-the-art methods by a large margin in terms of the accuracy and efficiency on KITTI and nuScenes datasets.\n\n- Although most techniques used in the paper are not new, combining these techniques yield effective results.\n\n- The idea of combining both the voxel-based features and the point cloud-based features helps with the domain difference between MLP-based and CNN-based features.\n\n- The writing of the paper is okay and is easy to follow.\n\nWEAKNESSES:\nMajor:\n\n- In the ablation study Table 3, the authors showed that the differentiable PnP solver improves the accuracy of the method. Could the authors further explain the reasons?\n\n- In Line 287-288, the authors mentioned that they did not exclude extreme cases when reporting the performance. However, the authors could report both the performance with/without extreme cases for a clear and fair comparison to other methods.\n\n- The paper uses non-differentiable PnP to solve pose during inference. What about other methods applying a fast PnP during inference? Will they achieve similar speedups?\n\n- Ablation study (Table 3): how about using 1 or 2 of the components to show a more comprehensive study of the effectiveness of the combination of the module?\n\n- Table 4 better adds efficiency comparison.\n\n- The limitations of the method should be discussed in the main paper.\n\n\n\nMinor:\n\n- The attached code does not show many intuitions. A pseudo code is preferred in the text to aid the understanding of the method.\n\n- There are some typos in the paper that need to attend to.\n\nQUESTIONS:\nPlease address my questions/concerns above. Specifically, my main concerns are some unclear/inconsistent arguments and experimental designs.\n\nLIMITATIONS:\nThe limitation of the method is not discussed. Some of the limitations that can be discussed are as follows: \n\n1. The method uses a non-differentiable fast PnP solver during the inference, which is different from the optimization process. How will it affect the performance? And will any fast PnP solver be suitable here? What is the limitation during inference?\n\n2. What is the limitation of the method when applied to different datasets/scenes?\n\n3. How to choose model parameters in order to balance accuracy and efficiency? The authors have included some ablation studies in both the main paper and the supplementary materials. However, the tradeoff between accuracy and efficiency could be further discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to register an image with its nearby Lidar scan, and it comes with three modules:\n\n1) A sparse 3D conv-net and point-net to extract 3D features; A 2D conv-net to extract 2D features;\n\n2)  An intersection detection module to discard non-matchable 2D and 3D points;\n\n3) Modified Circle, Probabilistic PnP, and pose losses are used to train the proposed triple network.\n\nSTRENGTHS:\n1) This paper is well-written and easy to follow;\n\n2) This paper makes good practice to combine multiple existing modules to address the image2Lidar registration problem;\n\n3) According to Table 1, the proposed method outperforms previous works addressing the image2Lidar registration problem.\n\nWEAKNESSES:\nThough this paper successfully combines multiple existing modules, some design choices are yet to be investigated. Specifically,\n\n1) Generalization ability. \n\nAccording to the paper, the proposed method focuses on autonomous driving datasets and conducts experiments on 3-DoF registration problem (Line 267). I have a concern that the network overfits on this specific configuration. To address my concern, please conduct the following experiments:\n\na) Cross-validation. Using a network trained on the KITTI dataset to test on the nuScenes dataset;\n\nb) Conducting 6-DoF registration, rather than 3-DoF registration. Please at least add rotations around the x-axis and y-axis;\n\n\n2) Pipelines.\n\na) The effectiveness of the modified Circle Loss. Please add the comparison with respect to the original Circle Loss in Table 3;\n\nb) Since metric-learning plays an important role in this paper, I would expect authors to conduct comparisons with off-the-shelf metric-learning losses other than the Circle Loss. Please refer to https://github.com/KevinMusgrave/pytorch-metric-learning;\n\nc) The effectiveness of the Probabilistic PnP and pose loss. I would expect a two-stage training strategy here by first training the network by only using the adaptive-weighted loss, and then adding the Probabilistic PnP and pose loss. This reminds me of the work [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization]. Please separate the impact of KL divergence loss and pose loss, and check their effectiveness independently in Table 3. It would be nice to further compare the proposed Probabilistic PnP with respect to the differentiable PnP module in [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization].\n\n\n3) Minor.\n\nI think processing 2D and 3D data with convolutions may not justify the claim of \"Sharing similar characteristics in feature space\" (Line 162), as 3D convolution is invariant to shift, viewpoint, distance, etc. In contrast, 2D convolution is vulnerable to viewpoint and scale differences.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver, which designs a triplet network to learn VoxelPoint-to-Pixel matching. The proposed method is trained in end-to-end manner by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. The experiments seem good.\n\nSTRENGTHS:\n(1). A framework to learn Image-to-Point Cloud registration by learning a\nstructured cross-modality latent space with adaptive-weighted optimization, together with\nan end-to-end training schema driven by a differentiable PnP solver.\n\n(2) The paper represents the 3D elements as the combination of voxels and points to\novercome the pattern gap between points and pixels, where a triplet network is designed to learn VoxelPoint-to-Pixel matching.\n\nWEAKNESSES:\n1. The motivation that use voxel information is unclear, as described in the paper, one of the bottleneck of previous methods is points and pixels are with different characteristics with patterns learned in different manners (MLP and CNN), however, domain gap is also exited between voxeled information and pixels, why does the paper use voxel information should be discussed.\n\n2. The ablation studies are not enough, \n(1). in 3.2 Adaptive-Weighted Optimization, a hyper-parameter radius r is used, which is should be discussed, because it affects the positive and negative pairs, experiments should be provided to verify the robustness.\n\n(2). In probabilistic PnP, the effectiveness of L_{pose} and L_{kl} should be verified separately.\n\n(3). I agree that the settings in CorrI2P is unsuitable that exclude RTE larger than 5m and RRE larger than 10◦, however, it is better to provide these results same with CorrI2P, since it would be better for readers to understand where does main improvement come from.\n\n(4). Limitatations and failure cases should be discussed.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nLimitations and failure cases should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}