Batch Differentiable Pose Refinement for In-The-Wild Camera/LiDAR Extrinsic Calibration
Authors: Lanke Frank Tarimo Fu
Maurice Fallon
Venue: CoRL 2023 Poster (2023)
Keywords: Sensor Fusion
Extrinsic Calibration
Differentiable Optimization

Abstract: Accurate camera to LiDAR (Light Detection and Ranging) extrinsic calibration is important for robotic tasks carrying out tight sensor fusion --- such as target tracking and odometry. Calibration is typically performed before deployment in controlled conditions using calibration targets, however, this limits scalability and subsequent recalibration. We propose a novel approach for target-free camera-LiDAR calibration using end-to-end direct alignment which doesn't need calibration targets. Our batched formulation enhances sample efficiency during training and robustness at inference time. We present experimental results, on publicly available real-world data, demonstrating 1.6cm/0.07° median accuracy when transferred to unseen sensors from held-out data sequences. We also show state-of-the-art zero-shot transfer to unseen cameras, LiDARs, and environments.

Batch Differentiable Pose Refinement for Th-The-Will Camera/LiDAR Extrinsic Calibration
Lanke Frank Tarimo Fu University of Oxford fu@robots. .ckuk
Maurice Fallon University Oxford mfallon@robots. ox ac. ac.uk
Misaligned
Aligned
Coarse
Fine
Figure 1: Coarse-to-fine refinement of the LiDAR-to-camera extrinsic parameters. The matching
appearance between the LiDAR and camera features trained using only ground-truth extrinsic parameters for self-supervision. During training, batched refinement helps retain difficult samples that individually would have been discarded. During inference, we show that batched refinement achieves state-of-the-arr zero-shot transfer. The rightmost column shows the refined overlay of Li- DAR points in the image.
Abstract: Accurate camera to LiDAR (Light Detection and Ranging) extrinsic calibration is important for robotic tasks carrying out tight sensor fusion such as target tracking and odometry. Calibration is typically performed before de- ployment controlled conditions using calibration targets, however, this limits scalability and subsequent recalibration. We propose a novel approach for target- free camera-LiDAR calibration using end-to-end direct alignment which doesn't t need calibration targets. Our batched formulation enhances sample efficiency dur- ing training and robustness at inference time. We present experimental results, on publicly available real-world data, demonstrating when transferred to unseen sensors from held-out data sequences. We also show statc-ofthhe-e zero-shot transfer to unseen cameras, LiDARs, and environments.
median accuracy
Keywords: Sensor Fusion, Extrinsic Calibration, Differentiable Optimization
1 Introduction
In many multi- robotic setups, information fusion between any two sensors requires accurate knowledge of the relative transformation between the sensors the extrinsic parameters. In the case of sensor fusion between a camera and a LiDAR, the extrinsic parameters along with the intrinsic parameters of the camera are used to determine point-to-pixel correspondence displayed Fig, 1. This correspondence enables fusion in downstream tasks such as object detection, tracking, and ego- meaning in uncontrolled
motion estimation. Camera/LiDDA extrinsic calibration 'in-the-wild' environments without specialized targets is difficult due to the domain gap between the two sensors. Cameras register textural information but can't directly measure geometry whereas LiDARs
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

measure geometry but can't register texture. Additionally, cameras are passive sensors and may capture illumination variations like shadows. In contrast, LiDARs, which produce their own light, don't detect shadows in the same visual range. Given these challenges, in-the-wild camera/LiDAR extrinsic calibration is still an active research question.
In this work, present a framework for camera/LiDAR extrinsic calibration that:
Can recover the accurate extrinsic parameters from a wide range of initial estimates.
Learns relevant features for pose alignment from both camera and LiDAR input automati- cally.
* Generalizes to sensors and environments not encountered during training.
We formulate the camera/LiDAR extrinsic calibration problem as one of batch differentiable direct alignment- aligning batch of learned features from the LiDAR domain their corresponding batch of deep image features, guided by deep feature gradients in the image. The end-to-end dif- ferentiable nature of our formulation relaxes the need for manual feature tuning. We show in our experiments that not only do we achieve state-of-the-es art performance on the training sensor suite (with median translation and rotation errors of 1. 1.6cm and 0.07%), but our method also generalizes to unseen sensor models and environments - with results demonstrated using a variety of common datasets.
2
Related Works
Classical solutions to the problem of target-free camera/LiDAR extrinsic calibration form two cat- egories: The first category exploits correlations between the image intensity value and the LiDAR reflectance value [1, 2, 3]. These methods perform well under assumptions of uniform lighting in the environment but still require an initial guess which is close to the ground truth extrinsic parameters. The other category of methods maps geometric (e.g. depth or normal) discontinuities in the LiDAR scan to image intensity discontinuities i.e. image gradients [4. 5, 6]. This use of local image gradient information can augmented by local intensity normalization, which improves the robustness of these methods in scenes with varying illumination sources. Still, their performance in cluttered en- vironments is limited by the abundance of local minima. Both these categories of elassical methods typically require manual parameter tuming for each unique sensor suite and as such struggle to be adapted to different configurations and manufacturers.
More recent works tackle the target-free calibration problem using deep-learned approaches. We classify these works into two categories: regression- methods [7, 8] that align camera and LiDAR features using parameters regressed by deep neural network. While these methods demon- strate impressive accuracy held-out datasets of their training environment, their regression-bastt nature makes zero-shot transfer to other datasets difficult since the neural network is biased to output extrinsic parameters in the distribution seen during training. Another, newer category of methods employs differentiable optimization for either (ii) indirect feature point matching via non- linear perspective-n-pott deep features to their corresponding pixel locations in the image. While both these methods gener- alize better than regression based methods, [10] only recovers calibration over a narrow region of operation and [9] achieves significantly less accurate results.
[9]: or (ii)) calibration-
refinement [10] to align LiDAR-extracter
RGKCNet [9] and DXQNet [10] are of different category of differentiable pose alignment com- pared to our method, in that they are forms of indirect alignment. RGKCNet learns 3-D point to 2-D pixel correspondences, and thus incorrectly models degenerate features such as lamp posts and thin trees which are line constraints and not point constraints. DXQNet learns a 2-dimensional weighting term which is used weight the per-point alignment along XY-axis aligned directions. This puts the burden the network to not just learn features but to also accurately model their orientations. Our method overcomes this challenge by explicitly computing deep image gradients, naturally dis-

---

Tí K
Canv2o
Image Gradient
T i+1
Warp
Batch set3)
Sperse
Conv3D
An overview of one iteration of the batched differentiable alignment. Batches of pairs of images and voxelized point clouds are passed through their respective U-Net feature extractors. Each sparse voxel feature is warped to the image using the latest transform parameters and the camera in- trinsic parameters and registers residual for its difference the image feature the corresponding pixel location. This residual along with the image gradient and the Jacobian of the projected pixel location with respect the transform parameters forms the signal for the optimization.
Figure 2:
tinguishing between line and point constraints allowing the network to focus simply learning useful features.
3 Method
We take inspiration from methods for scene-agnoster visual localization. Like PixLoc [11], we avoid overfitting pose estimation into scene memorization problem by performing deep feature alignment using differentiable optimization. This allows us to decouple the task of projective geometry, in pose refinement, from the task of learning features. doing so, we translate the task of achieving zero- shot transfer for camera/LiDAR extrinsic calibration, into the task of learning camera and LiDAR features that transfer to unseen environments, captured by different sensors.
3.1 Problem formulation
Given point cloud, set of N 3-D points, PL E 3 represented in the LIDAR reference frame (denoted by the subscript L) and a set of images, we want to determine the extrinsic parameters R E (3))), tf E IR3 such that,
Pc = R.P.+
(1)
Where Pc are the corresponding coordinates of Pi. from the camera's 's reference frame. For brevity, we use (R +C) = Ty. E SE(3) to denote the rigid body transform and omit the reference frame scripts T TC
3.2 Camera/LiDAR extrinsic calibration as differentiable direct alignment
With direct alignment, the transform parameters optimized to minimize the misalignment in the appearance of the signals registered by different sensors. In our case, our two sensors are the LiDAR and the camera. Instead of attempting to match the raw outputs from sensors, such as the intensity from LiDAR to image intensity, which has proven problematic in difficult lighting situations [3], we begin by extracting deep features from the raw inputs.
From the LiDAR side, map the 3-D points using the initial guess of the extrinsic parameters to get P = RPL t. We then voxelize these points and extract, using sparse 3-D CNN, set FP, w for each p E [1..., P) of P-level multi-scale feature pyramid. At each level, PP BaxA, the N, voxel centroids. and F?, D,,,N, the N, corresponding deep feature vectors each voxel centroid of dimension Dp. Lastly, w is the vector of N, learned weights with elements in the range [0. 1].

---

the learned From the camera side, we use 2-D CNN to extract from the camera image E RR xxx pyramid of deep features F E R x and corresponding weights W
C E R xH Similarly, p E [1,.... P| stands for one of the pyramid levels, each matching in scale with their corresponding LiDAR extractor pyramid levels.
At each i-th iteration of the optimization, the misalignment between the deep features extracted from the j-th LiDAR point and camera features at its corresponding projected point is given by,
r = -F7, - F (R.P t.)l) E Ps.
If is the projection function of a 3-D point in the camera reference frame onto the image plane at level p. and [-] denotes sub-pixel lookup by bi-linear interpolation. We've also introduced new variables R2 and i, which are optimization parameters that we iterate and initialize as RD = 13x3 and to =
Similar to [11], we formulate total cost function from these residuals in the form,
Np
(R, Σ wi, Wey ((Ir3])))))))
(3)
where wi, is the j-th element of w the LiDAR-lcamee weighting, and We, is the camera-learned weighting sub-pixel interpolated at the pixel location of the j-th LiDAR point projected into the image plane at level p. Lastly, p is the learnable robust cost function [12]. To robustly minimize this non-linear least-squares problem we perform the alignment solutions from the previous coarser level of the pyramid the starting point in the problem of the finer level. At each level,
coarse-to-fine fashion, using
algorithm [11], parameterizing transform updates using & E se(3). As such, stack all the residual terms at level p into r² PrN, and formulate each row of the Jacobian J R DDNpxx and the Hessian H E T with respect to & as,
J JkDD-1)D, = Or"
jt Op and H JTWJ, 0p 0g
(4)
where j [1, Nyp is the j-th 3-D point, &o Dpl denotes the k-th dimension of the deep-learned features at level p, and W E DDN,N DD,N, is block diagonal matrix with Np blocks of size D2 Dp each, where the j-th block has uniform diagonal weights Wi, Wey With these defined, we compute a Gauss-Newton gradient step in the direction of decreasing cost with & = -rr, and update our transform estimate using,
R 0
t++1 exp(E)
R. t, 1
(5)
We visualize one of these steps in Fig. 2
The training loss: During training, we perform these alignment updates for fixed M steps at each pyramid level, yielding for each level, Th We supervise these transforms with the ground truth using the reprojection error of each 3-D point:
L -= ==Σ>n P(IIIP + t) IP + +t
(6)
Note that since the points P3 are LiDAR points projected by the initial guess of the extrinsic pa- rameters, our supervision signal T given by, = TT-1, where T is the ground truth extrinsic parameters and T is the initial guess applied to the LiDAR points before voxelization.
3.3 Batch SE(3) alignment
At test time: When make the added assumption that solving for the same transform parameters across batch of image/point cloud pairs at test time, the change to algorithm is

---

Figure 3: PCA feature visualization of a Seature-deprive scene (top) and feature- rich scene (bot- tom). During training, our batch formulation keeps the harder example at the top from diverging. Consequently, we can still learn the sparse amount of features it does have e.g. the ground to hedge differences in the middle columns, and the salient post on the rightmost column.
rather straightforward. Where we used to have, at each pyramid level, the residual r E DrN, we now stack all the residuals across the batch together and get the vector r' E IR P2NNn where NB = -E NT is the total number of 3-D points in the batch at level p. Then, we update Eq. (4) accordingly and solve the optimization steps just as we did in the single sample case, This technique is deployed in most existing non-learning-ban
target-free calibration works [4, 3. 2]. where it is interpreted as making the calibration cost function Eq. (3) smoother and more convex. To our surprise, this simple yet very effective scheme has so far not been deployed in deep-learned camera- LiDAR extrinsic calibration.
At training time While at first glance it may seem impossible to perform batch pose alignment during training since we might encounter camera/LiDAR pairs with heterogeneous extrinsic param- eters, note that the pose alignment is performed relative to the initial guess (Eq. (5)). So while the actual camera/1 .iDAR relative transforms may differ across the batch, we can independently pick initial guesses for each sample in the batch such that the relative transform from the initial guess to the ground truth is identical. Mathematically, this gives for each b-th sample in the batch:
T = AT Tb.
(7)
Initializing the initial guess of each sample using Eq. (7) enables joint optimization across all sam- ples of the entire batch, allowing us learn features from even hard examples where individually the optimization would have diverged (see Fig. 3).
4 Training Setup
To test the capability of our framework to perform accurate zero-shot transfer to unseen environ- ments, we' ve set up our experiments to train solely on one dataset, and subsequently evaluate per- formances using other datasets with different sensors and test environments.
Dataset: Due to its popularity as benchmark for learning-based camera/L LiDAR calibration, the KITTI Odometry dataset [13] as sole training dataset. It consists of 22 sequences of sub- urban driving scenarios. During training, we only use camera "2" which is front-facing colored perspective camera and the top-mounted Velodyne HDL-64E LiDAR. Of the 22 sequences, we use sequences "01" - "21", leaving sequence "00" out for validation and testing.
Setup for LiDAR and camera input: Different LiDARs exhibit different spatial coverage, inten- sity profiles, and reference coordinate frames. To make our framework robust to these variations during zero-shot transfer, we perform pose and intensity augmentations to the LiDAR point cloud and crop augmentations to the camera image. We specify these details in Appendix A.1.
Model: To facilitate robust calibration from large initial offsets, we use a coarse-to- to-fine alignment scheme with levels. Two U-Net [14] architectures extract deep features from LiDAR and camera separately for each of these levels. To aid the correspondence of similar features from different domains, we further adapt the features from each domain with a single multi-layer perceptron. We provide details about model and its weight initialization in Appendix A.2. During training,
5

---

Table Results on the same camera of held-out sequence (values show component-wise
mean/median absolute values of translation/rotation along each axis).
Initial
Mcan/Median Åt (cm)
Mcan/Median AR (?)
Method
enror
roll
pitch
yaw
LCCNet
0.24/0.26
0.38/0.36
0.46/0.33
0.03/0.03
0.01/0.00 /
0.04/0.02
1.5mm -00
DXQNet
/
Ours (1)
8.77/1.76
5,50/1.45
9.25/1.80
0.36/0.08
0,43/0.07
0.53/0.07
Ours (8)
2.26/0.51
2.02/0.87
1.24/0.58
1010//002
0.21/0.04
0.16/0.03
LCCNet
0.24/0.15
0.48/0.26
1.11/0.47
0.02/0.02
0.17/0.10
0.03/0.03
+0.1m
DXQNet
0.75/0.53
0.48/0.51
1.09/0.78
0.05/0.03
0.05/0.03
0.03/0.02
=
Ours (1)
3.23/0.94
2.4
3.42/1.18
0.09/0.05
0.13/0.05
0.15/0.04
Ours (8)
0.42/0.32
0.82/0.83
0.59/0.46
0.02/0.02
0.04/0.04
0.02/0.02
use the Adam [15] optimizer with learning rate 10-5 5 8.
train for 20 epochs with batch size of
5 Results
All results showcased here are derived from models trained on a single camera/LiDAR pair as de- tailed in Section 4. To assess the capacity for zero-shot transfer, we incrementally test on more challenging scenarios, starting with held-out sequence of the training data and culminating tests on completely different cameras, LiDARs, and environments.
5.1 Extrinsic calibration in settings similar to training
Testing using the same camera: In this simple setting, we use images from the training camera (camera "02") but from held-out sequence "00" of the KITTI Odometry dataset which comprises 4541 image/point cloud samples. The differences between the extrinsic parameters in this sequence and the training sequences are negligible, so the key distinction from the training data is the novel scenes in this held-out sequence.
We compare against the regression- LCCNet [8] and the differentiable calibration flow method DXQNet [10]. When comparing against LCCNet, we pass to our model initial extrinsic parameter guesses sampled uniformly [0, and [0, 1.5]m around the ground truth value, using the scheme presented in Appendix A.I. Note that the initial angular errors used in this experiment are even larger than the values we used during training.
The upper section of Table shows LCCNet outperforming our method. Our method performs poorly on single image/point cloud pairs due to scenes with insufficient data for full 6-DoF pose observability. However, our method substantially improves when in batch optimization using pairs, achieving sub-centimeter median absolute error each translation axis and even surpassing LCCNet in median roll rotation accuracy. Remarkably, our model achieves this despite the fact that it has never encountered rotation perturbations up to 20% duriigg training.
DXQNet is designed to only recover calibration from small drifts [10] in the range [0, and [0, 0. 0.1Jm, Jm, so we sample initial calibration parameters in this same range when comparing our method against DXQNet. The lower section of Table shows that our method's single-sample performance is slightly worse than DXQNet in median metrics and notably worse in mean metrics since method diverges when single scene lacks sufficient structure. However, performing batch op- timization significantly enhances our method, achieving lower error than DXQNet along all axes except for Y-axis translation and median pitch-angle error.
While DXQNet is designed only for calibration from small initial errors of [0, 5] [0, 0. [10], in Table 1, our method competes even against fftee-f-thee-a regression based methods like LCCNet and recovers calibration from large initial errors of [0, 20] and [0, 1.5]m.

---

Table 2:
The performance change: trained on camera "2" and tested on camera "3" of a held-out
sequence (values show the mean/median magnitudes of the translation/rotation vector).
Initial enror
Mean At (cm)
Mcan AR (9)
Median At (cm)
Median AR (9)
Method
unseen
seen
unseen
seen
unscen
unseen
LCCNet
1.59
52.5
0.16
1.54
1.01
52.5
0.12
1.47
1.5mm
DXQNet
/
/
/
/
/
-00
Ours (1)
15.98
20.15
0,92
1.07
3.60
4.65
0.16
0.20
Ours (8)
3.09
3.77
0.15
0.30
1.39
1.69
0.07
0.07
LCCNet
1.29
52.5
0.18
1.52
0.61
52.5
0.12
1.47
+0.1m =
DXQNet
1.43
2.94
0.08
0.16
0.81
2.28
0.07
0.13
Ours (1)
6.25
8.32
0.25
0.31
2.21
2.86
0.10
0.13
Ours (8)
1.20
1.76
0.05
0.07
1.12
1.65
0.06
0.07
Testing using a different camera at a different vantage point:
While the methods are trained
on camera "2" (seen), this test, we perform calibration between the LiDAR and camera "3" (unseen). This is significant for generalization because cameras "3" and "2" are separated 50 apart. We also two ranges of initial errors in this experiment, larger to compare against LCCNet, and smaller one for DXQNet. In doing so, we test our model's ability to both recover calibration from large initial errors and also transfer to new sensors.
Unlike the case of testing on the seen camera, the upper section of Table 2 shows that, when tested on the unseen camera, our method consistently performs better than LCCNet whose mean and median translation error magnitudes (of more than 50em) lem error achieved with the seen camera. While our method also experiences drop in accuracy when tested on the unseen camera, our drop is significantly lower (from 3.60cm to 4.65cm median translation error), and even lower when run in batch optimization, from 1.39cm to 1.69cm median translation error magnitude- - only a 3mm drop.
than magnitude higher than the
Meanwhile, DXQNet, the learned calibration flow method, more robust when transferred the unseen camera. Seen the lower section of Table 2, the median translation accuracy of DXQNet drops 2.28cm the unseen camera, which is still relatively accurate when compared to LCCNet. While DXQNet performed better than our method in the single image/point cloud pair setting on the seen camera, once transferred to the unseen camera, the performance gap narrows down. In fact, shown on the lower section of Table 2, our method and DXQNet achieve the same level of median rotation accuracy.
We that running batched optimization with our method achieves the best mean/median rotation and translation accuracy compared to all other methods when transferred the wnseen camera. Additionally, the drop in accuracy (transferring from the seen to the unseen camera) is lower using our method - especially so in the batched alignment case. We highlight these facts in Fig. 4, where the slope of the graphs highlights the drop in accuracy.
Transiation error
Rotation error
Translation emor
Rolation error
1000m
+- LCCNet
Ours ttacch
Durs Sinple
Dura Batch
Ours Single
LoClet
Ours Single
Batth
Durs Butch
Ours Siigle
2cm
Socn
5oon
Lnnoon
30on
Unsoon
5oen
Figure 4: Slope graphs highlighting the error increase when the methods are tested on an unseen camera, having trained on the seen camera. All plots show median metries: (a) and (b) comparing translation and rotation eirors of our method versus LCCNet, and (c) and (d) comparing against DXQNet. Note that our method exhibits gentler slopes compared to the other methods, showing more robust transfer to the unseen camera.

---

Table 3: Zero-shot transfer performance on different datasets.
Method
At (cm)
AR (*)
Method
At (cm)
AR (*)
DXQNet
5.65/4.70
2.89/1.03
LCCNet
324/318 102/16.8 6.97/3.87
20.8/18.1
Ours Single
28.0/5.99 999
1.70/0.55
Ours Single
3.60/0.64
Ours Batch
3.67/3.61
0.51/0.51
Ours Batch
0.44/0.43
(a) KITTI-360 dataset. Initial error in the range: (b) Waymo dataset. Initial error in the range:
15. Ours Batch
batch size of
11.5m. Ours Batch
batch size of
5.2 Zero-shot transfer to different environments with different sensors
Transfer to different camera in different environment: The KITTI-360 [16] dataset is cap- tured in Karlruhe, just like the KITTI Odometry dataset used during training. However, the camera setup is different both in intrinsic parameters and its relative transform to the LiDAR.
We compare our calibration accuracy to DXQNet, as they have also reported their zero-shot transfer metries in the KITTI-360 setup. In Table 3a, we show that overall, the accuracy achieved by both methods worse rotation and in translation compared to their respective performances on the KITTI Odometry dataset. While our single sample optimization method performs better in rota- tion but worse in translation than DXQNet, our batched optimization method (batch of 8) performs significantly better than DXQNet in both rotation and translation.
Transfer to a different camera, LiDAR, and environment: We also test gencralizability to the Waymo dataset [17], which has higher resolution camera and a custom LiDAR. To match the training data image resolution, we halve the image size and camera intrinsics, allowing for consistent feature extraction. This flexibility further distinguishes optimization- calibration methods from regression-basee methods which can't explicitly reconfigure the camera projection parameters.
In Table 3b, we see that LCCNet does not generalize, with translation errors over 3m and rotation errors at 20.8 (mean) and 18.1 (mmedian)). Our method, run on single sample, poor the mean metric (just above Im and 3.6"), but is better the median metric (16.8em, 0.64%), suggesting that some outlier scenes impact calibration performance. Notably, batch optimizing image/point cloud pairs improves performance significantly, reducing the median translation error to 3.87cm and rotation errors to (.44((mea)) and 0.43% (median).
6 Limitations
Currently, our model assumes shared visibility between the LiDAR and the camera, enabling pose alignment from simultaneous image and point cloud pairs. sensor settings without shared vis- ibility, existing literature resolves this by creating a local map from several images and LiDAR scans [18]. Our model further presumes simultaneous image pixel and LiDAR point registration, necessitating ego-motion compensation for rotating LiDAR models. To overcome these limitations, in future work, we aim to tackle ego ago-motion estimation, inter-sensor temporal calibration and Li- DAR/camera extrinsic calibration jointly using differentiable representations of sensor relative pose as explored in [19].
7 Conclusion
We have presented a method for in-the-wild camera/LiDAR calibration that both recovers calibration from large initial errors ([O, 20] [0. 1.5]m) and transfers to unseen sensors and environments a trait that no existing method demonstrated. While these are promising results for in-the-wild calibration, our accuracy still falls short of target-based calibration methods. In future work, geometric priors such mapping/ego-mon motion consistency into feature
aim to incorporate learning to facilitate online tasks that require higher degrees of accuracy.

---

Acknowledgments
Support for this work has been provided by the Horizon Europe project DigiForest (101070405) and Royal Society University Research Fellowship (M. Fallon). This work has been carried out within the framework of the EUROfusion Consortium, funded by the European Union via the Euratom Research and Training Programme (Grant Agreement No 101052200 opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.
EUROfusion). Views and
References
[1] N. Williams, K.-L. Low, C. Hantak, M. Pollefeys, and A. Lastra. Automatic image alignment for 3d environment modeling. pages 388-399 395, 11 2004. ISBN 0-7695-2227-0. doi: 10::0..11 109/ SIBGRA.2004.135299
[2] G. Pascoe, W. Maddern, and Newman. Direct visual localisation and calibration for road vehicles in changing city environments. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, December 2015.
[3] G. Pandey, J. McBride, S. Savarese, and R. Fustice. Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information. Twenty-Sivi AAAI Conference on Artificial Intelligence, 26, 01 2012. doi:10. 1......//aaa.i.2..... sit.aaiiiiiil.83777
[4] J. Levinson and S. Thrun. Automatic online calibration of cameras and lasers. In Robotics; Science and Systems, 2013.
[5] A. Napier, P. Corke, and P. Newman. Cross-calibration push-broom 2d lidars and cameras in natural scenes. In 2013 IEEE International Conference on Robotics and Automation, pages 3679-3684, 2013. doi: 10.1 20.1100/CCA.2221..6633
[6] X. Liu, C. Yuan, and F. Zhang. Targetless extrinsic calibration of multiple small fov lidars and cameras using adaptive voxelization. IEEE Transactions on Instrumentation and Measurement, 71:1-12, 2022. 1oi:10. 209/TT..2022.311111
[7] G. Iyer, R. KarnikRam, K. M. Jatavallabhula, and K. M. Krishna. Calibnet: Geometrically supervised extrinsic calibration using 3d spatial transformer networks. 2018 JEEE/RS. Inter- national Conference Intelligent Robots and Systems (IROS), pages 1110-1117, 2018 2018.
[8] X. Lv, B. Wang, D. Ye, and S. Wang- Lidar and camera self-calibration using cost volume network. arXiv preprint arXiv:2012.13901, 2020.
[9] C. Ye, H. Pan, and H. Gao. Keypoint-based lidar-camera online calibration with robust ge- ometric network. IEEE Transactions on Instrumentation and Measurement, 71:1-11, 2022.
1229988
[10] X. Jing, X. Ding, R. Xiong, H. Deng, and Y. Wang. Dxq-net: Differentiable liderrcamera extrinsic calibration using quality-aware flow, 2022.
[11] P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, F. Kahl, and T. Sattler. Back to the Feature: Learning Robust Camera Localization from Pixels to Pose. In CVPR, 2021.
[12] T. Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4331 2019.
[13] A. Geiger, P. Lenz, and R. Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference Computer Vision and Pattern Recognition (CVPR), 2012.

---

[14] O. Ronneberger. P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
[15] D. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.
[16] Y. Liao, J. Xie, and A. Geiger. Kitti-360: novel dataset and benchmarks for urban scene understanding 2d and 3d, 2022.
[17] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, Shlens, Z. Chen, and D. Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[18] T. Scott, A. A. Morye, P. Piniés, L. M. Paz, I. Posner, and P. Newman. Choosing time and place for calibration of lidar-camera systems. 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4349-4356, 2016.
[19] Q Herau, N. Piasco, M. Bennehar, L. Roldão, D. Tsishkou, C. Migniot, P. Vasseur, and C. De- monceaux, Moisst: Multimodal optimization of implicit scene for spatiotemporal calibration, 2023.
[20] M. E. Muller. A note method for generating points uniformly n-dimensional spheres. Commun. ACM, 294):19-20, apr 1959, ISSN 0001-0782. (6::10.14553379999977966 URL https://doi. http:://doi.org/1..... 1115//779999///79999999
377946. 377944
[21] B. Graham, M. Engelcke, and L. van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[22] Contributors. Spconv: Spatially sparse convolution library.
https://github.com
traveller59/ spconv, /spooov 2022.
[23] K. Simonyan and A. 7isserman. Very deep convolutional networks for large-scale image recog- nition, 2015.
[24] T. Sattler, W. Maddern, C. Toft, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Oku- tomi, M. Pollefeys, J. Sivic, F. Kahl, and T. Pajdla. Benchmarking 6dof outdoor visual local- ization in changing conditions, 2018.
[25] X. Lai, Y. Chen, F. Lu, Liu, and J. Jia. Spherical transformer for lidar-based 3d recognition, 2023.
[26] Behley, Garbade, A. Milioto, Quenzel, S. Behnke, C. Stachniss, and J. Gall, Se- manticKITTI: Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
10

---

A Details on experimental setup
A.1 Setup for LiDAR and camera input
Setup for LiDAR input: To learn robust features for the calibration of different transforms, we train our network to recover the true calibration from variety of initial guesses T T. We perturb the ground truth T using uniformly sampled Tp, modeled by translation and angle-axis vector. We sample these vectors uniformly on the 3-sphere using [20] and scale them with uniformly distributed scalar. For training, the translation and rotation magnitudes are in the range [0, 1.5|m and [0, 15)" respectively.
To account for different intensity profiles produced by different LiDAR models, and to cater for the fact that some LiDAR drivers don't provide intensity readings, we augment the intensity channel of our LiDAR data to minimize our dependence on intensity information. We apply uniform random scalar perturbations in the range [0. 1.0) to the intensity channel of the LiDAR points, meaning that in some samples the intensity information is close to dropped out.
Lastly, to process the LiDAR data using sparse 3-D CNN, we voxelize the points using isotropic voxels of 2 em per side. We found this resolution to be reasonable since it is in the range of the measurement error reported by most automotive LiDAR manufacturers.
Setup for camera input: We've experimentally found that the camera feature extractor fails to learn generalizable geometric features unless spatial augmentations are applied. our training, we performed random crop augmentations of [512, 256] pixels in width and height the input image, and updated the camera intrinsic parameters accordingly.
A.2 Model setup
Both image and point cloud feature extractors in our model follow the U-Net [14] architecture with 5 layers of coarse-to-fine features, each layer being a factor of finer than the previous layer. We use features from layers for our alignment, the 1/16-scale, the 1/4-scale, and the 1-scale. The dimensionality of the feature these layers are 128, 128, and 32, respectively. To aid adaptation, at each pyramid level, the features from both domains passed through shared 2-layer MLP with the same imput and output dimensions at each layer and Leaky ReLu activation.
With recent advancements in 3-D convolutions [21], we' we've chosen the spconv [22] implementation of sparse 3-D CNNs for our LiDAR feature extractor, as can efficiently handle large point clouds, and can effectively manage the sparsity pattern of the data.
Learning features in the image domain is relatively straightforward, we' ve found that using the same U-Net [14] architecture (similar to Pixloc [11]) with the 2-D convolutional VGG [23] backbone was sufficient for image feature learning.
We initialize the visual extractor using weights from a pre-trained PixLoc [11] model trained on the CMU Seasons dataset [24], and the LiDAR extractor using only the sparse 3D CNN weights of SphereFormer [25] model pre-trained Semantic KITTI [26]. For the pose optimization, trained with M 5 iterations at every pyramid level.

---

B Additional Evaluations
B.1 Calibration recall as a function of initial error
To gauge the robustness of our method to initial calibration errors, conducted an experiment to measure the percentage of calibration trial that achieve errors less than 2cm and 0. 9n both transla- tion and rotation, respectively. In plotting this percentage as a function of the initial calibration error, we aim to show the sensitivity of our method to initial calibration errors both the single sample optimization setting and in the batched optimization setting. We used the unseen camera from the held-out KITTI Odometry data sequence. The results plotted in Figure show that batched opti- mization significantly boosts the robustness when there are large initial errors, achieving accurate calibration over 60 percent of the time even when the initial error is in the 22m, +20 range unseen during training.
Percentage of accurately calibrated samples
70
60
Peroentage (%6
50
batch single
40
30
20
10
0.3m 1'
0.5m S"
1 1.0m 10"
1.5m 15'
2.0m 20"
Initial Error Range
Figure 5: A plot of the percentage of calibration results that have errors less than 2cm and 0.1%, a function of the initial error range, tested on the unseen camera from the held-out sequences of the KITTI Odometry dataset. As seen, the batched optimization (batch size of 8) can estimate an accurate calibration over 60 percent of the time, even when tested in the initial error range of +2m, 2=20%hhch was not encountered during training.
B.2 Comparison against additional methods
We compared our method to LCCNet and DXQNet in our evaluations as they have the best perfor- mance for two different key traits - LCCNet excels in recovering from large initial errors, and can DXQNet transfer to unseen environments. Of our cited works, there only one other method based on differentiable alignment, RGKCNet, but this method has reported lower calibration accuracy than DXQNet, in slightly different experiment setting. For completeness, formance of our method tested in the RGKCNet experiment setting the KITTI Odometry dataset. The results, summarized in Table 4 show that our method significantly outperforms RGKCNet in both translation and rotation metrics. Note that the median metrics for CalibNet are not included because they were not made available by the authors.
also show the per-
Table 4: Comparison against RGKCNet and CalibNet in the KITTI Odometry setting. Initial error in the range: +
Mean/Median At (cm)
Mean/Median AR (")
Method
x
y
z
roll
pitch
yaw
CalibNet
12.0/-
3.5/_
7.9/_
0.18/_
0.9/_
0.15/_
RGKCNet
5.0/2.8
4.0/2.6
5.9/3.4
0.16/0.09
0.15/0.10
0.17/0.11
Ours (1)
3.2/1.0
2.7/1.0
3.4/1.2
0.09/0.05
0.13/0.05
0.14/0.04
Ours (8)
0.4/0.3
0.8/0.8
0.6/0.5
0.02/0.02
0.04/0.04
0.02/0.02
12

---

B.3 The impact of batch optimization
B.3.1 closer look at results from Table
In the case of using only single image and point cloud pair, the direct alignment can encounter outlier cases where the optimization diverges. In the absence of other sources of error, the median metric can be robust to these outlier cases. However, in addition to spurious diverging outlier cases, direct alignment with batch size autonomous driving scenarios, these local optima are prevalent along the translation axes. Due to the lack of features close the camera, changes the translation parameters create minimal visual parallax which make the optimization less sensitive to translation errors.
also affected by frequent convergence to local optima. In
This shortcoming of direct alignment is consistent with our findings reported in Table 2. Referring to the median rotation metric on the unseen camera, we see that Ours (1) performs just as well as DXQNet, and nearly an order of magnitude better than LCCNet (the regression-basee method). On the other hand, on the median translation metric for the unseen camera, Ours (1) performs an order of magnitude better than the regression-based method LCCNet (as expected), but is still 6 millimeters less accurate than DXQNet (the sparse flow-based approach).
In this context, optimizing over batch with size greater than 1, not only decreases the impact of outlier diverging samples but also improves the accuracy of all samples altogether. This can be seen the histograms of the error distributions for translation and rotation in Figure The error distribution of the batch optimization (size 8) exhibits shorter tail, it also has sharper peak closer zero.
Histogram of Translation Errors
Translation Batch 8 (count) Translation Single (count)
3000
2000
0
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
10.00
10.00
Distribution of abeolute value of transiation error (cm)
Histogram of Rotation Errors
Rotation Betch (count)
Rotation Single (count)
4000
3000
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.50
Distriution of absolute value of ratation error (dogroos)
Figure 6: The error histograms of our method on the unseen camera referred to in Table both translation and rotation error, the batched approach exhibits shorter tail and also peaks closer to zero than the single sample optimization. The last bin each plot aggregates all larger (outlier) values.
13

---

B.3.2 Experiment with 3 simple scenes
To demonstrate how batch optimization can improve accuracy, even on non-diverging samples, we performed small but instructive experiment using 3 image/point-cloud pairs from the validation set of KITTI odometry. Each of these samples contains sufficient information such that direct alignment does not diverge to extreme error. To further avoid divergence, sampled initial guess transforms relatively close to the ground truth (within 50 cm and 5 degrees).
(a) Scene
(b) Scene
(c) Scene 3
Figure 7: Scenes used experiment B.3.2: Scene has relatively more features close to the camera, whereas Scene 3 has the fewest features due to a large patch of under-exposed bush in the image.
Rumning individual optimizations from 100 different initializations yields the distribution of trans- lation errors seen in Figure 8.
Of the three scenes, performance is best Scene I as it has more features closer to the camera whereas performance is poorest in Scene 3 as it has the sparsest features. Aggregated statistics are shown in Table 5. The last column shows the mean and median values found by stacking all error values from scenes 2, and together. Note that the values the last column are close to the values computed from Scene 2, the mid ranking scene.
Table 5: Aggregate error statistics over 100 different runs of individual optimization each of the 3 scenes from Figure 7
Translation Errors on the Experiment with Scenes (em)
Scene
Scene 2
Scene 3
Scenes Stacked
Mean
1.49
3.10
6.52
3.72
Median
1.11
3.05
6.36
2.96
Performing the same experiment above, only running batch optimization with all 3 scenes yields significantly better results, in Figure 9.
14

---

Histogram of Translation
Scene (count)
Scene 2 (count)
Scene 3 (count)
60
40
20
1.50
4.50
6.00
7.50
9.00
12.00
15.00
16.50 17.50
Distribution absolute value translation (em)
Figure 8: Error distribution of optimizing 100 different initializations of each of the scenes in Fig- ure 7 individually.
Histogram of Translation errors
Batch Optimization (count)
Scene (count)
Scene 2 (count)
Scene (count)
80
60
20
0.00
1.50
3.00
4.50
6.00
7.50
9.00
10.50
12.00
12.50
Distribution of absolute value of translation errors (om)
Figure 9: Error distribution of optimizing 100 different initializations of all scenes in Figure jointly (Batch Optimization) compared against the individual optimizations. The distribution the batch optimization enrors exhibits shorter tail and is closer to zero.
By fusing information from all scenes, the overall eπror distribution is sharper and closer zero than the eтооr distribution of any of the individual scenes. Table 6 shows the aggregate statistics found by performing batch optimization and compares it to the individual optimization result shown previously.
The last columns Table 6 show very significant difference in mean and median error between performing batch optimization and individual per scene optimization. The mean error with batch optimization is better than the mean error achieved by any of the individual scenes. While the
15

---

Table 6: Aggregate error statistics over 100 different runs of batch optimization of all 3 scenes from Figure jointly (Scenes Batch Optimized), contrasted against individual optimization.
Translation Errors on the Experiment with Scenes (cm)
Scenes
Scene
Scene 2
Scene 3
Scenes Stacked
Batch Optimized
Mean (cm)
1.49
3.10
6.52
3.72
1.38
Median (cm)
1.11
3.05
6.36
2.96
1.28
median error of batch optimization is slightly worse than that of the best-performing individual scene, it is still significantly better than the median error of any other individual scene.
To conclude, the reason that batch optimization so significantly improves performance the rejection of outlier cases alone. Owing the effective fusion of features in direct alignment, batch optimization yields solutions that are close in accuracy to the best individual sample in the batch. This effectively raises the performance of many samples, not just the outlier cases.
due to
16


Reviews:
[{"review_id": "jF3W0jfHXMV", "reviewer": "Reviewer_3776", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a batch differentiable pose refinement network for camera lidar calibration. The proposed approach targets targetless calibration, argues for decoupling feature extraction and calibration to avoid scene memorization and overfitting, and therefore highlights its capability of zero-shot transfer to unknown camera, lidar and environment. The approach is an iterative one, refining previous/initial transformation, and determines the loss based on the Jacobian of transformed pixel location difference with respect to the transformation parameters themselves. Evaluations show batch version outperforms state-of-the-art approaches, and it is capable of zero-shot transfer to different sensors and datasets. ", "summary_of_recommendation": "This work proposed a batch differential pose refinement approach, advancing the state-of-the-art on targetless camera-lidar calibration. Evaluation is reasonably convincing and zero-shot transferability is a highlight of the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- While the authors have compared the proposed approach with two existing work, separately, with different perturbation parameters, it would be nice to make them consistent unless there is a convincing reason not to do so, to avoid the doubt of cherry picking results.\n- It would be nice if more baselines can be included beyond DXQNet, and LLCNet. Many cited in related work, but only two evaluated and compared against.\n- If space permitted, touching a bit on the requirement/dependency of good initial guess would also be greatly appreciated. Knowing how much of  a bad initial guess can be tolerated, and how faster the iteration can converge to good performance is valuable to real-world deployment,\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Originality: the paper is well motivated, pin-pointing to the unique novelty of the proposed approach with respect to existing literature. Related work is widely surveyed and well cited.\n- Quality: the proposed framework is technically sound, backed up by comprehensive evaluations. The zero-shot transfer results are impressive. \n- Clarity: the paper is well written, easy to follow.\n- Significance: automated lidar-camera calibration is an important problem. The paper has a potential of real-world deployment.\n- Relevance: the online mapping problem is critical in the automotive industry. The paper is definitely on the right path.\n- Limitations: Discussion section clearly states the assumption of shared visibility, the limitation of not leveraging ego motion, and lower accuracy compared to traditional offline methods.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "NVci0qVyAe", "reviewer": "Reviewer_sEmr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper tackles target-free camera-lidar extrinsics calibration with a learning-based approach. The problem is solved by direct alignment between deep features coming from a 2D CNN for the image and a sparse 3D CNN for the lidar scan. A key component of the method is to optimize the relative pose between sensors in a batch of image/scans pairs. Experiments show more accurate results than relevant competitors for several experimental scenarios, evens with sensors unobserved during training.", "summary_of_recommendation": "Overall, I think that this is a paper a good quality thanks to a polished writing, a well executed technical solution with some novelty and it is clearly relevant to the robotics community. My current recommendation is weak reject because of the poor results of the non-batched version, which seem counter intuitive and need to be explained more clearly in order to understand why the batched method performs well and better than competitors. I am disposed to increase my score if this problem is addressed.\n\nPost rebuttal : My concerns have been addressed thoroughly during the rebuttal and I am now convinced that the proposed method is worth publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I would really like more explanations and clarifications on the first weakness pointed out above. Am I missing something here ?\n I know that direct alignment can fail to converge when priors are too far from the solution, leading to failure cases and degrading the mean error, but even the median error is not very good for the optimization with batch size = 1. I find it suprising that increasing the batch size can improve the results that much. \nAddressing this weakness in the rebuttal is way more important in my opinion than emphasize on the second weakness, which is already mentioned in the limitations section and more a remark from my side than something that prevents the paper to be published.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths :**\n\n* The method shows a very good level of technical quality. Direct alignment on deep features has shown success in other areas such as camera relocalization and applying it to camera-lidar calibration is relevant and has not been done before to the best of my knowledge. Performing this alignment in a batched setup is a simple idea but well executed (using the same initial offset for different sensor setups during training is smart) and clearly beneficial.\n\n\n* The results with batched optimization are comparable or better than competitors and exhibit good generalization properties. The competitors are also well selected.\n\n\n* The paper is well writen and presented. \n\n**Weaknesses :**\n\n* I am really surprised by the difference between the error with and without batch optimization. It is intuitive that using more data brings more information and improves the optimization process. However, the results with a single pair of sensor data are really lower than expected. Competitors use only one pair of image/scan but exhibit better results. Why is the difference between \"Ours (1)\" and \"Ours (8)\" that large in all tables ? For example I would expect direct alignment to perform way better than the regression-based competitor on a new camera (Table 2).\n\n* As accurately pointed out in the limitations, the method assumes exact time synchronization between the camera image and lidar scan, which is usually the case in public dataset but much more difficult in practice (or it could be considered part of the calibration process). Recent work on the same topic [1] is also able to estimate the time offset between the sensors, is it something that could be improved in the future with this method ?\n\n[1] \"MOISST: Multi-modal Optimization of Implicit Scene for SpatioTemporal calibration\" Herau et al. arxiv 2023", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "5CUd2E25Agl", "reviewer": "Reviewer_StF3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work is about pose-refinement for self-driving scenarios, in special, the authors focus on zero-shot transfer of pose-refinement model: trained model can be applied on new scenes, new sensor placements and configurations.\nThe basic ideas are to align deep features between lidar and images.\nExperiments are conducted on KITTI, KITTI360 and Waymo to deonstrate the effectiveness of the method.\n", "summary_of_recommendation": "Overall I think the method makes sense, the results are good. But I think the improvement is a bit incremental compared to DXQNet.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address my concern on the weakness part. Besides:\n\n1. Why in setting ±1.5m/±20◦, only baseline LCCNet is applied and in ±0.5m\\±10◦, only baseline DXQNet is applied?\n2. What's the difference  between Mean ∆t in table 2 and  mean ∆t in table 1? Why the number is significant different. For example, 0.24 Vs 1.59 for LCCNet.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm not very familiar with this topic, espcially in the 2 baselines (LCCNet, DXQNet),  but I can understand the approach overall.\n## Strength\n1. An interesting and important task.\n2. The ability to restore the calibration error from >1m in input to a few cm is very impressive. Especially given that it can be applied on novel sensor placements and new dataset \n\n## Weakness\n1. The writing is not that clear to me (maybe because I'm not very familiar with this). It's not fluent either, e.g. the authors mix past tense and present tense here and there.\n2. The improvement seems a bit incremental compared to DXQNet,  The improvement of accuracy is  a few millimeter, and 0.2 degree -- seems negligible.  And this improvement is at the cost of using batched optimization. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a pose refinement network for camera/lidar extrinsic calibration. The method considers the targetless scenario and aims at zero-shot transfer for sensors and environments.

Strengths:
All reviewers agree that the problem is novel, sound, and relevant to the community. 

Weaknesses:
Minor issues in the evaluation and discussion have been pointed out by the reviewers:
1) Minor improvement with DXQNet
2) Consistency in the perturbation parameters for the different benchmarks
3) Extra baselines
4) Time synchronization

----- Post rebuttal -----

In the rebuttal period, all reviewers agree that the authors have addressed most of their comments with the new sets of experiments, minor corrections to the language, and discussion in the limitations section.
Decision: Accept (Poster)