Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work is about pose-refinement for self-driving scenarios, in special, the authors focus on zero-shot transfer of pose-refinement model: trained model can be applied on new scenes, new sensor placements and configurations.
The basic ideas are to align deep features between lidar and images.
Experiments are conducted on KITTI, KITTI360 and Waymo to deonstrate the effectiveness of the method.

review: I'm not very familiar with this topic, espcially in the 2 baselines (LCCNet, DXQNet),  but I can understand the approach overall.
## Strength
1. An interesting and important task.
2. The ability to restore the calibration error from >1m in input to a few cm is very impressive. Especially given that it can be applied on novel sensor placements and new dataset 

## Weakness
1. The writing is not that clear to me (maybe because I'm not very familiar with this). It's not fluent either, e.g. the authors mix past tense and present tense here and there.
2. The improvement seems a bit incremental compared to DXQNet,  The improvement of accuracy is  a few millimeter, and 0.2 degree -- seems negligible.  And this improvement is at the cost of using batched optimization. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I think the method makes sense, the results are good. But I think the improvement is a bit incremental compared to DXQNet.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposed a batch differentiable pose refinement network for camera lidar calibration. The proposed approach targets targetless calibration, argues for decoupling feature extraction and calibration to avoid scene memorization and overfitting, and therefore highlights its capability of zero-shot transfer to unknown camera, lidar and environment. The approach is an iterative one, refining previous/initial transformation, and determines the loss based on the Jacobian of transformed pixel location difference with respect to the transformation parameters themselves. Evaluations show batch version outperforms state-of-the-art approaches, and it is capable of zero-shot transfer to different sensors and datasets. 
review: - Originality: the paper is well motivated, pin-pointing to the unique novelty of the proposed approach with respect to existing literature. Related work is widely surveyed and well cited.
- Quality: the proposed framework is technically sound, backed up by comprehensive evaluations. The zero-shot transfer results are impressive. 
- Clarity: the paper is well written, easy to follow.
- Significance: automated lidar-camera calibration is an important problem. The paper has a potential of real-world deployment.
- Relevance: the online mapping problem is critical in the automotive industry. The paper is definitely on the right path.
- Limitations: Discussion section clearly states the assumption of shared visibility, the limitation of not leveraging ego motion, and lower accuracy compared to traditional offline methods.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This work proposed a batch differential pose refinement approach, advancing the state-of-the-art on targetless camera-lidar calibration. Evaluation is reasonably convincing and zero-shot transferability is a highlight of the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper tackles target-free camera-lidar extrinsics calibration with a learning-based approach. The problem is solved by direct alignment between deep features coming from a 2D CNN for the image and a sparse 3D CNN for the lidar scan. A key component of the method is to optimize the relative pose between sensors in a batch of image/scans pairs. Experiments show more accurate results than relevant competitors for several experimental scenarios, evens with sensors unobserved during training.
review: **Strengths :**

* The method shows a very good level of technical quality. Direct alignment on deep features has shown success in other areas such as camera relocalization and applying it to camera-lidar calibration is relevant and has not been done before to the best of my knowledge. Performing this alignment in a batched setup is a simple idea but well executed (using the same initial offset for different sensor setups during training is smart) and clearly beneficial.


* The results with batched optimization are comparable or better than competitors and exhibit good generalization properties. The competitors are also well selected.


* The paper is well writen and presented. 

**Weaknesses :**

* I am really surprised by the difference between the error with and without batch optimization. It is intuitive that using more data brings more information and improves the optimization process. However, the results with a single pair of sensor data are really lower than expected. Competitors use only one pair of image/scan but exhibit better results. Why is the difference between "Ours (1)" and "Ours (8)" that large in all tables ? For example I would expect direct alignment to perform way better than the regression-based competitor on a new camera (Table 2).

* As accurately pointed out in the limitations, the method assumes exact time synchronization between the camera image and lidar scan, which is usually the case in public dataset but much more difficult in practice (or it could be considered part of the calibration process). Recent work on the same topic [1] is also able to estimate the time offset between the sensors, is it something that could be improved in the future with this method ?

[1] "MOISST: Multi-modal Optimization of Implicit Scene for SpatioTemporal calibration" Herau et al. arxiv 2023
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I think that this is a paper a good quality thanks to a polished writing, a well executed technical solution with some novelty and it is clearly relevant to the robotics community. My current recommendation is weak reject because of the poor results of the non-batched version, which seem counter intuitive and need to be explained more clearly in order to understand why the batched method performs well and better than competitors. I am disposed to increase my score if this problem is addressed.

Post rebuttal : My concerns have been addressed thoroughly during the rebuttal and I am now convinced that the proposed method is worth publication.
review_title: null

Meta-review: This paper presents a pose refinement network for camera/lidar extrinsic calibration. The method considers the targetless scenario and aims at zero-shot transfer for sensors and environments.

Strengths:
All reviewers agree that the problem is novel, sound, and relevant to the community. 

Weaknesses:
Minor issues in the evaluation and discussion have been pointed out by the reviewers:
1) Minor improvement with DXQNet
2) Consistency in the perturbation parameters for the different benchmarks
3) Extra baselines
4) Time synchronization

----- Post rebuttal -----

In the rebuttal period, all reviewers agree that the authors have addressed most of their comments with the new sets of experiments, minor corrections to the language, and discussion in the limitations section.
Decision: Accept (Poster)