Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposed to study the problem of extracting correspondences for 3D point cloud registration. The paper presented CoFiNet (Coarse-to-Fine Network) to extract hierarchical correspondences from coarse to fine without detecting keypoints.  The model firstly learns to match down-sampled nodes and proposes node correspondences. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Patch correspondences are then refined to point level by a density-adaptive matching module. The proposed method has been evaluated on both indoor and outdoor standard benchmarks.
main_review: Originality: The paper extended the coarse-to-fine strategy in 2D vision tasks to 3D point cloud registration. The related work did not cite and discuss other category of point cloud registration approaches such as the correspondence-free approaches.

Quality: In the paper, the proposed coarse-to-fine pipeline works at a coarse and finer scale. Could this coarse-to-fine strategy be applied to multiple scales? 

Clarity: While the paper presented interesting ideas, the clarity is a major concern. For example, the differentiable density-adaptive matching module, which is a key component of the proposed framework, has not been made clear. 

Significance: 3d point cloud registration is an important task in computer vision. The paper extended the coarse-to-fine strategy in 2D vision tasks to 3D.
limitations_and_societal_impact: The paper did not address the limitations of the work. The paper aims at tackling partial overlap while it did not provide experiments with respect to different ratios of overlapping between input point clouds. It is interesting and critical to analyze the robustness with respect to different ratios of overlapping. The scalability issue has also not been analyzed in the paper. Could the proposed approach be applied to large scale point clouds with low overlapping ratios?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a coarse-to-fine network to extract the correspondences for point cloud registration.
main_review: Originality: The overall framework is similar to PREDATOR, but replace the sparse convolution with KPCov. The attention module follows PREDATOR's transformer. The encoder module is the coarse step and the decoder module is the fine step in this submission. Based on my understanding, the contribution is incremental and the performance achieves very limited improvement compared to PREDATOR. 

Quality: This paper is technically sound but I am still worried about the efficiency compared to PREDATOR. The authors have not discussed the weakness of their work. From Figure 3, we can find the final correspondences are located in the very limited regions. Why other regions are all outliers in this work. I am very worried about the robustness since most of the information are discarded roughly in the coarse step.

Clarity: The overall organization is well but could be improved. For example, I disagree with the claim in the abstract "Existing works widely adopt keypoint detection before establishing correspondences but struggle to propose repeatable keypoints". The recent point cloud registration methods, such as pointnetlk(CVPR2019), DGR (CVPR2020), FMR(CVPR2020), PREDATOR (CVPR2021) and PointDSC(CVPR2021), all do not adopt the keypoint detection. 

Significance: This work is an increment of PREDATOR by replacing the sparse convolution with KPCov. The significance of the scientific finding is very limited.
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an approach for finding correspondences for robust point cloud registration without detecting keypoints. The method is based on the hierarchical coarse-to-fine processing of the point clouds that is then used to find the sought rigid transformation by using RANSAC.
main_review: Strengths:
- Keypoints do not have to be detected to find the rigid transformation. This improves the repeatability of the solution. 
- The proposed weighting scheme helps in finding the correspondences efficiently.
- The experiments show that the proposed method leads to SOTA results on the 3DMatch and 3DLocMatch datasets. To Table 3, it is not SOTA on Kitti, thus, I suggest removing sentence "Results show that CoFiNet achieves state-of-the-art performance on
14 all evaluated datasets" from the abstract. 

Weaknesses:
- The paper (at least for me) was quite challenging to understand. Aside from the topic being far from me, I would suggest significantly improving the text since it is unclear in many places. For example, Fig.1 is supposed to help in understanding the method, however, it does not really help. 

Since I am not too familiar with the topic I can't judge if the method is new. 
The results seem good compared to recent approaches and, thus, I am suggesting acception. 

Suggestion:
- It would be interesting to see the performance if a modern robust estimator is used, e.g. [a,b], not RANSAC.

[a] Raguram, Rahul, et al. "USAC: A universal framework for random sample consensus." IEEE transactions on pattern analysis and machine intelligence 35.8 (2012): 2022-2038.

[b] Barath, Daniel, et al. "MAGSAC++, a fast, reliable and accurate robust estimator." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

limitations_and_societal_impact: -
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a novel learning method for point cloud registration by extracting hierarchical correspondences in a coarse to fine manner. There are two modules:

Coarse scale:  a network is first utilized to downsample and extract feature from input  point cloud. The features are then strengthened and used to compute a score matrix for coarse correspondence proposals.

Fine scale: coarse correspondence proposals are expanded to patch correspondences via grouping and refined with a density-adaptive matching module.
main_review: - It is a novel idea to treat point cloud registration as a coarse-to-fine correspondence problem. The approach is free of keypoint detection and thus keypoint repeatability issue, which enables robust registration.

- The method achieves state-of-the-art results on both indoor and outdoor benchmarks with fewer parameters. The experimental results are detailed.
- The paper is well-written.
limitations_and_societal_impact: - In the coarse scale, how the number of sample points affects the results?
- It would be better to add some examples of failure cases.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a coarse-to-fine approach to extract correspondences that can be used for a RANSAC-based point cloud registration.
They demonstrate on standardized datasets that they achieve a higher or similar inlier ratio, registration recall and feature matching recall than current state of the art with fewer parameters.
main_review: Interesting approach to employ a coarse-to-fine approach for refining matching points which improves on the two-stage RANSAC-based methods.

The paper is well-written though parts could be made clearer.


Some questions:

I'm a bit unclear on the changes you've made in the CPB block. Could you expand on how your correspondence proposal block compares to the one used in [24], is there a difference between their dustbins and your slack variables? And have you compared how the performance depends on the number of self- + cross-attention stacks you use (currently you use self- + cross- + self-attention)?

You've examined the use of a uniform sampling procedure as well as one based on the predicted scores but did you also try a voxel-based sampling procedure or similar to maintain the larger distribution of points that, as you discuss in the supplementary material, is necessary for a reliable registration?
Could this be the reason why the RR metric is similar with both the 'prob' and 'rand' sampling strategies while the IR is clearly in favor of the 'prob' approach?

The results on KITTI seems good, is the RTE and RRE calculated for the correctly registered point cloud or for all?

One comment is that the W and the B matrices used in the loss calculations probably should be defined in the paper not only verbally but similarly to what is done in the supplementary. It could also be clearer if the patch-to-patch correspondences are used or the point-to-point correspondences inside of two patches are compared since  $\tilde{S}^{(l)}$ is not defined.

Could you clarify the exact nature of the RANSAC procedure, is the GT transformation involved in selecting the correct transformation or is it only based on the number of inliers as would be done in a real-world scenario?


On the whole, it's a good paper that takes the concept of coarse-to-fine refinement from the 2D domain to the 3D case which results in a higher inlier ratio.

Update after rebuttal:

The authors addressed most issues from the reviews well. My already positive assessment remain unchanged.
limitations_and_societal_impact: Yes, as much as can be expected for the topic of this paper.
review_title: null

Meta-review: This papers introduces CoFiNet, a coarse-to-fine methods to extract hierarchical correspondences between 3D point clouds, which relaxes the need for keypoints detection. Experiments are conducted on datasets with high (3DMatch) and low (3DLoMatch) level of correspondence. 

The paper initially received three accept and two reject recommendation. The main concerns pointed out by reviewers were the lack of novelty of the proposed method, and the modest improvement compared to recent baselines (e.g. PREDATOR).
After rebuttal, the initially positive reviewers were convinced by the rebuttal and recommended acceptance. Especially, R5BJR who did a detailed review supported paper acceptance. On the other hand, Rem8a and R1Jc9 still considered the novelty insufficient and the experiments not convincing enough, and stuck on their rejection recommendation.

The AC carefully read the submission and authors' feedback. The AC considers that the idea of adapting keypoint-free coarse-to-fine refinement methods from the 2D to 3D is interesting. Although closely based on prior works, the adaptations and contributions are overall convincing, i.e. the weighting scheme and the differentiable density-adaptive matching. The experiments also show the relevance of the approach compared to recent baselines, e.g. PREDATOR. 
The AC thus recommends acceptance, but highly encourages the authors to carefully take into account and reviewers' comments and there feedback to improve the final version of the paper. Especially, the clarity of the paper should be improved to clarify and highlight the contribution and the positioning of the proposed method with respect to related works. The submission has been discussed with the senior area chair who agrees with the recommendation.

Decision: Accept (Poster)