Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a new encoder-decoder approach for correspondence-free point cloud registration with rotations. The encoder is a combination of PointNets which provides permutation invariance and the recently proposed Vector Neurons which provides SO3 equivariance. The decoder is an implicit function (occupancy network) that takes an encoding and a query position and predicts the occupancy value. Two point clouds are registered by treating their encodings as a pair of pseudo point clouds that are already pairwise matched, so that the registration problem admits a closed form solution.  Compared to two other baselines, when the initial rotational difference is large, the proposed method shows significantly lower error.
strengths_and_weaknesses: Strengths
 
- The paper is generally well written and proposes a novel point cloud rotational registration method.
- The improvements in rotational error seem quite significant particularly for large initial rotation angles.

Weaknesses

- Many details on network architecture and training are omitted and delegated to the appendix, making some sections a bit hard to follow.
- In Table 2 and Table 3, the error seems higher than the baselines for small initial rotation angles. How can this be fixed?
summary_of_recommendation: Point cloud registration has many applications in Robotics. Though a robotics application is not shown in this paper, the network design is particularly interesting in the way it addresses permutation and rotational invariance, and builds on implicit shape representations. The improvements seem convincing overall. 
issues: - Figure 1 and its caption could be improved to be more informative. 
- Overall, some technical details on architecture design are missing section 3.
- Modern autodiff engines allow Eqn 4 to be implemented and treated as a "layer" in a network. It was not clear whether the feature alignment step is part of the training or outside it?
- In Table 2 and Table 3, the error seems higher than the baselines for small initial rotation angles. How can this be fixed?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a correspondence-free Deep Neural Network (DNN)-based method for point cloud registration. The network exploits the "Vector Neurons" architecture, a SO(3)-equivariant DNN for point cloud processing, to extract a global feature vector from an input point cloud. A decoder network is used to reconstruct the original input from the global feature and a query position. Finally, the rotation between two point clouds is estimated by comparing their global feature vectors in a closed-form solution.
strengths_and_weaknesses: Strengths:
+ The paper is well written, well organized, and in general easy to read.
+ The proposed approach is very interesting, and addresses a relevant problem.
+ Different methods for point cloud registration are well presented and discussed.
+ The results show great improvement over existing correspondence-free DNN-based approaches, especially when the initial rotation between the point clouds is large.

Weaknesses:
- The paper does not provide a strong argument or evidence in support of using correspondence-free over correspondence-based methods. The argument that "matching (point) pairs may not exist in the input point clouds" (line 210) is weak, since robust estimators such as RANSAC can effectively overcome this issue. Recent correspondence-based point cloud registration methods [1, 2, 3] that combine DNN-based feature extractors with RANSAC achieved outstanding results and robustness to initial misalignment. A strong evidence, such as a comparison of the proposed approach with recent correspondence-based methods, or at least a strong argument on why correspondence-free methods should be preferred over correspondence-based ones, should be provided.
- The proposed method only estimates the rotation between the two point clouds, while most existing approaches estimate both rotation and translation.
- The proposed method is evaluated only on the synthetic ModelNet dataset, an experimental evaluation on a real-world dataset, such as 3DMatch or KITTI/KITTI-360, would give a better insight into the possible applications of the approach.

[1] Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L., & Tai, C. L. (2020). D3feat: Joint learning of dense detection and description of 3d local features.

[2] Cattaneo, D., Vaghi, M., & Valada, A. (2021). LCDNet: Deep Loop Closure Detection and Point Cloud Registration for LiDAR SLAM.

[3] Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., & Schindler, K. (2021). PREDATOR: Registration of 3D Point Clouds with Low Overlap.
summary_of_recommendation: Although the proposed method is interesting, a strong evidence supporting the claim that correspondence-free methods should be preferred over correspondence-based ones is missing. Therefore, my recommendation is weak reject.
issues: - A strong evidence, such as a comparison of the proposed approach with recent correspondence-based methods, or at least a strong argument on why correspondence-free methods should be preferred over correspondence-based ones, should be provided.
- An experimental evaluation on a real-world dataset, such as 3DMatch or KITTI/KITTI-360, would give a better insight into the possible applications of the approach.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to do point cloud registration in learned feature space. The features are extracted using a neural network that retains SO(3)-equivariance in points' features. Then the registration is done in the feature space and then propagated to every point. Experiments are done on the point clouds of the synthetic dataset ModelNet40. 
strengths_and_weaknesses: Strength: 

- The idea of doing point cloud registration in feature space is novel. This could be inspiring for the rest of the research community.
- The experiment results seem to be strong and outperform the baselines by a significant margin.

Weakness:
- The extracted deep features could be very unstable, especially when there is occlusion in the point cloud.
- The authors only evaluated two simple real-data effects: random noise in the point cloud and different densities. However, these two cases are both way too simple and easy, because if the point occupancy grid is used as representation, both cases do not change occupancy much. One very important experiment is missing: registration of $\textit{partial}$ 3D point scans, especially when the two scans have different partial pieces. This is a case where the proposed algorithm can fail completely because partial scans can dramatically change the deep features, therefore the registration step in the feature space can fail easily. The authors have to evaluate this case to support their claim in the paper.
- Since the authors used occupancy grid as point representation, the proposed method is unable to deal with large-scale point scans such as depth scans of the entire room, because the grid resolution will be severely limited.
- What is the loss function used to train the deep features? This is not clear from the paper.

Some other issues on the paper:
- Fig 2. visualization of equivariant features is unnecessary. 
- The related work section is too long. Most of the sentences in this section can be more concise.
- "Occupancy Network" is first mentioned in line 108, but is not cited until line 141. This is very confusing.
- In Fig 1., the author mentioned the output to be "Occupancy Field", but where is it defined in the text?
- How is "Rotation Error" defined in Tables 1, 2, and 3? Is it rotation angle difference, in terms of radians or degrees? If so, why isn't rotation axis difference evaluated? Or it is some combination of rotation axis and angle? It's not clear from the paper.
 

summary_of_recommendation: I'm not convinced that the proposed approach is actually useful, especially because the authors used point occupancy as representation. This results in the proposed approach being unable to deal with large-scale scans.

Besides, the presentation of this paper is very poor. A lot of necessary details are missing from the paper and are very hard to follow. It seems that this paper is written in a rush without careful check of logic and soundness.
issues: The authors can address the items under "Weakness" and "Some other issues" in my main review section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper describes a correspondence-free method for pointcloud registration.  The main idea is to learn an SO(3)-equivariant (and permutation-invariant) mapping from the (entire) input pointcloud into an abstract feature space.  The fact that the learned feature mapping is SO(3)-equivariant means that the desired rotation R aligning the pointclouds can be gotten as the optimal rotation aligning their *features*; this latter problem admits a simple direct closed-form solution using Horn’s method.

In detail, the paper shows how to “upgrade” the (permutation-invariant) PointNet to be SO(3)-equivariant by replacing its layers with the [SO(3)-equivariant] Vector Neurons proposed in reference [5].  This modified PointNet is then used as the encoder in an encoder-decoder network (based on OccupancyNet) that is designed to learn a (global) implicit shape descriptor for the pointcloud.  The idea behind the use of an implicit shape representation is to account for small nuisance variations in the sampled pointclouds (due to both measurement noise, and the fact that in general two scans of the same object will not sample *exactly* the same sets of points on the object).  The overall result is a network design that learns an implicit shape representation that is SO(3)-equivariant *by construction*.

Experimentally, the paper evaluates the proposed method in comparison with PCR-Net and Feature-Metric Registration (FMR) [two correspondence-free deep pointcloud registration methods] on three tasks requiring the registration of pairs of pointclouds sampled from CAD models taken from the ModelNet40 dataset.  Overall, the results show that (as expected) the accuracy of the registrations computed by the SO(3)-equivariant approach proposed in this work is *independent* of the magnitude of the rotation aligning the input pointclouds.  This is *not* true for PCR-Net and FMR, whose performance significantly degrades with increasing alignment angle because they make use of (local) iterative refinement to align the input pointclouds.  In absolute terms, the proposed method performs very well in the presence of large-magnitude rotations and Gaussian measurement noise, although its performance does seem to degrade significantly when comparing two pointclouds of differing density.
strengths_and_weaknesses: My overall impression of this work is very positive: the problem under consideration is certainly well-motivated, the proposed approach is very elegant, and the empirical results show that it achieves the desired result [namely, SO(3)-equivariance] -- as in fact it must, since the proposed method inherits this property *by construction*.  I would be very interested in using this method in my own work (which I think is about the highest praise a reader can bestow).  

The paper as a whole is also remarkably well-written; I particularly appreciated its clear and concise review of equivariance and Vector Neurons (the latter of which I had not seen before) and the thorough review and contextualization of related work in Section 5.

In my view the paper’s only substantial weakness (which it already acknowledges) is the significant degradation in performance in aligning pointclouds of different sampling densities.  I do not completely agree with the paper’s assertion (in lines 179-180) that pointcloud registration errors of ~20 degrees are “practical” -- I can easily imagine applications (e.g. object manipulation or localization) where such errors would be too large for the returned result to be practically useful.  [I would also expect this different-density scenario to arise fairly regularly, since it simply corresponds to scanning the same object from two different distances.]  Nevertheless, I still think that the SO(3)-equivariant design reported in this work is a very solid technical contribution that is likely to be of interest to the robotics community, despite this specific shortcoming.

A couple of other minor comments:

* I notice that in the results reported in Table 5.3, FMR seems to perform very well as long as the initial rotational offset is less than ~30 degrees.  Given that the proposed method is capable of generating registrations accurate to within about 20-25 degrees across *all* initial offsets, perhaps one way of improving robustness in the differing-densities case would be to treat the proposed method as an *initialization*, and then apply a subsequent (finer) refinement using FMR or something similar?  (Admittedly this lacks the elegance of having a single end-to-end SO(3)-equivariant method, but as a practical approach it might be very effective at addressing the (uniformly) higher errors reported in the differing-density case.)

* In lines 104-106, the paper mentions that this method is among the first to address the problems of measurement noise and inexact correspondences between the input pointclouds in deep correspondence-free registration methods.  If this is correct, I would encourage the authors to emphasize this point more strongly (e.g. in the abstract and introduction), as it strikes me as an additional significant feature of the method.


summary_of_recommendation: My recommendation for accepting this paper is based upon the following considerations:

* It addresses a fundamental problem in robotic perception (pointcloud alignment), which is likely to be of interest to a very broad subset of the community
* It develops a solution to this problem that provides a novel set of capabilities (DNN-based correspondence-free poincloud registration that is *also* SO(3)-equivariant) with respect to prior work
* The design process that leads to the proposed method is both mathematically rigorous and technically very interesting, and is based on principles [e.g. geometric analysis of the design of feature maps, equivariance, etc.] that can be generalized far beyond the specific problem and algorithm considered in this work.  I expect that many readers would find this independently interesting, and I could easily imagine that the presentation of these ideas in a nice "case study" like this paper could catalyze their application to many other problems.

Finally, I want to mention that I rated this paper's impact at "3" only because the criteria for 4 strikes me as unreasonably high (even the fraction of *published* papers that have "a major impact in robotics or machine learning" is vanishingly small -- well under 5%), and there was no intermediate rating between "incremental" and "major".  This should in no way be construed as a criticism of the paper's merit -- it's a very fine contribution, and I would like to see it accepted.
issues: The only issue that I would specifically encourage the authors to address during the response and revision period is to try to improve the performance of the method on the differing-density case: I think this would greatly strengthen the paper, as well as (more importantly) making the method significantly more useful.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for registering a pair of point clouds without correspondences (data association). The key idea is to use the so-called vector neurons to embed the original 3D point clouds into a 3D feature space that is equivariant to 3D rotations, and invariant to permutations. Precisely, if the original point cloud is rotated by an arbitrary rotation and permuted by an arbitrary permuation, then the learned features will be rotated by the same rotation, but will not be affected by the permutation. Using such an architecture, estimating the rotation can be performed in feature space using Horn's method in closed form.

To train the features, the paper proposes to use an encoder-decoder architecture with the encoder being the vector neurons and the decoder being an implicit shape representation for computing the losses more robustly. The proposed method is tested on ModelNet40, compared with two baselines, and shown to achieve generally better performance in the presence of noise and density variances.
strengths_and_weaknesses: Strengths:
- The SO(3)-equivariant network using vector neurons is a very interesting idea. I like this idea because, as written in the summary, it discards the permutation/ordering of the point cloud, but preserves the rotation so that registration can be done in the feature space.  In some sense, I think the network can be seen as simultaneously learning keypoints and correspondences, because the output features of the network, Q and Q' as in eq. (3), can be seen as a set of corresponding keypoints, where the keypoints do not necessarily belong to the original point cloud (kind of being hallucinated by the network). But I am OK with the authors calling this method "correspondence-free".

- This paper is very easy to understand, with very few typos. The preliminaries and methodology sections well explain the key novelty of the paper.

Weaknesses
- The major weakness of this paper is that experiments are too toy and simple to convince me that the proposed method is useful for any practical robotics or computer vision applications. The authors write in conclusions that "generalizing to deal with translation" and "generalizing to large-scale problems" are two directions of future work. However, I cannot agree with accepting a paper which proposes a new method without extensive evaluation on practical problems, and I think those two generalizations must be done before this paper can be accepted.  
- To be more specific, firstly, the ModelNet40 dataset is a toy dataset that only contains single objects without complex cluttered scenes typically encountered in real applications. Second, the experiments in this paper do NOT consider point clouds with partial overlapping, which is too ideal to be useful. In fact, Table 3 shows that the proposed method already gives poor performance under different densities. So I have reason to suspect this method will perform even worse under partial overlapping and cluttered scenes. Third, I think it is also important to compare the proposed method to correspondence-based methods.  Indeed, [Ref1] (see Table 1) shows that strong correspondence-based approaches like TEASER [Ref2] already achieves almost 100% recall on ModelNet40 and the problem without partial overlapping is considered more or less a solved problem.  Therefore, I suggest to test on the more realistic 3DMatch dataset [Ref3], with comparison to more baseline approaches to demonstrate the usefulness of the proposed method.
- I think Section 3.2 Deep implicit representation learning should be presented in more details for the reader to fully understand how the loss function is computed.

Technical suggestions
I want to provide a few known ideas that may help the proposed method.
- It is shown in [Ref2] that robust point cloud registration can be solved in a decoupled fashion (scale, rotation and then translation) by designing invariant measurements. Therefore, the idea in [Ref2] already gives one approach for generalizing the proposed method to handle translations.

- As I mentioned, the proposed method can be seen as simultaneously learning keypoints and correspondences. Therefore, using this correspondence-based perspective, it is straightforward to apply robust estimation methods to register the learned features. Precisely, currently the authors use Horn's method to estimate rotation from Q and Q' in eq. (3). However, Horn's method is known to be sensitive to outliers (wrong matches in Q and Q'). So I believe the authors could try using a robust method to estimate R from Q and Q', such as RANSAC (available in Open3D) or TEASER [Ref2], which perhaps can help get more accurate estimates.


[Ref1] Yuan, Wentao, Benjamin Eckart, Kihwan Kim, Varun Jampani, Dieter Fox, and Jan Kautz. "Deepgmr: Learning latent gaussian mixture models for registration." In European Conference on Computer Vision, pp. 733-750. Springer, Cham, 2020.

[Ref2] Yang, Heng, Jingnan Shi, and Luca Carlone. "Teaser: Fast and certifiable point cloud registration." IEEE Transactions on Robotics 37, no. 2 (2020): 314-333.

[Ref3] Zeng, Andy, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. "3dmatch: Learning local geometric descriptors from rgb-d reconstructions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1802-1811. 2017.

summary_of_recommendation: I think this paper proposes an interesting and novel technical idea that may have good impact.

However, the experiments are significantly lacking to convince me the practical value of the proposed method. In its current form, the paper cannot be accepted. 

**Post rebuttal**

The authors added new experiments on 7Scenes, which addressed my major concern. Although the experiments are still not particularly strong, considering the idea of this paper is interesting, I'd like to increase my score to Weak accept.
issues: See weaknesses.
review_title: null

Meta-review: The paper has obtained 5 expert reviews, which is above average for CORL and will hopefully provide useful inputs to the authors. 

The reviewers appreciated the following aspects: The paper is well written, proposes a novel and elegant registration method, and the results confirm such a method has lower errors under large rotations. The reviewers particularly appreciated the idea of mapping the problem into a rotation-equivariant and permutation-invariant feature space where registration becomes straightforward.

The main weaknesses are:
- For the paper to have an impact on the robotics community, the experiments should include evaluations on real-world datasets (such as 3DMatch and KITTI), and evaluate on partial/occluded point clouds. It is also recommended to include comparisons against correspondence-based methods (LCDNet, PREDATOR, TEASER or similar). 
- The approach seems to have slightly higher errors (compared to baselines) when the initial rotation angles are small. While this is not necessarily a problem, a discussion on why this is the case and how to resolve this issue is desirable. Similarly, the performance of the approach quickly degrades with the sampling density, which may limit real-world applicability. 
- The approach may not scale to large scenes due to the discretization and the size of the grid map. A discussion about scalability is in order.
- The discussion is hard to follow in some points (e.g., details on the architecture design in Section 3). This point should be easy to address.

*Post-rebuttal comments*:
The reviewers and area chair have carefully read the revised paper and rebuttal and there has been an interesting discussion about the merits of the paper. The reviewers mostly agree that the paper has an interesting contribution and advances the literature on deep-learning-based correspondence-free methods. The reviewers and AC also appreciated the additional experiments: they definitely strengthened the paper. On the downside, one reviewer is still concerned that the 7scenes evaluation is not fully satisfactory since it might still hide issues in the registration of point clouds with low overlap. While I share the same concern, I think no paper is perfect and I consider this paper interesting for the CoRL audience. A second issue that I consider very important is to provide a more transparent contextualization with respect to correspondence-based methods: while I agree that learning-based correspondence-free methods are less mature, it is important for the paper to acknowledge that, and to provide suitable pointers to the non-expert reader. Similarly, I would have liked to see a comparison against non-learning-based correspondence-free methods, such as ICP and variants. Since these are relatively minor changes, I trust the authors will add a few sentences to clarify these points in the final submission, therefore I vote to accept this paper. 

Minor comments:
- In the literature review, including ICP in the "correspondence-based" methods feels wrong. Correspondence-based methods take the correspondences as input or establish correspondences using descriptors. It should be included in the correspondence-free methods instead.
- Please rephrase the sentence “To our knowledge, we are one of the first to report registration results under this setting in correspondence-free methods” to mention “learning-based correspondence-free methods” (going back to the previous point, ICP has been extensively used for correspondence-free registration).




Decision: Accept (Poster)