Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the use of transformers architecture for shape matching and the method is applied to 3D point clouds. The main innovation with respect to standard attention architecture on point clouds is the weight introduced in the attention mechanism which accounts for the local density of the shape. Experiments show a clear gain of the method compared to other state of the art method, both quantitatively and qualitatively.
main_review: On the positive side, the proposed method seems to outperform state of the art methods in the field, although I am not an expert in this field.
The method does not require the use of a template which makes the method directly applicable.

My main concern is the writing style of the paper which is too loose and not accurate enough in my opinion. The first two sections are nicely written overviews discussing common generalities on attention and shape registration. Beginning of section 4 loosely describes the architecture with inaccuracies in formula (1). It is not possible to reimplement the method from reading out the main text.
The authors do not discuss the problem of long training time of transformers in their situation.

Questions: 
- Is the surface attention a particular case of classic attention in a particular formulation? Is it an inductive bias that could be learnt from enough data?
- How is the bottleneck of attention (quadratic in the input) dealt with in the context of hundreds of thousand of points ? it is not detailed enough.

Comments: 
- it seems that there are multiple typos in formula 1. The index summation $t$ is not present elsewhere in the formula, and the notations $\exp^{(x)}y$ and $\exp^{xy}$ are quite unusual. I would suggest to replace it with more standard notations.
- The running times and computational resources are difficult to find in the main text although the authors replied positively that it is stated in section 5. I have hard time to find it. It is not discussed in this section and only touched upon in the conclusion. It needs more discussion.
- Chamfer loss is said to be defined in section 4, but it is a pointer to another publication.

limitations_and_societal_impact: Limitations are not discussed enough, in particular the training time of transformers and ways to get around it.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a Transformer-based point cloud registration scheme. Given two point clouds, the target and an initial latent feature go through the encoder of a Transformer-like network, and the resulting latent vector is combined with the source and goes through the decoder to yield a transformed point cloud (that resembles the target). In the Transformer-like structure, iterative cross attention is used, which is inspired by [21]. There are some other details to handle the particular problem efficiently, e.g., density aware attention for handling the density differences in point clouds. The decoder and the loss function can also be used to refine the result with further optimization. The experiments show that the proposed method mostly achieves state-of-the-art results (sometimes with large gaps) for well-known datasets.
main_review: +- Somewhat derivative, but appropriately designed: The main ideas of the paper are inspired by other works, such as the recently popular Transformer structures and especially the iterative cross attention structure [21]. Architecture-wise, one might say that the work is somewhat derivative from [21]. But at the same time, some additional effort (e.g., density-adaptive attention) is appropriately made to handle the problem effectively. The method yields state-of-the-art performance, and sometimes, the gaps are quite large, which suggests that the proposed method is indeed effective.

** Why the Transformer for cross-modality?: The iterative cross attention inspired by [21] is originally for cross-modality. Point clouds are rather homogeneous in this sense. In my understanding, the authors are using this to easily pass on the latent feature (which can be smaller and contain essential information) from the target to the source. This can be indeed an efficient way to handle the problem. But at the same time, I think that a regular Transformer-like structure could also be used and there is a chance that the performance can be better. Why is the proposed method a better choice? More discussion is needed.

** How is the initial latent feature (latent probe) handled?: One thing that is not clearly explained in the paper is how the latent probe, which is one of the inputs of the network, is handled. The authors have provided a code as supplementary material, but providing this in the paper or in the supplementary document will be good for completeness.

[After rebuttal]
I have read all the other reviews and the authors' responses. All my previous questions have been answered. I stay with my original score.
limitations_and_societal_impact: ** What happens if we use somewhat heterogeneous point clouds for source and target? In my understanding, most of the experiments are about more-or-less homogeneous data. Discussing this for exploring the limitations of the method can be interesting.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel transformer based approach for 3D shape registration for non-rigid objects.
Transformers have been very successful in language modelling and recently also in vision applications. This paper introduces transformers for 3D shape registration which is an interesting step forward.
The approach also outperforms competing baselines.
main_review: + The approach is technically sound and makes intuitive sense.
+ Quantitative results look good.
+ Authors will release their code and data, which is appreciated.

I have minor questions regarding the work:

1. Surface attention, Eq. 1. The whole buzz around the transformer idea is centered around the fact that we no longer need to provide inductive bias to the network regarding how different components in the input are related. Eg: with CNNs the inductive bias takes the form of predefined kernel shapes that indicate to the network that some correlation exists within the receptive field of the kernel. Whereas with transformers no such assumption needs to be made. Despite this fact, why does the proposed method need explicit normalization for point density? Since SURREAL is a synthetic dataset, why can't it be augmented such that network learns to generalise to different sampling strategies. This is interesting because despite the proposed normalisation, there is a performance gap between performance on SURREAL and FAUST, which authors attribute to different surface distributions.

2. Authors mention that their method can be used to obtain correspondences between two shapes. Why is this not evaluated on the FAUST challenge?

3. Table 2: Are the baselines trained on the same data as the proposed method?

4. Authors currently restrict their problem setting to low resolution pointclouds (1000 points as input). How do the transformers perform with denser inputs?

5. Current limitations stated in the conclusion section are generic. Since this is a novel direction, it will be helpful to add a dedicated limitation and future works section with more detail. It would be good to also discuss other directions in 3D vision where transformer architectures would be beneficial.

Overall, I like the idea of using transformers for 3D shape registration and the authors also show good quantitative results.

limitations_and_societal_impact: Authors very briefly mention some limitations of their work but this can be further improved (see comments above).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary:

a) This paper proposes a transformer-based procedure to register non-rigid 3D point clouds or shapes;
b) Given a source point cloud, this paper aims to deform its coordinates to align to target point cloud;
c) Authors modifies the attention mechanism to make it aware of the point cloud density, by adding weight to the original attention score;
d) Experiments show the advantages of this paper over previous methods.


main_review: 
i) Strong points.

a) The use of the Transformer architecture for the task of non-rigid registration is new,  and seems to be the first as claimed by the authors;
b) For complete shape scans, it makes sense to modify the attention mechanism to make it aware of the point cloud local density;
c) Overall, this paper is well-written, and easy to follow.

ii) Weak points.

a) Architecture.

As noted by authors in line 109, the deep closest point [50] also use a transformer to register rigid shapes. In light of this,  it is highly beneficial if the author can provide the  following comparisons:

  * Trim the architecture of [50], add MLP to \Phi_X or \Phi_Y (Figure 2 in [50]) to estimate the deformed points. 

  * Use the above baseline to compare with your proposed architecture.

b) Partial scans.

This paper lacks experiments of partial--to-partial registration of non-rigid point clouds. Adding noise in experiment is good (Line 207).  I would expect this partial-to-partial registration are more  realistic in many real-world applications.   How will your method handle  significant portion of non-matchable outliers? 

Furthermore, the above experiment would also check the effectiveness and robustness of the proposed modified attention mechanism (Equation 1).

c) Quantitative comparisons between original attention and modified attention.

Giving qualitative comparisons in Figure 3 is good. However, I would like to see full quantitative comparisons besides the 50% claim on line 245.

d) Computation time.

Please specify the details of computation times, and compare them with respect to other methods (in Table 2).
As indicated on Line 301-302, the proposed method is computationally expensive/time consuming.  Why do you claim it is an "efficient registration" method (c.f. Line 1) ? 



limitations_and_societal_impact: Yes.
review_title: null

Meta-review: All reviewers agree that the idea of using a transformer-type architecture for (non-rigid) 3D point cloud registration is interesting and novel. While reviewers have raised questions or pointed out concerns about certain issues (e.g., being derivative), I am of the opinion that the authors sufficiently addressed those points in their answers (despite little or no serious discussion happening). I do, however, encourage the authors to follow the reviewer's advice on improving the writing style in some places and including all required clarifications as promised in the rebuttal. 
Decision: Accept (Poster)