Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a deep-learning framework to model shape deformations, especially in the context of modifying a surface with user-defined input locations shifted to their desired locations. The motivation is that prominent methods like ARAP are restrictive in their results, partly due to pure geometric action, and that learning these deformations from data allows for more richer semantic variability in the deformations.

The authors make use of a transformer-based architecture and implement a two-staged deformation learning paradigm. Specifically, a backward deformation network deforms the shape into a canonical position. A forward deformation network, inputs this canonicalized shape, and the desired target handle information to output a deformed mesh. The training is done in two stages, with three different loss functions in action accordingly.  

The authors demonstrate their method on the DeformingThing4D-Animals dataset and show favorable results both numerically and visually in comparison to competing prior works especially ARAP and NFGP.

strengths_and_weaknesses: Strengths 

- Overall, I found this to be an interesting paper, with good motivation, well-written and well-compiled experiments, especially in the supplementary 
- Clever use of the transformer system applied to modifying shapes with target handle information for shape editing 
- In addition, I found the limitation section to be honest and well explained


Weaknesses

- The notation is a bit cumbersome and hard to keep up with. Although I did catch up after multiple reads, section 3 could be written more clearly. I specifically suggest having a clear depiction for so many notations on various point clouds -   C, O, Q, P, and T
- As rightly pointed out in the paper, the method is essentially supervised requiring dense correspondences throughout. Although not a deal-breaker this must be noted as a drawback since availability of such data is scarce. 
- I found little attempt to combine the proposed learning-based approach with the optimization approaches like ARAP. Could there be a combination of these losses and does that lead to better generalization (and possibly fewer data requirements?) 
- From an evaluation perspective, this paper is very restrictive in its demonstration and the results are only shown for 1 dataset. As a result, the transfer learning capabilities (for e.g. to 4-legged animals in TOSCA)  are largely unknown.  
- Although there is a short commentary on robustness, very little is demonstrated apart from resampling the meshes. 
- Please do contrast, explain, and/or compare with a relevant recent work: Jiang, Chiyu, et al. "Shapeflow: Learnable deformation flows among 3d shapes." Advances in Neural Information Processing Systems 33 (2020): 9745-9757.
- Typo: Line 195, is that (Qs, Qc, and Qt)?

questions: Questions 

- In the limitations, what do you mean by model rotation? like equivariance to rotations?
- Is there a reason why you achieve very decent smooth results, especially for the canonical pose visualizations in the supplementary, despite no explicit imposition of some regularity?
- How exactly do you get ground truth matching between Qs and Qt? As far as I understand Qs, and Qt are the points sampled in space and in the vicinity of the shape (and not on it), how do you then establish correspondence between points in 3D space? when I presume: only surface-to-surface matching is available? 

limitations: See Weaknesses. Overall, I found this paper interesting and well compiled and I am inclined to weigh in positively as a pre-discussion rating. However, there are some outstanding issues that need clarification and will wait for the discussion phase to make a more informed opinion.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper deals with the problem of surface deformation by training a Transformer network on point cloud. The key idea is, they adopt a canonical model or space as often used in human modeling and design a backward and forward deformation network to deform from any source model to target model.
strengths_and_weaknesses: Strengths:
First, solving the mesh deformation under the movement of some user-specified handles is a typical problem in graphics and the proposed idea is straightfoward and pretty easy to understand. So the paper is well written and easy to follow. They have demonstrated better performance on the DeformingThing4D-Animals dataset.

Weakness:
They only demonstrated the performance on the DeformingThing4D-Animals dataset. Although they have tested on unseen identities, the models in the dataset are quite similar. But the mesh deformation itself is a general problem and we shouldn't assume the models are all animals. From this perspective, the generalization ability is a big issue for the proposed method. On the other hand, the compared methods are more general and can directly apply to various kinds of models. Therefore, the comparison is just not fair. From my understanding, the network is overfitted to this dataset.
The rotation is not included, instead the network only predicts the displacement of vertices. This could be a big issue for surface deformation. The authors have mentioned this in the future work, but as a fundemental problem, I think it should have first priority to be tackled.
questions: I have some questions or confuse about the experiments:
1) In Tab.1, the L2 error of the ARAP method increased quite a bit on unseen identities. Why would happen? ARAP is not a learning method, so it shouldn't get affected whether they have seen those identities before.
2) For the learning based baseline method, NFGP, I'm wondering whether it is re-trained under this DeformingThing4D-Animals dataset.
3) What is the major technical contribution the authors want to claim? The point transformer?
limitations: The authors have mentioned their limitations in the future work which is good, but those mentioned limitions are fundamental problems that have to be dealt with. It would be a bit better to demonstrate results on general models that are not from the DeformingThing4D-Animals dataset.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a neural deformation method, which utilize the deformation priors in a large-scale dataset. This method predicts a continuous deformation field in space. The input model can be of any pose. The proposed method first deforms the input model back to the canonical space, and then deforms it to the target that satisfies the constraints given by the user. The authors propose Transformer-based Deformation Networks (TD-Nets), which learns encoder-based local deformation fields on point cloud approximations of the input mesh and outputs the deformation.
strengths_and_weaknesses: Strengths
--The method is technically sound.
--The proposed method is carefully validated and has been compared to the representative deformation method ARAP and a neural-based method NFGP.

Weaknesses
-- The description needs to be improved.

questions: -- What is the difference between the randomly sampled surface point cloud P_S and the querying spatial points Q_S? More description about the querying point set should be given. 
Also in Line 195, ‘querying spatial points (P_S, P_C, P_T)’ should be ‘querying spatial points (Q_S, Q_C, Q_T)’.
--Are equations (6),(7),(8) all used during training, or equations (6) and (7) just for the derivation of equation (8)?
--There are many modules in the proposed network, PAB, PTB, Point transformer encoder, Attentive deformation decoder, and also many features and local codes denoted as Z. I am wondering how the data goes through these modules. It would be better to have a network diagram showing how the modules are combined.

limitations: The authors have discussed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper is introduced an approach to learn mesh deformations of dynamic bodies from user-provided handles. To manipulate the source meshes into different poses, the shape deformations are learned via canonicalization, i.e., the source mesh is first represented in the canonical space, and after that, that representation is transformed to the target space. To this end, two backward and forward deformation fields are considered, that are learned by transformer deformation networks. Both quantitative and qualitative analysis are provided, as well as a comparison with two competing methods, showing promising results. Additional ablation studies are also included in the work.  
strengths_and_weaknesses: In general, the paper is clear enough, it is well written, and all the technical details are given in the document. Motivation and related work are concise, and the authors clearly show their contribution. To be honest, I do not have many issues with this submission in its current form. Next, I have some comments and questions. 

The deformation priors are learned by the use of a canonical space, i.e., the shape transformation between two arbitrary poses is divided into two steps: a backward deformation to align the source mesh to the canonical space, and a forward deformation to map the canonical space in the target deformation space. While this process increases the complexity of the neural model, as two different models need to be learned, in practice the process is strongly simplified. This idea is simple yet effective. Once the transformation models are learned, a simple 3D displacement, via a deformation field, is considered to deform the mesh. 
 
Even trivial, all operators in Eq. (1) should be defined.  
 
For training, the method needs dense correspondences for the three deformation states (canonical, target, and source). While this can be an easy task for synthetic scenarios and CAD-based models, this could be a very hard task in real scenarios, where noisy point clouds could be considered with a variation of points in the representation. 
 
The authors claim they divide the test set in 143 sequences for seen train identities, and 55 sequences for unseen ones.  In my opinion, this should be clarified in the paper. How was that done? Note that some unseen identities could be the result of a simple linear combination of seen ones, i.e., the unseen motions are really seen ones. I would like to know as the authors can guarantee that division with no additional analysis. 
 
Regarding Figure 4. The authors claim their solution produces visibly the best results. I disagree with that. That conclusion is not actually easy, after checking the corresponding image. Maybe the authors could include the corresponding ground truth mesh, or color-based representation where every color displays a different error. 
 
Observing the rest of results, the proposed method outperforms, in both quantitatively and qualitatively, the competing approaches. 
 
It is worth noting that the user-specified handles could be non-realistic or non-physical-aware. In that case, my question is: could the proposed method obtain the deformation? This could help us to interpret a bit more the learned deformation priors. My doubt is the deformation priors could be just an algebraic representation with no meaning. Could the authors help me with this question?
 
More extreme poses could be considered as well as some realistic animal meshes (for instance, capturing the full mesh by a real vision sensor that means the mesh includes noisy and partial observations). 
questions: Please, see the strengths and weaknesses section for questions and suggestions. 
limitations: The authors have addressed the limitations properly.
review_title: null

Meta-review: metareview: While some of the scores on this paper are mixed, even the negative reviews highlight the quality and interest of the work and have specific (and somewhat debatable) technical concerns.  Overall, the AE recommends accept, especially in light of the detailed and thoughtful responses during the rebuttal phase.

In the camera ready, the authors are encouraged to see if they can squeeze some of the new results (e.g., transfer learning attempt in Figure 6 and comparisons to Shapeflow) in the main body of the paper, where they're more likely to be noticed.
confidence: Certain
Decision: Accept