Title: Skinned Motion Retargeting with Dense Geometric Interaction Perception
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Motion Processing

Motion Retargeting
Abstract: Capturing and maintaining geometric interactions among different body parts is crucial for successful motion retargeting in skinned characters. Existing approaches often overlook body geometries or add a geometry correction stage after skeletal motion retargeting. This results in conflicts between skeleton interaction and geometry correction, leading to issues such as jittery, interpenetration, and contact mismatches. To address these challenges, we introduce a new retargeting framework, MeshRet, which directly models the dense geometric interactions in motion retargeting. Initially, we establish dense mesh correspondences between characters using semantically consistent sensors (SCS), effective across diverse mesh topologies. Subsequently, we develop a novel spatio-temporal representation called the dense mesh interaction (DMI) field. This field, a collection of interacting SCS feature vectors, skillfully captures both contact and non-contact interactions between body geometries. By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpenetration and ensures contact preservation. Extensive experiments on the public Mixamo dataset and our newly-collected ScanRet dataset demonstrate that MeshRet achieves state-of-the-art performance. Code available at https://github.com/abcyzj/MeshRet.


Peer reviews:
comments: SUMMARY:
The paper proposes a data-driven method for character animation retargeting. The authors introduce Semantically Consistent Sensors (SCS), which are a set of surface points determined by casting rays from bones. Additionally, a Dense Mesh Interaction (DMI) field is proposed to measure the relative positions of sensors over time. They train a transformer that directly predicts motions of target characters, using source and target SCS, source motion, and source DMI. The training is conducted in an unsupervised manner, where the generated target DMI and target motion are matched to the source.

STRENGTHS:
Many previous methods are compared, and the proposed method achieves state-of-the-art (SOTA) performance according to these comparisons.

WEAKNESSES:
The contact resolution relies on clean training data without penetrations. There is no mechanism to explicitly prevent penetrations.

QUESTIONS:
Why does Contact Error depend on absolute scale? What if the retargeted character has a significantly different scale from the source character?

What if the bones have different relative lengths? To achieve the same motion, they may require different orientations (Q). How diverse are the skeletons in your dataset? It would be interesting to see how the method performs when the relative bone lengths are different.

Does the method assume that all characters in the training dataset share the same bone system? What if I would like to use the method on a character system with a different number of bones and connectivities? Do I need to collect a new dataset? How many modifications to the neural networks are required?

LIMITATIONS:
The limitation that the retargeting quality depends on inputs with clean contact is discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* The paper proposes a new motion retargeting method based on dense geometric information. This dense geometric information is gathered as dense interaction fields by pre-generated pseudo sensors, which are semantically embedded on the skin. The interaction fields are capable of representing the semantic information of different motions. Based on these ideas, the authors designed a new network to extract features from source motion, body shapes, and the newly designed interaction fields. The interaction fields are also used in the loss functions, together with GAN loss, reconstruction loss, and other loss functions. Experimental results show that the new method can effectively avoid contact loss and interpenetration while maintaining the semantics of the source motion.\n\n* The main contribution is the MeshRet method, which includes the Semantically Consistent Sensors, Dense Mesh Interaction Fields, a new network architecture, and losses designed based on the previous two modules. The authors also collected a dataset named ScanRet to improve the quality of motion retargeting datasets.\n\nSTRENGTHS:\n* The method is new, inspired by previous works on pseudo sensors and motion retargeting. There was a skeletal distance matrix in R2ET, and the DMI proposed in this paper can be seen as an improved, dense version of the distance matrix. R2ET used the distance matrix solely for motion semantics, while MeshRet uses the DMI for both semantics and geometrics.\n\n* DMI is a better representation of motion semantics compared to the distance matrix. The method has the potential to become the new SOTA. \n\n* The overall organization of the paper is good.\n\nWEAKNESSES:\nBoth qualitative and quantitative results illustrate the effectiveness of this method. However, there are some weaknesses in the experiments:\n\n* The ablation study focuses on close/far sensor pairs and the distance matrix, but does not address other aspects, such as different sensor arrangements.\n\n* The newly collected dataset, ScanRet, is only used as a source of motion. This raises the question of whether it can be used in conjunction with Mixamo.\n\n* The paper compares its results with SAN, PM-Net, and R2ET, but does not include comparisons with other works, such as the 2021 paper \"Contact-aware Retargeting of Skinned Motion.\"\n\n* Both ScanRet and Mixamo datasets are used in training. It would be interesting to see the results if only the Mixamo dataset is used.\n\nQUESTIONS:\nRefer to weaknesses part.\n\nLIMITATIONS:\n* The authors mention that one limitation is poor outcomes when the input is noisy. However, they didn't provide any failed cases or examples under noisy conditions to clarify this limitation. \n\n* Other limitations may include how to attach sensors for people with disabilities or different body shapes, and how to handle sensors for designed animated characters with more or fewer bones.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes MeshRet, a new solution that achieves geometry-aware motion retargeting across various mesh topologies in a single stage.\n\nThe author introduces semantically consistent sensors (SCS) and dense mesh interaction (DMI) field to guide the training of MeshRet, effectively achieving semantic alignment. SCS parametrically represents the relationship between points on the mesh surface and the skeleton. The DMI field expresses multi-frame motion features in a form similar to point cloud features.\n\nThe author presents a dataset called ScanRet, which is specifically tailored for evaluating motion retargeting techniques.\n\nSTRENGTHS:\nThe author introduces the concept of SCS as a feature representation and proposes the use of DMI field for learning motion retargeting alignment. These technical points demonstrate innovation.\n\nThe article is well-articulated and easy to follow, particularly in the task definition section where the input-output relationship of the overall task and the relationships between different modules are clearly represented.\n\nThe experiments conducted in the article provide evidence of the effectiveness of the proposed method.\n\nWEAKNESSES:\nOverall, there are no significant drawbacks. However, the method of selecting $K \\times L$ sensors from $S^2$ potential interacting sensors to balance complexity appears somewhat ad hoc.\n\nAdditionally, this method is only applicable to CG models and lacks robustness to real-world noise, a point the authors have also mentioned in the limitations section.\n\nQUESTIONS:\nRegarding the Semantically Consistent Sensors (SCS), how is the number S determined? Section 3.2 shows S, but later it is written as K. This part is rather confusing. Is it the total number versus the number after masking?\n\nLIMITATIONS:\nThe author has already addressed the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the issues such as jittery, interpenetration, and contact mismatches issues caused in the motion retargeting task, this paper proposes a new framework named MeshRet, directly modeling the dense geometric interactions in motion retargeting. It initially establishes dense mesh correspondences using semantically consistent sensors (SCS), and develops a spatio-temporal representation named dense mesh interaction field (DMI), which can capture the contact and non-contact interactions between body parts. The proposed model, MeshRet, achieves SOTA performance on a public dataset,Mixamo, and a newly proposed dataset, ScanRet.\n\nSTRENGTHS:\nThe proposed model, MeshRet, improves the geometric interaction-aware motion retargeting through a single pass. The proposed submodules, such as, SCS, DMI, can improve the contact and non-contact interaction semantics. \n\nThe SCS seems to be effective across various mesh topologies on different characters. \n\n The proposed model achieves SOTA performance on both a public dataset and a newly proposed dataset. The paper also provides good ablation studies, e.g., using nearest or farthest L sensors, and the idea of incorporating distance matrix loss.  \n\nThe paper proposes a new dataset specifically tailored for assessing motion retargeting, it has contact semantics and ensures smooth mesh interaction. This dataset will benefit the research society. The paper also conducts a user study to demonstrate the superiority of the proposed model.\n\nWEAKNESSES:\nThe ablation study suggests that the proximal sensor pairs are essential for minimizing interpenetration and preserving contact, whereas distal pairs provide insights into the non-contact spatial relationships among body parts. It’s a balancing problem to both maintain contact and at the same time to avoid penetration. Given a new unseen character for unknow motion retargeting, how to choose the “cls”, “far”, or “dm” version?\n\nQUESTIONS:\nGiven a new unseen character for unknow motion retargeting, how to choose the “cls”, “far”, or “dm” version? This seems to be challenging as it lacks of prior knowledge of how much to enforce contact and how much to avoid interpenetration.\n\nLIMITATIONS:\nThe model relies heavily on the accurate estimation of SCS, when the SCS extraction is affected by noisy input, complex clothes, it likes to decrease the performance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}