Skinned Motion Retargeting with Dense Geometric Interaction Perception

Authors: zijie  ye
jia-wei  liu
jia  jia
shikun  sun
mike zheng shou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 07:45:58

Abstract: Capturing and maintaining geometric interactions among different body parts is crucial for successful motion retargeting in skinned characters. Existing approaches often overlook body geometries or add a geometry correction stage after skeletal motion retargeting. This results in conflicts between skeleton interaction and geometry correction, leading to issues such as jittery, interpenetration, and contact mismatches. To address these challenges, we introduce a new retargeting framework, MeshRet, which directly models the dense geometric interactions in motion retargeting. Initially, we establish dense mesh correspondences between characters using semantically consistent sensors (SCS), effective across diverse mesh topologies. Subsequently, we develop a novel spatio-temporal representation called the dense mesh interaction (DMI) field. This field, a collection of interacting SCS feature vectors, skillfully captures both contact and non-contact interactions between body geometries. By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpenetration and ensures contact preservation. Extensive experiments on the public Mixamo dataset and our newly-collected ScanRet dataset demonstrate that MeshRet achieves state-of-the-art performance. Code available at https://github.com/abcyzj/MeshRet.

Main paper: 
 introduction: 
 Skinned character animation is prevalent in virtual reality [16] , game development [21] , and various other fields. However, animating these characters often presents significant challenges due to differences in body proportions between the motion source and the target character, leading to issues such as loss of motion semantics, mesh interpenetration, and contact mismatches. Consequently, motion retargeting is essential to adjust for these discrepancies in body proportions. This process is crucial for maintaining the integrity of the source motion's characteristics in the animation of the target character.Motion retargeting presents challenges due to the complex interactions among character limbs and the wide range of body geometries. Accurately preserving these interactions is crucial, as incorrect interactions can result in mesh interpenetration and contact mismatches. Prior research has typically addressed these interactions from two perspectives: skeleton interactions and geometry corrections. Early methods [1, 29, 15] employ cycle-consistency to implicitly align skeleton interaction semantics, yet they do not address the complexities of geometric interactions between different body parts. Villegas et al. [28] introduced mesh self-contact modeling; however, their approach does not extend The Proposed MeshRet Contradiction Figure 1 : Comparison with the existing method. Contrary to the earlier retargeting-correction approach [32] , which suffer from internal contradictions leading to interpenetration, jitter, and contact mismatches, our pipeline leverages the DMI field to accurately model complex geometric interactions.results. Consequently, recent studies have explored learning-based motion retargeting algorithms. Jang et al. [11] trained a motion retargeting network using a U-Net [26] architecture on paired motion data. Villegas et al. [29] introduced a recurrent neural network combined with cycle-consistency [35] for unsupervised motion retargeting. Lim, Chang, and Choi [15] propose to learn frame-by-frame poses and overall movements separately. Aberman et al. [1] develop differentiable operators for cross-structural motion retargeting among homeomorphic skeletons. However, these methods generally neglect the geometry of characters, leading to frequent contact mismatches and severe mesh interpenetrations.Geometry-aware motion retargeting Previous studies have generally processed character geometries through two approaches: contact preservation and interpenetration avoidance. Lyard and Magnenat-Thalmann [19] developed a heuristic optimization algorithm to maintain character selfcontact, while Ho, Komura, and Tai [9] proposed to maintain character interactions by minimizing the deformation of interaction meshes. Ho and Shum [10] introduced a spatio-temporal optimization framework to prevent self-collisions in robot motion retargeting. Jin, Kim, and Lee [12] employed a proxy volumetric mesh to preserve spatial relationships during retargeting. Subsequently, Basset et al. [4] combined both attraction and repulsion terms in an optimization-based method to avoid interpenetration and preserve contact. However, these methods necessitate per-vertex correspondence and involve costly optimization processes. More recently, Villegas et al. [28] attempted to retarget skinned motion through optimization in a latent space of a pretrained network, although their method does not accommodate non-contact interactions. Zhang et al. [32] implemented a two-stage pipeline that initially aligns skeleton interaction semantics and subsequently corrects geometric artifacts. Nevertheless, the inherent conflict between maintaining skeleton interaction semantics and correcting geometry often results in jittery movements and imprecise contacts. In a later study, Zhang et al. [30] added a stage that aligns visual semantics using a visual language model, but this approach requires extensive pair-by-pair fine-tuning due to the loss of spatial information when projecting 3D motion into 2D images.Existing geometry-aware motion retargeting methods either require expensive optimization or employ multi-stage strategies for skeleton and geometry semantics, resulting in a contradiction between stages that often leads to unsatisfactory results. In contrast, our method processes both contact and non-contact semantics using a dense mesh interaction field in a single stage.3 Method 
overview: 
 We introduce a novel geometric interaction-aware motion retargeting framework MeshRet, as illustrated in Figure 2 . Unlike previous methods that either overlook character geometries [1, 29, 15] or apply geometry correction after skeleton retargeting [32, 30] , our framework directly addresses dense geometric interactions with the Dense Mesh Interaction (DMI) field. This provides a detailed representation of the interactions within skinned character motions, preserving motion semantics by preventing mesh interpenetration and ensuring precise contact preservation. Task definition Given the source motion sequence m A , and the geometries G A and G B of the source and target characters in their T-poses, our objective is to generate the motion m B for the target character. This process aims to retain essential aspects of the source motion, including its semantics, contact preservation, and the avoidance of interpenetration.ℱ ! ℱ " ℱ ! Motion Encoder ℱ ! ℱ " 𝐐 # # 𝐐 $ 𝐃 # # 𝐃 $ 𝐌 %&' 𝐌 ()( ℱ * 𝐃 # 𝐃 $ 𝐒 # 𝐒 $Following the definition of the task, our MeshRet model initially derives Semantically Consistent Sensors (SCS) S ∈ R S×4×3 , which provide dense geometric correspondences essential for the retargeting process, where S = F s (G). S captures the sensor location and the sensor tangent space matrix, facilitating an enhanced perception of the geometry surface. Subsequently, we conduct sensor forward kinematics (FK) and pairwise interaction extraction to generate the source DMI fieldD A = F d (m A , S A ), where D A ∈ R T ×K×L×P .Here, K is the number of SCS in the DMI field, L represents a hyper-parameter of feature selection, and P indicates the feature dimension of the DMI. Lastly, a transformer-based network [27] ingests m A , D A , S A , and S B , and predicts a target motion sequence m B that aligns with the target character's geometry and the source DMI field. The entire pipeline is denoted as follows:EQUATIONTo facilitate dense geometric interactions, our MeshRet framework necessitates establishing dense mesh correspondence between source and target characters. Previous studies have typically derived correspondence from vertex coordinates [33] , virtual sensor [31] or through a bounding mesh [12] ; however, these methods are confined to template meshes sharing identical topology, such as MANO [25] or SMPL [18] . Villegas et al. [28] suggested determining vertex correspondence using nearest neighbor searches on predefined feature vectors. Nevertheless, this approach often lacks precision and brevity, resulting in inaccurate contact representations and substantial optimization burdens.In this study, we introduce Semantically Consistent Sensors (SCS) that are effective across various mesh topologies while ensuring precise semantic correspondence. Our approach draws inspiration Joints (by 𝑏)Sensor Tangent Space from the Medial Axis Inverse Transform (MAIT) [22] . We conceptualize the skeleton bones of each character as approximate medial axes of their limbs and torso. For each bone, a MAIT-like transform is applied to generate the corresponding SCS. This involves casting rays from the bone axis across a plane perpendicular to it. The origin parameter l and direction parameter ϕ of the rays, combined with the bone index b, establish the semantic coordinates of the SCS. The semantic coordinates describe connection between the sensor and the skeleton bones. A sensor is deemed valid if its ray intersects the mesh linked to the bone; otherwise, it is considered invalid. Through this method, we establish a dense geometric correspondence based on sparse skeletal correspondence. The procedure for deriving SCS is illustrated in Bone MatrixTo effectively represent the interactions between character limbs and the torso, we have developed the DMI field. Based on SCS detailed in Section 3.2, the DMI field comprehensively captures both contact and non-contact interactions across different body part geometries. Utilizing the DMI field allows for dense geometry interaction-aware motion retargeting, thereby eliminating the need for a geometry correction stage.Sensor forward kinematics For a given motion sequence, denoted as m, we initially conduct forward kinematics (FK) on S to derive sensor features S 1:T ∈ R T ×S×4×3 . Each S t encompasses the locations and tangent matrices for S sensors at frame t. The FK transformation for an individual sensor is expressed as:EQUATIONwhere G n (Q t ) ∈ SE(3) is the global transformation matrix for bone n, derived from its local rotation matrix, and ω(p i ) n represents the linear blend skinning (LBS) weight for sensor s i , determined through barycentric interpolation of its adjacent mesh vertices.Pairwise interaction feature Next, we model the geometric interactions as pairwise interaction features between sensors. Ideally, for each frame, we obtain a comprehensive DMI field, D t , representing pairwise vectors across K 2 sensor pairs:EQUATIONEQUATIONwhere t i ∈ R 3×3 is the tangent matrix if sensor i, and d t,i,j represents the relative position of target sensor j in the tangent space of observation sensor i. D t is composed of two components: the relative position of the sensor pair and the semantic coordinates of both the observation and target sensors. The use of semantic rather than spatial coordinates is essential, as it obviates the need for actual sensor positions, thereby making DMI suitable for motion retargeting applications.However, D t ∈ R S×S×P exhibits quadratic growth with respect to S because it includes S 2 sensor pairs, rendering it impractical when managing thousands of sensors. To address this, we implement two sparsification strategies for D t . Initially, we restrict interactions to critical body parts only, such as arm-torso, arm-head, arm-arm, and leg-leg, rather than between all sensor pairs, thereby restricting our focus to K observation sensors. Subsequently, for each observation sensor, we select L target sensors from each relevant body part, where L is a predetermined hyper-parameter. Specifically, we empirically choose L/2 nearest and L/2 furthest target sensors. We find that proximate sensor pairs are crucial for minimizing interpenetration and maintaining contact, while distant pairs delineate the overall spatial relationships between body parts, as shown in Figure 3 . These strategies lead to the formulation of the final DMI field D ∈ R K×L×P , with selected sensor pairs indicated by the sparse DMI mask M src ∈ R S×S shown in Figure 2 .To avoid the conflict between skeleton interaction and geometric correction, the proposed MeshRet employs the DMI field to model geometric interactions directly. As shown in Figure 2 , MeshRet initially extracts the DMI field D A from the source motion sequence m A , as described in Section 3.3. The field D A encapsulates interactions among various body parts within the source motion, encompassing both contact and non-contact interactions, further depicted in Figure 3 . The DMI field, composed of sensor pair feature vectors, possesses the unordered characteristics of a point cloud.Consequently, we implement a PointNet-like architecture [24] for our DMI encoder, which is divided into two components: the per-sensor encoder and the per-frame encoder. Given D A ∈ R T ×K×L×P , the per-sensor encoder initially processes it as T * K separate point clouds, producing representations H s A ∈ R T ×K×Dmodel for each observation sensor, where D model denotes the feature dimension. Subsequently, the per-frame encoder generates per-frame representations H f A ∈ R T ×Dmodel by encoding these T point clouds.Since DMI field D A lacks geometric information about characters, we introduced a geometry encoder F g to extract geometric features from their SCS. For each sensor, we form a feature vector by concatenating its rest-pose feature s i with its semantic coordinates (b i , l i , ϕ i ). The resultant geometric features are represented as C A ∈ R SA×C for character A and C B ∈ R SB×C for character B. The semantic coordinates of sensors act as intermediaries linking the DMI field to character geometry. The geometry encoder employs a PointNet-like architecture [24] to transform the geometric features C into a geometric latent code H g ∈ R Dmodel .The transformer-based retargeting network processes input features including the source DMI feature H f A , source joint rotation Q A , source geometry latent H g A , and target geometry latent H g B . Specifically, the encoder processes H f A and H g B , while the decoder processes Q A and H g A . The latents H g A and H g B serve as the initial tokens in the sequence, enabling both the encoder and decoder to operate over a sequence of length T + 1. The output sequence's final T frames are represented as QB .Due to the lack of paired ground-truth data, we employ the unsupervised method described by Lim, Chang, and Choi [15] . Our network utilizes four loss functions for training: reconstruction loss, DMI consistency loss, adversarial loss, and end-effector loss. Supervision signals are derived from the source motion. We maintain geometric interactions by aligning the source DMI field D A with the target DMI field DB . The target DMI field DB is generated by first applying sensor forward kinematics to QB , followed by selecting sensor pairs using the target sparse DMI mask M tgt ∈ R S×S . This mask, M tgt , is derived by excluding invalid sensors of the target character from M src . The DMI consistency loss is quantified as the cosine similarity loss between pair-wise relative positions in DB and D A :EQUATIONwhere c(k, l) takes the value 1 if sensor pair (k, l) is valid in both M src and M tgt , and 0 otherwise. The reconstruction loss serves as a regularization mechanism to minimize motion alterations during retargeting, defined as follows:L rec = || QB -Q A || 2 2 .(6) To facilitate realistic motion retargeting, a discriminator, denoted as δ(•), is employed. The adversarial loss is subsequently defined as:EQUATIONWe observed that the global orientation of end-effectors significantly influences user experience. Consequently, we introduced an end-effector loss to promote consistent orientations of end-effectors in the retargeted motion.EQUATIONwhere R(•) transforms local joint rotations into global rotations for joint i along the kinematic chain and X represents the set of end-effectors. Our MeshRet is trained by:EQUATION4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We removed textures from ScanRet to help preserve anonymity of participants. We acquired consent from every participant. All the participants are fairly paid.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
settings: 
 Datasets We trained and evaluated our method using the Mixamo dataset [2] and the newly curated ScanRet dataset. We downloaded 3,675 motion clips performed by 13 cartoon characters from the Mixamo dataset contains, while the ScanRet dataset consists of 8,298 clips executed by 100 human actors. Notably, the Mixamo dataset frequently features corrupted data due to interpenetration and contact mismatches. To overcome these issues, we created the ScanRet dataset, which provides detailed contact semantics and improved mesh interactions, with each clip being scrutinized by human animators. The training set comprises 90% of the motion clips from both datasets, involving nine characters from Mixamo and 90 from ScanRet. Our experiments tested the motion retargeting capabilities between cartoon characters and real humans, aligning closely with typical retargeting workflows. During inference, we adopted four data splits based on character and motion visibility: unseen character with unseen motion (UC+UM), unseen character with seen motion (UC+SM), seen character with unseen motion (SC+UM), and seen character with seen motion (SC+SM), as delineated by Zhang et al. [32] . We present the average results across these splits. Additional details available in Appendix A.The hyper-parameters λ rec , λ dmi , λ adv , λ ef , and L were empirically set to 1.0, 5.0, 1.0, 1.0, and 20, respectively. We use {0, 1, • • • , N body -1} × {0, 0.25, 0.5, 0.75} × {0, 0.5π, π, 1.5π} as the SCS semantic coordinates set, where N body = 18 is the number of body bones and × represents the Cartesian product. We employed the Adam optimizer [13] with a learning rate of 10 -4 to optimize our network. The training process required 36 epochs. For further details, please refer to Appendix C.We assess the effectiveness of our method through three metrics: joint accuracy, contact preservation, and geometric interpenetration. Joint accuracy is quantified by calculating the Mean Squared Error (MSE) between the retargeted joint positions and the ground-truth data provided by animators in ScanRet. This analysis considers both global and local joint positions, normalized by the character heights. Contact preservation is evaluated by measuring the Contact Error, defined as the mean squared distance between sensors that were originally in contact in the source motion clip. Geometric interpenetration is determined by the ratio of penetrated limb vertices to the total limb vertices per frame. Further details are available in Appendix B. during training. Most baseline methods, except R 2 ET [32] , fail to consider the geometry of characters, leading to significant geometric interpenetration and contact mismatches. Unlike these methods, R 2 ET [32] includes a geometry correction phase after skeleton-aware retargeting. However, this creates a conflict between the two stages, resulting in oscillations in R 2 ET's outcomes, which manifest as alternating contact misses and severe interpenetrations, as shown in the first two rows. Additionally, these oscillations appear variably across different frames within the same motion clip, producing jittery motion, as illustrated in Figure 1 and Figure 8 . A further limitation of R 2 ET is its neglect of hand contacts. In contrast, our method employs the innovative DMI field to preserve such detailed interactions, such as those observed in the "Praying" pose in the third row. Demo video available. that display incorrect interactions. Conversely, our method modifies the entire motion sequence, resulting in a higher MSE compared to the Copy strategy. Nevertheless, MSE remains a valuable auxiliary reference. In comparison to PMnet [15] , R 2 ET [32] , and SAN [1] , our method achieves MSE reductions of 65%, 29%, and 8%, respectively. These results demonstrate that our approach more closely aligns with the outputs produced by human animators.As shown in Table 1 , PMnet [15] and SAN [1] , exhibit high interpenetration ratios and contact errors due to their neglect of character geometries. R 2 ET [32] effectively reduces interpenetration through a geometry correction stage; nonetheless, it still encounters high contact errors stemming from conflicts between the retargeting and correction stages. Our approach explicitly models geometry interactions and thereby achieves low contact error and penetration ratio, illustrating the effectiveness of our proposed MeshRet in generating high-quality retargeted motions with detailed contact semantics and smooth mesh interactions. Additionally, we observe that retargeting using the mixed Mixamo+ dataset is more challenging than with the ScanRet dataset, attributable to significant body shape variations between cartoon characters and real person characters.We conducted ablation studies to demonstrate the significance of pairwise interaction feature selection and the implementation of DMI similarity loss. Initially, we evaluated the performance of a model trained exclusively with the nearest L sensor pairs, denoted as Ours cls , and another model trained solely with the farthest L sensor pairs, referred to as Ours f ar . As indicated in Table 1 and Figure 5 , Ours f ar compromises contact semantics and leads to significant interpenetration, while Ours cls also exhibits inferior performance. This outcome suggests that proximal sensor pairs are essential for minimizing interpenetration and preserving contact, whereas distal pairs provide insights into the noncontact spatial relationships among body parts. Further, we investigated the effect of incorporating a distance matrix loss, as proposed by Zhang et al. [32] , on our sensor pairs, designated as Ours dm .The results imply that the distance matrix loss fails to yield meaningful supervisory signals, likely because distance is non-directional and insufficient to discern the relative spatial positions among numerous sensors. We conducted a user study to assess the performance of our MeshRet model in comparison with the Copy strategy, PMnet [15] , SAN [1] , and R 2 ET [32] . Fifteen sets of motion videos were presented to participants, each consisting of one source skinned motion and five anonymized skinned results. Participants were requested to rate their preferences based on three criteria: semantic preservation, contact accuracy, and overall quality. Users were recruited from Amazon Mechanical Turk [3] ,resulting in a total of 600 comparative evaluations. As indicated in Table 2 , approximately 81% of the comparisons favored our results. Details can be found in Appendix D 
conclusion: 
 We introduce a novel framework for geometric interaction-aware motion retargeting, named MeshRet. This framework explicitly models the dense geometric interactions among various body parts by first establishing a dense mesh correspondence between characters using semantically consistent sensors. We then develop a unique spatio-temporal representation, termed the DMI field, which adeptly captures both contact and non-contact interactions between body geometries. By aligning this DMI field, MeshRet achieves detailed contact preservation and seamless geometric interaction.Performance evaluations using the Mixamo dataset and our newly compiled ScanRet dataset confirm that MeshRet offers state-of-the-art results.Limitations The primary limitation of MeshRet is its dependence on inputs with clean contact; motion clips exhibiting severe interpenetration yield poor outcomes. Consequently, it is unable to process noisy inputs effectively. Refer to Figure 12 and Figure 13 for failure cases under noisy inputs. Future efforts will focus on enhancing its robustness to noisy data. Additionally, SCS extraction can be compromised by noisy meshes, particularly those with complex clothing. A potential solution is to employ a Laplacian-smoothed proxy mesh for SCS extraction. Lastly, the method cannot handle characters with missing limbs. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss potential societal impacts in Appendix F Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work does not have such risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We properly cited papers and sources for existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: ScanRet details The primary motivation for collecting the ScanRet dataset stemmed from two main concerns. First, the data quality in the Mixamo [2] dataset was relatively low, suffering from significant issues such as interpenetration and contact mismatch. Second, the Mixamo dataset exclusively contained cartoon characters, whose body type distributions differed markedly from those of real human motion capture actors. In response, we developed the ScanRet dataset. We recruited 100 participants, evenly split between males and females, representing common ranges of height and BMI. Each participant underwent a 3D scan to create a T-pose mesh. We intentionally did not collect texture information for the body or face to protect privacy. Subsequently, we used motion capture equipment to build a library of 83 actions characterized by extensive physical contact. We enlisted human animators to map each action onto the 100 T-pose meshes, ensuring both semantic integrity and correct physical contact were maintained. All participants and animators received fair compensation. After discarding some invalid data, we compiled a total of 8,298 motion data entries. The ScanRet dataset is designed to simulate data obtained from real human motion capture, such as the MoSh [20, 17] algorithm, thus enhancing the realism of our evaluation process in the context of actual animation production workflows. Data splits We collected motion data for 13 characters from the Mixamo website, totaling 3,675 motion sequences, with each character having approximately the same number of sequences. The characters are: Aj, Amy, Kaya, Mousey, Ortiz, Remy, Sporty Granny, Swat, The Boss, Timmy, X Bot, and Y Bot. Among them, Ortiz, Kaya, X Bot, and Amy were not encountered by the network during training. Overall, our training set included motion data for 9 Mixamo characters and 90 randomly selected characters from the ScanNet dataset, where 90% of the motion sequences was randomly chosen from both datasets. Details regarding the train/test split for specific motion sequences and characters are provided in the code.Mixamo ScanRetWe evaluate the performance of our method from three perspectives: joint accuracy, contact preservation, and geometric interpenetration. In terms of joint accuracy, we calculate the Mean Squared Error (MSE) between the ground-truth joint positions X gt and the retargeted joint positions X, normalized by the character's height h:EQUATIONPrevious work [32] assessing the accuracy of self-contact measurements merely utilized the distance between hand vertices and the body surface to determine contact presence. Such experimental metrics fail to accurately reflect the precision of the contact location. Therefore, we adopted a metric similar to the vertex contact mean squared error (MSE) proposed by Villegas et al. [28] , termed "Contact Error". Specifically, we first identified sensor pairs where the distance between hand and body sensors in the source action was less than the arm's diameter d src . We then located the same sensor pairs in the retargeted motion. If the distance between these sensor pairs in the retargeted motion exceeded that in the source action, we calculated the MSE of the distance differences; otherwise, the contact error was zero. The formula is as follows:EQUATIONwhere d t,k,l A indicates the contact sensor pairs with ||d t,k,l A || 2 < d src , while R A and R B represent the radius of each character's arms.For geometric interpenetration, we assess the percentage of interpenetration, calculated as the ratio of penetrated vertices to the total vertices per frame. A lower ratio signifies reduced interpenetration. In our evaluation, we calculate the interpenetration ratio between arms (including hands) and the body. C Implementation Details SCS details As introduced in Section 3.2, we establish semantic correspondences between character meshes with different topologies using semantically consistent sensors. Specifically, given the semantic coordinates (b, l, ϕ) of a sensor, we can identify semantically consistent sensor positions on the meshes of different roles and obtain the feature vectors of the sensors. This process is detailed in Algorithm 1.Algorithm 1: Derive Semantically Consistent Sensors from Semantic Coordinate Network architecture The network architectures of both our DMI Encoder and Geometry Encoder resemble the structure of PointNet. However, since all our data is inherently situated within the canonical space, we have eliminated the T-Net from PointNet to reduce network complexity. Before being input into the encoder, sensor features pass through a sensor group embedding layer, which converts the bone index b into an 8-dimensional embedding vector. This embedding vector is updated during training. The Geometry Encoder consists of six PointNet layers with D model set at 256, and there is a distinct Geometry Encoder for the body, head, arms, and legs. The DMI Encoder comprises a per-sensor encoder and a per-frame encoder, each built with six PointNet layers, with each interaction pair having its own encoder. Specific interaction pairs include: [(Left Arm), (Right Arm, Head, Torso)], [(Right Arm), (Left Arm, Head, Torso)], [(Left Leg), (Right Leg, Torso)], and [(Right Leg), (Left Leg, Torso)]. The Motion Encoder is a multilayer perceptron (MLP). Both the Training details We implemented our network using PyTorch [23] , running on a machine equipped with an NVIDIA RTX A6000 GPU and an AMD EPYC 9654 CPU. The dataset was uniformly processed at a frame rate of 30 fps. During training, we randomly clipped a sequence of 30 frames from the dataset. The target character was set to be the same as the source character with a 50% probability, and different with a 50% probability, selected randomly from the dataset. On our system, training for 36 epochs required approximately 40 hours. During inference, our MeshRet model can achieve performance exceeding 30 fps.Input: Mesh O, joint locations J ∈ R N ×3 , bone index b ∈ {0, 1, • • • , N }, origin parameter l ∈ [0, 1), direction parameter ϕ ∈ [0, 2π) Output: Sensor feature s ∈ R 4×3 i parent ← bone_parent_joint(b), i child ← bone_child_joint(b); x parent ← J[i parent ], x child ← J[i child ]; o ← (1 -l)x parent + lx child ; /*We recruited participants via the Amazon Mechanical Turk [3] platform to partake in a user study. As shown in Figure 7 , during each session, subjects were presented with one source video and two retargeted motion videos: Video A and Video B. Participants were asked to watch all three videos and then compare Video A and Video B. At the conclusion of the viewing, they were requested to answer the following three questions:1. Which video better matches the source motion in terms of the overall meaning and intent of the motion?2. Which video has more accurate and detailed motion? Look for less self-interpenetration and better self-contact precision.3. Considering all factors, which video do you think is better overall?For each question answered, participants received a compensation of $0.04. We collected 600 comparison results in the end.𝑡 -1 𝑡 𝑡 + 1 Motion jitter comparison To better illustrate the jitter issue present in the results from the R 2 ET [32] method, we visualized consecutive frames generated by R 2 ET and our method in Figure 8 , and provided a line graph depicting the variations in height of the right-hand joint over time. These results demonstrate that R 2 ET is adversely affected by contradictions between skeletal retargeting and geometry correction phases, leading to significant motion jitter. In contrast, our method successfully avoids this problem. Qualitative comparison with Zhang et al. [30] Since Zhang et al. [30] did not open-source their code, we were unable to conduct a complete and fair comparison of their method with ours in our experiments. However, we endeavored to locate several examples presented in their paper and applied our MeshRet to the same motion sequences. The comparative results are displayed in Figure 9 . As observed in these examples, our method maintains the semantic integrity of the source motions, and it performs better in the Fireball case (the second motion sequence shown). This indicates that our method can achieve, and even surpass, the performance of their approach.𝑡 -1 𝑡 𝑡 + 1 𝑡 -1 𝑡 𝑡 + 1 Source R ! ET OursMetrics across different data splits Tables 3 and 4 present the contact error and penetration ratio of our method compared to the baseline method across four different data splits. A consistent pattern observed is that performance improves for seen characters or motions. It is evident that our method outperforms the baseline across all data splits. Ablation studies on ratios of proximal sensor pairs The full approach can be considered a mixed version of Ours f ar and Ours cls , utilizing an equal distribution of proximal and distal sensor pairs. To better illustrate this balance, we provide additional experimental results by testing different ratios of proximal to distal sensor pairs. Table 5 compares our method's performance with varying percentages of proximal sensor pairs under the Mixamo+ setting. As the percentage of proximal sensor pairs decreases, the interpenetration ratio fluctuates mildly, while the contact error initially decreases and then increases. Finally, with no proximal pairs (equivalent to the "far" version), the performance drops significantly. In Figure 10 , we present a qualitative comparison of our methods using different proximal sensor pair ratios. Except for the 100% Proximal version (equivalent to Ours cls ) and the 0% Proximal version (equivalent to Ours f ar ), our method demonstrates fair robustness to the proximal sensor ratio in the 25%-75% interval. Based on these results, we conclude that choosing 50% proximal sensor pairs strikes a reasonable balance for achieving good performance. Ablation studies on different sensor arragements We conducted further ablation studies on different sensor arrangements. Specifically, we evaluated the performance of a model trained with half the sample points in the ϕ space in SCS, denoted as Ours ϕ , and another model trained with half the sample points in the l space in SCS, referred to as Ours l . As shown in Table 6 , Ours ϕ compromises the interpenetration ratio, indicating that sufficient sample points in the space are crucial for avoiding interpenetration. We also found that both models introduce artifacts; please refer to Figure 11 . Our work can provide animation professionals with enhanced results in motion retargeting, thereby alleviating their workload and increasing productivity in fields such as virtual reality, game development, and animation production. Regarding potential negative social impacts, we believe the likelihood of misuse of our work is minimal. This is because our work is situated in the midstream phase of the animation production pipeline, whereas privacy-invading forgeries, such as DeepFake, primarily occur during the downstream rendering phase.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: Claimed contributions
 - Character motion retargeting respecting geometric interactions by
    1. Build Semantically Consistent Sensors (SCS) by casting rays perpendicular to bones and finding intersections with the mesh
    2. Dense Mesh Interaction (DMI) Field storing relative positions and semantics (bone information) of sparsified pairwise SCS
    3. MesRet network with a transformer architecture taking the interaction features from the source DMI and geometry features from the source SCS to retarget poses to a target character respecting the source interactions

Strengths
 - Novel ideas, especially with the DMI and its sparsification strategy
 - Convincing results for the self-contact-aware retargeting

Weaknesses
 - Not much - the rebuttal and the discussions seem to resolve all concerns
 - I may ask the authors to optionally add a reference to "Spatial Relationship Preserving Character Motion Adaptation" by Ho et al, 2010.

The paper proposes a solid solution to an important problem in mapping skeletal animations from one character to another while respecting the self-contact. The authors were sincere in resolving all concerns raised by the reviewers. Incorporating the discussions and clarifications in the discussion phase in the final revision will give extra value to the paper. Thus, I propose accepting the paper as a spotlight.

Reviews: review: paper_summary: paper_summary: To address the issues such as jittery, interpenetration, and contact mismatches issues caused in the motion retargeting task, this paper proposes a new framework named MeshRet, directly modeling the dense geometric interactions in motion retargeting. It initially establishes dense mesh correspondences using semantically consistent sensors (SCS), and develops a spatio-temporal representation named dense mesh interaction field (DMI), which can capture the contact and non-contact interactions between body parts. The proposed model, MeshRet, achieves SOTA performance on a public dataset,Mixamo, and a newly proposed dataset, ScanRet.
strength_weakness: strength_weakness: The ablation study suggests that the proximal sensor pairs are essential for minimizing interpenetration and preserving contact, whereas distal pairs provide insights into the non-contact spatial relationships among body parts. It’s a balancing problem to both maintain contact and at the same time to avoid penetration. Given a new unseen character for unknow motion retargeting, how to choose the “cls”, “far”, or “dm” version?
questions: questions: Given a new unseen character for unknow motion retargeting, how to choose the “cls”, “far”, or “dm” version? This seems to be challenging as it lacks of prior knowledge of how much to enforce contact and how much to avoid interpenetration.
limitations: limitations: The model relies heavily on the accurate estimation of SCS, when the SCS extraction is affected by noisy input, complex clothes, it likes to decrease the performance.
paper_summary: paper_summary: The author proposes MeshRet, a new solution that achieves geometry-aware motion retargeting across various mesh topologies in a single stage.

The author introduces semantically consistent sensors (SCS) and dense mesh interaction (DMI) field to guide the training of MeshRet, effectively achieving semantic alignment. SCS parametrically represents the relationship between points on the mesh surface and the skeleton. The DMI field expresses multi-frame motion features in a form similar to point cloud features.

The author presents a dataset called ScanRet, which is specifically tailored for evaluating motion retargeting techniques.
strength_weakness: strength_weakness: Overall, there are no significant drawbacks. However, the method of selecting $K \times L$ sensors from $S^2$ potential interacting sensors to balance complexity appears somewhat ad hoc.

Additionally, this method is only applicable to CG models and lacks robustness to real-world noise, a point the authors have also mentioned in the limitations section.
questions: questions: Regarding the Semantically Consistent Sensors (SCS), how is the number S determined? Section 3.2 shows S, but later it is written as K. This part is rather confusing. Is it the total number versus the number after masking?
limitations: limitations: The author has already addressed the limitations of the method.
paper_summary: paper_summary: * The paper proposes a new motion retargeting method based on dense geometric information. This dense geometric information is gathered as dense interaction fields by pre-generated pseudo sensors, which are semantically embedded on the skin. The interaction fields are capable of representing the semantic information of different motions. Based on these ideas, the authors designed a new network to extract features from source motion, body shapes, and the newly designed interaction fields. The interaction fields are also used in the loss functions, together with GAN loss, reconstruction loss, and other loss functions. Experimental results show that the new method can effectively avoid contact loss and interpenetration while maintaining the semantics of the source motion.

* The main contribution is the MeshRet method, which includes the Semantically Consistent Sensors, Dense Mesh Interaction Fields, a new network architecture, and losses designed based on the previous two modules. The authors also collected a dataset named ScanRet to improve the quality of motion retargeting datasets.
strength_weakness: strength_weakness: Both qualitative and quantitative results illustrate the effectiveness of this method. However, there are some weaknesses in the experiments:

* The ablation study focuses on close/far sensor pairs and the distance matrix, but does not address other aspects, such as different sensor arrangements.

* The newly collected dataset, ScanRet, is only used as a source of motion. This raises the question of whether it can be used in conjunction with Mixamo.

* The paper compares its results with SAN, PM-Net, and R2ET, but does not include comparisons with other works, such as the 2021 paper "Contact-aware Retargeting of Skinned Motion."

* Both ScanRet and Mixamo datasets are used in training. It would be interesting to see the results if only the Mixamo dataset is used.
questions: questions: Refer to weaknesses part.
limitations: limitations: * The authors mention that one limitation is poor outcomes when the input is noisy. However, they didn't provide any failed cases or examples under noisy conditions to clarify this limitation. 

* Other limitations may include how to attach sensors for people with disabilities or different body shapes, and how to handle sensors for designed animated characters with more or fewer bones.
paper_summary: paper_summary: The paper proposes a data-driven method for character animation retargeting. The authors introduce Semantically Consistent Sensors (SCS), which are a set of surface points determined by casting rays from bones. Additionally, a Dense Mesh Interaction (DMI) field is proposed to measure the relative positions of sensors over time. They train a transformer that directly predicts motions of target characters, using source and target SCS, source motion, and source DMI. The training is conducted in an unsupervised manner, where the generated target DMI and target motion are matched to the source.
strength_weakness: strength_weakness: The contact resolution relies on clean training data without penetrations. There is no mechanism to explicitly prevent penetrations.
questions: questions: Why does Contact Error depend on absolute scale? What if the retargeted character has a significantly different scale from the source character?

What if the bones have different relative lengths? To achieve the same motion, they may require different orientations (Q). How diverse are the skeletons in your dataset? It would be interesting to see how the method performs when the relative bone lengths are different.

Does the method assume that all characters in the training dataset share the same bone system? What if I would like to use the method on a character system with a different number of bones and connectivities? Do I need to collect a new dataset? How many modifications to the neural networks are required?
limitations: limitations: The limitation that the retargeting quality depends on inputs with clean contact is discussed in the paper.

Comments: title: Reply to Reviewer YXjG's further comment
Could you kindly review our response?
Could you kindly review our response?
Could you kindly review our response?
Thanks to Reviwer utNP
Could you kindly review our response?
Thanks to Reviewer WToy
comment: Thank you for your valuable feedback and support. We are pleased to confirm that we have thoroughly addressed your concerns. Given the limited time remaining in the rebuttal phase and the necessity of refactoring some of our code, we will strive to provide experimental results on characters with varying bone numbers. If we are unable to complete this before the rebuttal deadline, we commit to including experiments with varying bone counts in our final manuscript, accompanied by a comprehensive discussion of the results. All the discussion about our method's limitation will also be included.
Thank you for the response. I do not have further questions. Adding an experiment that can split or merge neighboring bones to handle different bone numbers would make the paper stronger. And please discuss the limitation as stated in Q3 in the revision.
Dear Ethics Reviewer,

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns?

We appreciate your time and input.

Sincerely,

The Authors
Dear Ethics Reviewer,

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns?

We appreciate your time and input.

Sincerely,

The Authors
Dear Reviewer,

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns?

We appreciate your time and input.

Sincerely,

The Authors
Thank you for taking the time to review our paper and rebuttal. We will thoroughly address all the concerns raised by the reviewers by incorporating additional results and improving explanations for clarity in the final manuscript.
Dear Reviewer,

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns?

We appreciate your time and input.

Sincerely,

The Authors
Thanks for the detailed response by the authors to my previous comments and questions. I appreciate the effort you’ve put into addressing the issues I raised, and I’m pleased to see the improvements made in your revised manuscript. I have no further concerns or questions at this time.
Thank you for taking your time to go through our paper and rebuttal. We will definitely address all the concerns raised by reviewers by including additional results and enhancing explanations for clarity in our final manuscript.
Thanks to the authors for their detailed feedback.

I don't have any new questions at this time. I hope the authors will consider all the concerns raised by the reviewers and address unclear expressions and other issues in the revised version.

I will continue to maintain my current score.
We extend our gratitude to Ethics Reviewer YeD9 for your insightful suggestions on discussing performance differences based on an actor's gender, height, or BMI.

# Impact of Race/Skin Tone on Scan Accuracy

We utilize a multi-camera array to perform 3D scans of the subjects. To the best of our knowledge, the impact of race or skin tone on the accuracy of these scans is minimal.

# Inclusion of Actors with Disabilities

As mentioned in our response to Reviewer utNP, our current method does not address the inclusion of actors with disabilities. Consequently, our ScanRet dataset does not include actors with disabilities. We acknowledge this as a limitation of our work and suggest it as an area for future research.

# Potential Performance Differences Based on Gender, Height, or BMI

Thank you for your suggestion. We conducted a preliminary analysis on our ScanRet dataset, calculating metrics for four distinct target character groups: Male, Female, Low-BMI, High-BMI, Short, and Tall. Overall, the performance differences were minimal. Specifically, the results for single-gender target characters were almost identical to the overall results. However, results for the Low-BMI/High-BMI and Short/Tall groups were slightly less accurate. This discrepancy is likely because the target characters within these specific BMI or height ranges exhibit less body shape variance, while we used all characters as source characters, which posed additional challenges for our retargeting model. Despite this, our findings indicate that the model does not exhibit a preference for any particular group.

| **Group**  | **Contact Error** &darr; | **Penetration** (\%)&darr; |
|---|---|-----|
Male | 0.281 | 1.60 |
Female | 0.291 | 1.58 |
Low-BMI (18-20) | 0.311 | 1.72 |
High-BMI (26-28) | 0.298 | 1.71 |
Short (156 cm-166 cm) | 0.297 | 1.63 |
Tall (178 cm-188 cm) | 0.304 | 1.70 |
Full | 0.284 | 1.59 |

# Compensation for Crowdsourced Workers

We appreciate your feedback regarding compensation. Before conducting our user study, we referred to a Quora response \[1\] which noted that "\\$6.50 per hour is common practice in academic projects." In our study, each participant completed 60 pairwise comparison questions within approximately 20-30 minutes, resulting in an hourly compensation of about \\$4.80-\\$7.20.

We sincerely apologize for any potential underpayment issues. Moving forward, we agree that participants should be compensated based on living wage standards.

\[1\] [How much should I pay on Mechanical Turk?](https://www.quora.com/How-much-should-I-pay-on-Mechanical-Turk)
We appreciate the valuable feedback provided by Ethics Reviewer ryee regarding the need for additional statistical reporting on our dataset and the importance of disclosing participant consent details.

# Basic Statistics of Participants in the ScanRet Dataset

Due to privacy considerations, detailed personal information about the subjects was not disclosed to us by the third-party company responsible for participant recruitment. We have access to limited demographic data, including height, BMI range, gender, and age. Of the 100 participants, 50 are male, and 50 are female. Participants' heights range from 156 cm to 188 cm, and their BMI ranges from 18 to 28. All participants are adults, with ages spanning from 19 to 75 years, and an average age of 37.25 years. We will provide the height and BMI data of each participant upon the release of the ScanRet dataset.

# Participant Consent for Data Release

Prior to data collection, we obtained explicit consent from each participant for the release of their data. Participants were fully informed that the collected data would be made publicly available.