Skinned Motion Retargeting with Dense Geometric Interaction Perception
Authors: Zijie Ye
Jia-Wei Liu
Jia Jia
Shikun Sun
Mike Zheng Shou
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Neural Motion Processing
Motion Retargeting
Primary area: generative_models
TL;DR: Geometry-aware motion retargeting in a single stage with no contradiction introduced by geometry correction.

Abstract: Capturing and maintaining geometric interactions among different body parts is crucial for successful motion retargeting in skinned characters. Existing approaches often overlook body geometries or add a geometry correction stage after skeletal motion retargeting. This results in conflicts between skeleton interaction and geometry correction, leading to issues such as jittery, interpenetration, and contact mismatches. To address these challenges, we introduce a new retargeting framework, MeshRet, which directly models the dense geometric interactions in motion retargeting. Initially, we establish dense mesh correspondences between characters using semantically consistent sensors (SCS), effective across diverse mesh topologies. Subsequently, we develop a novel spatio-temporal representation called the dense mesh interaction (DMI) field. This field, a collection of interacting SCS feature vectors, skillfully captures both contact and non-contact interactions between body geometries. By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpenetration and ensures contact preservation. Extensive experiments on the public Mixamo dataset and our newly-collected ScanRet dataset demonstrate that MeshRet achieves state-of-the-art performance. Code available at https://github.com/abcyzj/MeshRet.

Skinned Motion Retargeting with Dense Geometric Interaction Perception
Zijie Ye! Jia-WWel Liu', Jia Jia 12 Shikun Sun 12, Mike Zheng Shou
Department Computer Science and Technology. BNRist, Tsinghua University
Key Laboratory of Pervasive Computing. Ministry of Education Show Lab, National University of Singapore
Abstract
Capturing and maintaining geometric interactions among different body parts is crucial for successful motion retargeting in skinned characters. Existing approaches often overlook body geometries add a geometry correction stage after skeletal motion retargeting. This results in conflicts between skeleton interaction and ge- ometry correction, leading to issues such as jittery, interpenetration, and contact mismatches. To address these challenges, we introduce a new retargeting frame- work, MeshRet, which directly models the dense geometric interactions in motion retargeting. Initially, we establish dense mesh correspondences between characters using semantically consistent sensors (SCS), effective across diverse mesh topolo- gies. Subsequently, we develop novel spatio-temporal representation called the dense mesh interaction (DMI) field. This field, collection of interacting SCS feature vectors, skillfully captures both contact and non-contact interactions be- tween body geometrics. By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpeneetraionn and ensures contact preservation. Extensive experiments on the public Mixamo dataset and our newly-collected ScanRer dataset demonstrate that MeshRet achieves state-of-the performance. Code available at https//gtthb.oom//toyy//eeeeee
Introduction
Skinned character animation is prevalent virtual reality [T6), game development [21]. and various other fields. However, animating these characters often presents significant challenges due to differencee body proportions between the motion source and the target character, leading to issues such as loss of motion semanties, mesh interpenetration, and contact mismatches. Consequently, motion retargeting is essential to adjust for these discrepancies in body proportions. This process is crucial for maintaining the integrity of the source motion's 's characteristies in the animation of the target character.
Motion retargeting presents challenges due to the complex interactions among character limbs and wide range of body geometries. Accurately preserving these interactions is crucial, as incorrect interactions can result in mesh interpenetration and contact mismatches. Prior research has typically addressed these interactions from two perspectives: skeleton interactions and geometry corrections. employ cycle-consiseency to implicitly align skeleton interaction semantics, yet they do not address the complexities of geometric interactions between different body parts. Villegas et al. 28 introduced mesh self-contact modeling; however, their approach does not extend
Early methods [1 29 15
WWor partially done during visiting NUS. Cooresspodiigg Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to non-contact interactions. More recently, Zhang et al. [32] implemented two-stage pipeline that first aligns skeleton interaction semantics and then corrects geometric artifacts. Nonetheless, the inherent conflict between preserving skeleton interaction semantics and correcting geometry leads to jittery movements, severe interpenetration and imprecise contacts. Zhang et al. [30] subsequently proposed adding stage that aligns visual semantics with visual language model, but this requires detailed pair-by-pain finetuning due to the loss of spatial information when projecting 3D motion into 2D images.
To resolve the conflict between skeleton interaction and geometry correction, we propose new approach: focusing solely on dense geometric interaction for motion retargeting. Character animation videos, rendered from the skinned mesh, rely on geometric interactions to shape user perception. Skeleton interaction, in contrast, merely represents simplified, sparse form of geometric interaction. Therefore, maintaining correct interactions between different body part geometries not only preserves motion semantics but also prevents mesh interpenetration and ensures contact preservation, as illustrated in Figure
Given the significance of geometric interactions, we propose new framework, named MeshRet, for skinned motion retargeting. In contrast earlier methods that adjust skeletal motion retargeting outcomes, our approach models the intricate interactions among character meshes without depending on predefined vertex correspondences.
The design of MeshRer necessitates several technical innovations. Initially, there a requirement for dense mesh correspondence across different characters. Drawing inspiration from the medial axis inverse transform (MAIT) [22]. we have devised a technique, termed semantically consistent sensors (SCS), automatically derive dense mesh correspondence from sparse skeleton correspondence. This technique enables us to sample a point cloud of sensors on the mesh to represent each character. Following this, illustrate dense mesh interaction between body parts, we employ interacting mesh sensor pairs, maintaining generality. These pair-wise interactions are encoded within a novel spatial- temporal representation termed the Dense Mesh Interaction (DMI) field. The DMI field adeptly encapsulates both contact and non-contact interaction semantics. Finally, we proceed to learn motion manifold that aligns with the target character geometry and the source motion DMI field.
To align our evaluation process more closely with real animation production, we gathered an in- the-wild motion dataset, termed ScanRet, characterized by abundant contact semantics and minimal mesh interpenetration. ScanRet consists of 100 human actors ranging from bulky to skinny, each performing 83 motion clips scrutinized by human animators. The MeshRer model is trained on both the ScanRet dataset and the widely used Mixamo [2] dataset. We assessed our method across large variety of motions and diverse array of target characters. Both qualitative and quantitative analyses show that our MeshRel model significantly outperforms existing methods.
To summarize, we present the following contributions:
We introduce MeshRei, a pioneering solution that facilitates geometric interaction-aware motion retargeting across varied mesh topologies in a single pass.
We present the SCS and the novel DMI field to guide the training of MeshRet, effectively encapsulating both contact and non-contact interaction semantics.
We develop ScanRet, a novel dataset specifically tailored for assessing motion retarget- ing technologies, which includes detailed contact semanties and ensures smooth mesh interaction.
Our experiments demonstrate that MeshRer delivers exceptional performance, marked by accurate contact preservation and high-quality motion.
2
Related Work
Skeletal motion retargeting Motion retargeting seeks to preserve the characteristics of source motions when transferring them to different target character. Skeletal motion retargeting primarily addresses the challenge of differing bone ratios. Gleicher [8] initially formulated motion retargeting as spatio-temporal optimization problem, using source motion features as kinematic constraints. Subsequent researches (5) 7 [4] have focused on optimization- approaches with various con- straints. However, these methods, while requiring extensive optimization, often yield suboptimal

---

Existing Method
The Proposed Resh
Skeletoe-swiire
Geometry Correction
DMI Field Extraction
Geumstric Interaction-seenee Retargeting
Retergeting
Figure Comparison with the existing method. Contrary the earlier eetageting-correet approach [32]. which suffer from internal contradictions leading to interpenetration, jitter, and contact mismatches, our pipeline leverages the DMI field to accurately model complex geometric interactions.
results. Consequently, recent studies have explored learning dearning-based motion retargeting algorithms. Jang et 11 trained a motion retargeting network using a U-Net (26) architecture paired motion data. Villegas et al. (29 introduced a recurrent neural network combined with cycle-consistency (35) for unsupervised motion retargeting. Lim, Chang, and Choi [15] propose to learn frame-by-frane poses and overall movements separately. Aberman et al. [T] develop differentiable operators for cross-structural motion retargeting among homeomorphic skeletons. However, these methods gen- erally neglect the geometry of characters, leading to frequent contact mismatches and severe mesh interpenetrations.
Geometry-aware motion retargeting Previous studies have generally processed character ge- ometries through two approaches: contact preservation and interpenetration avoidance. Lyard and Magnenat-Thalmame Thaammnn [19] developed a heuristic optimization algorithm to maintain character self- contact, while Ho. Komura, and Tai 99 proposed maintain character interactions by minimizing the deformation of interaction meshes, Ho and Shum [10] introduced a spatio-temporal optimization framework prevent self-collisions in robot motion retargeting. Jin, Kim, and Lee [12] employed proxy volumetric mesh to preserve spatial relationships during retargeting. Subsequently, Basset al. combined both attraction and repulsion terms in an optimization-base -based method to avoid interpenetration and preserve contact. However, these methods necessitate per-vertex correspondence and involve costly optimization processes. More recently, Villegas et al. (28) attempted to retarget skinned motion through optimization in a latent space of pretrained network, although their method does not accommodate non-contact interactions. Zhang et al. [32] implemented two-stage pipeline that initially aligns skeleton interaction semantics and subsequently corrects geometric artifacts. Nevertheless, the inherent confiict between maintaining skeleton interaction semantics and correcting geometry often results in jittery movements and imprecise contacts. In a later study. Zhang al. [30) added a stage that aligns visual semantics using a visual language model, but this approach requires extensive pair-by-pair fine-tuning due the loss of spatial information when projecting 31) motion into 2D images.
Existing geometry-awen motion retargeting methods either require expensive optimization or employ multi-stage strategies for skeleton and geometry semantics, resulting in a contradiction between stages that often leads to unsatisfactory results. In contrast, our method processes both contact and non-contact semantics using dense mesh interaction field in a single stage.
3 Method
3.1 Overview
We introduce novel geometric interaction-awann motion retargeting framework MeshRet, as illus-
trated in Figure Unlike previous methods that either overlook character geometries [T 29 15)
apply geometry correction after skeleton retargeting 332 30,, our framework directly addresses dense geometric interactions with the Dense Mesh Interaction (DMI) field. This provides a detailed representation of the interactions within skinned character motions, preserving motion semanties by preventing mesh interpenetration and ensuring precise contact preservation.
Motion geometry representations Assume the motion sequence has frames and the character has N' skeletal joints. The motion sequence m is represented by the global root translation X B8 7x3

---

Source Motion
Qa
Source DMI
D
Fx
D
Fd
A
Mgrc
Motion Encoder
DMI Encoder
DMI Consistency
Transformer Decoder
Transformer Encoder
Fg
M Mat
Target Motion
Q
Fk
DB
Fe
Target DMI
Figure 2: Overview the proposed MeshRet. The pipeline begins with the extraction of the DMI field using sensor forward kinematics, denoted as FR, and pairwise interaction feature selection, represented by Fa. This DMI field, in conjunction with geometric features derived from Fg. is fed into encoder-decoder network. The network predicts the target motion sequence, which is aligned with the target character's geometry and the original DMI field.
and the local joint rotation Q RRTxNNX where adopt the 6D representation 34 for the joint rotations. The rest-pose geometry G of the character is represented by the rest-pose mesh O and the rest-pose joint locations J E R3
Task definition Given the source motion sequence mA, and the geometries G, and Gg of the source and target characters in their T-poses, objective is generate the motion mg for the target character. This process aims to retain essential aspects of the source motion, including its semantics, contact preservation, and the avoidance of interpenetration.
Following the definition the task, our MeshRet model initially derives Semantically Consistent Sensors (SCS) S Sx44x which provide dense geometric correspondences essential for the retargeting process, where $ FG) S captures sensor location and the sensor tangent space matrix, facilitating enhanced perception of the geometry surface. Subsequently, we conduct sensor forward kinematics (FK) and pairwise interaction extraction to generate the source DMI field DA Fa Sx), where DA R xx Here, K is the number of SCS in the DMI field, L represents a
feature selection, and P indicates the feature dimension of the DMI. Lastly, transformer-baat based network (27] ingests mA, DA SΛ, and Sp, and predicts target motion sequence my that aligns with the target character's geometry and the source DMI field. The entire pipeline is denoted as follows:
mB = Fr(mA, DA, SA, Sp.)
3.2 Semantically consistent sensors
To facilitate dense geometric interactions, our MeshRet framework necessitates establishing dense mesh correspondence between source and target characters. Previous studies have typically derived
correspondence from vertex coordinates 333 virtual sensor [31] through bounding mesh [12]:
however, these methods are confined to template meshes sharing identical topology, such as MANO [25] or SMPL 18]. Villegas et al. (88 suggested determining vertex correspondence using nearest neighbor searches on predefined feature vectors. Nevertheless, this approach often lacks precision and brevity, resulting in inaccurate contact representations and substantial optimization burdens.
In this study, we introduce Semantically Consistent Sensors (SCS) that are effective across various mesh topologies while ensuring precise semantic correspondence. Our approach draws inspiration

---

Joints (by b)
Tangent
Space
Matrix
Figure 3: Left: Illustration of the method to derive sensor feature s from the semantic coordinate 16,1,6) across different characters. The red line represents the projected ray. The feature encom- passes the sensor's location and its tangent space matrix. Right: The DMI field effectively captures both contact and non-contact interactions. Red lines represent d in the DMI field. the second example, the body sensors (yellow points) are located in the tangent plane of the hand sensors (blue points), signifying contact interaction.
from the Medial Axis Inverse Transform (MAIT) [22] We conceptualize the skeleton bones of each character as approximate medial axes of their limbs and torso. For each bone, a MAIT-like transform is applied to generate the corresponding SCS. This involves casting rays from the bone axis across a plane perpendicular to it. The origin parameter and direction parameter & of the rays, combined with the bone index b, establish the semantic coordinates of the SCS. The semantic coordinates describe connection between the sensor and the skeleton bones. A sensor is deemed valid if its ray intersects the mesh linked to the bone; otherwise, it is considered invalid. Through this method, we establish a dense geometric correspondence based on sparse skeletal correspondence, The procedure for deriving SCS is illustrated in Figure Given a unified of SCS semantic coordinates {(b,, (1, Φ1), (b2, l2, ...)..... , (bg, Lg, Φs)}. we can derive SCS feature S {81, $2, S for each character. Further details can be found in Algorithm
3.3 Dense mesh interaction field
To effectively represent the interactions between character limbs and the torso, we have developed the DMI field. Based on SCS detailed Section 3.2, the DMI field comprehensively captures both contact and non-contact interactions across different body part geometries. Utilizing the DMI field allows for dense geometry interaction-awas motion retargeting, thereby eliminating the need for geometry correction stage.
Sensor forward kinematies For a given motion sequence, denoted as m, we initially conduct forward kinematics (FK) on S to derive sensor features S1:T¹ E x5'XXxx3 Each S' encompasses the locations and tangent matrices for S sensors frame t. The FK transformation for an individual sensor is expressed as:
Σ (Q))
n=1
where G, (Q') SE(3) is the global transformation matrix for bone derived from its local rotation matrix, and w(P)), represents the linear blend skinning (LBS) weight for sensor s,, determined through barycentric interpolation of its adjacent mesh vertices.
Pairwise interaction feature Next, we model the geometric interactions as pairwise interaction features between sensors. Ideally, for each frame, obtain comprehensive DMI field, D' representing pairwise vectors across K² sensor pairs:
d 1, (p)) p!).
(3)
D' (dd'',, b,, b, le, L, di H.
(4)
5

---

where t. E R3x33 is the tangent matrix if sensor i, and d' represents the relative position of target sensor j in the tangent space of observation sensor i. D' is composed of two components: the relative position of the sensor pair and the semantic coordinates of both the observation and target sensors. The use of semantic rather than spatial coordinates is essential, as it obviates the need for actual sensor positions, thereby making DMI suitable for motion retargeting applications.
However, D E SxSXP exhibits quadratic growth with respect to S because it includes S sensor pairs, rendering it impractical when managing thousands of sensors. To address this, we implement
two sparsification strategies for D Initially, we restrict interactions critical body parts only, such as arm-torso, arm-head, arm-arm, and leg-leg, rather than between all sensor pairs, thereby restricting our focus to K observation sensors. Subsequently, for each observation sensor, we select L target sensors from each relevant body part, where L is predetermined hyper-parametee empirically choose L/2 nearest and L/2 furthest target sensors. We find that proximate sensor pairs are crucial for minimizing interpenetration and maintaining contact, while distant pairs delineate the overall spatial relationships between body parts, as shown in Figure 3 These strategies lead the formulation of the final DMI field D E R LxX" with selected sensor pairs indicated by the sparse DMI mask Mrc sxS shown in Figure 2
Specifically, we
3.4 Geometry interaction-awant motion retargeting
To avoid the conftict between skeleton interaction and geometric correction, the proposed MeshRet employs the DMI field to model geometric interactions directly. As shown in Figure 2 MeshRer initially extracts the DMI field DA from the source motion sequence MA as described in Section 3.3 The field DA encapsulates interactions among various body parts within the source motion, encom- passing both contact and non-contact interactions, further depicted in Figure 3 The DMI field, composed of sensor pair feature vectors, possesses the unordered characteristics of point cloud. Consequently, we implement a PointNet-like architecture (24) for our DMI encoder, which is divided into two components: the per-sensor encoder and the per-irame encoder. Given D E T xxXX the per-sensor encoder initially processes it as T separate point clouds, producing representations HA E T Dima for each observation sensor, where Dendel denotes the feature dimension. Sub- sequently, the per-frame encoder generates per-frame representations H E T by encoding these T point clouds.
Since DMI field DA lacks geometric information about characters, we introduced a geometry encoder F, extract geometric features from their SCS. For each sensor, we form a feature vector by concatenating its rest-pose feature S; with its semantic coordinates (bi, 4i Φ.). The resultant geometric features are represented as CA E S for character A and Cg E S for character The semantic coordinates of sensors act as intermediaries linking the DMI field to character geometry. The geometry encoder employs a PointNet-like architecture [24] to transform the geometric features C into geometric latent code H E R Pootal
The transformer-be basedd retargeting network processes input features including the source DMI feature H source joint rotation Qs, source geometry latent H and target geometry latent H3 Specifically, the encoder processes H and H while the decoder processes QA and H The latents H and H serve as the initial tokens in the sequence, enabling both the encoder and decoder to operate over a sequence of length 7' + 1. The output sequence's 's final T' frames are represented as QB.
Due to the lack of paired ground-trui truth data, employ the unsupervised method described by Lim, Chang, and Choi [15]. Our network utilizes four loss functions for training: reconstruction loss, DMI consistency loss, adversarial loss, and end-effector loss. Supervision signals are derived from the source motion. We maintain geometric interactions by aligning the source DMI field D with the target DMI field Dg. The target DMI field Dg is generated by first applying sensor forward kinematics to Qg. followed by selecting sensor pairs using the target sparse DMI mask Mgg E S This mask. Mgt. is derived by excluding invalid sensors of the target character from M The DMI consistency loss is quantified as the cosine similarity loss between wair-wise relative positions in DR and DA:
T
K
d...k.
31.,11
Lami
c(k,
1)
(5)
||
||2
| k 11

---

where e(k, takes the value if sensor pair (k. () is valid in both Mac and Mgt. and 0 otherwise. The reconstruction loss serves as a regularization mechanism to minimize motion alterations during retargeting, defined as follows:
Lree Q - QAI
(6)
To facilitate realistic motion retargeting, a discriminator, denoted as 8(.), is employed. The adversarial loss is subsequently defined as:
Ladv Bo- [log 8(Q) + Q [log(1 6(Q))
We observed that the global orientation of end-effectors significantly influences user experience. Consequently, we introduced end-effector loss promote consistent orientations of end-effectors in the retargeted motion.
Let T ΣE [IR(Q). R( i)ll,
(8)
where R(-) transforms local joint rotations into global rotations for joint i along the kinematic chain and X' represents the set of end-effectors. Our MeshRet trained by:
Lutal = Arec free + Admi Lami + Auao Ladv + Aet
Experiments
4.1 Settings
Datasets We trained and evaluated our method using the Mixamo dataset (2) and the newly curated ScanRei dataset. We downloaded 3,675 motion clips performed by 13 cartoon characters from the Mixamo dataset contains, while the ScanRer dataset consists of 8,298 clips executed by 100 human actors. Notably, the Mixamo dataset firquuetly features corrupted data due to interpenetration and contact mismatches. To overcome these issues, we created the ScanRer dataset, which provides detailed contact semantics and improved mesh interactions, with each clip being scrutinized by human animators. The training set comprises 90% of the motion clips from both datasets, involving nine characters from Mixamo and 90 from ScanRet. Our experiments tested the motion retargeting capabilities between cartoon characters and real humans, aligning closely with typical retargeting workflows. During inference, adopted four data splits based character and motion visibility: unseen character with unseen motion (UC+UM), unseen character with seen motion (UC+SM), character with unseen motion (SC+UM), and seen character with seen motion (SC+SM), as delineated by Zhang et al. (32]. We present the average results across these splits. Additional details available in Appendix A
Implementation details The to 1.0, 5.0, 1.0, 1.0, and 20. respectively. We use {0, 1,... .Noody 1} x {0, 0.25, 0.5, 0.75} x (0, 0.5m. x, 1.5T). as the SCS semantic coordinates set, where Nbuty = 18 is the number of body bones and x represents the Cartesian product. We employed the Adam optimizer [13] with learning rate of 10-4 to optimize our network. The training process required 36 epochs. For further details, please refer Appendix C
Arec. Admi- Andv. det and were empirically set
We assess the effectiveness of our method through three metrics: joint accuracy, contact preservation, and geometrie interpenetration. Joint accuracy is quantified by calculating the Mean Squared Error (MSE) between the retargeted joint positions and the ground- -tuut data provided by animators in ScanRet. This analysis considers both global and local joint positions, normalized by the character heights. Contact preservation is evaluated by measuring the Contact Error, defined as the mean squared distance between sensors that were originally in contact in the source motion clip. Geometric interpenetration is determined by the ratio of penetrated limb vertices to the total limb vertices per frame. Further details are available in Appendix B
Evaluation metrics
4.2 Comparison with state-of-the-arts
Qualitative results Figure demonnsrats the performance of skinned motion retargeting across characters with diverse body shapes, where the motion sequences novel to the target characters
7

---

Souree
Copy
PMnet
SAN
E²E
Ours
Figure 4: Qualitative comparison with baseline methods. Our method ensures precise contact preservation and minimal geometric interpenetration.
during training. Most baseline methods, except R²ET 32 fail to consider the geometry of characters, leading significant geometric interpenetration and contact mismatches. Unlike these methods, RºET (22 includes a geometry correction phase after skeleton-aw aware retargeting. However, this creates conflict between the two stages, resulting in oscillations R²ET's outcomes, which manifest as alternating contact misses and severe interpenetrations, as shown in the first two rows. Additionally, these oscillations appear variably across different frames within the same motion clip, producing jittery motion, as illustrated in Figure and Figure 8 A further limitation of R²ET is its neglect of hand contacts. contrast, our method employs the innovative DMI field to preserve such detailed interactions, such as those observed in the "Praying" pose in the third row.
Table I: Quantitative comparison between our method and state-of-the-arts. Mixamo+ represents the mixed dataset of Mixamo and ScanRet. MSE denotes the local MSE
Metric
MSEI
MSE
Contact Error
Penetration %%)
Dataset
ScanRet
ScanRet
Mixamot
ScanRet
Mixamo+
ScanRet
Source Copy
0.234
3.04
1.37
0.026
0.006
1.702
0.387
5.26
2.16
PMnet
15
0.130
0.029
2.716
0.890
5.23
2.23
SAN
0.049
0.011
2.432
0.627
4.95
1.72
32
0.063
0.017
2.209
0.589
4.21
2.01
OurSels
0.048
0.013
0.800
0.426
3.35
1.73
Ours far
0.045
0,010
1.642
0,610
5.37
1.77
Oursam
0.048
0,010
2.568
0.797
4.69
1.78
Ours
0.047
0.009
0.772
0.284
3.45
1.59
Table presents comparison between our methods and ttaee-of-the-arts. We initially measure the joint location error using MSE and MSE on ScwnRet. The ground truth in ScanNet is established by human animators. Our observations indicate that human animators typically retarget motions by initially replicating joint rotations and subsequently modifying frames
Quantitative results

---

Source
Ours
Our
Ours far
OurS dm
Source
Ours
OurS,ts
Ours for
OurSym
Figure 5: Qualitative comparison of ablation studies, A red circle highlights areas of interpenetration, while a red rectangle identifies errors in non-contact semantics.
that display incorrect interactions. Conversely, our method modifies the entire motion sequence, resulting in a higher MSE compared to the Copy strategy. Nevertheless, MSE remains valuable auxiliary reference. In comparison to PMnet [15], I R 32), and SAN (I. our method achieves
MSE reductions of 65%, 29%, and 8%, respectively. These results demonstrate that our approach more closely aligns with the outputs produced by human animators.
As shown in Table PMnet [5] and SAN [T. exhibit high interpenetration ratios and contact errors due to their neglect of character geometries. E2ET (3) effectively reduces interpenetration through a geometry correction stage; nonetheless, it still encounters high contact errors stemming from conflicts between the retargeting and correction stages, Our approach explicitly models geometry interactions and thereby achieves low contact error and penetration ratio, illustrating the effectiveness of our proposed MeshRet in generating high-quality retargeted motions with detailed contact semantics and smooth mesh interactions. Additionally, dataset is more challenging than with the ScanRer dataset, attributable to significant body shape variations between cartoon characters and real person characters.
observe that retargeting using the mixed Mixamo+
4.3 Ablation Studies
We conducted ablation studies to demonstrate the significance of pairwise interaction feature selection and the implementation of DMI similarity loss. Initially, evaluated the performance of model trained exclusively with the nearest L sensor pairs, denoted as Oursats. and another model trained solely with the farthest L sensor pairs, referred to Ours far- As indicated in Table and Figure 5 Ours for compromises contact semantics and leads to significant interpenetration, while Ours, also exhibits inferior performance. This outcome suggests that proximal sensor pairs essential for minimizing interpenetration and preserving contact, whereas distal pairs provide insights into the non- contact spatial relationships among body parts. Further, we investigated the effect of incorporating distance matrix loss, as proposed by Zhang et al. [32], on our sensor pairs, designated as Oursam- The results imply that the distance matrix loss fails to yield meaningful supervisory signals, likely because distance is non-directional and insufficient to discern the relative spatial positions among numerous sensors.
Table 2: Human preferences between our method and baselines.
Methods
Semantics Preservation
Contact Accuracy
Overall Quality
Copy
20.7% v.s. 79.3 % (Ours)
22.7% v.s. 77.3% 90%%%co0 (Ours)
18.7% v.s. 81.3 % (Ours)
PMnet
15
2.7% v.s. 97.3 (Ours)
5.3% v.s. 94.7% 7.% (Ours)
1.3% v.s. 98.7 % (Ours)
SAN
9.3% v.s. 90.7 (0
15.3% v.s. 84.7% 1 (Ours)
7.3% v.s. 92.7 % (Ours)
R²ET
32
14.6% v.s. 85.4% (Ours)
16.0% v.s. (0urs
13.3% v.s. 86.7 (Ours)
4.4
User study
We conducted a user study to assess the performance of our MeshRet model in comparison with the Copy strategy, PMnet [15), SAN [1), and R*ET (32). Fifteen sets motion videos were presented to participants, each consisting of one source skinned motion and five anonymized skinned results.
Participants were requested to rate their preferences based on three criteria: semantic preservation, contact accuracy, and overall quality. Users were recruited from Amazon Mechanical Turk (3)

---

resulting in total of 600 comparative evaluations. As indicated in Table the comparisons favored our results. Details can be found in Appendix D
2 approximately 81% of
Conclusion
We introduce novel framework for geometric interaction-awars motion retargeting, named MeshRet. This framework explicitly models the dense geometric interactions among various body parts by first establishing dense mesh correspondence between characters using semantically consistent sensors. We then develop unique spatio-temport representation, termed the DMI field, which adeptly captures both contact and non-contact interactions between body geometries. By aligning this DMI field, MeshRer achieves detailed contact preservation and seamless geometric interaction. Performance evaluations using the Mixamo dataset and our newly compiled ScanRet dataset confirm that MeshRet offers state-of-the-art results.
Limitations The primary limitation of MeshRel is its dependence inputs with clean contact; motion clips exhibiting severe interpenetration yield poor outcomes. Consequently, is unable to process noisy inputs effectively. Refer to Figure Figure[12nnd and Figure 13 for failure cases under noisy inputs. Future efforts will focus on enhancing its robustness to noisy data. Additionally, SCS extraction can be compromised by noisy meshes, particularly those with complex clothing. A potential solution is to employ a Laplacian-smootte characters with missing limbs.
proxy mesh for SCS extraction. Lastly, the method cannot handle
Acknowledgments and Disclosure of Funding
This work is supported by the National Key R&D Program of China under Grant No. D024QY1400 the National Natural Science Foundation of China No. 62425604, and Tsinghua University Initiative Scientific Research Program. Mike Shou does not receive any funding for this work.
10

---

References
[1] Kfir Aberman et al. "Skeleton-awart networks for deep motion retargeting". In: ACM Trans. Graph. 39.4 (2020), p. 62.
[2] Adobe. Mixamo. https//www.mixamo.com 2018.
[3] Amazon. Amazon Mechanical Turk. https:/wwww.mtur..com
[4] Jean Basset et al. "Contact preserving shape transfer: Retargeting motion from one shape to another". In: Computers & Graphics 89 (2020). pp. 11-23.
[5] Antonin Bernardin et al. "Normalized Euclidean distance matrices for human motion retar- geting". In: Proceedings of the 10th International Conference on Motion in Games. 2017,
pp. 1-6.
[6] Yingruo Fan et al. "Faceformer: Speech-driven 3d facial animation with transformers". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022,
pp. 18770-18780.
[7] Andrew Feng et al. "Automating the transfer of a generic set of behaviors onto a virtual character". In: Motion in Games; 5th International Conference, MIG 2012, Rennes, France, November 15-17, 2012. Proceedings 5. Springer. 2012. pp. 134-145.
[8] Michael Gleicher. "Retargetting motion to new characters". In: Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 1998, pp. 33-42.
[9] Edmond S. L.. Ho, Taku Komura, and Chiew-l Lan Tai. "Spatial relationship preserving character motion adaptation". In: ACM Trans. Graph. 29.4 (2010), 33:1-33:8.
[10] Edmond SL Ho and Hubert PH Shum. "Motion adaptation for humanoid robots in constrained enviromments" . In: 2013 IEEE International Conference on Robotics and Automation. IEEE, 2013. pp. 3813-3818.
[11] Hanyoung Jang et al. "A variational u-net for motion retargeting". In: SIGGRAPH Asia 2018
Posters. 2018, pp. 1-2.
[12] Taeil Jin, Meekyoung Kim, and Sung-Hee Lee. "Aura mesh: Motion retargeting preserve the spatial relationships between skinned characters", In: Computer Graphics Forum. Vol. 37.
2. Wiley Online Library. 2018, pp. 311-320.
[13] Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015.
[14] Jehee Lee and Sung Yong Shin. "A hierarchical approach to interactive motion editing for human-like figures". In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 1999, pp. 39-48.
[15] Jongin Lim, Hyung Jin Chang, and Jin Young Choi. "PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting." " In: BMVC. Vol. 2019, p. 7.
[16] Jinghuai Lin and Mare Erich Latoschik. "Digital body, identity and privacy in social virtual reality: A systematic review". In: Frontiers in Virtual Reality 3 (2022), p. 974652.
[17] Matthew Loper, Naureen Mahmood, and Michael J Black. "MoSh: motion and shape capture from sparse markers." In: ACM Trans. Graph. 33.6 (2014), pp. 220-1.
[18] Matthew Loper et al. "SMPL: skinned multi-person linear model".. . In: ACM Trans. Graph. 34.6 (2015), 248:1 1-248:16 16.
[19] Etienne Lyard and Nadia Magnenat-Thalmane "Motion adaptation based on character shape".
In: Computer Animation and Virtual Worlds 19.3-4 (2008), pp. 189-198.
[20] Naureen Mahmood al. "AMASS: Archive of motion capture as surface shapes". In: Pro- ceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 5442- 5451.
[21] Lucas Mourot et al. "A survey on deep learning for skeleton-bart human animation". In: Computer Graphics Forum. Vol. 41. 1. Wiley Online Library. 2022, pp. 122-157.
[22] Henning Naß al. "Medial axis (inverse) transform complete 3-dimensional Riemannian manifolds". In: 2007 International Conference on Cyberworlds (CW'07). IEEE. 2007, pp. 386- 395.
[23]
Adam Paszke et al. "Pytorch: An imperative style, high-performance deep learning library". In: Advances in neural information processing systems 32 (2019).

---

[24] Charles R Qi et al. "Pointnet: Deep learning point sets for 3d classification and segmen- tation". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 652-660.
[25] Javier Romero, Dimitrios Tzionas, and Michael J. Black. "Embodied hands: modeling and capturing hands and bodies together". In: ACM Trans. Graph. 36.6 (2017), 245:1-245:17.
[26] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation". In: Medical image computing and computer-assisted intervention-MICCA/ 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer. 2015, pp. 234-241.
[27] Ashish Vaswani et al. "Attention is all you need". In: Advances in neural information process- ing systems 30 (2017).
[28] Ruben Villegas et al. "Contact-aware retargeting of skinned motion". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 9720-9729.
[29] Ruben Villegas et al. "Neural kinematic networks for unsupervised motion retargetting". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 8639-8645
[30] Haodong Zhang et al. "Semantics-awarc Motion Retargeting with Vision- Vision-Language Models".
In: arXiv preprint 2rXiv:2312.00990 (2023).
[31]
He Zhang et al. "ManipNet: neural manipulation synthesis with a hand-object spatial represen- tation". In: ACM Trans. Graph. 40.4 (2021), 121:1-121:14.
[32] Jiaxu Zhang et al. "Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry" In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2023, pp. 13864-13872.
[33] Keyang Zhou et al. "Toch: Spatio-temporal object-to-hand correspondence for motion refine- ment". In: European Conference on Computer Vision. Springer. 2022, pp. 1-19.
[34] Yi Zhou al. "On the continuity of rotation representations in neural networks". In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 5745-5753.
[35] Jun- Yan Zhu et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks". In: Proceedings of the IEEE international conference computer vision. 2017, pp. 2223-2233
12

---

A Dataset Details
ScanRer details The primary motivation for collecting the ScanRer dataset stemmed from two main concerns. First, the data quality in the Mixamo 2 dataset was relatively low, suffering from significant issues such as interpenetration and contact mismatch. Second, the Mixamo dataset exclusively contained cartoon characters, whose body type distributions differed markedly from those of real human motion capture actors. response, we developed the ScanRer dataset. We recruited 100 participants, evenly split between males and females, representing common ranges of height and BMI. Each participant underwent 3D sean create a T-pose mesh. We intentionally did not collect texture information for the body or face to protect privacy. Subsequently, we used motion capture equipment to build a library of 83 actions characterized by extensive physical contact. We enlisted human animators to map each action onto the 100 T-pose meshes, ensuring both semantic integrity and correct physical contact were maintained. All participants and animators received fair compensation. After discarding some invalid data, we compiled a total of 8,298 motion data entries. The ScanRet dataset designed to simulate data obtained from real human motion capture, such as the MoSh [20 17) algorithm, thus enhancing the realism of our evaluation process in the context of actual animation production workflows.
Mixamo
ScanRet
Figure 6: Left: Characters of varying body types in the Mixamo dataset do not always maintain reasonable hand contact during clapping actions. Right: In our ScanRet dataset, characters of diverse body types consistently maintain appropriate hand contact while performing the same clapping actions.
Data splits We collected motion data for 13 characters from the Mixamo website, totaling 3,675 motion sequences, with each character having approximately the same number of sequences. The characters are: Aj. Amy, Kaya, Mousey, Ortiz, Remy, Sporty Granny, Swat, The Boss, Timmy, Bot, and Y Bot. Among them, Ortiz, Kaya. X Bot, and Amy were not encountered by the network during training. Overall, our training set included motion data for 9 Mixamo characters and 90 randomly selected characters from the ScanNet dataset, where 90% of the motion sequences was randomly chosen from both datasets. Details regarding the train/test split for specific motion sequences and characters are provided in the code.
B Evaluation metric details
We evaluate the performance of our method from three perspectives: joint accuracy, contact preserva- tion, and geometric interpenetration. In terms of joint accuracy, we calculate the Mean Squared Error (MSE) between the ground-truth joint positions Xg, and the retargeted joint positions X, normalized by the character's height h:
1 11X, X11
MSE =
(10)
assessing the accuracy of self-contact measurements merely utilized the distance between hand vertices and the body surface to determine contact presence. Such experimental metrics fail to accurately reflect the precision of the contact location. Therefore, we adopted metric similar to the vertex contact mean squared error (MSE) proposed by Villegas al. (28), termed "Contact Error", Specifically, we first identified sensor pairs where the distance between hand and body sensors
Previous work 32
13

---

in the source action was less than the arm's 's diameter dare. We then located the same sensor pairs in the retargeted motion. If the distance between these sensor pairs in the retargeted motion exceeded that in the source action, we calculated the MSE of the distance differences; otherwise, the contact error was zero. The formula as follows:
{(11 A | dh
RA 11- Ru 112)
if]
112 >
Contact Error
RA
Rg
(11)
0,
otherwise,
where t indicates the contact sensor pairs with ||d] 112 < dares while RA and Rg represent the radius of each character's arms.
For geometric interpenetration. we assess the percentage of interpenetration, calculated the ratio of penetrated vertices to the total vertices per frame. A lower ratio signifies reduced interpenetration. In our evaluation, we calculate the interpenetration ratio between arms (including hands) and the body.
Number of penetrated arm vertices
Penetration
(12)
Total number of arm vertices
C Implementation Details
SCS details As introduced in Section 3.2 we establish semantic correspondences between character meshes with different topologies using semantically consistent sensors. Specifically, given the semantic coordinates (b, I, ()) of sensor, we can identify semantically consistent sensor positions on the meshes of different roles and obtain the feature vectors of the sensors. This process is detailed in Algorithm
Algorithm 1: Derive Semantically Consistent Sensors from Semantic Coordinate
Input: Mesh O, joint locations J Vx3 bone index b 10,1,'' .. NN,, origin parameter
1 e [0, 1). direction parameter [0, 2T)
Output: Sensor feature E IR 1x3
parent f- tareee_paeen_jjiieeeee
ichild cone_child_joimee
Xparent f J parent] Xchild + [ichial];
+ (1 X)pprrttt + IXchild
/* Ray origin */
d forward + forward Sowpaarddddrcccoo(o_o
(O)
/* Face forward direction */
sone normalize (Xehaidd Xparent:
/* Bone unit direction vector */
dother - droeward& X dbone:
cos(1)) (ollaad (in)@))aaeee
/+ Ray direction +/
B - bone_moeee (oone_mesh(..))
/* Bone associated mesh
nay
< joy__mesh__em inttesietttteo(B.,... r);
P then
t + caugent_matrixee B);
S concat(p, t);
else
8 +- 0:
end
Network architecture The network architectures of both our DMI Encoder and Geometry Encoder resemble the structure of PointNet. However, since all our data is inherently situated within the canonical space, we have eliminated the T-Net from PointNet to reduce network complexity. Before being input into the encoder, sensor features pass through sensor group embedding layer, which converts the bone index b into an 8 8-dimensional embedding vector. This embedding vector is updated during training. The Geometry Encoder consists of six PointNet layers with Dmoael set at 256, and there is distinct Geometry Encoder for the body, head, arms, and legs. The DMI Encoder comprises a per-sensor encoder and a fer-frame encoder, each built with six PointNet layers, with each interaction pair having its own encoder. Specific interaction pairs include: [(Left Arm), (Right Arm, Head, Torso)], [(Right Arm), (Left Arm, Head, Torso)]. [[Left Leg), (Right Leg, Torso)], and [(Rigtt Leg), (Left Leg, Torso)]. The Motion Encoder is a multilayer perceptron (MLP). Both the
14

---

Instuctiona
Coneder
Figure 7: User interface presented to participants during the user study.
Transformer Encoder Transformer Decoder have eight layers, with the number of heads set to four and the we employ an alignment mask proposed by Fan et al. (6]. which ensures that each frame feature in the decoder attends only to the corresponding DMI frame and initial token, thereby aligning the network's ' output motion sequence with the input features.
size 256. Between the Transformer Encoder and Transformer Decoder,
Training details We implemented our network using Py PoTochh running on a machine equipped with an NVIDIA RTX A6000 GPU and an AMD EPYC 9654 CPU. The dataset was uniformly processed at a frame rate of 30 fps. During training, we randomly clipped a sequence 30 frames from the dataset. The target character was set to be the same the source character with a 50% probability, and different with a 50% probability, selected randomly from the dataset. our system, training for 36 epochs required approximately 40 hours. During inference, our MeshRet model can achieve performance exceeding 30 fps.
D User study details
We recruited participants via the Amazon Mechanical Turk platform to partake in user study. As shown in Figure during each session, subjects were presented with one source video and two retargeted motion videos: Video A and Video B. Participants were asked watch all three videos and then compare Video A and Video B. At the conclusion of the viewing, they were requested to answer the following three questions:
Which video better matches the source motion in terms of the overall meaning and intent of the motion?
15

---

2. Which video has more accurate and detailed motion? Look for less self interpptepennnnioonnn and better self-contact precision.
3. Considering all factors, which video do you think is better overall?
For each question answered, participants received compensation of $0.04. We collected 600 comparison results the end.
E
Additional results
t+1
t+1
t 1
Heiht Gamn
Source
R²ET
Ours
Time
Figure 8: Left: We visualized three consecutive frames within motion sequence. It is evident that while there was no jitter in the motion source, significant jitter occurred in the (-th frame of the R²ET (32) results, which right-hand height for this segment of the sequence. The results indicate that the jitter in the R²ET output was pronounced.
the case with method. Right: We visualize the corresponding
Motion jitter comparison To better illustrate the jitter issue present in the results from the EºET (22 method, we visualized consecutive frames generated by E² and our method Figure 8 and provided a line graph depicting the variations in height of the right-hand joint over time. These results demonstrate that ET is adversely affected by contradictions between skeletal retargeting and geometry correction phases, leading to significant motion jitter. In contrast, our method successfully avoids this problem.
Source
Zhang et al.
Ours
Source
Zhang et al.
Ours
Source
Zhang et al.
Ours
Figure 9: Qualitative comparison with Zhang et al. 30
Qualitative comparison with Zhang et al. (30] Since Zhang et al. (30 did not open-source their code, were unable to conduct complete and fair comparison of their method with ours in experiments. However, we endeavored to locate several examples presented their paper and applied our MeshRet the same motion sequences. The comparative results displayed in Figure 9 As observed in these examples, our method maintains the semantic integrity of the source motions, and it performs better in the Fireball case (the second motion sequence shown). This indicates that method can achieve, and even surpass, the performance of their approach.
Metries across different data splits Tables and present the contact error and penetration ratio of our method compared to the baseline method across four different data splits. A consistent pattern observed is that performance improves for seen characters or motions. It is evident that our method outperforms the baseline across all data splits.
Ablation studies on ratios of proximal sensor pairs The full approach can considered a mixed version of far and Oursele: utilizing equal distribution of proximal and distal sensor pairs. To
16

---

Table Contact errors of MeshRer and baselines across all data splits on Mixamo+.
Metric
Contact Error
Data Split
UC+UM
SC+UM
SC+SM UC+SM
Copy
1.462
1.188
2.477
1.682
PMnet
15
1.826
1.774
4.134
3.132
SAN [
1,416
1.181
4.229
2.902
RºET [32]
1.653
1.498
3.372
2.314
Ours
0.573
0.837
1.248
0.432
Table 4: Penetration ratios of MeshRet and baselines across all data splits on Mixamot.
Metric
Penetration( %)
Data Split
UC+UM
SC+UM
UC+SM
SC+SM
Copy
1.57
4.16
5.78
9.56
PMnet
15
1.43
4.20
5.71
9.56
SAN 1
1.81
5.52
4.66
7.81
R
32
1.54
4.66
4.92
5.71
Ours
1.55
2.63
4.60
5.04
better illustrate this balance, we provide additional experimental results by testing different ratios of proximal to distal sensor pairs. Tabper5 our method's performance with varying percentages of proximal sensor pairs under the Mixamo+ setting. As the percentage of proximal sensor pairs decreases, the interpenetration ratio fluctuates mildly, while the contact error initially decreases and then increases. Finally, with proximal pairs (equivalent to the "far" version), the performance drops significantly. In Figure 1O we present a qualitative comparison of our methods using different proximal sensor pair ratios. Except for the 100% Proximal version (equivalent to Oursats) and the 0% Proximal version (equivalent to Oursfar) our method demonstrates fair robustness to the proximal sensor ratio in the 25%-75% interval. Based these results, conclude that choosing 50% proximal sensor pairs strikes reasonable balance for achieving good performance.
Table 5: Quantitative comparison between methods with varing percentages of proximal sensor pairs under the Mixamo+ setting.
Method
Contact Error
Penetration %
100%
Proximal Pairs
0,800
3.35
75%
Proximal Pairs
0.909
3.61
50%
Proximal Pairs
0.772
3.45
25%
Proximal Pairs
0.781
3.29
0% Proximal Pairs
1.642
5.37
Ablation studies on different sensor arragements We conducted further ablation studies on different sensor arrangements. Specifically, we evaluated the performance of model trained with half the sample points in the & space in SCS, denoted as Ours and another model trained with half the sample points in the space in SCS, referred to as Ours]. As shown in Table 6 Ours compromises the interpenetration ratio, indicating that sufficient sample points in the space are crucial for avoiding interpenetration. We also found that both models introduce artifacts; please refer to Figure
Failure cases with noisy inputs We provide resutls with clean and noisy inputs in Figure and Figurd [3 The results of MeshRet exhibit interpenetration with noisy inputs.
17

---

Source 100%
75%
50%
25%
0%
Proximal
Proximal
Proximal Proximal Proximal
Source 100%
75%
50%
25%
0%
Proximal
Proximal Proximal Proximal Proximal
Figure 10: Qualitative results with different proximal sensor pair ratios.
Table 6: Quantitative comparison between methods with different sensor arrangements.
Metric
MSE
MSE
Contact Error
Penetration %%
Dataset
ScanRet
ScanRet
Mixamo+
ScanRet
Mixamo+
ScanRet
Ours
0.052
0.011
0.793
0.410
3.90
1.60
Ours
0.049
0.010
0.805
0.293
3,46
1.56
Ours
0.047
0.009
0.772
0.284
3.45
1.59
Source
Ours
Ours
Ours;
Source
Ours
Ours,
Ours;
Figure 11: Qualitative comparison of additional ablation studies on sensor arrangements. The red
rectangles identify artifacts introduced by different sensor arrangements.
Clean Source
Noisy Source
Noisy Result
Clean Source
Clean Result
Noisy Source
Noisy Result
Result
Figure 12: Qualitative results on the Mixamo dataset with clean and noisy inputs. A red rectangle indicates interpenetration.
18

---

Clean Source
Clean Result
Noisy Source
Noisy Result
Clean Source
Clean Result
Noisy Source
Noisy Result
Figure 13:
Qualitative results the Mixamo dataset with ScanRel characters as targets. A red
rectangle indicates interpenetration.
More cases 15 16 and deppict four motion sequences retargeted from the source character distinct target characters. These examples illustrate that our MeshRer is capable of generating high-quality motion sequences on target characters with diverse body shapes.
We present additional cases to validate the effectiveness of our MeshRet. Figures 14
Source
Target
Target 2
Target 3
Figure 14: Snapshots of motion sequence 4 in ScanRet, retargeted from the source character to three distinct characters.
19

---

Source
Target
Target
Target
Figure 15: Snapshots of motion sequence 43 in ScanRet, retargeted from the source character to three distinct characters.
Source
Target
Target 2
Target 3
Figure 16: Snapshots of motion sequence 9 in ScanRet, retargeted from the source character to three distinct characters.
20

---

Source
Target
Target
Target
Figure 17: Snapshots of motion sequence 45 in ScanRet, retargeted from the source character to three distinct characters.
Broader impacts
Our work can provide animation professionals with enhanced results in motion retargeting, thereby alleviating their workload and increasing productivity in fields such as virtual reality, game devel- opment, and animation production. Regarding potential negative social impacts, we believe the likelihood of misuse of our work is minimal. This is because our work is situated in the midstream phase of the animation production pipeline, whereas privacy-invading forgeries, such as DeepFake, primarily occur during the downstream rendering phase.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our results clearly support our main claims made in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not contain theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide experimental details Section 4.1 and Appendix A data will be made public if this paper gets accepted.
Code and
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: The Mixamo dataset is accessible to the public. Our ScanRer dataset will also be made public this puper gets accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guidest httpe////ooooouuebesssssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co
wetgguddddeeouuubbuneeee ssiniioiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental settings in Section 4.1 Appendix and Appendix C
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We omitted error bars from our analysis due the excessive computational expense involved in enumerating all data splits. For example, calculating the penetration ratio solely using ScanRer requires dozens of hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide resource requirements Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We removed textures from ScanRet to help preserve anonymity of participants. We acquired consent from every participant. All the participants are fairly paid.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in Appendix F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cited papers and sources for existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Due the limited capacity of currently available network storage tools that support anonymous link sharing, we are unable to host our entire dataset during the anonymous review phase. However, we have placed single data example in the 'artifact/scanret" folder of our code in the supplementary material, which can be loaded using Python's 's pickle library.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We provide these information in the Appendix D
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We performed our research accordance with local laws.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "EsLzy7VzwF", "reviewer": "Reviewer_yJ9L", "summary": "To address the issues such as jittery, interpenetration, and contact mismatches issues caused in the motion retargeting task, this paper proposes a new framework named MeshRet, directly modeling the dense geometric interactions in motion retargeting. It initially establishes dense mesh correspondences using semantically consistent sensors (SCS), and develops a spatio-temporal representation named dense mesh interaction field (DMI), which can capture the contact and non-contact interactions between body parts. The proposed model, MeshRet, achieves SOTA performance on a public dataset,Mixamo, and a newly proposed dataset, ScanRet.", "questions": "Given a new unseen character for unknow motion retargeting, how to choose the “cls”, “far”, or “dm” version? This seems to be challenging as it lacks of prior knowledge of how much to enforce contact and how much to avoid interpenetration.", "limitations": "The model relies heavily on the accurate estimation of SCS, when the SCS extraction is affected by noisy input, complex clothes, it likes to decrease the performance.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed model, MeshRet, improves the geometric interaction-aware motion retargeting through a single pass. The proposed submodules, such as, SCS, DMI, can improve the contact and non-contact interaction semantics. \n\nThe SCS seems to be effective across various mesh topologies on different characters. \n\n The proposed model achieves SOTA performance on both a public dataset and a newly proposed dataset. The paper also provides good ablation studies, e.g., using nearest or farthest L sensors, and the idea of incorporating distance matrix loss.  \n\nThe paper proposes a new dataset specifically tailored for assessing motion retargeting, it has contact semantics and ensures smooth mesh interaction. This dataset will benefit the research society. The paper also conducts a user study to demonstrate the superiority of the proposed model.", "weaknesses": "The ablation study suggests that the proximal sensor pairs are essential for minimizing interpenetration and preserving contact, whereas distal pairs provide insights into the non-contact spatial relationships among body parts. It’s a balancing problem to both maintain contact and at the same time to avoid penetration. Given a new unseen character for unknow motion retargeting, how to choose the “cls”, “far”, or “dm” version?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to Reviewer yJ9L for recognizing the novelty and state-of-the-art performance of our method. Your guidance on balancing proximal and distal sensor pairs has been particularly insightful.\n\n# Q1: How to choose the “cls”, “far”, or “dm” version?\n\nWe appreciate your suggestion to discuss the balance problem between proximal and distal sensor pairs. In the ablation study presented in Section 4.3, the \"dm\" version ($Ours_{dm}$), the \"cls\" version ($\\mathrm{Ours}_{cls}$), and the \"far\" version ($\\mathrm{Ours}\\_{far}$) all exhibit worse performance compared to the full approach (Ours). The full approach can be considered a mixed version of $\\mathrm{Ours}\\_{cls}$ and $\\mathrm{Ours}\\_{far}$, utilizing an equal distribution of proximal and distal sensor pairs. To better illustrate this balance, we provide additional experimental results by testing different ratios of proximal to distal sensor pairs. The following table compares our method's performance with varying percentages of proximal sensor pairs under the Mixamo+ setting. As the percentage of proximal sensor pairs decreases, the interpenetration ratio fluctuates mildly, while the contact error initially decreases and then increases. Finally, with no proximal pairs (equivalent to the \"far\" version), the performance drops significantly.\n\n| **Method**  | **Contact Error** &darr; | **Penetration** (\\%) &darr; |\n|---|---|-----|\n100% Proximal Pairs | 0.800 | 3.35 |\n75% Proximal Pairs | 0.909 | 3.61 |\n50% Proximal Pairs | 0.772 | 3.45 |\n25% Proximal Pairs | 0.781 | 3.29 |\n0% Proximal Pairs | 1.642 | 5.37 |\n\nIn Figure 18 of our rebuttal PDF, we present a qualitative comparison of our methods using different proximal sensor pair ratios. Except for the 100% Proximal version (equivalent to $\\mathrm{Ours}\\_{cls}$) and the 0% Proximal version (equivalent to $\\mathrm{Ours}\\_{far}$), our method demonstrates fair robustness to the proximal sensor ratio in the 25%-75% interval.\n\nBased on these results, we conclude that choosing 50% proximal sensor pairs strikes a reasonable balance for achieving good performance. We will include the additional results in the final manuscript.\n\nAs for the balance between enforcing contact and avoiding interpenetration, our method directly captures the interaction between mesh surfaces, as illustrated in Figure 3. Therefore, there is no explicit contradiction between preserving contact and avoiding interpenetration.\n\n# L1: The SCS extraction may be affected by noisy mesh input.\n\nThank you for your valuable comments. Noisy mesh surfaces can indeed pose challenges. One potential solution is to adopt a Laplacian-smoothed proxy mesh of the character for the SCS extraction. We have not implemented this due to the limited time available during the rebuttal phase but plan to discuss this further in our final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NBFVHg72gi", "reviewer": "Reviewer_WToy", "summary": "The author proposes MeshRet, a new solution that achieves geometry-aware motion retargeting across various mesh topologies in a single stage.\n\nThe author introduces semantically consistent sensors (SCS) and dense mesh interaction (DMI) field to guide the training of MeshRet, effectively achieving semantic alignment. SCS parametrically represents the relationship between points on the mesh surface and the skeleton. The DMI field expresses multi-frame motion features in a form similar to point cloud features.\n\nThe author presents a dataset called ScanRet, which is specifically tailored for evaluating motion retargeting techniques.", "questions": "Regarding the Semantically Consistent Sensors (SCS), how is the number S determined? Section 3.2 shows S, but later it is written as K. This part is rather confusing. Is it the total number versus the number after masking?", "limitations": "The author has already addressed the limitations of the method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The author introduces the concept of SCS as a feature representation and proposes the use of DMI field for learning motion retargeting alignment. These technical points demonstrate innovation.\n\nThe article is well-articulated and easy to follow, particularly in the task definition section where the input-output relationship of the overall task and the relationships between different modules are clearly represented.\n\nThe experiments conducted in the article provide evidence of the effectiveness of the proposed method.", "weaknesses": "Overall, there are no significant drawbacks. However, the method of selecting $K \\times L$ sensors from $S^2$ potential interacting sensors to balance complexity appears somewhat ad hoc.\n\nAdditionally, this method is only applicable to CG models and lacks robustness to real-world noise, a point the authors have also mentioned in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer WToy for acknowledging our novel technical contributions and the effectiveness of our experimental evidence. We also appreciate your guidance in clarifying the notation in the DMI field description.\n\n# Q1: What are the number $S$ and $K$?\n\nWe apologize for any confusion. The number $S$ is determined by the size of the SCS semantic coordinates set. Please refer to Line 257 for the specific semantic coordinates set used in our experiment, where $S = 288$. Your understanding of $K$ is correct. After masking the full DMI field $\\overline{\\textbf{D}} \\in \\mathbb{R}^{S \\times S \\times P}$ with $\\mathbf{M}_{\\text{src}} \\in \\mathbb{R}^{S \\times S}$, we obtain the final DMI field $\\mathbf{D} \\in \\mathbb{R}^{K \\times L \\times P}$, where $K$ represents the number of observation sensors in $\\mathbf{D}$. We will include a more detailed explanation in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VndY5AgtdJ", "reviewer": "Reviewer_utNP", "summary": "* The paper proposes a new motion retargeting method based on dense geometric information. This dense geometric information is gathered as dense interaction fields by pre-generated pseudo sensors, which are semantically embedded on the skin. The interaction fields are capable of representing the semantic information of different motions. Based on these ideas, the authors designed a new network to extract features from source motion, body shapes, and the newly designed interaction fields. The interaction fields are also used in the loss functions, together with GAN loss, reconstruction loss, and other loss functions. Experimental results show that the new method can effectively avoid contact loss and interpenetration while maintaining the semantics of the source motion.\n\n* The main contribution is the MeshRet method, which includes the Semantically Consistent Sensors, Dense Mesh Interaction Fields, a new network architecture, and losses designed based on the previous two modules. The authors also collected a dataset named ScanRet to improve the quality of motion retargeting datasets.", "questions": "Refer to weaknesses part.", "limitations": "* The authors mention that one limitation is poor outcomes when the input is noisy. However, they didn't provide any failed cases or examples under noisy conditions to clarify this limitation. \n\n* Other limitations may include how to attach sensors for people with disabilities or different body shapes, and how to handle sensors for designed animated characters with more or fewer bones.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The method is new, inspired by previous works on pseudo sensors and motion retargeting. There was a skeletal distance matrix in R2ET, and the DMI proposed in this paper can be seen as an improved, dense version of the distance matrix. R2ET used the distance matrix solely for motion semantics, while MeshRet uses the DMI for both semantics and geometrics.\n\n* DMI is a better representation of motion semantics compared to the distance matrix. The method has the potential to become the new SOTA. \n\n* The overall organization of the paper is good.", "weaknesses": "Both qualitative and quantitative results illustrate the effectiveness of this method. However, there are some weaknesses in the experiments:\n\n* The ablation study focuses on close/far sensor pairs and the distance matrix, but does not address other aspects, such as different sensor arrangements.\n\n* The newly collected dataset, ScanRet, is only used as a source of motion. This raises the question of whether it can be used in conjunction with Mixamo.\n\n* The paper compares its results with SAN, PM-Net, and R2ET, but does not include comparisons with other works, such as the 2021 paper \"Contact-aware Retargeting of Skinned Motion.\"\n\n* Both ScanRet and Mixamo datasets are used in training. It would be interesting to see the results if only the Mixamo dataset is used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our appreciation to Reviewer utNP for recognizing the novelty of our method and the effectiveness of the proposed DMI field. We sincerely value your recommendation to include additional experimental results.\n\n# Q1: More ablation studies on different sensor arrangements.\n\nWe provide additional ablation experiments here. We conducted further studies on different sensor arrangements. Specifically, we evaluated the performance of a model trained with half the sample points in the $\\phi$ space in SCS, denoted as ${Ours}\\_\\phi$, and another model trained with half the sample points in the $l$ space in SCS, referred to as ${Ours}\\_l$. As shown in the following table, ${Ours}_\\phi$ compromises the interpenetration ratio, indicating that sufficient sample points in the $\\phi$ space are crucial for avoiding interpenetration. We also found that both models introduce artifacts; please refer to Figure 14 in our rebuttal PDF.\n\n| **Metric**  | **MSE** &darr; | **MSE $^{lc}$** &darr; | **Contact Error** &darr; | **Contact Error** &darr; | **Penetration** (\\%) &darr; | **Penetration** (\\%) &darr; |\n|---|---|---|---|-----| -----| -----|\n| **Dataset** | ScanRet | ScanRet | Mixamo+ | ScanRet | Mixamo+ | ScanRet |\n| Ours $_{\\phi}$ | 0.052 | 0.011 | 0.793 | 0.410 | 3.90 | 1.60 |\n| Ours $_{l}$ | 0.049 | 0.010 | 0.805 | 0.293 | 3.46 | **1.56** |\n| Ours | **0.047** | **0.009** | **0.772** | **0.284** | **3.45** | 1.59 |\n\n\nWe will add the results to the final manuscript.\n\n\n# Q2: More results with ScanRet characters as target characters.\n\nWe provide additional qualitative results using ScanRet characters as target characters. Please refer to Figure 17 in our rebuttal PDF. For quantitative results, the metrics in Table 1 are computed using both ScanRet characters and Mixamo characters as targets.\n\n# Q3: Comparison with \"Contact-aware Retargeting of Skinned Motion.\"\n\nWe value your advice on including more baseline methods. We intended to compare our method to \"Contact-aware Retargeting of Skinned Motion,\" but we were unable to do so because the authors did not release their code, and some implementation details, such as the vertex correspondence feature, were not described in detail. Notably, the authors of R2ET did not compare their method to this one in their paper. In general, we have made our best effort to provide a comprehensive comparison with existing methods, including the currently open-sourced state-of-the-art method, R2ET.\n\n# Q4: The results only trained on the Mixamo dataset.\n\nWe value your advice on including more results. The following table presents a comparison between our methods and state-of-the-art methods trained only on the Mixamo dataset.\n\n| **Method**  | **Contact Error** &darr; | **Penetration** (\\%)&darr; |\n|---|---|-----|\nCopy | 2.77 | 4.07 |\nPMnet | 2.76 | 4.17 |\nSAN | 2.75 | 4.19 |\nR2ET | 2.81 | 4.15 |\nOurs | **2.07** | **3.47** |\n\nBecause the Mixamo dataset does not always provide clean contact input, the performance of our method is affected. However, our method still achieves much lower Contact Error and Penetration Ratio compared to baseline methods. Please refer to Figure 16 in our rebuttal PDF for qualitative results.\n\n\n# L1: Failure cases under noisy conditions.\n\nWe provide some failure cases in Figures 16 and 17 in our rebuttal PDF. We will include these in our final manuscript.\n\n# L2: How to attach sensors for disabled people with missing limbs and characters with different bone numbers?\n\nTo process characters with different bone counts, such as varying spine bone numbers, we can first split or merge neighboring bones and then attach sensors to the modified virtual bone system. However, our method cannot process characters with missing limbs. We will address this limitation in our final manuscript and leave it to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMbZzKFgWk", "reviewer": "Reviewer_YXjG", "summary": "The paper proposes a data-driven method for character animation retargeting. The authors introduce Semantically Consistent Sensors (SCS), which are a set of surface points determined by casting rays from bones. Additionally, a Dense Mesh Interaction (DMI) field is proposed to measure the relative positions of sensors over time. They train a transformer that directly predicts motions of target characters, using source and target SCS, source motion, and source DMI. The training is conducted in an unsupervised manner, where the generated target DMI and target motion are matched to the source.", "questions": "Why does Contact Error depend on absolute scale? What if the retargeted character has a significantly different scale from the source character?\n\nWhat if the bones have different relative lengths? To achieve the same motion, they may require different orientations (Q). How diverse are the skeletons in your dataset? It would be interesting to see how the method performs when the relative bone lengths are different.\n\nDoes the method assume that all characters in the training dataset share the same bone system? What if I would like to use the method on a character system with a different number of bones and connectivities? Do I need to collect a new dataset? How many modifications to the neural networks are required?", "limitations": "The limitation that the retargeting quality depends on inputs with clean contact is discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Many previous methods are compared, and the proposed method achieves state-of-the-art (SOTA) performance according to these comparisons.", "weaknesses": "The contact resolution relies on clean training data without penetrations. There is no mechanism to explicitly prevent penetrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to Reviewer YXjG for acknowledging our extensive experiments and state-of-the-art performance. Your assistance in refining metric definitions is highly appreciated.\n\n# Q1: Contact Error depends on absolute scale.\n\nWe agree that Contact Error should not depend on the absolute scale. Therefore, before calculating Contact Error, we normalize the DMI field by the limb radius of the character. Please refer to line 511 in `MeshRetCode/model/retnet.py` from our supplemental zip file for this detail.\n\nThe precise definition of Contact Error should be:\n$$\n\\text{Contact Error} = (||\\frac{\\mathbf{d}_A^{t,k,l}}{R_A}||_2 - ||\\frac{\\mathbf{d}_B^{t,k,l}}{R_B}||_2)^2 \\quad \\text{if}  ||\\frac{\\mathbf{d}_A^{t,k,l}}{R_A}||_2 > ||\\frac{\\mathbf{d}_B^{t,k,l}}{R_B}||_2 \\quad \\text{else} \\quad 0,\n$$\nwhere $R_A$ and $R_B$ represent the radius of each character's arms.\n\nWe apologize for not including this implementation detail in our paper and will correct this definition in our final manuscript.\n\n# Q2: How diverse are the skeletons in your dataset? Need for results on characters with different relative bone lengths.\n\nWe plot the skeleton length distribution in Figure 15 of our rebuttal PDF. In the ScanRet dataset, the arm lengths of characters range from 51.9 cm to 67.0 cm, body heights range from 153.4 cm to 185.1 cm, and arm-height ratios range from 0.32 to 0.39. In the ScanRet dataset, the arm lengths of characters range from 35.8 cm to 134.2 cm, body heights range from 120.4 cm to 368.3 cm, and arm-height ratios range from 0.28 to 0.45.\n\nTo see results on characters with different relative bone lengths, please refer to Figures 10-13 in our paper and the last two clips in our demo video from the supplemental zip file. In these results, our method performs well on characters with different relative bone lengths.\n\n# Q3: What if I would like to use the method on a character system with a different number of bones and connectivities?\n\nTo process characters with different bone numbers, such as varying spine bone numbers, we can split or merge neighboring bones and then attach sensors to the modified virtual bone system. This approach requires no training or modification of the neural network. However, to process characters with different bone connectivities, we need to collect a new dataset and retrain the neural network. Additionally, some hyper-parameters, such as the dimensions of certain embedding layers, must be modified. Notably, previous state-of-the-art methods like R2ET and SAN also require characters to share a common skeleton topology.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable comments.

We extend our gratitude to Reviewer yJ9L for recognizing the novelty and state-of-the-art performance of our method. Your guidance on balancing proximal and distal sensor pairs has been particularly insightful.

We are grateful to Reviewer WToy for acknowledging our novel technical contributions and the effectiveness of our experimental evidence. We also appreciate your guidance in clarifying the notation in the DMI field description.

We extend our appreciation to Reviewer utNP for recognizing the novelty of our method and the effectiveness of the proposed DMI field. We sincerely value your recommendation to include additional experimental results.

We express our gratitude to Reviewer YXjG for acknowledging our extensive experiments and state-of-the-art performance. Your assistance in refining metric definitions is highly appreciated.

We have included additional figures in the global rebuttal PDF. Please refer to the PDF for further results.

Meta-review: Claimed contributions
 - Character motion retargeting respecting geometric interactions by
    1. Build Semantically Consistent Sensors (SCS) by casting rays perpendicular to bones and finding intersections with the mesh
    2. Dense Mesh Interaction (DMI) Field storing relative positions and semantics (bone information) of sparsified pairwise SCS
    3. MesRet network with a transformer architecture taking the interaction features from the source DMI and geometry features from the source SCS to retarget poses to a target character respecting the source interactions

Strengths
 - Novel ideas, especially with the DMI and its sparsification strategy
 - Convincing results for the self-contact-aware retargeting

Weaknesses
 - Not much - the rebuttal and the discussions seem to resolve all concerns
 - I may ask the authors to optionally add a reference to "Spatial Relationship Preserving Character Motion Adaptation" by Ho et al, 2010.

The paper proposes a solid solution to an important problem in mapping skeletal animations from one character to another while respecting the self-contact. The authors were sincere in resolving all concerns raised by the reviewers. Incorporating the discussions and clarifications in the discussion phase in the final revision will give extra value to the paper. Thus, I propose accepting the paper as a spotlight.
Decision: Accept (spotlight)