TapMo: Shape-aware Motion Generation of Skeleton-free Characters
Authors: Jiaxu Zhang
Shaoli Huang
Zhigang Tu
Xin Chen
Xiaohang Zhan
Gang YU
Ying Shan
Venue: ICLR 2024 poster (2024)
Keywords: 3D animation
text-driven animation
motion generation
mesh deformation
Primary area: generative models
TL;DR: We present TapMo, a text-based animation pipeline for generating motion in a wide variety of skeleton-free characters.

Abstract: Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged characters. In this work, we present TapMo, a Text-driven Animation PIpeline for synthesizing Motion in a broad spectrum of skeleton-free 3D characters. The pivotal innovation in TapMo is its use of shape deformation-aware features as a condition to guide the diffusion model, thereby enabling the generation of mesh-specific motions for various characters. Specifically, TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module. Mesh Handle Predictor predicts the skinning weights and clusters mesh vertices into adaptive handles for deformation control, which eliminates the need for traditional skeletal rigging. Shape-aware Motion Diffusion synthesizes motion with mesh-specific adaptations. This module employs text-guided motions and mesh features extracted during the first stage, preserving the geometric integrity of the animations by accounting for the character's shape and deformation. Trained in a weakly-supervised manner, TapMo can accommodate a multitude of non-human meshes, both with and without associated text motions. We demonstrate the effectiveness and generalizability of TapMo through rigorous qualitative and quantitative experiments. Our results reveal that TapMo consistently outperforms existing auto-animation methods, delivering superior-quality animations for both seen or unseen heterogeneous 3D characters.

Published as a conference paper at ICLR 2024
TAPMo: SHAPE-AWARE MOTION GENERATION OF SKELETON-FREE CHARACTERS
Jiaxu Zhangl.2; Shaoli Huang Zhigang Tu Xin Chen², Xiaohang Zhan", Gang Yu², Ying Shan
Wuhan University, Teeceet Al Lab, Tencent PCG
ABSTRACT
Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged char- acters. In this work, we present TapMo, a Text-driven Animation Pipeline for synthesizing Motion in broad spectrum of skeleton-free 3D characters. The pivotal innovation in TapMo is its use of shape deformation-aawrre features as a condition guide the diffusion model, thereby enabling the generation of mesh- specific motions for various characters. Specifically. TapMo comprises two main components Mesh Handle Predictor and Shape-aware Diffusion Module. Mesh Handle Predictor predicts the skinning weights and clusters mesh vertices into adaptive handles for deformation control, which eliminates the need for tradi- tional skeletal rigging. Shape-aware Motion Diffusion synthesizes motion with mesh-specific adaptations. This module employs text-guided motions and mesh features extracted during the first stage, preserving the geometric integrity of the animations by accounting for the character's shape and deformation. Trained in suaely-s manner, TapMo can accommodate multitude of non- human meshes, both with and without associated text motions. We demon- strate the effectiveness and generalizability of TapMo through rigorous qualita- tive and quantitative experiments. Our results reveal that TapMo consistently outperforms existing anto-animation methods, delivering superior-quality anima- tions for both seen or unseen heterogeneous 3D characters. The project page: mttps::/ssmare iicmantticchh wwt.tig..tt..aaaaaaa.. 1o/TapMo
INTRODUCTION
The advent of 3D animation has revolutionized the digital world, providing a vibrant platform for storytelling and visual design. However, the intricacy and time-consuming nature of traditional animation processes poses significant barriers to entry. In recent years, learning-based methods have emerged as promising solution for animation. Motion generation models (Guo et al., 2020) Zhang et al. 2022) and auto-rigging methods (Xu et al. 2020 2022b) are among the techniques that offer unprecedented results for motion synthesis and mesh control, respectively, in both quality and generalization. Nevertheless, they still fall short of providing comprehensive solution.
Existing motion generation methods (Tevet et al. 2022 Zhang et al. 2022 Chen et al. 2023 facil- itate animation by leveraging the SMPL model (Loper al. 2015). parametric 3D model of the human form equipped with a uniform skeletal rigging system. However, these approaches oversim- plify the task by assuming a fixed mesh topology and skeletal structures, and they fail to account for the geometric shapes of skeleton-fie free characters. This inherent assumption significantly restricts the creation of motions for diverse, non-rigged 3D characters. Moreover, the predominant focus of existing motion datasets is on humanoid characters, which consequently limits the potential for data- driven methods to be applicable to non-standard characters. Therefore, significant challenge lies in extending motion generation models from human to non-human characters, enabling the model to accurately perceive their geometry even in the absence of pre-rigged skeletons.
"Most of this work done during Jiaxu's internship at Tencent Al Lab. Email: zjiaxu @ whu.edu.cn
1 Jiaxu Zhang and Shaoli Huang contributed equally. i "Corresponding author: tuzhigang @whu.edu.cn

---

Published as a conference paper at ICLR 2024
Rise from falling pose
Walk forword
Doing jumping jocks
Existing methods
The proposed TapMo
Figure 1: We present TapMo, a text-based animation pipeline for generating motion in wide variety of skeleton frreen-freeeee characters.
On the other hand, auto-rigging methods have attempted to enable mesh deformation control by skeletonizing and rigging 3D characters using optimization techniques (Baran & Popović 2007) or deep models (Xu et al.] 2020; 2019). Despite their attempts, these methods tend to generate diverse skeleton structures for different characters, hindering the possibility of generating uniform motion representation for animation. is worth highlighting that certain methods have suggested the utilization of pre-defined skeleton template for rigging characters (Liu et al.) 2209bb Li et al., (02T). However, these approaches are incapable of handling meshes that exhibit substantial variations in shape and topology. Consequently, the resulting rigging often necessitates manual adjustments or custom motions, which undermines the objective of achieving automatic motion synthesis.
To overcome these limitations, we introduce a new Text-driven Animation Pipeline (TapMo) capable of generating realistic and anthropomorphic motion for a wide range of skeleton-fre free 3D characters as Figure [ shows. TapMo only requires the input motion descriptions in natural language and can animate skeleton-free characters by perceiving their shapes and generating mesh-specific motions. To achieve this goal, we explore two key components of TapMo, namely the Mesh Handle Predictor and the Shape-aware Motion Diffusion. These two components synergistically automate the entire animation creation process, establishing a more accessible way generating animations beyond SMPL-based humans, so to democratize animation creation on heterogeneous characters.
The Mesh Handle Predictor addresses the mesh deformation control in a unified manner for diverse 3D characters. Inspired by SiPT (Liao et al.) 2022), we utilize GCN-based network that adap- tively locates control handles in the mesh, regulating their movements based on predicted skinning weights. Each predicted handle independently controls its assigned semantic part, while the first handle is fixed as the character's ' root for global control. More importantly, the module also serves a dual purpose by offering a mesh deformation feature for future shape-aware motion generation.
The Shape-aware Motion Diffusion copes with the challenge of perceiving the mesh shape and gen- erating realistic motions for heterogeneous 3D characters. Language descriptors are regarded as a user-friendly interface for people to interact with computers (Manaris) 1998). Thus, we design a text-driven motion generation model with respect to the Diffusion Model (Sohl-Dickstein et al.) 2015 Ho et al. (0200) that has achieved impressive results in the generation of complex data distri- bution recently. Different from previous Motion Diffusion Models (Tevet et al., [022] /Zhag al.] 2022), our model generates a universal motion representation that does not rely on the character's skeletal rigging structure, enabling it to animate wide range of characters. Notably, we use the mesh deformation feature encoded by the Mesh Handle Predictor erate motion adaptations that are flexible for the specific mesh to preserve the geometric integrity of the animations. Finally, Linear Blend Skinning is used to animate the mesh with the generated motion and the predicted handles.
additional condition gen-
In response the challenge posed by the absence of ground-truth handles and motions for diverse 3D characters, we present a novel, weakly-supervise training approach for TapMo. Our strategy leverages shape constraints and motion priors auxiliary supervision signals enhance the learn- ing of handle prediction and motion generation in TapMo.
We evaluate our TapMo on a variety of 3D characters and complex motion descriptions. The qual- itative and quantitative results show that our TapMo is effective on seen and unseen 3D characters. The generated animation of TapMo achieves higher motion and geometry quality compared with existing learning-basse based methods.
Our contributions are listed below:
We present a comprehensive solution TapMo that, to our knowledge, represents the first attempt to enable the motion generation of generic skeleton-free characters using text descriptions.
2

---

Published as a conference paper at ICLR 2024
We design two key components in TapMo Le., the Mesh Handle Predictor and the Shape-aware Motion Diffusion. These components collaborate to govern mesh deformation and generate plau- sible motions while accounting for diverse non-rigged mesh shapes.
A weakly-supervised training strategy is presented to enable the geometry learning of TapMo with limited ground-truth data of both
characters and text-relevant motions.
Extensive experiments demonstrate the effectiveness and generalizability of TapMo qualitatively and quantitatively, and the animation results of TapMo are superior to existing learning-band methods both on motion and geometry.
2 RELATED WORK
Motion generation remains a persistent challenge within the realm of computer graphics (Guo et al. (013)) Recently, an upsurge of interest has been seen in learning-based models for motion generation (Holden al. 2015] 2016). These cutting-edge models often explore various conditions for human motion generation, which include but are not limited to action labels (Guo et al., 2020 Petrovich et al., 202T), audio (Xu et al., 2022a Dabral et al. 2022), and text (Tevet et al. Chen et al. 2023 Zhang et al.] 2022). Among all these conditioned modalities, text descriptors are identified as the most user-friendly and convenient, thereby positioning text-to-motion as a rising area of research. While the existing advancements pave the way for the motion generation of hu- mans equipped with a pre-defined skeletal rigging system, they do not cater to various 3D characters lacking this skeletal system. Moreover, several large-scale datasets, providing human motions quences of 3D skeletons (Liu et al.] 2019a Guo et al.] 2020: 2022) or SMPL parameters (Mahmood al. 2019). However, these datasets also primarily cater to human motion learning and lack the in- clusion of a broad spectrum of heterogeneous characters. Consequently, methods trained directly on these datasets struggle adapt the task of generating motions for heterogeneous characters. Ac- knowledging this limitation, our research sets sail into previously unexplored territory: text-driven motion generation for skeleton-fre 3D characters. Distinetly different from previous studies, the 3D characters in our approach are non-rigged and feature wide array of mesh topologies.
2022
Skeleton-free mesh deformation. The skeletal rigging system widely used in animation (Mag- nenat et al.] 1988), but the process of skeletonization and rigging is not accessible to laypeople. To address this issue, some researchers have explored skeleton- mesh deformation methods. Tra- ditional methods rely on manual landmark annotations (Ben-Chen et al., 2009 [Sumner & Popović, 2004]. Recently, Yifan et al. (2020) proposed a learnable neural cage for detail- -peeeriipeerviinggg shape de- formation, which works well for rigid objects but not for articulated characters. Other approaches, such as those by Tan et al.) (2018). Yang et al. (021), embed the mesh into latent spaces to analyze the mesh deformation primitives. However, these latent spaces not shared across subjects, so they cannot be used for animating different characters. [Wang et al. (2023b) propose a zero-shot implicit deformation module, which is incompatible with our explicit motion generation approach.
The recent SfPT (Liao et al. (022) and HMC (Wang et al. 2023a) generate consistent deforma- tion parts across different character meshes, allowing for pose transfer through the corresponding parts. However, the character's ' root is ignored in the deformation parts, leading to failure in motion transfer. In this work, we are inspired by SfPT and introduce a Mesh Handle Predictor to animate skeleton-free characters in handle-based manner. This manner represents motion using the transla- tion and rotation of predicted mesh handles, which differs from traditional skeleton-based methods. Our approach diverges from SIPT in two key respects: first, unlike SIPT, we incorporate root handle prediction to enable more intricate motions. Second, our handle predictor also functions to deliver a mesh deformation-avwe aware representation that is crucial for subsequent mesh motion generation.
METHOD
As illustrated in Figure 2 given a non-rigged mesh and motion description in natural language, our goal is to synthesize realistic animation for the character according to the motion description k. Specificallyy, the input mesh of the character & is fed into the Mesh Handle Predictor X(-) to predict the mesh handle h and the skinning weight of the vertices. Besides, a low-dimensional mesh deformation feature fo of the character ó is extracted to represent the shape of the mesh. Let 0x denote the learnable parameter of the Mesh Handle Predictor, this process is formulated as:
A (6: 0x) (h, 8, f))-
3

---

Published as a conference paper at ICLR 2024
Handle Predictor A(0: 0z)
Handle- brsebaeed drive
2D @ 8, 8,
Motion description
Motion Diffusion t, f pp)
de Mesh
0: Lenmable parameter
x: Motion sequence
s: Skinning weight
Fully supervised
Motion description
f: Feature embodding
8: Motion adaptation
Mesh handle
Weakly supervised
Figure 2: An overview of the TapMo pipeline. Given non-rigged mesh and motion description input by the user, the Mesh Handle Predictor X(.) predicts mesh handles and skinning weights to control the mesh. The Shape-aware Motion Diffusioon pa.)) generates rext-guided and mesh-specific motion for the character using the motion description and the mesh deformation feature fs extracted by the Mesh Handle Predictor.
Subsequently, in Shape-aware Motion Diffusion, the mesh-deformation feature fo and the text em- bedding of the motion description fx are used as the conditions to generate a text-guided motion do and mesh-specific adaptation 8. The text embedding 2, is extracted by the CLIP (Radford et al. 202T model. This inverse diffuse process is formulated as:
p (Ii, fr føi 0p) + (20. 6),
(2)
where pu-) is the conditional denoiser of Diffusion Model and at is the motion of noising step t. 0, is the learnable parameter of иd-). Following Tevet et al. (2022). we predict the real motion Tt. rather than the noise é, Finally, after T 1, 000 times of the inverse diffuse process, the sum of the generated motion Φy and the adaptation 8 are applied the character via a handle-based driving method (-)) with the predicted h and s.
3.1 MESI HANDLE PREDICTOR
Given mesh consisting of V vertices, we designate K mesh handles (ha JK to govern its defor- mation. Consequently, every vertex is associated with --dimensional skinning weight associated with K handles. Overall, for the mesh Φ, the skinning weight is defined as E RV*K where S $1,A S 1 and Ek=1 Si,k 1. is the vertex index. Since characters may have varying shapes and topologies, each handle is dynamically assigned to vertices with the same semantics across dif- ferent meshes, except that the first handle is fixed at the root of the mesh. This adaptive assignment approach enables effective control of local and global motion over heterogeneous 3D characters.
K
Based on the definition of the mesh handles, the motion of the mesh can be expressed as the combination of the translations and rotations of the handles, which involves local translations T' TR. (K 11xx3 local rotations r' E ( global translation 7 E TR:3 and global rota- tion r 3 The rotations are represented by the 6D features proposed by Zhou et al. (2019). This motion can be denoted as 2o = (r', r', -5, and the mesh can be driven by the motion in handle-based manner D(-) according to the following formula:
K
V = 7
Σ
8i,k (ri(V, - hx) + +b) + h2 -hs/ +
(3)
where V E Vx3 the positions of the mesh vertices. The rotations in this formula are represented by the rotation matrix.
In our TapMo, following Liao et al. (022), we employ the Mesh Handle Predictor based on a GCN to locate the handles for the mesh The input feature is the positions and normal of the mesh vertices, which can be represented as o E IR V x6 The outputs are the skinning weight of mesh vertices and the handle positions in terms of the average position of vertices weighted by the skinning weight.
Adaptive Handle Learning. The existing 3D character datasets (Xu et al.] 2019 Adobe) lack handle annotations, and the skinning weights defined independently for each character, which lacks consistency. Therefore, these datasets cannot be directly used as ground-truth. To overcome this limitation, introduce three losses, i.e, Skinning Loss L.. Pose Loss Lp. and Root Loss L,
4

---

Published as a conference paper at ICLR 2024
3,06 Sampled ask
Motton
Limeat
Skeleton free drive
Skeleton
Conditio iskem
Foud bandu
Mil
Spring Loss
Transformer
rech setice
rard martion
Limort
ExtxA
Linear
Lincar
Discmminatoo
Mesh
Real fake? Adversurial
Motion Diffasion
+Motion
Figure 3: The structure and the training strategy of Shape-aware Motion Diffusion. The Mo- tion Diffusion lears to generate shape-aware motions based on the textual descriptions, as well as the mesh deformation features with only the ground truth text-motion data of the SMPL model. The parameters of the Handle Predictor are frozen during the second stage to provide stable mesh deformation features. th is random motion sequence in skeletal representation.
to achieve adaptive handle learning. Among them, the Skinning Loss L, and Pose Loss Cp stem FromLiao et al. (022) and will detailed in the Appendix.
Root Loss L, is to enforce the first handle point close to the character's center hip, which is an important step in animation for controlling the global motion independently. [iao et al. (2022) do not take into account the global motion of the character and thus produce severe mesh distortion when applied to drive complex motions. We observe that in the traditional skeleton rigging system, the first joint of the character is always located in the hip part, which is also considered the root joint of the character. Thus, we use D, to make the predicted position of the first handle h1 close to the position of the root joint b in the character. L, is formulated as:
C, = |//
b1 b1ll2
Σ
i=1 St.1 V
(4)
b
1im1 81,1
With the three losses introduced above, the Handle Predictor can be trained by:
min L. + uphop + 14 Lr, 0x
(5)
where 24 and Up are the loss balancing factor.
3.2 SHAPE-AWARE MOTION DIFFUSION
The Mesh Handle Predictor allows for controlling heterogeneous characters using a unified mo- tion representation described 3 However, directly generating this motion representation to synthesize animation can lead to significant mesh distortion due to large variations in body shape among different characters. To address this issue, the Shape-aware Motion Diffusion generates mesh-specific adaptation parameters in addition to the handle motion. This approach preserves the geometric integrity of the animations.
The left part of Figure 3 illustrates the structure of the Shape-aware Motion Diffusion. The model takes the noising step f, the motion description feature fx. and the mesh deformation feature fs as conditions while the t-step noised motion Hy as input to generate the text-guided motion a @!:N of N frames and the mesh-specific adaptation 81:N that are suitable for the mesh á. This process is defined in 2 For brevity, and when there is danger of confusion, we omit the superscripts.
We employ Transformer Decoder (Vaswani et al., 2017) to construct the denoiser of the Diffusion Model and use two linear layers to map the output feature into motion and adaptation, respectively. Unlike MDM (Tevet et al. 2022). which uses only one text token from the last layer of CLIP, we use multiple text tokens from the penultimate layer of CLIP, the number of which is the same as the number of words in the input motion prompt. We have observed that the motion distortion is primarily caused by the generated local translations in our motion representation. Therefore, we only apply adaptations to the local translations of the generated motion. Thus, the dimension of the motion adaptation in one frame is K x 3. The final local translations the motion are obtained by summing up the generated local translations and the mesh mesh-specific adaptations.
To train the Diffusion Model, we fully utilize the existing motion lanioag dataset (Guo et al. 2022) by converting SMPL motions to our Handle-based motion representation using the analytical method
5

---

Published as a conference paper at ICLR 2024
introduced in Besl & McKay (1992) the ground-truth motion Zo- Originating from the simple
loss of DDPM (Ho et al. 2020). a fully-supervised Motion Loss L. is formulated as:
Lm f. ||| .o112
(6)
where the motion to is generated as Eq according to the conditions t, fr. and fø-
Mesh-specific Motion Learning. The existing motion-language dataset only involves human mo- tions of SMPL model, which cannot be directly used to learn shape- aware motions for heteroge- neous characters. To surmount this constraint, we randomly select characters from the 3D character datasets during training and use weakly-supervised manner to assist the model in learning the mesh-specific information from the mesh deformation feature extracted by the Mesh Handle Predic- tor. As the right part of Figure [ shows, in conjunction with the fully-supervised Motion Loss Lm, we designed two reakly-supervised losses, i.e., Spring Loss LA and Adversarial Loss La to achieve mesh-specific motion learning.
The traditional skeletal rigging system of 3D characters has an important property: the length of the bones typically remains consistent during motion. This property ensures that the character's body structure remains stable throughout the animation. Drawing inspiration from this property, we propose the Spring Loss Lp. We model the handles the character's s limbs spring system and penalize the length deformation of this system. Specifically, we pre-define the common adjacency relationships between the handles of the character's 's limbs and use the distance between adjacent handles in the rest-pose as a reference to penalize the distance changes during the character's 's motion:
Lb= = Σ
(E(H).19)+0) [E E h +
IJEA
( kT) - ((
where E() is the Euclidean Distance Function and hyper-parameter. i and j are the indices of the adjacent handles. 1o is the handle position of the rest-pose, h" is that of the n-frame pose. (S00)A9))+)) is an adaptive spring coefficient.
The Adversarial Loss La is inspired by GAN (Goodfellow et al., 2020) and is used to encourage the generated mesh motion to conform to the prior of the skeleton-driven mesh motion on various characters. To reduce computation costs, we randomly sample 100 mesh vertices on the arms in each frame to calculate the Adversarial Loss for each character. This is based on the observation that motion distortion mainly occurs in the arm parts of the characters during movement. We utilize a Discriminator to differentiate the generated handle-driven mesh motions from the skeleton-driven ones. La is designed based on the adversarial training:
La BV BV~p(V-kk log Disc( [V)] +
EV~)(Vo) log (1 Disc( [))]
in which p(-) represents the distribution of the skeleton-driven s or generated V mesh motions.
With the three losses introduced above, the Shape-aware Motion Diffusion can be trained by:
min L. + Un Cn + PaC.: 0,
where and Va are the balancing factors. La and C. are conducted on the motion adaptation 6.
Motion Adaptation Fine-tuning. To further enhance the compatibility of the generated motion for heterogeneous characters and minimize the mesh distortion caused by the motion, we utilize pseudo-labels to finetune the linear mapping layer of the motion adaptation. These pseudo-labels are obtained by optimizing the motion adaptation through an se-rigidda--possibee (ARAP) objective defined as:
O Vo
Σ |E(V VT" E(V, 17>1 1|6' - -8113. 1.jED
(10)
Here, represents the set of edges on the mesh, and 5' represents the optimized motion adaptation at the previous step. v, is the balancing factor.
The linear mapping layer of the motion adaptation is then finetuned by:
min 0
| - 8
(II)
where 5l is the pseudo-labels of the motion adaptations, 0% is the learnable parameters.
6

---

Published as a conference paper at ICLR 2024
Runs with o footbol! and then aimost tripped over stone.
Lylng on the ground and crowling forward in straight line.
MDM SFR MDM-SR
TapMo (Tun))
forword
Running and dribbling
MDM-SR
TapMin (Duss MDM-SPR
Figure 4: Qualitative comparison of the generated animations. The red circle indicates that this sample suffers from severe mesh distortion. The black wireframe indicates that this sample does not align with the motion description.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Baselines. To the best of our knowledge, TapMo is the first pipeline to generate animations for skeleton-free characters in a text-driven manner. We design two reasonable baselines to evaluate the effectiveness of our TapMo. et al. 2022) is popular motion generation method for the SMPL model. We use MDM to generate human motion based on the text, then automatically rig the skeleton template of the SMPL model to the character by Baran & Popović (2007), and finally retarget the human motion to the character by motion copy. 2) MDM + SIPT (MDM-SFR). We use MDM to generate human motion, then use 2022 to transfer the human motion to the character frame by frame.
MDM skeleton-based retargeting (MDM-SR), MDM (Tevet
SfPT (Liao et al.
Datasets. Four public datasets are used to train and evaluate our TapMo, i.e., AMASS (Mahmood et al. 2019), Mixamo (Adobe), SondelsResourceRRiigie (Xu et al. 2019), and HumanML3D (Guo ct al. 2022. Please see the Appendix for more details about datasets.
Metrics. We quantitatively evaluate TapMo from two aspects, i.e., motion quality and geometry quality. For evaluating the motion quality, we follow the approach presented in Guo et al. 2022), where motion representations and text descriptions extractor and then evaluated using five metrics. However, as the motion representation in our TapMo adopt the same configurations
first embedded using traitra feature
differs from that in Guo al. (2022), feature extractor to align with our motion representation. For the geometry quality, we evaluate from mesh vertices and mesh handles using two metrics as follows: 1) ARAP-Loss is used to gauge the level of local distortion of the mesh during the characters' motion, which can be calculated by:
theirs and retrain the
CARAP Σ "IE((V." EVV, V i.j0
(12)
where is the set of edges on the mesh ARAP-Loss is similar to the ARAP objective function proposed in [Q but without the second objective item. 2) Handle-FID. To assess the global geometry quality, we employ two SMPL human models that vary significantly in body size by adjusting the shape parameters. Using FID, measure the distribution distances between the handle positions of the generated motion and the real motion on these two models.
We would encourage the reviewers more about the implementation details in the Appendix.
7

---

Published as a conference paper at ICLR 2024
Figure 5: Visualization of the mesh handles and their corresponding skinning weights. The skinning weights that correspond to the root handle are highlighted in red.
wio WY &
w/o 8 w/ 5
w/o Wi wio
Figure 6: Illustration of the impact of the mesh-specific motion in TapMo. "w/ 8" and "w/o 8" means the animation results with or without equipping the motion adaptations. The mesh-specific motion adaptations generated by TapMo can effectively alleviate the mesh distortion in animations.
4.2 QUALITATIVE RESULTS
Figure iillustrats the overall qualitative results of our TapMo pipeline, demonstrating its impres- sive capabilities in generating reasonable motions and animating a diverse range of 3D characters based textual descriptions. In comparison baseline models, TapMo's motions more closely align with textual descriptions, exhibit less mesh distortion, and offer higher-quality animation. For instance, as highlighted by the red circle in Figure the results produced by MDM-SR and MDM- SFR demonstrate significant mesh distortion, while TapMo preserves the geometry details better. Additionally, the black wireframe displayed in Figure demonstrates that baseline methods tend to generate motions that do not align with the description, while our TapMo benefits from improved Diffusion Model structure, resulting generated motion that is more consistent with the description.
Figure 5 provides a visual representation of the adaptive mesh handles and their corresponding skinning weights for various characters estimated by TapMo's Mesh Handle Predictor: Handles not assigned to any mesh vertices are not drawn in this figure. In humanoid characters, the Mesh Handle Predictor accurately assigns handles to the positions that are near the skeleton joints the human body, resulting in natural and realistic humanoid movements. For non-humanoid characters, the Mesh Handle Predictor assigns mesh vertices to different numbers of handles based on the semantics of the mesh parts. It is worth noting that all characters have root handle located near the center of their bodies, which serves as the key to driving global movement.
Figure 6 iillustrases how mesh-speciiic motion adaptations impact the animation results in TapMo. Due to the significant variation in the shape and topology of characters, the original motion generated by the Diffusion Model may lead to severe mesh distortion. The results exhibit local mesh distortion and inter-penetrations without motion adaptations. The mesh-specific motion adaptations generated by TapMo can effectively alleviate these issues and enhance the geometry quality.
4.3 QUANTITATIVE RESULTS
Table provides a quantitative comparison between TapMo and previous motion generation models, including T2G (Bhattacharya et al.) 2021). Hier (Ghosh et al. 2021). TEMOS (Petrovich et al.
2022). T2M (Guo et al.) 2022). MDM (Tevet et al. (022). and ReMoDiffuse (Zhang et al. 2023). The results demonstrate that our TapMo outperforms previous models significantly in terms of both R-Precision and FID metrics. We introduced multi-toker and Transformer Decoder strategies the Shape-aware Motion Diffusion, which showed a stable improvement in all metrics. Notably. TapMo (Ours) achieved remarkable improvement of over 50% in the FID metric compared the TapMo (MDM) implementation, with a score of 0.515 compared to 1.058. These findings demonstrate the superior performance of our TapMo and the effectiveness of the adaptations we made to the Diffusion Model structure, which allowed for more accurate and higher-quality motion generation.
Table provides comprehensive comparison between TapMo and the baseline methods, focusing on the geometry quality of the generated animations. TapMo demonstrates remarkable improve- ments, achieving a significant reduction ARAP-Loss both on seen and unseen characters. Specif- ically, TapMo achieves 69.8% (0.271 vs. 0.897) lower scores compared to MDM-SR and 73. % (0.271 vs. 1.006) lower scores compared to MDM-SFR on seen characters. For unseen characters, TapMo achieves 58.2% (0.329 vs. 0.787) lower scores compared to MDM-SR and 63.8% (0.329 vs. 0.910) lower scores compared to MDM-SFR, These results indicate that TapMo excels in gen- erating animations with superior geometry details and smoother mesh surfaces compared to the
8

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of text-based motion generation on the HumanML3D dataset. "Real is the real distribution that is extracted from our motion representation by the retrained feature extrac- tor. "TapMo (MDM)" is a implementation that employs the same diffusion structure as MDM.
Method
R-Prec. (
(1op-3),
FID;
Multimodal Dist,
Diversity
Multimodality:
Real
0,511 t.003
0.797 .002
0.002
2.974
9.503
T2G
t.on
0,345:t.00z
7.664
6.030
6.409=-071
(:0]4.000
0.5524.004
6,5322-024
5.0122.018
8.332 042
TEMOS
0-424+-002
0.722
3.734 3.734+.022
3,703 +000
8.973 -973-0071
0.368 +-018
T2M
0.457
0.740+t.ons
1.677t.002
3.340
9.188
2.090
MotionDiiluse
0.491 20001
0.782
0.630 .001
3.113
9.410
1.553
MDM
0.320
0.61 00-007
0,5442.000 -084
3.566-2.0m 027
9.559 9.559 086
2,799 077
ReMoDiffuse
0,510 .055
0,7954.000 0004
0.103 .004
2.974 016
9.018 075
1,795 .08
Realf
0.559 .07
0.828
0.002 .000
4.371
17.176 .178
TapMo (MDM)
0.498 on0e
0.801 0.801+.008 +.ona
1.058 1.058 .500
5.126
16,553 16.553 4.11
2.74
TapMo (Ours)
0.542t.000
0.805 t000
0.515
115.t.00 .88
5.058
16.724
2.55
Table 2: Comparison of mesh distortion. "w/o ft" means the results without adaptation fine- tuning. "A-L" is the ARAP-Loss. "S" and "U"
MDM-SR
MDM-SER
TapMo (Ours)
2:49
2.43
Average Rank
2,4
represent the seen and unseen characters.
2.2
Methods
(S);
A-L
Handle-FID,
1.94
1.99
1.98
MDM-SR MDM-SFR
0.897
0.787
0.012
1.8
1.006
0,910
0.163
1.59
1.54
TopMo (wfo 6)
0.310
0.356
0.055
TapMo (w/ Cx)
0.303
0.347
0.048
Q-Q
T-M
G-Q
TopMo (w/ Ca)
0.307
0.342
0.056
Table 3: User Study of the animation quality.
TopMo (wro 6 (t)
0,301
0.335
0.048
We show the average ranking results where the first is the best.
TapMo (Ours)
0,271
0.329
baseline methods. Moreover, TapMo exhibits superior global geometry preservation performance, with Handle-FID scores 71.7% (0.046 vs 0.163) lower than that of MDM-SFR, Notably, MDM-SR achieves the best Handle-FID (0.012) due to its utilization of standard SMPL skeleton structure, resulting in minimal global distortion when driving the SMPL model. Additionally, the ablative study underscores the effectiveness of the special designs employed in our TapMo pipeline.
4.4
USER STUDY
We conducted a user study to evaluate the visual effects of our Tap against the baseline methods. We invited 100 volunteers and gave them 15 videos. Each video includes one textual description and three anonymous animation results. We ask users to rank the three results in three aspects: overall quality (O-Q), text-motion matching degree (T-M), and geometry quality (G-Q). After excluding abnormal questionnaires, rank of the methods is summarized in Figure Our TapMo outperforms the baseline methods by large margin and more than 84% of users prefer the animation generated by our TapMo.
collect 1,335 ranking comparison statistics in total, and the average
CONCLUSION
In this work, propose novel text-driven animation pipeline called TapMo, which enables non- experts to create their own animations without requiring professional knowledge. In TapMo, two key components
exploited to control and animate wide range of skeleton-free characters. The first is the Mesh Handle Predictor, which clusters mesh vertices into adaptive handles for semantic control. The second is Shape-aware Motion Diffusion, which generates text-guided motions con- sidering the specific deformation properties of the mesh ensuring coherent and plausible character animation without introducing mesh distortion. To train TapMo with limited ground-truth data of 3D characters and text-relevant motions, we propose seaerie-upeerviedd training strategy for adaptive mesh handle learning and mesh-specific motion learning. We con- duct extensive experiments to validate the effectiveness of our method and show that it achieves state-of-ther the-----tttttttt performance compared to baseline methods.
both handle-annotatee
Acknowledgements. This work supported by the fund of Tencent AI Lab RBFR2022012. and the Natural Science Fund for Distinguished Young Scholars of Hubei Province under Grant 2022CFA075.

---

Published as a conference paper at ICLR 2024
REFERENCES
Adobe. Mixamo. https: /ww::///wwww. mi xamo com/
Iyy Baran and Jovan Popović. Automatic rigging and animation of 3d characters. ACM Transactions on graphics (TOG), 26(3):72-es, 2007.
Mirela Ben-Chen, Ofir Weber, and Craig Gotsman. Spatial deformation transfer. In Proceedings of the 2009 ACM SIGGRAAPH/Eungrrphiiss Symposium on Computer Animation, pp. 67-74, 2009.
Paul Besl and Neil D McKay. Method for registration of 3-d shapes. In censorfusion IV: control parudigms and data structures, volume 1611, pp. 586-606. Spie, 1992.
Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: transformer-basee network for generating emotive body ges- tures for virtual agents. In 2021 IEEE virtual reality and 3D user interfaces (VR), pp. 1-10. IEEE, 2021.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang. Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Ex- ecuting your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mofusion: A framework for denoising- diffusion- based motion synthesis. arXiv preprint DrXiv:2212.044455 2022.
Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek. Syn- thesis of compositional animations from textual descriptions. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1396-1406, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11)1139-144 2020.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference Multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5152-5161, 2022.
Shihui Guo, Richard Southern, Jian Chang, David Greer, and Jian Jun Zhang. Adaptive motion synthesis for virtual characters: a survey, The Visual Computer, 31:497-512, 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 technical briefs, pp. 1-4. 2015.
Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG), 35(4):1-11, 2016.
Peizhuo Li, Kfrr Aberman, Rana Hanocka, Libin Liu, Olga Sorkine-Horuung, and Baoquan Chen. Learning skeletal articulations with neural blend shapes. ACM Transactions on Graphics (TOG), 40(4):1-15, 2021.
Zhouyingcheng Liao, Jimei Yang, Jun Saito, Gerard Pons-Moll, and Yang Zhou. Skeleton-free pose transfer for stylized 3d characters. In Computer Vision-ECCe 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part II, pp. 640 656. Springer, 2022.
10

---

Published as a conference paper at ICLR 2024
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprini arXiv:2211.106 2022.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+ 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions pattern analysis and machine intelligence, 42(10))2684-22) 27014 2019a.
Lijuan Liu,, Youyi Zheng, Di Tang, Yi Yuan, Changjie Fan, and Kun Zhou. Neuroskinning: Auto- matic skin binding for production characters with deep graph networks. ACM Transactions on Graphics (TOG), (6644)1-12, 2019b.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned
linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Thalmann Magnenat, Richard Laperrière, and Daniel Thalmann. Joint- Joint-dependent local deformations for hand animation and object grasping. Technical report, Canadian Inf. Process. Soc, 1988.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 5442-5451, 2019.
Bill Manaris. Natural language processing: A human-computer interaction perspective. In Advances in Computers, volume 47, pp. 1-66. Elsevier, 1998.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10985-10995, 2021.
Mathis Petrovich, Michael Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In Computer Vision ECconn-CVVVAA 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXII, pp. 480-497. Springer, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.149900 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMIR, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Robert Sumner and Jovan Popović. Deformation transfer for triangle meshes. ACM Transactions graphics (TOG), 23(3):399-405, 2004.
Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. Variational autoencoders for deforming 3d mesh models. I Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5841-58500 2018
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Haoyu Wang, Shaoli Huang, Fang Zhao, Chun Yuan, and Ying Shan. Hmc: Hicrarchical mesh coarsening for skeleton-free motion retargeting. arXiv preprint arXiv:2303. 1004, 2023a.
11

---

Published as a conference paper at ICLR 2024
Jiashun Wang. Xueting Li, Sifei Liu, Shalini De Mello, Orazio Gallo, Xiaolong Wang, and Jan
Kautz. Zero-shot pose transfer for unrigged stylized 3d characters. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8704-8714, 2023b.
Jing Xu, Wei Zhang, Yalong Bai, Qibin Sun, and Tao Mei. Freeform body motion generation from speech. arXiv preprint arXiv:2203.0223) 0.2991, 2022a.
Zhan Xu, Yang Zhou, Evangelos Kalogerakis, and Karan Singh. Predicting animation skeletons for 3d articulated models via volumetric nets. In 2019 international conference on 3D vision (3DV), pp. 298-307. IEEE, 2019.
Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. Rignet: Neural rigging for articulated characters. ACM transactions on graphics, 39(4), 2020.
Zhan Xu, Yang Zhou, Li Yi, and Evangelos Kalogerakis. Morig: Motion-aware rigging of character meshes from point clouds. In SIGGRAPH Asia 2022 Conference Papers, pp. 1 1-9, 2022b.
Jie Yang, Lin Gao, Qingyang Tan, Huang Yihua, Shihong Xia, and Yu-Kun Lai. Multiscale mesh deformation component analysis with attention- -ased autoencoders. IEEE Transactions Visu- alization and Computer Graphics, 2021.
Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Sorkine- Sorming-Homuung Neural cages for detail-prees prreerring 3d deformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 75-83, 2020.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiifuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval- -ugmeeete motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations neural networks. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019.
A APPENDIX
A.1 SUPPLEMENTARY ON METHOD
We introduce three losses. i.c., Skinning Loss L., Root Loss Cy, and Pose Loss Cp achieve an adaptive handle learning for our Handle Predictor.
Following Liao al. (202)). the assumption of L, that if two mesh vertices belong to the same body part based on the ground-trutt skinning weight, they should also be controlled by the same handle the predicted skinning. We select mesh vertices with 8i,8 > 0.9, 3k and use the KL divergence to enforce similarity between skinning weights of these two vertices:
K
L.
TH.
Σ
(si, log(si,)) Si,k log(s;+))
(13)
where i and j indicate two randomly sampled vertices. 7 is indicator function defined as follows:
Vigj = 1 if i and belong to the same part in the ground-truth skinning weight and 7..j = =- if not.
Pose Loss Lp is ensure that the posed mesh driven by the predicted handle and skinning weight can be as similar as possible to the shape of the mesh driven by the ground-truth skeleton, which helps the predicted handle more in line with the character's 's articulation structure. To achieve this, we randomly select a skeletal pose for the character and drive the mesh using its skeleton to obtain a posed mesh VP. Then, the local translation and local rotation of our representation can be analyt- ically calculated according to the rest-pose mesh and the posed mesh (Besl & McKay 1992). Next, we apply the local motion to the mesh using Eq without the global items to obtain a skeleton-free posed mesh VP. The Cp can be calculated by:
Cp VP 1
(14)
12

---

Published as a conference paper at ICLR 2024
A.2 SUPPLEMENTARY ON EXPERIMENTAL SETUP
Implementation Details. We implement our pipeline using PyTorch framework Paszke et al, 2019). The Mesh Handle Predictor (refer Section B.I) consists of three-layer GCN followed by a three-layer MLP with a softmar activation function. The input feature dimension of the Han- dle Predictor V 6, and the output dimension of the skinning weight is V x K. The handle positions, with a dimension of K x 3, are calculated by averaging the positions of vertices weighted by the skinning weight. The number of the mesh handle K is set as 30. The architecture of the Diffusion Model in our TapMo (see Section 3.2) is illustrated in the left part of Figure 3 with the number of attention heads set to 4. The mesh deformation feature fa is formed by concatenating three-level features within the Mesh Handle Predictor, each with dimensions of 64, 128, and 256 respectively. Subsequently, a linear layer is employed in the Diffusion Model to map the mesh deformation feature to a dimension of 512.
In training, the motion sequence length N is padded to 196 frames, and the text length is padded to 30 words. The text feature and the mesh feature are concatenated form the condition token, with dimension of 31 x 512. Padding masks for the motion sequence and text are utilized during training to prevent mode collapse. The loss balancing factors Vy, Vy, 2h, and Vo are set as 0.1, 1.0, 0.001, and 0.1, respectively. We use an Adam optimizer with a learning rate of le-4 to train the Handle Predictor. The batch size is 4, and the training epoch is 500. To train the Diffusion Model, we also use an Adam optimizer with learning rate of le-4. The batch size is 32 and the training step is 800,000. The finetune step is set as 100,000. The optimization process involves simultaneously optimizing the Diffusion Model and the mesh motion discriminator. In addition, margin parameter of 0.3 is introduced, dictating that the discriminator is optimized only when the score of the fake sample exceeds the margin parameter, and vice versa.
Datasets. Four public datasets are used to train and evaluate our TapMo, i.e., AMASS (Mahmood et al. 2019), Mixamo (Adobe), ModelsRessourceRiigee RigNet (Xu al. 2019), and HumanML3D (Guo et al.
2022).
AMASS is a comprehensive human motion dataset comprising over 11,000 motion sequences. This dataset was created by harmonizing diverse motion capture data using a standardized pa- rameterization of the SMPI model. The SMPI, model disentangles and parameterizes human pose and shape, allowing for the generation of a range of body shapes by manipulating the shape parameters.
Mixamo is animation repository consisting of multiple 3D virtual characters with diverse skele- tons and shapes. In our training process, we utilize 100 characters and 1,112 motion sequences from this dataset. By matching the corresponding joints of different characters, these motion se- quences can be applied to any character in the dataset, allowing for a wide range of animations to be used in our pipeline.
ModelsRessurcce RigNet is 3D character dataset that contains 2,703 rigged characters with large shape variety. Each character has one mesh in the rest pose and has its individual skele- tal rigging system with skinning weight. The meshes of these characters are heterogeneous in number, order, and topology of the vertices. We follow the train-test split protocol in Xu et al. 2019).
HumanML.3D ML3D is a large-scale motion-languaan dataset that textually re-annotating motion capture data from the AMASS and HumanAct12 (Guo et al., 2020) collections. It contains 14,616 motion sequences annotated by 44,970 textual descriptions. The motion sequences in Human are all fit for the SMPL model and represented by concatenation of root velocity, joint positions, joint velocities. rotations, and the foot contact binary labels. We convert these SMPL motions to our ground-truth motions using the analytical method introduced in Besl & McKay (9922) and
follow the train-test split protocol in Guo al. (2020).
Metries. The five metrics Guo et al. 2022 for evaluating motion quality are listed as follows:
R-Precision. We provide single motion sequence and 32 text descriptions, consisting of one ground truth and 31 randomly selected mismatched descriptions. We calculate the Euclidean distances between the motion and text embeddings and rank them. We report the accuracy of motion-to-ted textet retrieval at Top-1, Top-2, and Top-3 accuracy.
Frechet Inception Distance (FID). Calculating the distribution distance between the generated and real motion using FID (Heusel et al. 2017) the extracted motion features.
13

---

Published as a conference paper at ICLR 2024
faces first on the ground and swims doing dolphin
jogged olong a zigoog line, then Stopped and roised honds.
Bina jellyinn
Arphsst with long tongue
Aquivkeard
hond
AB overweight superman
A gentio in white suit
Figure 7: Qualitative results of executing the same motion descriptions with characters of dif- ferent shapes. TapMo can generate mesh-specific motions for characters without causing noticeable mesh distortion while utilizing adaptive handles to drive the characters naturally.
Run in place.
Stop forward and look left and right.
Sourte medinn
Roise and spread orms repeutedly, (edil upper body)
Wolk forward carefully, (odit lower body)
Figure 8: Examples of animation editing. The left part and the right part of this figure are the generated results of editing the upper body and lower body of the character, respectively. Please note that we edit the upper body (the left one) and the lower body (the right one) based on different text inputs.
Multimodal Distance (MM-Dist). The average Euclidean distances between each text feature and the generated motion feature corresponding to this text.
Diversity. We randomly select 300 motion pairs from a set, extract their features, and calculate the average Euclidean distances between the pairs to quantify motion diversity within the set.
Multimodality (MModality). We generate 20 motion sequences for each text description, result- ing in 10 pairs motion. For each pair, we extract motion features and calculate average Euclidean distance between them. The reported metric is the average distance over all text de- scriptions.
A.3 SUPPLEMENTARY ON EXPERIMENTS
Figure shows the results of TapMo in generating animations that match the same motion descrip- tions for characters with vastly different body shapes. TapMo can generate mesh-specific motions for characters without causing noticeable mesh distortion while utilizing adaptive handles drive the characters naturally. For instance, as depicted in the left portion of Figure 7 TapMo accurately replicates the dynamic postures of both jellyfish and humans swimming. In the example on the right side of Figure 7 ghost with legs, hand, as well as gorilla, can all execute the running and raising hands motion. Overall, the motion generated by TapMo plausible and lifelike.
We have implemented an additional application of TapMo, which is animation editing. This appli- cation allows users to make edits to the animation during the motion generation process without requiring any additional training- Users can fix handles that they do not want to edit and let the model generate the rest of the animation. We experiment with editing the upper and lower body mo- tion of the characters, which is achieved by overwriting Eo with part of the input motion during the sampling process of the Diffusion Model. Figure g demonstrates that TapMo is capable of editing animations according to the text description while maintaining motion consistency.
14

---

Published as a conference paper at ICLR 2024
Rest pose
Standing
Figure 9: A failure case. The generated animation exhibits obvious mesh distortion and breakage when the rest pose the character is non-standard.
Figure demonstrates failure case of TapMo, where the character animation exhibits significant mesh distortion and breakage. This issue arises due to the non-standard rest pose of the character, which deviates from the typical T-pose used in our training data. Consequently, TapMo faces limi- tations in generating animations for characters with non-standard rest poses. It is important to note method, traditional animation
poses is unique creation also requires consistent rest poses for target characters. Addressing this bias and finding solutions for character rest pose variations an essential area for further research in the field of baaming-based animation.
that this bias related different
Generalizability. TapMo achieves high-quality geometry animations for unseen characters, as dicated by the quantitative results in Table Moreover, the green photo elf and the Sponge Bob in Figure the polygon tree Figure and hand Figure ters. Most of the characters shown in experiments lacked ground-truth motion data during training. Thus, our TapMo exhibits strong generalizability across diverse mesh topologies, consistently de- livering superior-quality animations for both seen and unseen 3D characters.
internet-sourced wild charac-
Limitations this work are listed as follows:
Due to the weakly-supervised training strategy on SMPL motions, our TapMo is limited in gener- ating rich animations for quadrupeds or characters with significantly different locomotion patterns from humans. possible solution this issue to collect larger and more diverse dataset of language-motion pairs for wide range of characters.
Before the motion generation of TapMo, generating meshes from textual descriptions using ex- isting mesh generation methods like DreamFusion (Poole al. 2022) Magic3D (Lin al.) 2022) is preferable approach to achieve a comprehensive text- bxtt-baed animation pipeline. Unfor- tunately, current mesh generation methods are constrained to producing simple objects like tables, chairs, or airplanes due to the lack of character-langee dataset. Generating complex animated characters from textual descriptions remains an exciting area for future research.
15


Reviews:
[{"review_id": "6m2N29nRqF", "reviewer": "Reviewer_6xCW", "summary": "The main contribution of this paper is a motion diffusion model that can take shape-deformation features as inputs to generate shape-aware motions. This function is desirable in the computer animation, which can save a lot of efforts in the animation production. The proposed pipeline has two components: mesh handle predictors to predict skinning weights and underlying skeletons and shape-aware motion diffusion models that can synthesizes motion with mesh-specific adaptations.", "questions": "Please clarify how the adversarial loss is trained in the rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  the generated animations for a variety of 3D characters are impressive. The structure of the 3D meshes are well recognized when associating it with the animations. \n\n2. The application of shape deformation feature in animation is nice.", "weaknesses": "There are still penetrations between foot and ground in the generated animations, which downgrade the animation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Azu24TAzzU", "reviewer": "Reviewer_JBvr", "summary": "TapMo is a text driven motion synthesis framework for skeleton-free 3D characters. Addressing limitations of relying on pre-rigged character models, TapMo introduces the Mesh Handle Predictor and the Shape-aware Motion Diffusion. These components enable the framework to generate motions of skeleton-free characters using text descriptions. Specifically, Mesh Handle Predictor predict the skinning weights and can clusters mesh vertices into adaptive handles. Then the Shape-aware Motion Diffusion takes mesh deformation feature and output handles' motion. Trained in a weakly-supervised manner, TapMo demonstrates its performance in generating animations for novel non-human characters.", "questions": "Compared to Eq1 in SfPT, if we ignore the global translation and rotation of the root, authors added $\\tau^{l}_k$ and $h_k$, which seems different from SfPT Eq1. Could the authors explain the purpose of this?\n\nFor local translations and local rotations, their first dimension is K. Does this include the root (first) handle? Is k=1 represent root handle in Eq. 3? Or there are K-1 local translations/rotations from k=2 to k=K.\n\nIn Eq.7, defining handle adjacent handles is required, is this something that needs to be done separately for each character? For instance, shapes with large differences as in Figure 6. Is this only necessary during training or during inference? And how many characters need to be specifically defined?\n\nIn the first paragraph of the Method section, does \"skinning weight s of the handle\" refer to the skinning weights of the vertices?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The research addresses an interesting and promising problem, as far as I know it is the first attempt to enable text-driven motion synthesis for skeleton-free characters.\n\nComprehensive experiments are conducted, yielding impressive results across diverse shapes. Supplementary videos and a user study further validate the naturalness of the generated results.\n\nThe combination of diffusion-based motion synthesis and skeleton-free mesh deformation is interesting and novel.", "weaknesses": "Some details are not clearly explained, such as the mesh deformation feature, what exactly is f_ and how it's obtained, and its dimensions, which are not reflected in the main text. From the appendix, it seems to be a 512-dimensional vector. Further explanation from the authors is desired. And how does mesh-specific adaptation affect the vertices, it is not included in the equations. How is the Discriminator implemented? Are the two modules trained jointly or separately?\n\nWhat are the visualization and qualitative results for Handle-FID? Specifically, what are the two SMPL human models and what do the results look like? On the other hand, are two models enough? Why don't authors try MGN (\"Multi-garment net: Learning to dress 3d people from images\") which includes clothed human models and can use SMPL pose parameter to drive.\n\nI recommend the authors discuss the following skeleton-free papers: \"Zero-shot Pose Transfer for Unrigged Stylized 3D Characters\" and \"HMC: Hierarchical Mesh Coarsening for Skeleton-free Motion Retargeting\". The self-supervised shape understanding method in the former could potentially strengthen handle prediction including mesh deformation feature extraction.\n\nThe authors could try using the PMD metric in SfPT and the previous works, which allows for direct comparison with ground truth vertices and can judge both motion and mesh quality, this is suitable for SMPL-based models.\n\nCan the authors provide qualitative results before and after Motion Adaptation Fine-tuning? The post-process usually results in big differences in naturalness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Doe5UOxFWc", "reviewer": "Reviewer_e3Gp", "summary": "This work focuses on an interesting research topic - synthesizing motions for skeleton-free 3D characters, with two main modules: 1. Mesh Handle Predictor, and 2. Shape-aware Motion Diffusion Module.  In addition, this work utilizes the shape deformation-aware features as a condition to guide the motion generation for specific character models. The proposed method could show impressive generated animations for both seen and unseen characters.", "questions": "The authors could answer my questions asked in the weakness section first.\nBesides, I would like to suggest the authors could improve its readability and also highlight the key design.\nFor the experiments, I would like to see more generated motions, especially for non-humanoid characters and unseen characters.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. it is good to study generating shape-aware motions, especially for non-humanoid 3D characters.\n2. The proposed method seems to be reasonable and might be promising to generate motions for unseen characters.", "weaknesses": "1. The proposed mesh handle predictor is simple and straightforward, but it is not clear how the proposed method resolves different characters that have different topologies with different semantics.  Currently, the manuscript mentions that \"each handle is dynamically assigned to vertices with the same semantics across different meshes\", but it is not clear how the method will select those handles.\nAlso, it is unclear how the method will choose the number of handles since different topologies tend to have different numbers of handles. It is unclear how the proposed method could achieve training with different numbers of handles.\n\n2. The proposed Shape-aware Motion Diffusion seems to be simple modifications for existing methods (e.g., MDM), but the current presentation makes it overcomplicated, and difficult for readers to get the key designs that could highly improve the motion generation quality. I am not sure if considering the character shapes is the major factor that improves the quality, and the others could make further improvements.\n\n3. The experiments seem to be insufficient.  HumanML3D dataset is the only benchmark that is used to report quantitative comparisons.  However, recent methods, such as MotionDiffuse[1], ReMoDiffuse[2], and T2M-GPT[3] have not been discussed.\n\n[1] Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022). Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001.\n\n[2] Zhang, M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., ... & Liu, Z. (2023). ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. ICCV 2023\n\n[3] Zhang, J., Zhang, Y., Cun, X., Huang, S., Zhang, Y., Zhao, H., ... & Shen, X. T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7O5j9tpXMm", "reviewer": "Reviewer_qE61", "summary": "The paper (TapMo) tackles text-guided character motion generation in a skeleton-free manner. It can be thought of as the intersection of  MDM (a human-specific motion generation method) and SfPT (a category-agnostic per-frame pose transfer method). Combining the two enables new capabilities -- text-guided motion generation for generic shapes.\n\nMethod-wise, instead of simply combining the two prior works, TapMo made innovations on (1) modeling root movement by an additional root handle (2) taking shape into account when generating handle movements (3) introducing a delta term to account for motion that cannot be explained by a fixed number of handles.\n\nThe video nicely demonstrates the capability of motion generation beyond human characters.", "questions": "1. The model is trained on text-motion pairs of human motion but seems to be able to generalize to other categories like quadruped animals. This is interesting. Is the text description transferable over bipeds and quadrupeds? Has the model seems quadruped motion before? \n\n2. The methods use both diffusion-based reconstruction loss and GAN loss. In which case is the GAN loss necessary/complementary to diffusion loss? What happens if GAN loss is removed?\n\n3. In Table 1, the TapMo variants have a much higher Diversity metric than top rule methods. Is there an explanation? Additionally, the Multimodal Dist and Multimodality results are not explained in the paper. Explanations on why certain method performs better/worse would be useful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Significance**\n- The paper is a nice first step toward motion generation for generic characters. \n\n**Quality**\n- The method is sound and relatively simple.\n- The motion generation results are impressive, especially for the walking hand and animals. \n\n**Presentation**\n- The visual illustrations are well done, and the paper is easy to follow. I enjoyed reading them.", "weaknesses": "**Data**\n- The metrics are reported on a human dataset (HumanML3D), which does not show off the cross-category generation ability of the proposed method. Evaluating the method on a dataset with characters beyond humans would be beneficial, such as deforming things 4D [A]. \n\n**Method**\n- Driving signal. The driving signals seem to be limited to text in the current form. However, there are motions that cannot be described purely by natural language, such as body gestures, facial expressions, hair movements, etc. This is not necessarily a weakness, as the paper already made the setup clear.\n- Representation. To generalize to fine-grained motion, such as cloth deformation and hair, the handle-based deformation with a limited number of handles (K=30) seems not enough. \n\n\n[A] Li, Yang, et al. \"4dcomplete: Non-rigid motion estimation beyond the observable surface.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)