Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs
Authors: Peng Jin
Yang Wu
Yanbo Fan
Zhongqian Sun
Yang Wei
Li Yuan
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-driven Motion Synthesis
Diffusion Models
Graph networks
TL;DR: We propose hierarchical semantic graphs for fine-grained text-to-motion generation. This approach enhances the fine-grained correspondence between textual data and motion sequences and achieves better controllability.

Abstract: Most text-driven human motion generation methods employ sequential modeling approaches, e.g., transformer, to extract sentence-level text representations automatically and implicitly for human motion synthesis. However, these compact text representations may overemphasize the action names at the expense of other important properties and lack fine-grained details to guide the synthesis of subtly distinct motion. In this paper, we propose hierarchical semantic graphs for fine-grained control over motion generation. Specifically, we disentangle motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Such global-to-local structures facilitate a comprehensive understanding of motion description and fine-grained control of motion generation. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments on two benchmark human motion datasets, including HumanML3D and KIT, with superior performances, justify the efficacy of our method. More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact on the community. Code and pre-trained weights are available at https://github.com/jpthu17/GraphMotion.

Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs
Peng Jin 1.4
Yang 3u
Yanbo Fan
Zhongqian Sun
Yang Wei
Li Yuan 12,44 1.,4
School of Electronic and Computer Engineering, Peking University, Shenzhen, China Peng Cheng Laboratory, Shenzhen, China Tencent AI Lab, China
4 Al for Science (AleS)-Preferred Program, Peking University Shenzhen Graduate School, China
jp21 @stu.pku.edu.cn dylan.yangwuugg.com
yuanli-ece pku.edu..cn
Abstract
Most text-driven human motion generation methods employ sequential modeling approaches, e.g., transformer, to extract sentence-level text representations auto- matically and implicitly for human motion synthesis. However, these compact text representations may overemphasize the action names the expense of other important properties and lack tineggraine details to guide the synthesis of sub- tly distinct motion. this paper, we propose hierarchical semantic graphs for fine-grained control over motion generation. Specifically, we disentangle motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Such global-to-local structures facilitate comprehensive understanding of motion description and fine- fine-grained control motion generation. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the texi-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments on two benchmark human motion datasets, in- cluding HumanML3D and KIT, with superior performances, justify the efficacy of our method. More encouragingly, by modifying the edge weights of hierarchical semantic graphs, method can continuously refine the generated motion, which may have a far-reaching impact on the community. Code and pre-trained weights are available https://gihubb.oo//ppp 7/GraphMoin/r.
Introduction
Human motion generation is fundamental task in computer animation [54, 4] and has many practical applications across various industries including gaming, film production, virtual reality, and robotics [61, 8]. With the progress made in recent years, text-driven human motion generation has allowed for the synthesis of a variety of human motion sequences based on natural language descriptions. Therefore, there is growing interest in generating manipulable, plausible, diverse, and realistic sequences of human motion from flexible natural language descriptions.
Existing tootoooooonnn generation methods [42, 61, 8, 1, 63] mainly rely on sentence-level repre- sentations of texts and directly learn the mapping from the high-level language space the motion sequences. Recently, some works [54, 8, 62] propose conditional diffusion models for human motion synthesis and further improve the synthesized quality and diversity. Although these methods have made encouraging progress, they are still deficient in the following two aspects. (i) Imbalance. The model, which directly uses the transformers [55] to extract text features automatically and implicitly, may overemphasize the action names at the expense of other important properties like direction and
"Corresponingg author: Yang Wu, Li Yuan.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Mocions
A man walks forward, then squals to pick something up with both hands, stands back up, and resumes walking.
Overall
Motion
Coarse-to- fine Diffusion
refine
actipe-
action
action
action
Actions
walks
squais
pick
stands
resumes
Local
Actions
disection ageut
time
agent
agent
agent
parlent
refine
agent
instumenn, benefactive
patlent
manger
Action
Specifics
a man
forward
then
something
with both hands
back
walking
Specifics
Figure 1: We propose hierarchical semantic graphs, a fine-grained control signal, for text-to- motion generation and factorize text-fo-motion generation into hierarchical levels including motions, actions, and specifics to form a coarse-to-fine structure. This approach enhances the fine- grained correspondence between textual data and motion sequences and achieves better controllability conditioning on hierarchical semantic graphs than carcfully designed baselines.
intensity. As a typical consequence of this unbalanced learning, the network is insensitive to the subtle changes in the input text and lacks mne-grained controllability. (ii) Coarseness. On the one hand, motion descriptions frequently refer to multiple actions and attributes. However, the compact sentence-level representations extracted by current works usually fail to convey the clarity and detail needed fully understand the text, leading to a lack of fine-gree fine-grained grained details guide the synthesis of subtly distinct motion. On the other hand, mapping directly of existing works from the high-level language space to motion sequences further hinders the generation of fine-grained details. Therefore, we argue that it is time to seek a more precise and detailed text-driven human motion generation method to ensure an accurate synthesis of complex human motions.
To this end, we propose a more fine-grained control signal, hierarchical semantic graphs, to represent different intentions for controllable motion generation and design a coarse-to-fine motion diffusion model, called GraphMotion. As shown in Fig. I, motion descriptions inherently possess hierarchical structures and can be represented as hierarchical graphs composed of three types of abstract nodes, namely motions, actions, and specifics. Concretely, the overall sentence describes the global motion involving multiple actions, e.g., "walk", "pick", and "stand" in Fig. 1, which occur in sequential order. Each action consists of different specifics that act as its attributes, such as the agent and patient of the action. Such global-to-local structures contribute to a reliable and comprehensive understanding of motion descriptions. Correspondingly, to take full advantage of this fine-grained control signal, we decompose the text-to-motion diffusion process into three semantic levels from coarse to fine, which are responsible for capturing the overall motion, local actions, and action specifics, respectively.
The proposed GraphMotion has three compelling advantages: First, the explicit factorization of the language embedding space enables us to build a fine-grained correspondence between textual data and motion sequences, which avoids the imbalanced learning of different textual components and coarse- -raiined control signal representation. Second, the hierarchical refinement property of GraphMotion allows the model to progressively enhance the generated results from coarse to fine, which avoids the coarse -arriee generated results. Third, to further fine-tune the generated results for more fine-grained control, our method can continuously refine the generated motion by modifying the edge weights of the hierarchical semantic graph. Experimental results on two benchmark datasets for text-to-motion generation, including HLmanML3D [14] and KIT [43], demonstrate the advantages of GraphMotion. The main contributions of this work are as follows:
To the best of our knowledge, we are the first to propose hierarchical semantic graphs, a fine-grained control signal, for text-to-motion generation. It decomposes motion descriptions into global-to-local three types of abstract nodes, namely motions, actions, and specifics.
Correspondingly, we decompose the eoxttoo-motion diffusion process into three semantic levels. This allows the model to gradually refine results from coarse to fine. Experiments show that our method achieves new state-of-the-nn results on two text-to-motion datasets.
More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact.

---

2 Related Work
Text-driven Human Motion Generation. Text-driven human motion generation aims generate 3D human motion based on text descriptions. Due to the user-friendliness and convenience natural language [22]. text-driven human motion generation is gaining significant attention and has many applications. Recently, two categories of motion generation methods have emerged: joint-latent models [2, 42] and diffusion models [8, 62, 47]. Joint-latent models, e.g., TEMOS [42], typically learn a motion variational autoencoder and a text variational autoencoder. These models then constrain the text and motion encoders into shared latent space using the Kullback-Leible divergences [29] loss. The latter category of methods, e.g., MDM [54], proposes a conditional diffussin model for human motion generation to learn powerful probabilistic mapping from the textual descriptors human motion sequences. Although these methods have made encouraging progress, they still suffer from two major deficiencies: unbalanced text learning and grasse- generated results. In this paper, we propose to leverage the inherent structure of language [6, 58] for motion generation. Specifically, introduce hierarchical semantic graphs more effective control signal for representing different intentions and design a coarse-to-fine motion diffusion model using this signal.
Diffusion Generative Models. Diffusion generative models [49, 18, 11, 21, 50] type of neural generative model that uses the stochastic diffusion process, which based on thermodynamics. The process involves gradually adding noise to sample from the data distribution, and then training a neural network to reverse this process by gradually removing the noise. In recent years, diffusion models have shown promise in variety of tasks such image generation [18, 50, 11, 19, 57], natural language generation [3, 36, 13], as well as visual tasks [9]. Some other works [25, 32] have attempted adapt diffusion models for pross-modal retrieval [33, 34, 35, 23, 24]. Inspired by the success of diffusion generative models, some works [62, 54, 8] have applied diffusion models to human motion generation. However, these methods typically learn a one-stage mapping from the high-level language space to motion sequences, which hinders the generation of fine-grained details. In this paper, we decompose the text-to-motion diffusion process into three semantic levels from coarse to fine. The resultant levels are responsible for capturing overall motion, local actions, and action specifics, which enhances the generated results progressively from coarse to fine.
Graph-based Reasoning. The graph convolutional network [28] is originally proposed to recognize graph data. It uses convolution on the neighborhoods of each node to produce outputs. Graph attention networks [56] further enhance graph-based reasoning by dynamically attending to the features of neighborhoods. The graph- reasoning has shown great potential in many tasks, such as scene graph generation [60]. visual question answering [20, 31, 30], natural language generation [6]. and cross-modal retrieval [7]. In this paper, we focus on reasoning over hierarchical semantic graphs on motion descriptions for aineggraieed control of human motion generation.
3 Methodology
In this paper, we tackle the tasks of text-driven human motion generation. Concretely, given an arbitrary motion description, our goal is to synthesize a human motion æl:l = =(x" {(x)) H of length L. The overview of the proposed GraphMotion is shown in Fig. 2.
3.1 Hierarchical Semantic Graph Modeling
Existing works for text-driven human motion generation typically directly use the transformer [55] to extract text features automatically and implicitly. However, motion descriptions inherently possess hierarchical structures that can be divided into three sorts of abstract nodes, including motions, actions, and specifics. Compared with the sequential structure, such global-to-local structure contributes to a reliable and comprehensive understanding of the semantic meaning of motion descriptions and is promising fine mee-griined control signal for
generation.
Semantic Role Parsing. To obtain actions, attributes of action as well as the semantic role of each attribute to the corresponding action, we implement a semantic parser of motion descriptions based on a semantic role parsing toolkit [48, 7]. We extract three types (motions, actions, and specifics) of nodes and twelve types of edges to represent various associations among the nodes. For details about the types of nodes and edges, please refer to our supplementary material.
3

---

Hierarchical Semantic
Coarse-to-Fine Motion Diffusion Model
Graph Reasoning
ymm
Semantic Role
Parsing
Mation level
VT
Deration
Transformer
Motion
Embeskdings
yum
p
Action Embeskdings
V2
VS
V4
Deration y times
Transfomer
Action level
Teli Molion decoder D2
ym
Spocific
Embesldings
Di
Iseration
Transforer
7 Eimes
Specific lonel
Figure 2: The overview of the proposed GraphMotion for text-driven human motion generation. We factorize motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Conrespondingly, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics.
Specifically, given the motion description, the parser extracts verbs that appeared in the sentence and attribute phrases corresponding verb, and the semantic role of each attribute phrase. The overall sentence is treated as the global motion node in the hierarchical graph. The verbs are considered as action nodes and connected to the motion node with direct edges, allowing for implicit learning of the temporal relationships among various actions during graph reasoning. The attribute phrases are specific nodes that are connected with action nodes. The edge type between action and specific nodes is determined by the semantic role of the specifics in relation to the action.
Graph Node Representation. Given the motion description, we follow previous works [53, 54, 61, 8] and leverage the text encoder of CLIP [45] to extract the text embedding, For the global motion node ym we utilize the [CLS] token to summarize the salient event described in the sentence. For the action node v", we use the token of the corresponding verb as the action node representation. For the specific node v², we apply mean-pooling over tokens of each word in the attribute phrase.
Action-aware Graph Reasoning. The interactions across different levels in the constructed graph not only explain the properties of local actions and how local actions compose the global motion, but also reduce ambiguity at each node. For example, the verb "pick" in Fig. 1 can represent different actions without context, but the context "with both hands" constrains its semantics, so that represents the action of "pick up with both hands" rather than "pick up with one hand." Therefore, we propose to reason over interactions in the graph to obtain hicrarchical textual representations.
We utilize graph attention networks [56] (GAT) to model interactions in a graph. Specifically, given the initialized node embeddings fu 2a v* p,), first transform the input node embeddings into higeer-level embeddings h = = h", h1 by:
hmh Wy.. W h = W
(1)
where W E P> is shared linear transformation and D is the dimension of node representation. For each pair (h,, hy } of connected nodes, we concatenate the node h4 E RR" with its neighbor node hy E RR, generating the input data h.j = [h,, h,] IR2D of the graph attention module.
However, in graph with multiple types of edges, the vanilla graph networks need learn separate transformation matrices for each edge type. This can be inefficient when learning from limited motion data, and prone over-fitting on rare edge types.
To this end, we propose to factorize multi-relational weights into two parts: common transformation matrix M E 222xx that is shared for all edge types and relationship embedding matrix M, 2D that is specific for different edges, where N is the number of edge types. Following GAT [56],
we apply LeakyReLU [39] in the calculation of attention coefficients and set the negative input slope
to 0.2. The attention coefficient ēzg is formulated as:
exp(eij)
etj LeakyReLU(M h.y)) + LeakyReLU M hey), ēs ΣLEN, exp(ex)
(2)

---

where Ray E ERIxM is one-hot vector denoting the type of edge between node i and j. N2 is the set of neighborhood nodes of node i. To alleviate over-smoothing [59] graph networks, we apply skip connection when calculating output embeddings. The output embeddings V are formulated as:
Vi o Enght) Us:
JEN;
where a is nonlinear function. Following GAT [56], we use ELU [10] as the nonlinear function σ.
3.2 Coarse-to-Fine Motion Diffusion Model for Graphs
To leverage the coarse-to-fine topology of hierarchical semantic graphs during generation, we also decompose the to--toootiion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. During the reverse denoising process, the fine-grained semantic layer generates results based on the results from the coarse-grained semantic layer. This allows for detailed and plausible representation of the intended motion.
Motion Representation. Following previous works [42, 8, 61], we first encode the motion into the latent space with a motion variational autoencoder [27] and then use diffusion models to learn the mapping from hierarchical semantic graphs the motion latent space.
Specifically, we build the motion encoder and decoder based on the transformer [55, 41]. For the motion encoder to generate motion latent embeddings E Cx''' where D' is the dimension of latent representation. For the motion decoder DD we take the latent embeddings E C'' and the motion query tokens as the input to generate a human motion sequence x. I. {x*) j== with L frames.
take C' leamable query tokens and motion sequence Iاد {x*) L inputs
The loss LVae of the motion variational autoencoder can be divided into two parts. First, we use the mean squared error (MSE) to reconstruct the original input. Second, we use the Kullback-Lebbeer divergences (KL) loss [29] narrow the distance between the distribution of latent space ((2(2r) and the standard Gaussian distribution (O,,I), I). The full loss LAV is formulated as:
LVAE E (x)
|| ((E()))1 + (N (1aa()))))
MSE
KL
where À is the trade-off mean (4s and variance Σg estimated by the model. To generate motion from coarse to fine step by
g= N( E,) is obtained by sampling based on the
step, encode motion independently into three latent representation spaces 2 E RC DD' 2a xC'xD and = RC*xD' where the number of tokens gradually increases, i.e., C S C S C
Hierarchical Graph-to-Motion Diffusion. Corresponding to the three-level structure of the hierarchical semantic graphs, we decompose the diffusion process into three semantic levels and build three transformer-basee denoising models, which correspond to motions, actions, and specifics.
For the motion level model 6m. our goal is to learn the diffusion process from global motion node noised sampled
ym to motion latent representation 2 "" forward diffusion process from Gaussian distribution is added to ground truth data distribution 20" at every noise level t:
T
q<<;"<=;-)) N(2 /1 -,22 3,1),
q(2 20 = II g( 2TT1),
(5)
where Be is the step size which gradually increases. 7 is the length of the Markov chain. We sample
z!" by - Var=l" + /I Cr,"", where ã, -I (1 B.). em is a noise sampled from (0, 1). We follow previous works [18, 8] and predict the noise component "", i.e., e Qm("" 1m ymm).
For the action level model da. to leverage the results generated by the motion level, we concatenate the action node V the motion node ymi and the result zd generated by the motion level together as
the input of the action level denoising network, i.e., ea ea 1aa,, tt" [Vm V". 1))
For the specific level model é., leverage the results generated by the action level and nodes at all semantic levels to predict the noise component, i.c., e O2 t 1 Va V 2!)).
5

---

Table 1: Comparisons to current state-of-the-art methods on the HumanML33 test set. "f" denotes that higher is better. "4" denotes that lower is better. "-" denotes that results are better if the metric is closer to the real motion. We repeat all the evaluations 20 times and report the average with a 95% confidence interval. Bold and underlined indicate the best and second-best results, respectively. For fair comparisons, we report results with total diffusion steps TTT T + T of 50 and 150.
Methods
R- R-Precision f
FID 4
MM-Dist L
Diversity + MModality 1
Top-1
Top-2
Top-3
Real motion
0.511 2
0.703
0.7974.000
0.002-.000
2.974
9.5033
Tex12Gesture [5]
0.165
0.2674.002
0.345
7.664#-080
6.030=
6.409
Seg2Seq [44]
0.180 0012
0.300
0.396 1.00
11.75 1o0
5.529 007
6.2234-001
Language2Pose [2]
0.246 0011
0.387
0.486
11.02#0008
5.296 008
7.676
Hier [12]
0.301 #
0,425 00g
0.552 004
6.5322#
5.012 018
8. 8.3312# 042
MDM [54]
0.320 oos
0.498 004
0.611 1-000
0.5444 oc1
5. 8.566= 097
9.559 nww
2.799
TEMOS [42]
0.424
0.612 002
0.722
3.734
3.703 cos
8.973 071
0.368
TM2T [15]
0.424 003
0.618 m03
0,729 1.000
1.501 1..07
3. 3-467--011 011
8. 5.9 o2w
2.424 -00
T2M [14]
0.457 # 0012
0.639
+.000
1.067 0002
3.340=-00s 000
9.188 0012
2.090= 085
MLD [8]
(0.481 1
0.673
0.772
0.473
3.196#-610
9.721 4.02
2.413
MitionDiffuse [62]
0.491 + .011
0.681
0.782
0.630
3.113
9.410
1.553 042
T2M-GPT [61]
0.492 001
0.67y 002
0.775
0.141 000
3.121 009
9.722 002
1.831 048
GraphMation 50)
0.496 0013
h.68
0.778
0.118
3.143 000
9.796
2.603#-005
GraphMotion (5tep=150)
0.504 .088
0.699=.008
0.785 400
0.116 007
3.070 008
9.692 067
2. 2.7665.006
Table 2: Comparisons to current state-of-the-art methods on the KIT test set. We repeat all the evaluations 20 times and report the average with a 95% confidence interval.
Methods
R- P-Poccision 1
FID |
MM-Dist L
Diversity +
MModality 1
Top- 1
Tup-2
Top-3
Real motion
0.424 608
0.649 0.00
0.77g
0.031
2:00
2.788 012
11.08
Sog2Seq [44]]
0.103
0.178 005
0.211
21.86
7.960 ca1
6.711
Text2Gesture [5]
0.156 cos
0.255 004
0.338 1.000
12. 12.1
6.9 6.9464-020 029
9.33441009
MDM [54]
0.164 004
0.201 4.004
t.00
#.021
9.191=-022
10.85
1.907
1 Longuage2Pose [?]
0.221 à
0.373
0.483 1.000
6.545 6.545..072
5. 5.147
9.073
Hier [12]
0.255 008
0.432 007
0.531 #.007
5.203 L07
4.986#.027 027
9.563 072
2. 2.090=.08 085
TM2T [15]
0.280) 005
0.163 006
0,587 000
3.599 153
4 4.591 026
9.173+ 117
3. 3,292 081
TEMOS [42]]
0.3333
0.561
0.687#.006
3.717##0051
3.417#.019
10.9 10.91 221 10.84 100
0.5:32=004
T2M [14]
0.370 ons
0.569 007
0. 0.693 007
2.770 109
3. 3/101##00m cos
1/482 063
MLD
0.390 o08
0.609 .088
0.734
0.404 -227
3.204 022
10.50
2. 2.192
T2M-GPT [61]
0.416 006
0.627 n06
0.7454.000
0.5144#.029
3.007 095
10.92 108
1.570 039
MotionDiffuse [62]
0. 41717 117:2.000 0004
0.621 $1.044
0,739 h.03
1.951
2.95g2t.005
11.10
0.730 0115
GraphMation 50))
0.417 0018
0.635
0.755
f.222
3. 3.085#-081 001
11.21 #:000
3.568#1182
GraphMotion (5tep=150)
0.429 007
0.6188#000
0.769-t.000
0.313
3.076* 022
12:12
$.627#-113
Finally, the training objective of the diffusion models can be defined as:
CDFF = -E.--N(0.11 ||e" (m" tmm, VTT 212 + l|e" - 00(2", tn, 1,m,, vo, 2=1
Motion level
Action level
(6)
||e" do (2 ; t mm Vo, Vo, 2 DIB
Specific level
During the training stage, the motion variational autoencoders are frozen. During the inference stage, the fine-grained semantie layer generates results based on the results from the coarse- coarse-grained semantic layer. We take the output at the specific level as the final result and use the motion decoder to decode the latent representation into the motion sequence.
is worth noting that our method is as efficient as the one-stage diffusion methods during the inference stage, even though we decompose the diffusion process into three parts. This is because we can control the total number 7 + T + T of iterations by restricting to be the same as those of the one-stage diffusion methods, which we will discuss in experiments (see Tab. 5).
4 Experiments
Datasets, Metrics and Implementation Details. Datasets. We compare the proposed method with other methods on two commonly used public benchmarks: HumanML3D [14] and KIT [43].

---

Real
Ours
MDM (2C)
MLD (CVPR23)
MifionDiffuse (arXii22))
"A person takes step forward, moves to the right, then continues forward with their right hand rail."
"The person wulked forward and lifted something up turned around and walked back."
"A person was running by making the circle and return."
Figure 3: Qualitative comparison of the existing methods. We provide the motion results from three text prompts. The darker colors indicate the later in time. The generated results of our method better match the descriptions, while others have downgraded motions or improper semantics, demonstrating that our method achieves superior controllability compared well-designed baseline models.
HumanML3D [14] is currently the largest 3D human motion dataset that originates from and textually reannotates the HumanAct12 [16] and AMASS [40] datasets. This dataset comprises 14,616 human motions and 44,970 text descriptions, with each motion accompanied by at least three precise descriptions. The lengths of these descriptions are around 12 words. KIT [43] contains 3,911 human motion sequences and 6,278 textual annotations. Each motion sequence is accompanied by one to four sentences, with an average description length of 8 words.
Metrics. Following previous works, we use the following five metrics to measure the performance of the model. (1) R-Precision. Under the feature space of the pre-trained network in [14], given one motion sequence and 32 text descriptions (1 ground-trutt -ruth and 31 randomly selected mismatched descriptions), motion-retrieval precision calculates the text and motion Top 1/2/3 matching accuracy. (2) Frechet Inception Distance (FID). We measure the distribution distance between the generated and real motion using FID [17] on the extracted motion features [14]. (3) Multimodal Distance (MM- Dist). We calculate the average Euclidean distances between each text feature and the generated motion feature from that text. (4) Diversity. All generated motions are randomly sampled to two subsets of the same size. Then, we extract motion features [14] and compute the average Euclidean distances between the two subsets. (5) Multimodality (MModality). For each text description, we generate 20 motion sequences, forming 10 pairs of motions. We extract motion features and calculate the average Euclidean distance between each pair. We report the average of all text descriptions.
Implementation Detuils. For the motion variational autoencoder, motion encoder & and decoder D consist layers and heads with skip connection [46]. Following MLD [8], we utilize a frozen text encoder of the representation is set to 768. The dimension of latent embedding D' is set to 256. We set the token sizes C to 2, C to 4, and C* to 8. We set À to 1c-4. All our models are trained with the AdamW [26, 38] optimizer using a fixed learning rate of Ic-4, We use 4 Tesla V100 GPUs for the training, and there are 128 samples each GPU, so the total batch size is 512. For the HumanML3D dataset, the model is trained for 6,000 epochs during the motion variational autoencoder stage and 3,000 epochs during the diffusion stage. The number of diffusion steps of each level is 1,000 during training, and the step sizes 32 are scaled linearly from 8.5x le-4 to 0.012. For runtime, training tasks 16 hours for motion variational autoencoder and 24 hours for denoiser on 4 Tesla V100 GPUs.
[45] model for text representation. The dimension of node
Comparisons State-of-the-art. We compare the proposed GraphMotion with other methods on two benchmarks. In Tab. 1, we show the results the HumanML3D test set. Tab. 2 shows the results
7

---

Table 3: Ablation study about each part of our method on the HumanML3D test set. "f" denotes
Table 4: Ablation study of the coarse-to- fine motion diffusion model on the Hu- manML3D test set. "f" denotes that higher is better. "4" denotes that lower is better.
that higher is better. "V" denotes that lower is better.
Semantic Graph
Course-to-Fine
R-Precision Top-1
FID
Graph Reasoning Diffusion
Motion Action Specific|
R-Precision Top-1 1
FID
0.485=.003
0.418=-013
level level
level
0.49110003 003
0.281#-011
0.4855..003
0.418-1.013
0.494#.003 003
0.212#.015
0.488=t0008
0,217##009
0.490
0.196#.012
0.494
0.150=
0.504
0.116
.003
0.116#.000
Table Ablation study about the total number of diffusion steps on the HumanMML3D test set.
"1" denotes that higher is better. "L" denotes that lower is better. We repeat all the evaluations 20 times and report the average with a 95% confidence interval. "x" denotes that this method does not
apply this parameter. To speed up the sampling process, we use DDIM in practice following MLD.
Diffusion Steps
R-Procision t
Methods
FID J
Motion T
Action T
Specific T
Top-1
Top-2
Top-3
The total number of diffusion steps 1000 with DDPM (18)
MDM [54]
1000
X
0.320 005
0.498800004
0.611 000
0.544 +.44
MitionDiffuse
1000
x
0.491 001
0.681=-001
0.782 001
0. 0.630
total number of diffusion steps 50
[50]
MLD [S]
50
X
0,481
0.673#0003
0.772 +002
0. 0.473
GraphMation (Ours)
20
15
0.489 003
0.676=.002
0.771 0000
0.131
GraphMotion (Ours)
15
0.496
0.686
0.778#.0002
0.118:t.00
total number of diffusion steps with DDIM [50]
MLD
150
x
0.461 + 002
0.649
0.797 002
0.457+.011
GraphMotion (Ours)
50
50
50
0.501 .00
0.699
0.785 002
0.116
on the KIT test set. Our model consistently outperforms state-of-the-as art methods on both benchmarks, which validates the effectiveness of our method. Moreover, provide qualitative motion results in Fig. 3. Compared to other methods, our method generates motions that match the text descriptions better, indicating that our method is more sensitive to subtle differences in texts.
1.2
R-Poodsionn Top
0,500
Actier
Aztion leod
0:475
0.8
Specific lovel
MML Dist
3.4
Specdic lovd.
0.400
0.8
0.425
Moonee lovel
Action levd)
3.2
0.400
Specific levi
0.2
200
400
600
100
1030
1200
1400
200
400
600
800
1000
1200
1400
200
400
600
800
1000
1200
1400
Epoch
Epoch
Epoch
Performance at each level on the HumanML.3D test set. We repeat all experiments 10 times and report the average with 95% confidence interval. The performance of the specific level is the best, which confirms the effectiveness of the coarse-to-fine motion diffusion model.
Figure 4:
Ablative Analysis. Effect of each part of our method. To explore the impact of each part of our method, we provide the ablation results in Tab, 3. "Semantic Graph" refers to the decomposition of the original motion description into nodes of three levels, namely motion, action, and specific. These nodes are then directly fed into the one-stage motion diffusion model to generate the final motion. "Graph Reasoning" utilizes graph network reason and update nodes in the hicrarchical semantic graph. "Coarse-to- three semantic levels. from coarse to fine, which capture the overall motion, local actions, and action specifics, respectively. If not otherwise specified, all ablation experiments are performed using diffusion step setting of 150. As shown in Tab. 3, we find that "Semantic Graph" can avoid the imbalanced learning of different textual components, thus significantly improving the semantic understanding ability of the model and the overall quality of motion generation. "Graph Reasoning" further enhances the semantic reasoning ability of the model. In addition, "Coarse-to- -o-nne Diffusion" can significantly improve the quality of motion generation. Our full model achieves the best performance, which demonstrates that the three parts are beneficial for motion generation.
Diffussio"" decomposes the text-to-motion diffusion process into
8

---

Motions
Figure walks forward then zags left before walking right.
action
actoo
action
clions
walks
zags
w walking
ageed
direction
agent
Bgéett
Liure
patentt
direction
Specifics
figure
forward
then
to left
before walking right
right
Base
Moxiifying the edge
zags
left.
weaken
Base
strengthen
Modifying the edge
walking
weaken
Base
strengthen
Qualitative analysis of refining motion results. The darker colors indicate the later in time. The trajectory of the human body is indicated by an arrow. The trajectory associated with the modified edge is highlighted in red, and other parts are identified in blue.
Figure 5:
Table 6: Quantitative experiment of the imbalance problem on the HumanML3D test set. "f" denotes that higher is better. "L" denotes that lower better.
Table 7: User studies for quantitative comparison. We show the preference rate of GraphMotion over the compared model.
Methods
FID L. MM-Dist 4 Diversity + MModality f
Methods Compared
Preference Rate
MDM
5,622
7.163
8.713
3,578
GruphMotion vs. MotionDiffuse
64.10%
MLD
3,492
5.632
8.874
3.596
GraphMotion vs. MLD
56.41% %
GraphMotion
1.826
5.530
9.284
3.699
GraphMotion vs. Ground Truth
48.72%
Analysis of the course-to-fine motion diffusion model. In Tab. 4, we provide the ablation study of the coarse-to- to-eine motion diffusion model on the HumanML3D test set. These results prove that coarse-to- to--ttofnnn generation is beneficial to motion generation. addition, we show the performance at each level in Fig. 4. Among the three levels, the performance of the specific level is the best, which confirms the effectiveness of the coarse-to-fine motion diffusion model.
Effect of the diffusion steps. In Tab. 5, we show the ablation study of the total number of diffusion steps on the HumanML3D models [50] (DDIM) during interference. As shown in Tab. 5, our method consistently outperforms the existing state-of-the-a
set. Following MLD [8], we adopt the denoising diffusion implicit
methods with the same total number of diffusion steps, which demonstrates the efficiency of our method. With the increase of the total diffusion steps, the performance of our method is further improved, while the performance of MLD saturates. find that the number of diffusion steps at the higher level (e.g., specific level) has greater impact on the result. Therefore, in scenarios requiring high efficiency, we recommend allocating more diffusion steps to the higher level.
Quantitative and Qualitative Discussion. Quantitative experiment of the imbalance problem. In this experiment, we mask the verbs and action names the motion description force the model to generate motion only from action specifics. For example, given the motion description "a person walks several steps forward a straight line.", we would mask "walks". Transformer extracts text features automatically and implicitly. However, it may encourage the model to take shortcuts, such as overemphasizing the action name "walks" the expense of other important properties. Therefore, when the verbs and action names are masked, the other models, which directly the transformer

---

Motions
person walks several steps forward striight line.
action
ctions
walks
agezt
Cinettion
Specifies
person
several steps forward
in a struight line
Base
Bose
Replace the node with [mask]
Modify the node
Delele the nodtes
Add a new node at stumbling pace
walks
[mosk]]
walks
Tuns
S
3
1
6
Figure 6: Additional qualitative analysis of refining motion results. The qualitative results demonstrate that our approach provides a novel method of refining generated motions.
to extract text features, fail to generate motion well. By contrast, the hierarchical semantic graph explicitly extracts action specifics. The explicit factorization of the language embedding space facilitates comprehensive understanding of motion description. It allows the model to infer from action specifics such as "several steps forward" and "in straight line" that the overall motion is "walking forward". As shown in Tab. 6, our method synthesize motion by relying only action specifics, while other methods fail to generate motion well. These results indicate that our method avoids the imbalance problem of other methods.
Human evaluation. study. As shown in Tab. 7, GraphMotion is preferred over the other models most of the time.
our evaluation, we randomly selected 39 motion descriptions for the user
Qualitative analysis of refining motion results. To fine-tune the generated results for more fine- grained control, our method can continuously refine the generated motion by modifying the edge weights of the hierarchical semantic graph. As illustrated in Fig. 5, we alter the action attributes by manipulating the weights of the edges of the action node and the specific node, For example, by increasing the weights of the edges of "zags" and "to left". the human body will move farther the left. Morcover, by fine-tuning the weights of the edges of the global motion node and the action node, we can fine-tune the duration of the corresponding action in the whole motion. For example, by enhancing the weights of the edges of the global motion node and "walking", the length of the walk will be increased. In Fig. 6, we provide additional qualitative analysis of refining motion results. Specifically, we perform the additional operations on the hierarchical semantic graphs: (1) masking
node by replacing it with the MASK token; (2) modifying the node; (3) deleting nodes; (4) adding new node. The qualitative results demonstrate that approach provides a novel method of refining generated motions, which may have far-reaching impact on the community.
Conclusion
In this paper, we focus on improving the controllability of text-driven human motion generation. To provide fine-grained control over motion details, we propose novel control signal called the hierarchical semantic graph, which consists of three kinds abstract nodes, namely motions, actions, and specifics. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the rel-to-mmttion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments demonstrate that our method achieves better controllability than the existing oftte-of-the-at methods. More encouragingly, method can continuously refine the generated motion by modifying the edge weights hierarchical semantic graphs, which may have far far-reaching impact on the community.
Acknowledgements. This work was supported by the National Key R&D Program of China (20227D01 18101), Nature Science Foundation China (No.6220201 4), and Shenzhen Basic Re- search Program NN.JCY22022081311177660000
10

---

References
[1] Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, and Songhwai Oh. Text2action: Generative adversarial synthesis from language action. In ICRA, pages 5915-5920, 2018.
[2] Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose forecasting. In 3DV, pages 719-728, 2019.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. In NewrIPS, pages 17981-17993, 2021.
[4] Norman Badler, Cary Phillips, and Bonnie Lynn Webber. Simulating humans: computer graphics animation and control. Oxford University Press, 1993.
[5] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Tex:gestures: A transformer-based network for generating emotive body gestures for virtual agents. In VR, pages 1-10, 2021.
[6] Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control image caption generation with abstract scene graphs. In CVPR, pages 9962-9971, 2020.
[7] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, pages 10638-10647, 2020.
[8] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your commands via motion diffusion in latent space. In CVPR, pages 18000-18010, 2023.
[9] Zesen Cheng, Kehan Li, Peng Jin, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Parallel vertex diffusion for unified visual grounding. arXiv preprint arXiv:2303.07216, 2023.
[10] Ajórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). arXiv preprint 1rXiv:1511.07289, 2015.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, pages 8780-8794, 2021.
[12] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek. Synthesis of compositional animations from textual descriptions. In ICCV, pages 1396-1406 2021.
[13] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence sequence text generation with diffusion models. arXiv preprint arXiv:2210.089) 00.0333,, 2022.
[14] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, pages 5152-5161 2022.
[15] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, pages 580-597, 2022.
[16] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation 3d human motions. In ACM MM, pages 2021-20000 0021-2029, 2020.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, pages 6840-6851, 2020.
[19] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. axX1v:220..12598, 2022.
arXiv preprint

---

[20] Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-conditioned graph networks for relational reasoning. In ICCV, pages 10294-10303, 2019.
[21] Hyeonho Jeong, Gibyun Kwon, and Jong Chul Ye. Zero-shot generation of coherent storybook from plain text story using diffusion models. arXiv preprint 1.XIi::3302.33900 2023.
[22] Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. Expectation-
pppectationmmxximizzaioo maximization contrastive learning for compact video-and and-language representations.
In NeurIPS. pages 30291-30306 2022.
[23] Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text game players: Hierarchical banzhaf interaction for cross-modal representation learning. In CVPR, pages 2472-2482, 2023.
[24] Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. Text-video retrieval with disentangled conceptualization and set-to-set alignment. In IJCAI, pages
8 2023.
[25] Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Diffusionret: Generative text-video retrieval with diffusion model. ICCV, pages 2470-2481, 2023.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[27] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:13 14, 2013.
[28] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
[29] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
[30] Hao Li, Xu Li, Belhal Karimi, Jie Chen, and Mingming Sun. Joint learning of object graph and relation graph for visual question answering. In ICME, pages 01-06, 2022.
[31] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-awan graph attention network for visual question answering. In ICCV, pages 10313-10322, 2019.
[32] Pandeng Li, Chen-Wee Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momenidiff: Generative video moment retrieval from random to real. arXiv preprint arXiv:2307.0m 02869 2023.
[33] Pandeng Li, Chen- Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive apatio-tempore prototype matching for text-video retrieval. In ICCV, pages 4100-4110, 2023.
[34] Pandeng Li, Hongtao Xie, Jiannan Ge. Lei Zhang, Shaobo Min, and Yongdong Zhang. Dual- stream knowledge- prreeervvgggg hashing for unsupervised video retrieval. In ECCV, pages 181- 197, 2022.
[35] Pandeng Li, Hongtao Xie, Shaobo Min, Jiannan Ge, Xun Chen, and Yongdong Zhang. Deep fourier ranking quantization for semi-supervised image retrieval. TIP, 31:5909-5999 5922, 2022.
[36] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205.1422) 14211, 2022.
[37] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Moll, and Michael J Black. Smpl: A skinned multi-person linear model. TOG, 34(6):1-16, 2015.
[38] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.
[39] Andrew Maas, Awni Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.
12

---

[40] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. In ICCV, pages
2019.
[41] Mathis Petrovich, Michael Black, and Gül Varol. cetion-conditioned 3d human motion synthesis with transformer vae. In ICCV, pages 10985-10995, 2021.
[42] Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In ECCV, pages 480-497, 2022.
[43] Matthias Plappert. Christian Mandery, and Tamim Asfour. The kit motion-languee dataset.
Big data, 4(4):236 252., 2016.
[44] Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems, 109: 109:13 3 26. 2018.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, pages 234-241, 2015.
[47] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as generative prior. arXiv preprint arXiv:2303.01418, 2023.
[48] Peng Shi and Jimmy Lin. Simple bert models for relation extraction and semantic role labeling. arXiv preprint (eX.v0:90.002255 2019.
[49] Jascha Sohh-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In /CML, pages 2256-2263 2015.
[50] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[51] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-199 1958. 2014.
[52] Ömer Terlemez, Stefan Ulbrich, Christian Mandery, Martin Do, Nikolaus Vahrenkamp, and and toolkit for capturing, represent- ing, and reproducing human motion on humanoid robots. In 2014 BEEE-RAS International Conference Humanoid Robots, pages 894-901, 2014.
Tamim Asfour. Master motor map
[53] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionelip: Exposing human motion generation to clip space. In ECCV, pages 358-374, 2022.
[54] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. In ICLR, 2023.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[56] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
[57] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint 12Xiv:2212.004900 2022.
[58] Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual and language learning. CVPR, 2023.
[59] Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisiting over-smoothing in deep gens. arXiv preprint arXhv:2003.13663 2020.
13

---

[60] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cmn for scene graph generation. ECCV, pages 670-685, 2018.
[61] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In CVPR, 2023.
[62] Mingyuan Zhang. Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208, 15001, 2022.
[63] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmenne motion diffusion model. arXiv preprint 16 2023.
14

---

Abstract
This appendix provides additional discussions (Sec. A), implementation details (Sec. B), more qualitative results (Sec. C), several additional experiments (Sec. D), details of motion represen- tations and metric definitions (Sec. E).
Code
Code is available https://github.comm hwtph//thco///uuuu/cco/m 17/GraphMotton. In this code, we provide the process of the training and evaluation of the proposed method, and the pre-trained weights.
A
Additional Discussions
A.1 Potential Negative Societal Impacts
While our work effectively enhances the quality of human motion synthesis, there a potential risk that it may be used for generating fake content, such as generating fake news, which can pose threat to information security. Moreover, when factoring in energy consumption, there is a possibility that the widespread use of generative models for synthesizing human motions may contribute increased carbon emissions and exacerbate global warming.
A.2 Limitations of our Work
Although our method makes some progress, there are still many limitations worth further study, (I) The proposed GraphMotion inherits the randomness diffusion models. This property benefits diversity but may yield undesirable results sometimes. (2) The human motion synthesis capabilities the pre-trained motion variational autoencoders,
of GraphMotion are limited by the performance which we will discuss in experiments (Tab. D and Tab. E). This defect also exists in the existing state-of- thetrh-art methods, such as MLD [8] and T2M-GPT [61]. which also use motion variational computations, still limited by the slow inference speed of existing diffusion models. We will discuss the inference time in experiments (Tab. C). This defect also exists in the existing state-of-the-ar methods, such as MDM [54] and MLD [8], which also use diffusion models.
autoencoder. (3) Though the proposed GraphMotion brings negligible extra
A.3 Future Work
In this paper, we focus improving the controllability of text-driven human motion generation. Recently, large language models have made remarkable progress, making large language models a promising text extractor for human motion generation. However, despite their strengths in general reasoning and broad applicability, large language models may not be optimized for extracting subtle motion nuances. In future research, we will incorporate the features of large-scale languages into our model, using hierarchical semantic graphs to give large language models the ability to extract fine-grained motion description structures.
B
Implementation Details
Table A: Node types and edge types in the parsed hierarchical semantic graph. Each edge type corresponds to type of semantic role.
B.1 Details of Hierarchical Semantic Graphs
To obtain actions, attributes of action as well as the semantic role of each attribute to the corresponding action, we imple- ment a semantic parser of motion descriptions based on a semantic role parsing toolkit [48, 7]. Specifically, given the motion description, the parser extracts verbs that appeared in the sentence and attribute phrases corresponding verb, and the semantic role of each attribute phrase. The overall sentence treated as the global motion node in the hierar- chical graph. The verbs are considered as action nodes and connected to the motion node with direct edges, allowing for implicit learning of the temporal relationships among var- ious actions during graph reasoning. The attribute phrases are specific nodes that are connected with action nodes. The edge type between action and specific nodes is determined by the semantic role of the specifics in relation to the action.
Node type
Description
Motion
global motion description
Action
verb
Specitic
attribute of action
Edge type
Description
ARGO
agent
ARGI
parient
instrument, benefactive
ARG3
start point
ARG4
end point
AROM-LOC
location (where)
ARGM-MNR
manner (how)
ARGM-TMP
time (when)
ARGM-DIR
direction (where tr/frrm)
ARCM-ADV
miscellaneuus
ARGM-MA
motion-actian dependencios
OTHERS
other argument types, c.g., action
15

---

Real
Our
MDM (ICLR23
MLD (CVPR23
MotionDiffuse (aXXiv22)
"Using both hands, a person bends over to pat the ground
"A person sitting on the floor scratches their head and then leans bn their right hand
"A person gets on all fours and crawis forward, then turns around and cravis back
"Person kicks left legs backand forth "
Figure A: Qualitative comparison of the existing methods. The darker colors indicate the later in time. The generated results of our method better match the descriptions, while others have downgraded motions or improper semantics, demonstrating that our method achieves superior controllability compared to well-designed baseline models. We have provided a supplemental video in our supplementary material. In the supplemental video, we show comparisons of text-driven motion generation. We suggest the reader watch this video for dynamic motion results.
As shown in Tab. A, extract three types (motions, actions, and specifics) of nodes and twelve types of edges to represent various associations among the nodes.
B.2 Classifier-free Diffusion Guidance
Following MLD [8], our denoiser network is learned with classifier-free diffusion guidance [19]. The classifier-free diffusion guidance improves the quality samples by reducing diversity in conditional diffusion models. Concretely, it learns both the conditioned and the unconditioned distribution (10% dropout [5i]) the samples. Finally, we perform a linear combination in the following manner, which is formulated
scale = a @mm("", ,.m VmT) + (1 )m ". 1)) @ dcaate a da(2", ,t" IVmm va =) + (1 a (2" Ø), scaale = a (z",", ,t", I va, Vo, 20) + (1 a' )o.(2")
(A)
16

---

person is doing jumping jackissjjcck..
A person takes a jump sideways to their right.
person is doing a dance.
A person hops up and down in place.
A person looks to the right then kicks something with their left foot.
The person is clapping.
D Dancnn randomly like an indian traditional dance and mix of western culture.
A perso bend the knee and leand to the floor and start walked.
A person walking forward then walks back.
A person steps forward and reaches down to grab or place something with their right hand.
Figure B: Additional qualitative motion results generated with text prompts of the Hu- manML3D test set. The darker colors indicate the later in time. These results demonstrate that our method can generate diverse and accurate motion sequences.
Where o' is the guidance scale and a' in practice following MLD. Please refer to our code for more details.
strengthen the effect of guidance [8]. We set a' 7.5
B.3 Implementation Details for Different Datasets
Following MLDD [8], we utilize a frozen text encoder of the CLIP-ViT-L-14 [45] model for text representation. The dimension of node representation D is set to 768. The dimension of latent embedding D' is set to 256. For the motion variational autoencoder, motion encoder & and decoder D all consist of 9 layers and 4 heads with skip connection [46]. We set the token sizes C to 2, C
17

---

Real
Ours
MDM (ICLIZ3)
MLD (CVPR22)
MotionDiffuse (aXXV22))
"The person pick something up and tilted it to the right." "
"The person mmakk] something up and [makk] it to the right."
Figure C: Qualitative analysis on the imbalance problem. The darker colors indicate the later in time. When the verbs and action names are masked, existing models tend generate motion randomly. In contrast, our method can generate motion based solely the action specifics, These results show that our method not overly focused on the verbs and action names.
Motion level
Action level
Specific level
"A man performs a squat while lifting arms to shoulder height and hands above his head."
"A figure standing, turns to look right and vaises the right arm perpendicular to the ground, then looks left and raises left arm perpendicular to the ground."
Figure Qualitative comparison of different hierarchies. The output at the higher level (e-g.. specific level) has more action details. Specifically, the motion level generates only coarse-grained overall motion. The action level generates local actions better than the motion level but lacks action specifics. The specific level generates more action specifics than the action level.
4, and C* 8. We set À to le-4, All our models are trained with the AdamW [26, 38] optimizer using a fixed learning rate of lc-4. We use 4 Tesla V100 GPUs for the training, and there are 128 samples on each GPU, so the total batch size is 512. The number of diffusion steps of each level is 1,000 during training, and the step sizes B3 are scaled linearly from 8.5x le-4 to 0.012. We keep running similar number of iterations on different data sets. For the HumanML33 dataset, the model is trained for 6,000 epochs during the motion variational autoencoder stage and 3,000 epochs during the diffusion stage. For the KIT dataset, the model trained for 30,000 epochs during the motion variational autoencoder stage and 15,000 epochs during the diffusion stage. Code is available at https://github. http://gihhub.com/pphiin com/jpthu wwt.:pthul1//rrappphoo /7/rraph
this code, we provide the process of the training and evaluation of the proposed method, and the pre- traitedined model.
18

---

Table B: Ablation study about the total number of diffusion steps on the HumanML3D test set. "T" denotes that higher is better. "4" denotes that lower is better. We repeat all the evaluations 20 times and report the average with 95% confidence interval. "x" denotes that this method does not
apply this parameter. To speed up the sampling process, we use DDIM in practice following MLD.
Diffassoo Steps
R Precision
Methods
FID J
Motion 1
Action 7
Specilic T
Top-1
Top-2
Top-3
The total number diffasion steps 1000 with DDPM [18]
MDM [54]
1000
X
0.320 000
0.498 0.498=
0.611 +:
0. 0.6444.004
MifionDiffuse [62]
0.491 0000 000
0.681..00)
0.782
0.630
The total number of diffasion steps 50 with DDIM [50]
MLD
50
0,481 003
0.673 003
0.7724: 002
0,473 013
GraphMotion (Ours)
2U
15
15
0.4892 003
0.676mm-022
0.771
0.131
GraphMotion (Ours)
15
15
0.498-t.008
1.888.0000
0.778 tt .002
1.118:0000
The total number of diffusion steps 150 with DDIM [50]
MLD
150
0.461 000
0.649==000
0.797 002
0,4374-011 001
GraphMotion (Ours)
50
50
50
0.501
0.6993.0002
0.785=2.002
0.116
The total number of diffusion steps 300 with DDIM [50]
MI.D [8]
300
x
0.473 002
0.684m.003
0.765
0.403
GraphMotioo (Ours)
100
0.486=t.00
4:004
0.767# 000
4.006
total number of diffusion steps 1000 with [50]
MLD [S]
1000
X
X
0.452#-002
0.639=-005
0.751
0.460
GraphMotion (Ours)
400
300
300
0.475#.008
0.659
07756+000
0.136+000
GraphMotion (Ours)
0.484#t.008
0.694:t.00s
0.787=t.o0s
0.132
C Additional Qualitative Analysis
C.1 Qualitative Comparison of the Existing Methods
We provide additional qualitative motion results in Fig. A. Compared to other methods, our method generates motions that match the text descriptions better, indicating that our method is more sensitive to subtle differences in texts. The generated results demonstrate that our method achieves superior controllability compared to well-designed baseline models.
C.2 Additional Visualization Results
In Fig. B, we provide additional qualitative motion results which are generated with text prompts of the HumanML3D test set. These results demonstrate that our method can generate diverse and accurate motion sequences from variety of motion descriptions.
C.3 Qualitative Analysis the Imbalance Problem
To demonstrate the imbalance problem of other methods and prove that our method does not have this problem, we mask the verbs and action names the motion description to force the model to generate motion only from action specifics. As shown in Fig. C, when the verbs and action names are masked, existing models tend to generate motion randomly. motion that matches the description based solely on the action specifics. These results show that our method is not overly focused on the verbs and action names.
contrast, our method can generate
C.4 Qualitative Comparison of Different Hierarchies
We provide different levels qualitative comparison in Fig. D. The results show that the output at the higher level (e.g., specific level) has more action details. Specifically, the motion level generates only coarse-graines overall motion. The action level generates local actions better than the motion level but lacks action specifies. The specific level generates more action specifics than the action level.
19

---

Table C: Evaluation of Inference time costs the HumanML.3D test set. We evaluate the average time per sample with different diffusion schedules and FID. "4" denotes that lower is better. Please note the bad FID of MDM with DDIM is mentioned in their GitHub issues #76. "x" denotes that this method does not apply this parameter. We use DDIM in practice following MLD.
Diffusion Steps
Methods
Reference
Averuge time sample (s)
FID
Motion T
Action T
Specific T
The total number difflusion steps 1000 with DDPM [18]
MDM [54]
ICLR 2023
1000
X
x
178.7699 5.5045
0.544
MLD [8]
CVPR 2023
1000
X
x
0.568
The total number of diffuusin steps is 50 with DDIM [50]
MDM [54]
ICLR 2023
x
20,5078
7.334
MLD
CVPR 2023
X
X
0.9349
0.473
GruphMotion
Ours
15
15
0.9094
0.131
GraphMotion
Ours
15
15
20
0.7758
0.118
The total number of diffiusion sleps is 150 with DDIM [50]
MLD [8]
CVPR 2023
2,4998
0.457
GraphMotion
50
2.5518
0.116
The total number of diffusion steps is 1000 with DDIM (50)
MLD [8]
CVPR 2023
1000
x
x
16.6654
0.460
GraphMotion
Ours
300
300
22.1238
0.136
GraphMotion
Ours
300
400
17.0912
0.132
D Additional Experiments
D.1 Analysis of the Diffusion Steps
In Tab. B, we show the ablation study of the total number of diffusion steps on the HumanML3D test set. Following MLD [8]. we adopt the denoising diffusion implicit models [50] (DDIM) during interference. As shown in Tab. B, our method consistently outperforms the existing state-of-the-ard methods with the same total number of diffusion steps, which demonstrates the efficiency of our method. We find that the number of diffusion steps at the higher level (e.g., specific level) has greater impact on the result. Therefore, in scenarios requiring high efficiency, we recommend allocating more diffusion steps to the higher level. Moreover, with the increase of the total diffusion steps, the performance of our method further improved, while the performance MLD saturates. These results further prove the superiority of our design.
D.2 Analysis of the Inference Time
In Tab. we provide the evaluation of inference time costs. Our method as efficient as the one-stage diffusion methods during the inference stage, even though we decompose the diffusion process into three parts. This is because we can control the total number 7 + T + T of iterations by restricting it to be the same as those of the one-stage diffusion methods. As shown in Tab. C, the inference speed our method is comparable to that of the existing state-of-the of-the-art methods with the same total number of diffusion steps, which demonstrates the efficiency of our method.
D.3 Analysis of the motion VAE models
We provide the evaluation of the motion VAE models. In Tab. D, show the results the HumanML.3D test set. Tab. E shows the results on the KIT test set. Among the three levels, the performance of the specific level is the best, which indicates that increasing the token size improve the reconstruction ability of the motion VAE models.
E
Motion Representations and Metric Definitions
E.1
Motion Representations
Motion representation can be summarized into the following four categories, and we follow the previous work of representing motion in latent space.
20

---

Table D: Evaluation of the VAE models the motion part of the HumanMMLD test set. "I" denotes that higher is better. "4" denotes that lower is better. "-" denotes that results are better if the metric is closer to the real motion. The performance of the specific level is the best.
R-Precision
Methods
Token Size
FID
Diversity
Top-1
Top-2
Top-3
Real motion
0.511
0.703
0.797
0.002
9.503
Motion Level
2
0.498
0.692
0.791
1.906
9.675
Action Level
0.514
0.703
0.793
0.068
9.610
Specific Level
8
0.525
0.708
0.800
0.019
9.863
Table E: Evaluation of the VAE models on the motion part of the KIT test set. "1" denotes that higher is better. "L" denotes that lower is better. "-"" denotes that results are better if the metric is closer to the real motion. The performance of the specific level is the best.
R- R-Precision
1
Methods
Token Size
FID 4
Diversity
Top-1
Top-2
Top-3
Real motion
0.424
0.649
0.779
0.031
11.08
Motion Level
2
0.431
0.623
0.745
1.196
10.66
Action Level
0.413
0.644
0.770
0.396
10.85
Specific Level
8
0.414
0.640
0.760
0.361
10.86
Latent Format. Following previous works [42, 8, 61], we encode the motion into the latent space with motion variational autoencoder [27]. The latent representation is formulated as:
I.... =D(=), E.x
(B)
HumanML3D Format. HumanML3D [14] proposes a motion representation x I:L inspired by motion features in character control. This motion representation well-suited for neural networks. To be specific, the ieh pose x is defined by a tuple consisting of the root angular velocity r E IR along the Y-axis, root linear velocities (r p R) on the XZ-plane, root height r R, local joints positions j" E R3N, velocities j E RaN,, and rotations j" e IR&,, in root space. and binary foot-ground contact features cf 4 obtained by thresholding the heel and toe joint velocities. Here, Ny denotes the joint number. Finally, the HumanML3D format can be defined as:
x² (r x M. j.. j j"" c..
(C)
SMPL-based Format.
SMPL [37] is one of the most widely used parametric human models. SMPL and its variants propose motion parameters 9 and shape parameters
33x2343 is rotation
vectors for 23 joints and a root, while 3 represents the weights for linear blended shapes. The global translation is also incorporated formulate the representation as follows:
x² {r. 0, B).
(D)
MMM Format. Master Motor Map [52] (MMM) representations propose joint angle parameters based on uniform skeleton structure with 50 degrees of freedom (DoFs). In text-to text-to-motion tasks, recent methods [2, 12, 42] converts joint rotation angles into 21 joint XYZ coordinates. Given the global trajectory broot and Pm E 3J the preprocessed representation is formulated as:
= (pm, troot]-
F.2 Metrie Definitions
Following previous works, we use the following five metrics to measure the performance of the model. Note that global representations of motion and descriptions are first extracted with the pre-trained network in [14].
21

---

R-Precision. Under the feature space of the pre- pre- trainriind network in [14], given one motion sequence and 32 text descriptions ground-truth and randomly selected mismatched descriptions), motion- retrieval precision calculates the text and motion Top 1/2/3 matching accuracy.
Frechet Inception Distance (FID). We measure the distribution distance between the generated and real motion using FID [17] on the extracted motion features [14]. The FID is calculated as:
FID
Pgt Mpreall Tr(-igt
Spred 2(2 E)red))),
where E is the covariance matrix. Tr denotes the trace of matrix. Mgt and ppred are the mean of ground-truth motion features and generated motion features.
Multimodal Distance (MM-Dist). Given N randomly generated samples, we calculate the average Euclidean distances between each text feature ft and the generated motion feature fm from that text. The multimodal distance is calculated as:
MM-Dist
lff -m.ill
(G)
where Jt,s and fm, are the features of the ith text-motion pair.
Diversity. All generated motions are randomly sampled to two subsets (fx1, (f ..., xx, } and { Tzg...., ..., x of the same size Xg. Then, we extract motion features [14] and compute the average Euclidean distances between the two subsets:
X 2//l-
Diversity = X
1
(H)
Multimodality (MModality). We randomly sample set of text descriptions with size Jm from all descriptions. For each text description, we generate 2 x Xm motion sequences, forming Xm pairs of motions. We extract motion features and calculate the average Euclidean distance between each pair. We report the average of all text descriptions. We define features of the jth pair of the dea text description as (z J.). The multimodality is calculated as:
X
MModality
2 2.,all
(I)
Jm x X m j=1
22


Reviews:
[{"review_id": "DtwLhygDFj", "reviewer": "Reviewer_DRsM", "summary": "This paper presents a novel motion generation pipeline that utilizes a 3-level hierarchical semantic graph. The entire reverse process of the motion diffusion model is divided into three stages: overall motion, local actions, and action specifics. The semantic graph is extracted through semantic role parsing and further enhanced using a Graph Attention Network. The obtained node features are subsequently input into a single transformer based on their semantic level.", "questions": "1. Why is the video length of MotionDiffuse shorter than other methods in the demo video?\n\n2. Why is there a significant decrease in the FID metric on KIT-ML when the number of steps increases from 50 to 150?\n\n3. Can GraphMotion perform low-level motion editing or motion composition similar to MotionDiffuse and MDM?", "limitations": "Limitations have been well discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed GraphMotion approach demonstrates excellent performance in terms of numerical metrics on two datasets, showing significant improvements compared to existing methods. Particularly noteworthy is the detailed comparison of GraphMotion at different diffusion steps with other state-of-the-art methods, which adds more persuasive evidence to the results.\n\n2. The motion refinement aspect is intriguing, as users are able to achieve a certain degree of motion editing by modifying the content of nodes. This adds an interesting and interactive element to the method.\n\n3. The motivation and objectives of the entire approach are well-explained, making it easy to understand the rationale behind the proposed method.", "weaknesses": "1. It is recommended that the authors conduct user studies to quantitatively compare their method with other existing works based on qualitative results. The current demo video lacks sufficient comparisons, and additional comparisons should be included to provide a comprehensive evaluation. Furthermore, it would be beneficial to showcase the qualitative results of motion refinement in the demo video to provide a more complete demonstration of the method's capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive comments, and for noting that our method is \"novel\", \"intriguing\" and \"well-explained\". We address the questions as below.\n\n&nbsp;\n\n**Q1**: Why is the video length of MotionDiffuse shorter than other methods in the demo video?\n\n**A1**: This is because we refer to the settings provided by the repository of MotionDiffuse. The repository of MotionDiffuse recommends setting the generated motion sequence length to 60 frames, so we follow this setting when visualizing, which makes the generated video short.\n\n&nbsp;\n\n**Q2**: Why is there a significant decrease in the FID metric on KIT-ML when the number of steps increases from 50 to 150?\n\n **A2**: We explain this phenomenon from two perspectives:\n\n* Due to the relatively small scale of the KIT-ML dataset, motion sequences can be generated well when the number of diffusion steps is 50.\n* Since we use DDIM instead of DDPM, increasing the number of diffusion steps does not introduce additional randomness and may reduce the diversity of the generated results. We find that when the number of DDIM diffusion steps increase, even though the fluency of generated motions increases, the diversity of generated motions reduces, leading to a decrease in the FID metric.\n\nTherefore, to balance generation quality and generation diversity, setting the number of steps to 50 on the KIT-ML dataset is a better choice than setting the number of steps to 150.\n\n&nbsp;\n\n**Q3**: Can GraphMotion perform low-level motion editing or motion composition similar to MotionDiffuse and MDM?\n\n**A3**: GraphMotion can perform in-betweening motion editing. Unlike MotionDiffuse and MDM, which implement motion composition during the reverse diffusion process, GraphMotion can use the motion decoder $\\mathcal{D}$ to fill new motion frames from the motion latent space and fixed motion frames.\n\nIn addition, since GraphMotion performs a diffusion process on the motion latent space rather than on the raw motion sequences, GraphMotion is more suitable for high-level motion editing, such as style transfer. Specifically, the input motion is first encoded into the motion latent space by the motion encoder $\\mathcal{E}$. Then we can use the energy-guided diffusion process [1] to make overall motion editing according to the text.\n\n[1] Yu, Jiwen, et al. \"Freedom: Training-free energy-guided conditional diffusion model.\" *arXiv preprint arXiv:2303.09833* (2023).\n\n&nbsp;\n\n**Q4**: It is recommended that the authors conduct user studies for quantitative comparison.\n\n**A4**: Thanks for your valuable suggestion. We randomly selected 39 motion descriptions for the user study. The results are shown in the following table. Each row represents the preference rate of GraphMotion over the compared model. GraphMotion is preferred over the other models most of the time.\n\n| **Methods**                   | Preference Rate of GraphMotion|\n| ----------------------------- | :-------------: |\n| GraphMotion vs. MotionDiffuse |     64.10%      |\n| GraphMotion vs. MLD           |     56.41%      |\n| GraphMotion vs. Ground Truth  |     48.72%      |\n\n&nbsp;\n\n**Q5**: It would be beneficial to showcase the qualitative results of motion refinement in the demo video to provide a more complete demonstration of the method's capabilities.\n\n**A5**: Thanks for your advice. We will showcase the qualitative results of motion refinement in the demo video, such as modifying the edge. In addition, we will also showcase the qualitative results of new operations in the global response (Figure III), i.e., **(1)** masking the node by replacing it with the MASK token; **(2)** modifying the node; **(3)** deleting nodes; **(4)** adding a new node.\n\n&nbsp;\n\nWe sincerely thank you for your constructive comments. We will add the above important discussions in the final manuscript and highlight them. Thanks again for taking the time and effort on our paper."}, {"review_id": "wm3hbkYET3", "reviewer": "Reviewer_xtyr", "summary": "This paper proposes a coarse-to-fine diffusion model coupled with a hierarchical semantic graphs to address the text-to-motion generation problem. To preserve the fine-grained control signals from captions, three-level textual features are extracted through GAT. Then, three diffusion models are adopted to recover the latent motion representations, which is decoded into motion sequences via VAE.", "questions": "Q1: Please present more examples to prove that imbalance problem exists in the other methods and how your method addresses it.\n\nQ2: According to Table D and E in your supplementary material. R-Precision of your proposed VAE models outperforms Real motion, does this mean that R-Precision is not suitable for evaluating these two motion datasets?", "limitations": "The authors have discussed the potential negative societal impacts.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Semantic graphs are introduced to model the coarse-to-fine textual descriptions.\n2. Hierarchical diffusion model is technically sound to learn the latent motion distributions.\n3. The experimental results outperforms sota methods.\n4. Fine-grained controllability of the proposed model is interesting.", "weaknesses": "1. The motivation behind the imbalance problem needs to be more convincing. In line 31-33, deficience of the existing method boils down to imbalance and coarseness. I am curious about if there are some examples could prove that imbalance problem exists in the other methods.\n2. Many descriptions in this paper are misleading or incorrect, and it needs further explaination. In line 176, the term ``codebook size'' is ambiguous because it is generallly associated with VQVAE. In line 180-181, the purpose of diffusion process should be learning a mapping from gaussian noise sampled from $\\mathcal{N}(0,1)$ to motion latent representation $z^m$, and the condition is the global motion node $V^m$. In line 183, $\\beta_t$ is not step size, it is the noise schedule.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time and effort when reading our paper and providing constructive comments. We address the questions below.\n\n&emsp;\n\n**Q1**: Please present more examples to prove that the imbalance problem exists in the other methods and how your method addresses it.\n\n**A1**: We will prove the imbalance problem of other methods from quantitative and qualitative perspectives respectively, and how our method addresses it.\n\n* From the quantitative perspective, we mask the verbs and action names in the motion description to force the model to generate motion only from action specifics. For example, given the motion description \"a person walks several steps forward in a straight line.\", we would mask \"walks\". \n\n  Transformer extracts text features automatically and implicitly. However, **it may encourage the model to take shortcuts, such as overemphasizing the action name \"walks\" at the expense of other important properties**. Therefore, when the verbs and action names are masked, the other models, which direct use the transformer to extract text features, fail to generate motion well.\n\n  By contrast, our method explicitly extracts the action specifics.  The explicit factorization of the language embedding space discourages the model from taking shortcuts. When the verbs and action names are masked, our method **can infer from action specifics such as \"several steps forward\" and \"in a straight line\" that the overall motion is \"walking forward\"**. \n\n  As shown in the following table, our method can synthesize motion by relying only on action specifics, while other methods fail to generate motion well. These results indicate that our method avoids the imbalance problem of other methods.\n\n  | Methods            |    FID    |  MM-Dist  | Diversity | MModality |\n  | ------------------ | :-------: | :-------: | :-------: | :-------: |\n  | MDM (ICLR 2023)    |   5.622   |   7.163   |   8.713   |   3.578   |\n  | MLD (CVPR 2023)    |   3.492   |   5.632   |   8.874   |   3.596   |\n  | GraphMotion (Ours) | **1.826** | **5.530** | **9.284** | **3.699** |\n\n\n\n* From the qualitative perspective, we provide qualitative comparison of the imbalance problem in the global response (Figure I). As shown in Figure I in the attached PDF, when the verbs and action names are masked, existing models tend to generate motion randomly. In contrast, our method can generate motion that matches the description based solely on the action specifics. These results show that our method is not overly focused on the verbs and action names.\n\n&emsp;\n\n**Q2**: According to Table D and E in your supplementary material. R-Precision of your proposed VAE models outperforms Real motion, does this mean that R-Precision is not suitable for evaluating these two motion datasets?\n\n**A2**: Since the existing methods are already very close to real motion in the R-Precision, we also believe that the R-Precision is not a good metric to evaluate these two motion datasets. For the motion generation task, it may be necessary to integrate a variety of metrics to evaluate the model performance.\n\n&emsp;\n\n**Q3**: Many descriptions in this paper are misleading or incorrect, and it needs further explanation. \n\n**A3**: Thank you for your helpful comments. We have corrected these descriptions in the revision. Specifically, we have made the following corrections:\n\n* We have amended \"codebook size\" to \"token size\" in line 176. \n* We have rewritten lines 180-181 to \"our goal is to learn the diffusion process from Gaussian noise sampled  from $\\mathcal{N}(0,1)$ to motion latent representation $z^m$, conditioned on the global motion node $\\mathcal{V}^{m}$.\" \n* We have explained ${\\beta}_t$ in line 183 by \"${\\beta}_t$ is the noise schedule.\"\n\nBesides, we have proofread our paper carefully in revision.\n\n&emsp;\n\nWe sincerely thank you for your valuable comments. We will add the above important discussions in the final manuscript and highlight them. If you have further questions, please feel free to contact us. Thanks again for taking the time and effort on our paper."}, {"review_id": "RMSCgfnopo", "reviewer": "Reviewer_Xx1r", "summary": "This work decomposes the motion description into three levels including motion, action and specifics, and proposes hierarchical semantic graphs to achieve fine-grained control of motion generation. \nExperiments with the proposed method on HumanML3D and KIT datasets demonstrate better motion generation and more sensitive to subtle differences in texts than existing techniques.\n", "questions": "See the above weakness.", "limitations": "The authors fully explain the social impact of the work in the appendix, but only elaborate on the shortcomings of the used models (e.g., diffusion model and VAE) in terms of limitations, and do not well explain their own limitations and failure cases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation is clear, and the proposed solutions are to the raised issues. \nThe results seem to be impressive and the experimental setup is somewhat reasonable.\nThe ability to continuously refine the generated motion is meaningful and helpful to the community.", "weaknesses": "1. The paper states several times that Transformer may overemphasize action names, but I think that the action level in the hierarchical semantic graph is also another kind of emphasis. Therefore, if possible, I hope the authors can experimentally demonstrate the \"overemphasis\" of the Transformer and prove that the hierarchical semantic graph does not have this problem.\n2. Why is the last one in the action level of Figure 1 resumes instead of walking? How does it handle \"squats to picks\" and distinguish \"walks and walking\"? Also, the hierarchical semantic graph is built on a valid semantic parsing tool, and I'm not sure whether the tool will greatly affect the overall robustness of the model.\n3. What are the results in the first row of Tables 3 and 4 in what configuration? By my understanding, these two rows should be the complete ablation of their respective parts, so they should be numerically different.\n4. Is the modification in Figure 5 implemented in other latest works as well? Does it only modify the weights of the edges and can it modify the nodes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing constructive feedback, and for noting that \"the ability to continuously refine the generated motion is meaningful and helpful to the community.\" We address the questions below.\n\n&nbsp;\n\n**Q1**: Experimentally demonstrating the \"overemphasis\" of the Transformer and proving that the hierarchical semantic graph does not have this problem.\n\n**A1**: To demonstrate the \"overemphasis\" of the Transformer and prove that the hierarchical semantic graph does not have this problem, we mask the verbs and action names in the motion description to force the model to generate motion only from action specifics. For example, given the motion description \"a person walks several steps forward in a straight line.\", we would mask \"walks\". \n\nTransformer extracts text features automatically and implicitly. However, **it may encourage the model to take shortcuts, such as overemphasizing the action name \"walks\" at the expense of other important properties**. Therefore, when the verbs and action names are masked, the other models, which direct use the transformer to extract text features, fail to generate motion well.\n\nBy contrast, the hierarchical semantic graph **explicitly extracts the action specifics**. The explicit factorization of the language embedding space facilitates a comprehensive understanding of motion description. It allows the model to **infer from action specifics such as \"several steps forward\" and \"in a straight line\" that the overall motion is \"walking forward\"**.\n\nAs shown in the following table, our method can synthesize motion by relying only on action specifics, while other methods, which direct use the transformer, fail to generate motion well. These results indicate that the hierarchical semantic graph avoids the imbalance problem.\n\n| Methods            |    FID    |  MM-Dist  | Diversity | MModality |\n| ------------------ | :-------: | :-------: | :-------: | :-------: |\n| MDM (ICLR 2023)    |   5.622   |   7.163   |   8.713   |   3.578   |\n| MLD (CVPR 2023)    |   3.492   |   5.632   |   8.874   |   3.596   |\n| GraphMotion (Ours) | **1.826** | **5.530** | **9.284** | **3.699** |\n\nBesides, we also provide qualitative comparison in the global response (Figure I). When the verbs and action names are masked, the existing models, which direct use the transformer to extract text features, tend to generate motion randomly. In contrast, the generated results of our method better match the descriptions.\n\n&nbsp;\n\n**Q2**: Why is the last one in the action level of Figure 1 resumes instead of walking?\n\n**A2**: The last part of the original video is walking. But for ease of visualization, we sample 8 frames from the video and merge them into one figure. The walking part is not sampled and therefore fails to be shown. We are sorry for any confusion caused by this figure. In the revision, we have sampled more frames from the video to eliminate this confusion.\n\n&nbsp;\n\n**Q3**: How does it handle \"squats to picks\" and distinguish \"walks and walking\"?\n\n**A3**: In \"squats to picks\", both \"squats\" and \"picks\" are considered verbs and are therefore extracted as local actions. The time relationship between \"squats\" and \"picks\" is provided by the overall sentence (i.e., the global motion node). For \"walks and walking\", \"walks\" is considered a verb. For the semantic role parsing toolkit we use, \"walking\" might be considered action specific. For example, for \"resumes walking\", \"walking\" is taken as the action specific of the local action \"resumes\".\n\n&nbsp;\n\n**Q4**: The hierarchical semantic graph is built on a valid semantic parsing tool, and I'm not sure whether the tool will greatly affect the overall robustness of the model.\n\n**A4**: We explain the robustness of our method from two aspects. **First**, referring to the experiment in **A1**, our method can synthesize motion even after masking all verbs, which indicates our robustness. **Second**, semantic parsing tools are well-developed to handle simple texts in motion generation. Therefore, the robustness of our method is not a particular concern.\n\n&nbsp;\n\n**Q5**: What are the results in the first row of Tables 3 and 4 in what configuration? \n\n**A5**: Both the first row of Tables 3 and 4 represent the result of motion generation based on the vanilla Transformer and single-stage diffusion models. For comparison purposes, we present this baseline result in both Tables 3 and 4. We have added detailed descriptions in the revision to guide the readers.\n\n&nbsp;\n\n**Q6**: Is the modification in Figure 5 implemented in other latest works as well? \n\n**A6**: No, no other method can achieve this modification. To the best of our knowledge, we are the first to propose hierarchical semantic graphs, a fine-grained control signal, for text-to-motion generation. Besides, we are the first to propose modifying the edge weights of hierarchical semantic graphs to refine the generated results.\n\n&nbsp;\n\n**Q7**: Does it only modify the weights of the edges and can it modify the nodes?\n\n**A7**: It can modify the nodes. We provide additional qualitative analysis of refining motion results in the global response (Figure III). Specifically, we perform the following operations on the hierarchical semantic graphs: **(1)** masking the node by replacing it with the MASK token; **(2)** modifying the node; **(3)** deleting nodes; **(4)** adding a new node. The qualitative results demonstrate that our approach provides a novel method of refining generated motions.\n\n&nbsp;\n\nWe sincerely thank you for your insightful comments. We will add the above important discussions in the final manuscript and highlight them. Thanks again for your great effort in improving our paper."}, {"review_id": "Yqm5yxh36H", "reviewer": "Reviewer_E76p", "summary": "The authors identify two major issues of text-to-motion generation as overemphasis on action names and the coarseness of sentence-level representations.\nTo this end, hierarchical semantic graphs are adopted to factorize coarse sentences into fine-grained action concepts, thus refining the generated motion from coarse-to-fine.\nFurthermore, the hierarchical design allows flexible control of the generation procedure.\nExtensive experiments show the efficacy of the proposed method.", "questions": "Is it possible to generate with Action level or Specific level only? It would be interesting to see the corresponding results, given there is a trend that the more diffusion steps for Specific level brings better performance.\n\nThe hierarchical nature of the method might be suitable for long sequence generation. Comparison on this would be an interesting thing to do.\n\nFor motion refinement, is it possible to modify the graph beyond changing edge weights, i.e., adding/deleting nodes? Or tuning some nodes' weight to zero. What would happen?\n\n", "limitations": "The author described some limitations of the methods, while it would be more helpful to discuss whether the current top-down paradigm could be extended in a bottom-up manner. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed three tiers of node as motions, actions and specifics are interesting. \n\nThe hierarchical diffusion is well-designed in disentangling different level of granularity. Also, the experimental results show reasonable improvements with more and levels.\n\nThe performance is impressive.\n\nThe motion refinement application is attractive and promising.\n\nThe analysis on distribution of diffusion steps might be interesting in identifying the performance bottleneck among different semantic levels.\n\nExtensive ablation experiments provide a nice view of the effectiveness of different components.", "weaknesses": "Qualitative comparison of different hierarchies is missed. It would be helpful to visualize motion generated from different level of latent embeddings, since fundamentally human is the only gold standard to evaluate motion generation, given the rather close numeric metrics of different levels.\n\nThe ablation on the design of the semantic graph is missed.\n\nA more detailed description of the motion refining procedure, like a figure, an algorithm, or some equations might be added for better clarity.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments, and for noting that our method is \"interesting\". We address the questions as below.\n\n&nbsp;\n\n**Q1**: Is it possible to generate with action level or specific level only?\n\n**A1**: As suggested, we train networks to generate only the action level and specific level. As shown in the following table, the hierarchical diffusion model performs better on the FID metric with the same number of diffusion steps. We consider that it is because the hierarchical generation method can generate more detail than the single-stage generation method.\n\n| Methods             | Total steps |   R-Precision Top 1/2/3   |    FID    |  MM-Dist  | Diversity |\n| ------------------- | :---------: | :-----------------------: | :-------: | :-------: | :-------: |\n| GraphMotion         |     50      |   0.496/0.686/**0.778**   | **0.118** |   3.143   | **9.792** |\n| Action level only   |     50      | **0.507**/**0.692**/0.763 |   0.190   |   3.133   |   9.634   |\n| Specific level only |     50      |     0.492/0.686/0.772     |   0.144   | **3.090** |   9.710   |\n\n&nbsp;\n\n**Q2**: The hierarchical nature of the method might be suitable for long sequence generation.\n\n**A2**: Thanks for your insightful advice. We select motions with sequence frame lengths greater than 150 in the HumanML3D test set to evaluate the long sequence generation ability of the models. As shown in the following table, our method performs better on the FID metric than the baseline methods in generating long sequences. The results show that our method is significantly suitable for long sequence generation.\n\n| Methods            |     R-Precision Top 1/2/3     |    FID    |  MM-Dist  | Diversity | MModality |\n| ------------------ | :---------------------------: | :-------: | :-------: | :-------: | :-------: |\n| MDM (ICLR 2023)    |       0.295/0.467/0.572       |   0.594   |   5.497   |   8.994   | **2.874** |\n| MLD (CVPR 2023)    |       0.420/0.607/0.709       |   0.824   |   3.532   | **9.429** |   2.700   |\n| GraphMotion (Ours) | **0.442**/**0.644**/**0.744** | **0.260** | **3.411** |   9.167   |   2.702   |\n\n&nbsp;\n\n**Q3**: For motion refinement, is it possible to modify the graph beyond changing edge weights, i.e., adding/deleting nodes? Or tuning some nodes' weight to zero. What would happen?\n\n**A3**: Yes, we also perform the following operations on the hierarchical semantic graphs: **(1)** masking the node by replacing it with the MASK token; **(2)** modifying the node; **(3)** deleting nodes; **(4)** adding a new node. Please see Figure III in our global response for the details. The qualitative results demonstrate that our approach provides a novel method of refining generated motions. \n\n&nbsp;\n\n**Q4**: Qualitative comparison of different hierarchies is missed.\n\n**A4**: Thanks for your valuable suggestion. We provide different levels of qualitative comparison in the global response (Figure II). The results show that the output at the higher level (e.g., specific level) has more action details. Specifically, the motion level generates only coarse-grained overall motion. The action level generates local actions better than the motion level but lacks action specifics. The specific level generates more action specifics than the action level.\n\n&nbsp;\n\n**Q5**: The ablation on the design of the semantic graph is missed.\n\n**A5**: Thanks for your advice. We also tried another semantic graph based on the Stanford Scene Graph Parser. Unlike the proposed GraphMotion extracts three types (motions, actions, and specifics) of nodes, the Stanford Scene Graph Parser extracts triplet relationships (e.g., \"woman\"->\"in\"->\"room\"). We construct the semantic graph using the overall sentence as the first semantic level, the relation (e.g., \"in\") as the second semantic level, and the entities (e.g., \"woman\" and \"room\") as the third semantic level. \n\nAs shown in the following table, the method based on Stanford Scene Graph Parser is not suitable for motion generation. We consider that it is because the triplet relation emphasizes the inter-entity relationship rather than actions, and the proposed GraphMotion builds the semantic graph with actions as the core, so GraphMotion is more suitable for motion generation.\n\n\n| Methods                                  | Total steps |     R-Precision Top 1/2/3     |    FID    |  MM-Dist  | Diversity |\n| ---------------------------------------- | :---------: | :---------------------------: | :-------: | :-------: | :-------: |\n| Stanford Scene Graph Parser based method |     150     |       0.497/0.683/0.781       |   0.212   |   3.105   | **9.789** |\n| GraphMotion                              |     150     | **0.504**/**0.699**/**0.785** | **0.116** | **3.070** |   9.692   |\n\n&nbsp;\n\n**Q6**: A more detailed description of the motion refining procedure, like a figure, an algorithm, or some equations might be added for better clarity.\n\n**A6**: Thanks for your suggestion. We have added detailed descriptions of the motion refining procedure in the revision. Specifically, for modifying the edge, we modify the attention coefficient $e$ as $e'=\\gamma e+\\delta$, where $\\gamma$ and $ \\delta$ are adjustable parameters. For modifying the node, we first extract text embedding through CLIP, and then replace the original node with this embedding or add it to the graph as a new node. Then the graph network updates all nodes. Finally, we use the updated nodes to generate new motions.\n\n&nbsp;\n\n**Q7**: It would be more helpful to discuss whether the current top-down paradigm could be extended in a bottom-up manner.\n\n**A7**: It is an interesting topic. A possible idea is the retrieval enhancement method, which generates motions based on action details retrieved from graph nodes.\n\n&nbsp;\n\nWe sincerely thank you for your helpful comments. We will add the above important discussions in the final manuscript and highlight them. Thanks again for spending a huge amount of time on our paper."}]

Author rebuttal:
# Global Response

We sincerely thank all PCs, SACs, ACs, and Reviewers for their time and efforts when handling our paper.

&nbsp;

All reviewers appreciate the contributions of our method:

* Both Reviewers DRsM and Xx1r point out that "**the motivation is clear**, and **the proposed solutions are to the raised issues**."
* Reviewers DRsM, Xx1r, and E76p all comment that "**the experimental setup is somewhat reasonable**" and "**the detailed comparison** of GraphMotion at different diffusion steps with other state-of-the-art methods, which adds **more persuasive evidence to the results**."
* Reviewers DRsM, xtyr, Xx1r, and E76p all mention that "the proposed GraphMotion approach demonstrates **excellent performance** in terms of numerical metrics on two datasets, showing **significant improvements** compared to existing methods."
* Reviewers DRsM, xtyr, Xx1r, and E76p all point out that the motion refinement aspect is "**intriguing**", and "**meaningful and helpful to the community**".

&nbsp;

As suggested by the reviewers, we provide the following in the attached PDF:

* **Figure I: Qualitative comparison between our method and other methods on the imbalance problem.** To demonstrate the imbalance problem of other methods and prove that our method does not have this problem, we mask the verbs and action names in the motion description to force the model to generate motion only from action specifics. When the verbs and action names are masked, existing models tend to generate motion randomly. In contrast, our method can generate motion that matches the description based solely on the action specifics. These results show that our method is not overly focused on the verbs and action names.
* **Figure II: Qualitative comparison of different hierarchies.** The results show that the output at the higher level (e.g., specific level) has more action details. Specifically, the motion level generates only coarse-grained overall motion. The action level generates local actions better than the motion level but lacks action specifics. The specific level generates more action specifics than the action level.
* **Figure III: Additional qualitative analysis of refining motion results.** We perform the following operations on the hierarchical semantic graphs: **(1)** masking the node by replacing it with the MASK token; **(2)** modifying the node; **(3)** deleting nodes; **(4)** adding a new node. The qualitative results demonstrate that our approach provides a novel method of refining generated motions, which may be meaningful and helpful to the community.

&nbsp;

We will add the above important qualitative analysis in the final manuscript and highlight them. Thanks again for taking the time and effort on our paper.




Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)