Title: Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Text-driven Motion Synthesis

Diffusion Models

Graph networks
Abstract: Most text-driven human motion generation methods employ sequential modeling approaches, e.g., transformer, to extract sentence-level text representations automatically and implicitly for human motion synthesis. However, these compact text representations may overemphasize the action names at the expense of other important properties and lack fine-grained details to guide the synthesis of subtly distinct motion. In this paper, we propose hierarchical semantic graphs for fine-grained control over motion generation. Specifically, we disentangle motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Such global-to-local structures facilitate a comprehensive understanding of motion description and fine-grained control of motion generation. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments on two benchmark human motion datasets, including HumanML3D and KIT, with superior performances, justify the efficacy of our method. More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact on the community. Code and pre-trained weights are available at https://github.com/jpthu17/GraphMotion.

Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs

Abstract

Most text-driven human motion generation methods employ sequential modeling approaches, e.g., transformer, to extract sentence-level text representations automatically and implicitly for human motion synthesis. However, these compact text representations may overemphasize the action names at the expense of other important properties and lack fine-grained details to guide the synthesis of subtly distinct motion. In this paper, we propose hierarchical semantic graphs for fine-grained control over motion generation. Specifically, we disentangle motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Such global-to-local structures facilitate a comprehensive understanding of motion description and fine-grained control of motion generation. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments on two benchmark human motion datasets, including HumanML3D and KIT, with superior performances, justify the efficacy of our method. More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact on the community. Code and pre-trained weights are available at https://github.com/jpthu17/GraphMotion.

1 Introduction

Human motion generation is a fundamental task in computer animation [54, 4] and has many practical applications across various industries including gaming, film production, virtual reality, and robotics [61, 8]. With the progress made in recent years, text-driven human motion generation has allowed for the synthesis of a variety of human motion sequences based on natural language descriptions. Therefore, there is a growing interest in generating manipulable, plausible, diverse, and realistic sequences of human motion from flexible natural language descriptions.

Existing text-to-motion generation methods [42, 61, 8, 1, 63] mainly rely on sentence-level representations of texts and directly learn the mapping from the high-level language space to the motion sequences. Recently, some works [54, 8, 62] propose conditional diffusion models for human motion synthesis and further improve the synthesized quality and diversity. Although these methods have made encouraging progress, they are still deficient in the following two aspects. (i) Imbalance. The model, which directly uses the transformers [55] to extract text features automatically and implicitly, may overemphasize the action names at the expense of other important properties like direction and

Action

Specifics

Overall

Motion

Local

Actions

a man

forward

then

something

pick

resumes

with both hands

back

walking

instrument,

benefactive

patient

manner

patient

time

direction

agent

action

action

action

action

walks

A man walks forward, then squats to pick something up with both hands,

stands back up, and resumes walking.

Coarse-to-fine Diffusion

Actions

Specifics

Motions

refine

refine

agent

agent

stands

agent

action

agent

squats

intensity. As a typical consequence of this unbalanced learning, the network is insensitive to the subtle changes in the input text and lacks fine-grained controllability. (ii) Coarseness. On the one hand, motion descriptions frequently refer to multiple actions and attributes. However, the compact sentence-level representations extracted by current works usually fail to convey the clarity and detail needed to fully understand the text, leading to a lack of fine-grained details to guide the synthesis of subtly distinct motion. On the other hand, mapping directly of existing works from the high-level language space to motion sequences further hinders the generation of fine-grained details. Therefore, we argue that it is time to seek a more precise and detailed text-driven human motion generation method to ensure an accurate synthesis of complex human motions.

To this end, we propose a more fine-grained control signal, hierarchical semantic graphs, to represent different intentions for controllable motion generation and design a coarse-to-fine motion diffusion model, called GraphMotion. As shown in Fig. 1, motion descriptions inherently possess hierarchical structures and can be represented as hierarchical graphs composed of three types of abstract nodes, namely motions, actions, and specifics. Concretely, the overall sentence describes the global motion involving multiple actions, e.g., 'walk', 'pick', and 'stand' in Fig. 1, which occur in sequential order. Each action consists of different specifics that act as its attributes, such as the agent and patient of the action. Such global-to-local structures contribute to a reliable and comprehensive understanding of motion descriptions. Correspondingly, to take full advantage of this fine-grained control signal, we decompose the text-to-motion diffusion process into three semantic levels from coarse to fine, which are responsible for capturing the overall motion, local actions, and action specifics, respectively.

The proposed GraphMotion has three compelling advantages: First , the explicit factorization of the language embedding space enables us to build a fine-grained correspondence between textual data and motion sequences, which avoids the imbalanced learning of different textual components and coarse-grained control signal representation. Second , the hierarchical refinement property of GraphMotion allows the model to progressively enhance the generated results from coarse to fine, which avoids the coarse-grained generated results. Third , to further fine-tune the generated results for more fine-grained control, our method can continuously refine the generated motion by modifying the edge weights of the hierarchical semantic graph. Experimental results on two benchmark datasets for text-to-motion generation, including HumanML3D [14] and KIT [43], demonstrate the advantages of GraphMotion. The main contributions of this work are as follows:

To the best of our knowledge, we are the first to propose hierarchical semantic graphs, a fine-grained control signal, for text-to-motion generation. It decomposes motion descriptions into global-to-local three types of abstract nodes, namely motions, actions, and specifics.

Correspondingly, we decompose the text-to-motion diffusion process into three semantic levels. This allows the model to gradually refine results from coarse to fine. Experiments show that our method achieves new state-of-the-art results on two text-to-motion datasets.

More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact.

2 Related Work

Text-driven Human Motion Generation. Text-driven human motion generation aims to generate 3D human motion based on text descriptions. Due to the user-friendliness and convenience of natural language [22], text-driven human motion generation is gaining significant attention and has many applications. Recently, two categories of motion generation methods have emerged: joint-latent models [2, 42] and diffusion models [8, 62, 47]. Joint-latent models, e.g., TEMOS [42], typically learn a motion variational autoencoder and a text variational autoencoder. These models then constrain the text and motion encoders into a shared latent space using the Kullback-Leibler divergences [29] loss. The latter category of methods, e.g., MDM [54], proposes a conditional diffusion model for human motion generation to learn a powerful probabilistic mapping from the textual descriptors to human motion sequences. Although these methods have made encouraging progress, they still suffer from two major deficiencies: unbalanced text learning and coarse-grained generated results. In this paper, we propose to leverage the inherent structure of language [6, 58] for motion generation. Specifically, we introduce hierarchical semantic graphs as a more effective control signal for representing different intentions and design a coarse-to-fine motion diffusion model using this signal.

Diffusion Generative Models. Diffusion generative models [49, 18, 11, 21, 50] are a type of neural generative model that uses the stochastic diffusion process, which is based on thermodynamics. The process involves gradually adding noise to a sample from the data distribution, and then training a neural network to reverse this process by gradually removing the noise. In recent years, diffusion models have shown promise in a variety of tasks such as image generation [18, 50, 11, 19, 57], natural language generation [3, 36, 13], as well as visual tasks [9]. Some other works [25, 32] have attempted to adapt diffusion models for cross-modal retrieval [33, 34, 35, 23, 24]. Inspired by the success of diffusion generative models, some works [62, 54, 8] have applied diffusion models to human motion generation. However, these methods typically learn a one-stage mapping from the high-level language space to motion sequences, which hinders the generation of fine-grained details. In this paper, we decompose the text-to-motion diffusion process into three semantic levels from coarse to fine. The resultant levels are responsible for capturing overall motion, local actions, and action specifics, which enhances the generated results progressively from coarse to fine.

Graph-based Reasoning. The graph convolutional network [28] is originally proposed to recognize graph data. It uses convolution on the neighborhoods of each node to produce outputs. Graph attention networks [56] further enhance graph-based reasoning by dynamically attending to the features of neighborhoods. The graph-based reasoning has shown great potential in many tasks, such as scene graph generation [60], visual question answering [20, 31, 30], natural language generation [6], and cross-modal retrieval [7]. In this paper, we focus on reasoning over hierarchical semantic graphs on motion descriptions for fine-grained control of human motion generation.

3 Methodology

In this paper, we tackle the tasks of text-driven human motion generation. Concretely, given an arbitrary motion description, our goal is to synthesize a human motion x 1: L = { x i } L i =1 of length L . The overview of the proposed GraphMotion is shown in Fig. 2.

3.1 Hierarchical Semantic Graph Modeling

Existing works for text-driven human motion generation typically directly use the transformer [55] to extract text features automatically and implicitly. However, motion descriptions inherently possess hierarchical structures that can be divided into three sorts of abstract nodes, including motions, actions, and specifics. Compared with the sequential structure, such global-to-local structure contributes to a reliable and comprehensive understanding of the semantic meaning of motion descriptions and is a promising fine-grained control signal for text-to-motion generation.

Semantic Role Parsing. To obtain actions, attributes of action as well as the semantic role of each attribute to the corresponding action, we implement a semantic parser of motion descriptions based on a semantic role parsing toolkit [48, 7]. We extract three types (motions, actions, and specifics) of nodes and twelve types of edges to represent various associations among the nodes. For details about the types of nodes and edges, please refer to our supplementary material.

Motion

decoder

𝒟

Action

Embeddings

Specific

Embeddings

Motion

Embeddings

𝒛 𝒎

Iteration

𝑻 𝒎

times

𝓥 𝒎

Transformer

𝒛 𝒂

Iteration

𝑻 𝒂

times

𝓥 𝒎

Transformer

𝒛 𝒔

Iteration

𝑻 𝒔

times

𝓥 𝒎

Transformer

𝓥 𝒂

𝒛 𝒎

𝓥 𝒂

𝓥 𝒔

𝒛 𝒂

Coarse-to-Fine Motion Diffusion Model

Hierarchical Semantic

Semantic Role

Parsing

𝒛 𝒔

𝓥𝟏

𝒎

𝓥𝟐

𝒂

𝓥𝟑

𝒂

𝓥𝟒

𝒂

𝓥𝟓

𝒔

𝓥𝟔

𝒔

𝓥𝟕

𝒔

𝓥𝟖

𝒔

𝓥𝟗

𝒔

𝓥𝟏𝟎

𝒔

𝓥𝟏𝟏

𝒔

𝒔

Motionlevel

O

Actionlevel

Specifically, given the motion description, the parser extracts verbs that appeared in the sentence and attribute phrases corresponding verb, and the semantic role of each attribute phrase. The overall sentence is treated as the global motion node in the hierarchical graph. The verbs are considered as action nodes and connected to the motion node with direct edges, allowing for implicit learning of the temporal relationships among various actions during graph reasoning. The attribute phrases are specific nodes that are connected with action nodes. The edge type between action and specific nodes is determined by the semantic role of the specifics in relation to the action.

Graph Node Representation. Given the motion description, we follow previous works [53, 54, 61, 8] and leverage the text encoder of CLIP [45] to extract the text embedding. For the global motion node v m , we utilize the [CLS] token to summarize the salient event described in the sentence. For the action node v a , we use the token of the corresponding verb as the action node representation. For the specific node v s , we apply mean-pooling over tokens of each word in the attribute phrase.

Action-aware Graph Reasoning. The interactions across different levels in the constructed graph not only explain the properties of local actions and how local actions compose the global motion, but also reduce ambiguity at each node. For example, the verb 'pick' in Fig. 1 can represent different actions without context, but the context 'with both hands' constrains its semantics, so that it represents the action of 'pick up with both hands' rather than 'pick up with one hand.' Therefore, we propose to reason over interactions in the graph to obtain hierarchical textual representations.

We utilize graph attention networks [56] (GAT) to model interactions in a graph. Specifically, given the initialized node embeddings v = { v m , v a , v s } , we first transform the input node embeddings into higher-level embeddings h = { h m , h a , h s } by:

where W ∈ R D × D is a shared linear transformation and D is the dimension of node representation. For each pair { h i , h j } of connected nodes, we concatenate the node h i ∈ R D with its neighbor node h j ∈ R D , generating the input data ˜ h ij = [ h i , h j ] ∈ R 2 D of the graph attention module.

However, in a graph with multiple types of edges, the vanilla graph networks need to learn separate transformation matrices for each edge type. This can be inefficient when learning from limited motion data, and prone to over-fitting on rare edge types.

To this end, we propose to factorize multi-relational weights into two parts: a common transformation matrix M ∈ R 2 D × 1 that is shared for all edge types and a relationship embedding matrix M r ∈ R 2 D × N that is specific for different edges, where N is the number of edge types. Following GAT [56], we apply LeakyReLU [39] in the calculation of attention coefficients and set the negative input slope to 0.2. The attention coefficient ˜ e ij is formulated as:

where R ij ∈ R 1 × N is a one-hot vector denoting the type of edge between node i and j . N i is the set of neighborhood nodes of node i . To alleviate over-smoothing [59] in graph networks, we apply skip connection when calculating output embeddings. The output embeddings V are formulated as:

where σ is a nonlinear function. Following GAT [56], we use ELU [10] as the nonlinear function σ .

3.2 Coarse-to-Fine Motion Diffusion Model for Graphs

To leverage the coarse-to-fine topology of hierarchical semantic graphs during generation, we also decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. During the reverse denoising process, the fine-grained semantic layer generates results based on the results from the coarse-grained semantic layer. This allows for a detailed and plausible representation of the intended motion.

Motion Representation. Following previous works [42, 8, 61], we first encode the motion into the latent space with a motion variational autoencoder [27] and then use diffusion models to learn the mapping from hierarchical semantic graphs to the motion latent space.

Specifically, we build the motion encoder E and decoder D based on the transformer [55, 41]. For the motion encoder E , we take C learnable query tokens and motion sequence x 1: L = { x i } L i =1 as inputs to generate motion latent embeddings z ∈ R C × D ′ , where D ′ is the dimension of latent representation. For the motion decoder D , we take the latent embeddings z ∈ R C × D ′ and the motion query tokens as the input to generate a human motion sequence x 1: L = { x i } L i =1 with L frames.

The loss L VAE of the motion variational autoencoder can be divided into two parts. First, we use the mean squared error (MSE) to reconstruct the original input. Second, we use the Kullback-Leibler divergences (KL) loss [29] to narrow the distance between the distribution of latent space q ( z | x ) and the standard Gaussian distribution N (0 , I ) . The full loss L VAE is formulated as:

where λ is the trade-off hyper-parameter. q ( z | x ) = N ( µ z , Σ z ) is obtained by sampling based on the mean µ z and variance Σ z estimated by the model. To generate motion from coarse to fine step by step, we encode motion independently into three latent representation spaces z m ∈ R C m × D ′ , z a ∈ R C a × D ′ and z s ∈ R C s × D ′ , where the number of tokens gradually increases, i.e., C m ≤ C a ≤ C s .

Hierarchical Graph-to-Motion Diffusion. Corresponding to the three-level structure of the hierarchical semantic graphs, we decompose the diffusion process into three semantic levels and build three transformer-based denoising models, which correspond to motions, actions, and specifics.

For the motion level model ϕ m , our goal is to learn the diffusion process from global motion node V m to motion latent representation z m . In a forward diffusion process q ( z t m | z t m -1 ) , noised sampled from Gaussian distribution is added to a ground truth data distribution z m 0 at every noise level t :

where β t is the step size which gradually increases. T is the length of the Markov chain. We sample z t m by z t m = √ ¯ α t z m 0 + √ 1 -¯ α t ϵ m , where ¯ α t = ∏ t i =1 (1 -β i ) . ϵ m is a noise sampled from N (0 , 1) . We follow previous works [18, 8] and predict the noise component ϵ m , i.e., ̂ ϵ m = ϕ m ( z m , t m , V m ) .

For the action level model ϕ a , to leverage the results generated by the motion level, we concatenate the action node V a , the motion node V m , and the result z m generated by the motion level together as the input of the action level denoising network, i.e., ̂ ϵ a = ϕ a ( z a , t a , [ V m , V a , z m ]) .

For the specific level model ϕ s , we leverage the results generated by the action level and nodes at all semantic levels to predict the noise component, i.e., ̂ ϵ s = ϕ s ( z s , t s , [ V m , V a , V s , z a ]) .

Finally, the training objective of the diffusion models can be defined as:

During the training stage, the motion variational autoencoders are frozen. During the inference stage, the fine-grained semantic layer generates results based on the results from the coarse-grained semantic layer. We take the output at the specific level as the final result and use the motion decoder to decode the latent representation into the motion sequence.

It is worth noting that our method is as efficient as the one-stage diffusion methods during the inference stage, even though we decompose the diffusion process into three parts. This is because we can control the total number T m + T a + T s of iterations by restricting it to be the same as those of the one-stage diffusion methods, which we will discuss in experiments (see Tab. 5).

4 Experiments

Datasets, Metrics and Implementation Details. Datasets. We compare the proposed method with other methods on two commonly used public benchmarks: HumanML3D [14] and KIT [43].

'A person takes a step forward, moves to the right, then continues forward with their right hand on a rail.'

Real

Ours

MDM (ICLR23)

MLD (CVPR23)

MotionDiffuse (arXiv22)

'The person walked forward and lifted something up turned around and walked back.'

'A person was running by making the circle and return.'

HumanML3D [14] is currently the largest 3D human motion dataset that originates from and textually reannotates the HumanAct12 [16] and AMASS [40] datasets. This dataset comprises 14,616 human motions and 44,970 text descriptions, with each motion accompanied by at least three precise descriptions. The lengths of these descriptions are around 12 words. KIT [43] contains 3,911 human motion sequences and 6,278 textual annotations. Each motion sequence is accompanied by one to four sentences, with an average description length of 8 words.

Metrics. Following previous works, we use the following five metrics to measure the performance of the model. (1) R-Precision. Under the feature space of the pre-trained network in [14], given one motion sequence and 32 text descriptions (1 ground-truth and 31 randomly selected mismatched descriptions), motion-retrieval precision calculates the text and motion Top 1/2/3 matching accuracy. (2) Frechet Inception Distance (FID). We measure the distribution distance between the generated and real motion using FID [17] on the extracted motion features [14]. (3) Multimodal Distance (MMDist). We calculate the average Euclidean distances between each text feature and the generated motion feature from that text. (4) Diversity. All generated motions are randomly sampled to two subsets of the same size. Then, we extract motion features [14] and compute the average Euclidean distances between the two subsets. (5) Multimodality (MModality). For each text description, we generate 20 motion sequences, forming 10 pairs of motions. We extract motion features and calculate the average Euclidean distance between each pair. We report the average of all text descriptions.

Implementation Details. For the motion variational autoencoder, motion encoder E and decoder D all consist of 9 layers and 4 heads with skip connection [46]. Following MLD [8], we utilize a frozen text encoder of the CLIP-ViT-L-14 [45] model for text representation. The dimension of node representation D is set to 768. The dimension of latent embedding D ′ is set to 256. We set the token sizes C m to 2, C a to 4, and C s to 8. We set λ to 1e-4. All our models are trained with the AdamW [26, 38] optimizer using a fixed learning rate of 1e-4. We use 4 Tesla V100 GPUs for the training, and there are 128 samples on each GPU, so the total batch size is 512. For the HumanML3D dataset, the model is trained for 6,000 epochs during the motion variational autoencoder stage and 3,000 epochs during the diffusion stage. The number of diffusion steps of each level is 1,000 during training, and the step sizes β t are scaled linearly from 8.5 × 1e-4 to 0.012. For runtime, training tasks 16 hours for motion variational autoencoder and 24 hours for denoiser on 4 Tesla V100 GPUs.

Comparisons to State-of-the-art. We compare the proposed GraphMotion with other methods on two benchmarks. In Tab. 1, we show the results on the HumanML3D test set. Tab. 2 shows the results

on the KIT test set. Our model consistently outperforms state-of-the-art methods on both benchmarks, which validates the effectiveness of our method. Moreover, we provide qualitative motion results in Fig. 3. Compared to other methods, our method generates motions that match the text descriptions better, indicating that our method is more sensitive to subtle differences in texts.

R-Precision Top-1

FID

MM-Dist

Epoch

Epoch

Epoch

Ablative Analysis. Effect of each part of our method. To explore the impact of each part of our method, we provide the ablation results in Tab. 3. 'Semantic Graph' refers to the decomposition of the original motion description into nodes of three levels, namely motion, action, and specific. These nodes are then directly fed into the one-stage motion diffusion model to generate the final motion. 'Graph Reasoning' utilizes a graph network to reason and update nodes in the hierarchical semantic graph. 'Coarse-to-Fine Diffusion' decomposes the text-to-motion diffusion process into three semantic levels, from coarse to fine, which capture the overall motion, local actions, and action specifics, respectively. If not otherwise specified, all ablation experiments are performed using a diffusion step setting of 150. As shown in Tab. 3, we find that 'Semantic Graph' can avoid the imbalanced learning of different textual components, thus significantly improving the semantic understanding ability of the model and the overall quality of motion generation. 'Graph Reasoning' further enhances the semantic reasoning ability of the model. In addition, 'Coarse-to-Fine Diffusion' can significantly improve the quality of motion generation. Our full model achieves the best performance, which demonstrates that the three parts are beneficial for motion generation.

patient

time

direction

agent

action

action

action

Actions

Specifics

Motions

time

direction

agent

agent

Base

figure

forward

then

to left

before walking right

right

walking

zags

walks

Figure walks forward then zags to left before walking right.

Base

strengthen

weaken

1

2

3

4

5

6

7

8

9

10

Modifying the edge

zags

to left

1

2

3

4

5

6

7

8

9

10

Modifying the edge

𝓥𝟏

𝒎

walking

Base

strengthen

weaken

Analysis of the coarse-to-fine motion diffusion model. In Tab. 4, we provide the ablation study of the coarse-to-fine motion diffusion model on the HumanML3D test set. These results prove that coarse-to-fine generation is beneficial to motion generation. In addition, we show the performance at each level in Fig. 4. Among the three levels, the performance of the specific level is the best, which confirms the effectiveness of the coarse-to-fine motion diffusion model.

Effect of the diffusion steps. In Tab. 5, we show the ablation study of the total number of diffusion steps on the HumanML3D test set. Following MLD [8], we adopt the denoising diffusion implicit models [50] (DDIM) during interference. As shown in Tab. 5, our method consistently outperforms the existing state-of-the-art methods with the same total number of diffusion steps, which demonstrates the efficiency of our method. With the increase of the total diffusion steps, the performance of our method is further improved, while the performance of MLD saturates. We find that the number of diffusion steps at the higher level (e.g., specific level) has a greater impact on the result. Therefore, in scenarios requiring high efficiency, we recommend allocating more diffusion steps to the higher level.

Quantitative and Qualitative Discussion. Quantitative experiment of the imbalance problem. In this experiment, we mask the verbs and action names in the motion description to force the model to generate motion only from action specifics. For example, given the motion description 'a person walks several steps forward in a straight line.', we would mask 'walks'. Transformer extracts text features automatically and implicitly. However, it may encourage the model to take shortcuts, such as overemphasizing the action name 'walks' at the expense of other important properties. Therefore, when the verbs and action names are masked, the other models, which directly use the transformer

1

2

3

4

5

direction

action

Actions

Specifics

Motions

agent

a person

A person walks several steps forward in a straight line.

walks

several steps forward

in a straight line

manner

Base

Replace the node with [mask]

walks

[mask]

2

4

5

3

1

Modify the node

walks

2

4

5

3

1

Delete the nodes

3

1

Add a new node

4

5

3

1

runs

2

4

5

at a stumbling pace

6

2

Base

?

to extract text features, fail to generate motion well. By contrast, the hierarchical semantic graph explicitly extracts the action specifics. The explicit factorization of the language embedding space facilitates a comprehensive understanding of motion description. It allows the model to infer from action specifics such as 'several steps forward' and 'in a straight line' that the overall motion is 'walking forward'. As shown in Tab. 6, our method can synthesize motion by relying only on action specifics, while other methods fail to generate motion well. These results indicate that our method avoids the imbalance problem of other methods.

Human evaluation. In our evaluation, we randomly selected 39 motion descriptions for the user study. As shown in Tab. 7, GraphMotion is preferred over the other models most of the time.

Qualitative analysis of refining motion results. To fine-tune the generated results for more finegrained control, our method can continuously refine the generated motion by modifying the edge weights of the hierarchical semantic graph. As illustrated in Fig. 5, we can alter the action attributes by manipulating the weights of the edges of the action node and the specific node. For example, by increasing the weights of the edges of 'zags' and 'to left', the human body will move farther to the left. Moreover, by fine-tuning the weights of the edges of the global motion node and the action node, we can fine-tune the duration of the corresponding action in the whole motion. For example, by enhancing the weights of the edges of the global motion node and 'walking', the length of the walk will be increased. In Fig. 6, we provide additional qualitative analysis of refining motion results. Specifically, we perform the additional operations on the hierarchical semantic graphs: (1) masking the node by replacing it with the MASK token; (2) modifying the node; (3) deleting nodes; (4) adding a new node. The qualitative results demonstrate that our approach provides a novel method of refining generated motions, which may have a far-reaching impact on the community.

5 Conclusion

In this paper, we focus on improving the controllability of text-driven human motion generation. To provide fine-grained control over motion details, we propose a novel control signal called the hierarchical semantic graph, which consists of three kinds of abstract nodes, namely motions, actions, and specifics. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments demonstrate that our method achieves better controllability than the existing state-of-the-art methods. More encouragingly, our method can continuously refine the generated motion by modifying the edge weights of hierarchical semantic graphs, which may have a far-reaching impact on the community.

Acknowledgements. This work was supported by the National Key R&D Program of China (2022ZD0118101), Nature Science Foundation of China (No.62202014), and Shenzhen Basic Research Program (No.JCYJ20220813151736001).

References

Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, and Songhwai Oh. Text2action: Generative adversarial synthesis from language to action. In ICRA , pages 5915-5920, 2018.

Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose forecasting. In 3DV , pages 719-728, 2019.

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurIPS , pages 17981-17993, 2021.

Norman I Badler, Cary B Phillips, and Bonnie Lynn Webber. Simulating humans: computer graphics animation and control . Oxford University Press, 1993.

Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents. In VR , pages 1-10, 2021.

Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In CVPR , pages 9962-9971, 2020.

Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR , pages 10638-10647, 2020.

Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your commands via motion diffusion in latent space. In CVPR , pages 18000-18010, 2023.

Zesen Cheng, Kehan Li, Peng Jin, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Parallel vertex diffusion for unified visual grounding. arXiv preprint arXiv:2303.07216 , 2023.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS , pages 8780-8794, 2021.

Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek. Synthesis of compositional animations from textual descriptions. In ICCV , pages 1396-1406, 2021.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933 , 2022.

Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR , pages 5152-5161, 2022.

Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV , pages 580-597, 2022.

Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In ACM MM , pages 2021-2029, 2020.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , pages 6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-conditioned graph networks for relational reasoning. In ICCV , pages 10294-10303, 2019.

Hyeonho Jeong, Gihyun Kwon, and Jong Chul Ye. Zero-shot generation of coherent storybook from plain text story using diffusion models. arXiv preprint arXiv:2302.03900 , 2023.

Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. Expectation-maximization contrastive learning for compact video-and-language representations. In NeurIPS , pages 30291-30306, 2022.

Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In CVPR , pages 2472-2482, 2023.

Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. Text-video retrieval with disentangled conceptualization and set-to-set alignment. In IJCAI , pages 938-946, 8 2023.

Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Diffusionret: Generative text-video retrieval with diffusion model. In ICCV , pages 2470-2481, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Solomon Kullback. Information theory and statistics . Courier Corporation, 1997.

Hao Li, Xu Li, Belhal Karimi, Jie Chen, and Mingming Sun. Joint learning of object graph and relation graph for visual question answering. In ICME , pages 01-06, 2022.

Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. In ICCV , pages 10313-10322, 2019.

Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. arXiv preprint arXiv:2307.02869 , 2023.

Pandeng Li, Chen-Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive spatio-temporal prototype matching for text-video retrieval. In ICCV , pages 4100-4110, 2023.

Pandeng Li, Hongtao Xie, Jiannan Ge, Lei Zhang, Shaobo Min, and Yongdong Zhang. Dualstream knowledge-preserving hashing for unsupervised video retrieval. In ECCV , pages 181197, 2022.

Pandeng Li, Hongtao Xie, Shaobo Min, Jiannan Ge, Xun Chen, and Yongdong Zhang. Deep fourier ranking quantization for semi-supervised image retrieval. TIP , 31:5909-5922, 2022.

Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. arXiv preprint arXiv:2205.14217 , 2022.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. TOG , 34(6):1-16, 2015.

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In ICML , 2013.

Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In ICCV , pages 5442-5451, 2019.

Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. In ICCV , pages 10985-10995, 2021.

Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In ECCV , pages 480-497, 2022.

Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data , 4(4):236-252, 2016.

Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems , 109:13-26, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , pages 234-241, 2015.

Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418 , 2023.

Peng Shi and Jimmy Lin. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255 , 2019.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , pages 2256-2265, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR , 15(1):1929-1958, 2014.

Ömer Terlemez, Stefan Ulbrich, Christian Mandery, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour. Master motor map (mmm)-framework and toolkit for capturing, representing, and reproducing human motion on humanoid robots. In 2014 IEEE-RAS International Conference on Humanoid Robots , pages 894-901, 2014.

Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In ECCV , pages 358-374, 2022.

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. In ICLR , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR , 2018.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 , 2022.

Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual and language learning. In CVPR , 2023.

Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisiting over-smoothing in deep gcns. arXiv preprint arXiv:2003.13663 , 2020.

Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV , pages 670-685, 2018.

Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In CVPR , 2023.

Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001 , 2022.

Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. arXiv preprint arXiv:2304.01116 , 2023.

Abstract This appendix provides additional discussions (Sec. A), implementation details (Sec. B), more qualitative results (Sec. C), several additional experiments (Sec. D), details of motion representations and metric definitions (Sec. E).

Code Code is available at https://github.com/jpthu17/GraphMotion. In this code, we provide the process of the training and evaluation of the proposed method, and the pre-trained weights.

A Additional Discussions

A.1 Potential Negative Societal Impacts

While our work effectively enhances the quality of human motion synthesis, there is a potential risk that it may be used for generating fake content, such as generating fake news, which can pose a threat to information security. Moreover, when factoring in energy consumption, there is a possibility that the widespread use of generative models for synthesizing human motions may contribute to increased carbon emissions and exacerbate global warming.

A.2 Limitations of our Work

Although our method makes some progress, there are still many limitations worth further study. (1) The proposed GraphMotion inherits the randomness of diffusion models. This property benefits diversity but may yield undesirable results sometimes. (2) The human motion synthesis capabilities of GraphMotion are limited by the performance of the pre-trained motion variational autoencoders, which we will discuss in experiments (Tab. D and Tab. E). This defect also exists in the existing state-of-the-art methods, such as MLD [8] and T2M-GPT [61], which also use motion variational autoencoder. (3) Though the proposed GraphMotion brings negligible extra cost on computations, it is still limited by the slow inference speed of existing diffusion models. We will discuss the inference time in experiments (Tab. C). This defect also exists in the existing state-of-the-art methods, such as MDM[54] and MLD [8], which also use diffusion models.

A.3 Future Work

In this paper, we focus on improving the controllability of text-driven human motion generation. Recently, large language models have made remarkable progress, making large language models a promising text extractor for human motion generation. However, despite their strengths in general reasoning and broad applicability, large language models may not be optimized for extracting subtle motion nuances. In future research, we will incorporate the features of large-scale languages into our model, using hierarchical semantic graphs to give large language models the ability to extract fine-grained motion description structures.

B Implementation Details

B.1 Details of Hierarchical Semantic Graphs

To obtain actions, attributes of action as well as the semantic role of each attribute to the corresponding action, we implement a semantic parser of motion descriptions based on a semantic role parsing toolkit [48, 7]. Specifically, given the motion description, the parser extracts verbs that appeared in the sentence and attribute phrases corresponding verb, and the semantic role of each attribute phrase. The overall sentence is treated as the global motion node in the hierarchical graph. The verbs are considered as action nodes and connected to the motion node with direct edges, allowing for implicit learning of the temporal relationships among various actions during graph reasoning. The attribute phrases are specific nodes that are connected with action nodes. The edge type between action and specific nodes is determined by the semantic role of the specifics in relation to the action.

Real

Ours

MDM (ICLR23)

MLD (CVPR23)

MotionDiffuse (arXiv22)

'Using both hands, a person bends over to pat the ground.'

As shown in Tab. A, we extract three types (motions, actions, and specifics) of nodes and twelve types of edges to represent various associations among the nodes.

B.2 Classifier-free Diffusion Guidance

Following MLD [8], our denoiser network is learned with classifier-free diffusion guidance [19]. The classifier-free diffusion guidance improves the quality of samples by reducing diversity in conditional diffusion models. Concretely, it learns both the conditioned and the unconditioned distribution (10% dropout [51]) of the samples. Finally, we perform a linear combination in the following manner, which is formulated as:

A person is doing

jumping jacks.

A person is doing a

dance.

A person takes a jump

sideways to their right.

A person hops

up and down in

place.

A person looks to the right

then kicks something with

their left foot.

The

person is

clapping.

A person walking

forward then

walks back.

A perso bend the knee and

leand to the floor and

start walked.

Dancing randomly like an

indian traditional dance

and mix of western culture.

A person steps forward and

reaches down to grab or place

something with their right hand.

Where α ′ is the guidance scale and α ′ > 1 can strengthen the effect of guidance [8]. We set α ′ to 7.5 in practice following MLD. Please refer to our code for more details.

B.3 Implementation Details for Different Datasets

Following MLD [8], we utilize a frozen text encoder of the CLIP-ViT-L-14 [45] model for text representation. The dimension of node representation D is set to 768. The dimension of latent embedding D ′ is set to 256. For the motion variational autoencoder, motion encoder E and decoder D all consist of 9 layers and 4 heads with skip connection [46]. We set the token sizes C m to 2, C a

Real

Ours

MDM (ICLR23)

MLD (CVPR23)

MotionDiffuse (arXiv22)

'The person pick something up and tilted it to the right.'

'The person [mask] something up and [mask] it to the right.'

Motion level

Action level

Specific level

'A man performs a squat while lifting his arms to shoulder height and hands above his head.'

to 4, and C s to 8. We set λ to 1e-4. All our models are trained with the AdamW [26, 38] optimizer using a fixed learning rate of 1e-4. We use 4 Tesla V100 GPUs for the training, and there are 128 samples on each GPU, so the total batch size is 512. The number of diffusion steps of each level is 1,000 during training, and the step sizes β t are scaled linearly from 8.5 × 1e-4 to 0.012. We keep running a similar number of iterations on different data sets. For the HumanML3D dataset, the model is trained for 6,000 epochs during the motion variational autoencoder stage and 3,000 epochs during the diffusion stage. For the KIT dataset, the model is trained for 30,000 epochs during the motion variational autoencoder stage and 15,000 epochs during the diffusion stage. Code is available at https://github.com/jpthu17/GraphMotion. In this code, we provide the process of the training and evaluation of the proposed method, and the pre-trained model.

C Additional Qualitative Analysis

C.1 Qualitative Comparison of the Existing Methods

We provide additional qualitative motion results in Fig. A. Compared to other methods, our method generates motions that match the text descriptions better, indicating that our method is more sensitive to subtle differences in texts. The generated results demonstrate that our method achieves superior controllability compared to well-designed baseline models.

C.2 Additional Visualization Results

In Fig. B, we provide additional qualitative motion results which are generated with text prompts of the HumanML3D test set. These results demonstrate that our method can generate diverse and accurate motion sequences from a variety of motion descriptions.

C.3 Qualitative Analysis on the Imbalance Problem

To demonstrate the imbalance problem of other methods and prove that our method does not have this problem, we mask the verbs and action names in the motion description to force the model to generate motion only from action specifics. As shown in Fig. C, when the verbs and action names are masked, existing models tend to generate motion randomly. In contrast, our method can generate motion that matches the description based solely on the action specifics. These results show that our method is not overly focused on the verbs and action names.

C.4 Qualitative Comparison of Different Hierarchies

We provide different levels of qualitative comparison in Fig. D. The results show that the output at the higher level (e.g., specific level) has more action details. Specifically, the motion level generates only coarse-grained overall motion. The action level generates local actions better than the motion level but lacks action specifics. The specific level generates more action specifics than the action level.

D Additional Experiments

D.1 Analysis of the Diffusion Steps

In Tab. B, we show the ablation study of the total number of diffusion steps on the HumanML3D test set. Following MLD [8], we adopt the denoising diffusion implicit models [50] (DDIM) during interference. As shown in Tab. B, our method consistently outperforms the existing state-of-the-art methods with the same total number of diffusion steps, which demonstrates the efficiency of our method. We find that the number of diffusion steps at the higher level (e.g., specific level) has a greater impact on the result. Therefore, in scenarios requiring high efficiency, we recommend allocating more diffusion steps to the higher level. Moreover, with the increase of the total diffusion steps, the performance of our method is further improved, while the performance of MLD saturates. These results further prove the superiority of our design.

D.2 Analysis of the Inference Time

In Tab. C, we provide the evaluation of inference time costs. Our method is as efficient as the one-stage diffusion methods during the inference stage, even though we decompose the diffusion process into three parts. This is because we can control the total number T m + T a + T s of iterations by restricting it to be the same as those of the one-stage diffusion methods. As shown in Tab. C, the inference speed of our method is comparable to that of the existing state-of-the-art methods with the same total number of diffusion steps, which demonstrates the efficiency of our method.

D.3 Analysis of the motion VAE models

We provide the evaluation of the motion VAE models. In Tab. D, we show the results on the HumanML3D test set. Tab. E shows the results on the KIT test set. Among the three levels, the performance of the specific level is the best, which indicates that increasing the token size can improve the reconstruction ability of the motion V AE models.

E Motion Representations and Metric Definitions

E.1 Motion Representations

Motion representation can be summarized into the following four categories, and we follow the previous work of representing motion in latent space.

Latent Format. Following previous works [42, 8, 61], we encode the motion into the latent space with a motion variational autoencoder [27]. The latent representation is formulated as:

HumanML3D Format. HumanML3D [14] proposes a motion representation x 1: L inspired by motion features in character control. This motion representation is well-suited for neural networks. To be specific, the i th pose x i is defined by a tuple consisting of the root angular velocity r a ∈ R along the Y-axis, root linear velocities ( r x , r z ∈ R ) on the XZ-plane, root height r y ∈ R , local joints positions j p ∈ R 3 N j , velocities j v ∈ R 3 N j , and rotations j r ∈ R 6 N j in root space, and binary foot-ground contact features c f ∈ R 4 obtained by thresholding the heel and toe joint velocities. Here, N j denotes the joint number. Finally, the HumanML3D format can be defined as:

SMPL-based Format. SMPL [37] is one of the most widely used parametric human models. SMPL and its variants propose motion parameters θ and shape parameters β . θ ∈ R 3 × 23+3 is rotation vectors for 23 joints and a root, while β represents the weights for linear blended shapes. The global translation r is also incorporated to formulate the representation as follows:

MMMFormat. Master Motor Map [52] (MMM) representations propose joint angle parameters based on a uniform skeleton structure with 50 degrees of freedom (DoFs). In text-to-motion tasks, recent methods [2, 12, 42] converts joint rotation angles into J = 21 joint XYZ coordinates. Given the global trajectory t root and p m ∈ R 3 J , the preprocessed representation is formulated as:

E.2 Metric Definitions

Following previous works, we use the following five metrics to measure the performance of the model. Note that global representations of motion and text descriptions are first extracted with the pre-trained network in [14].

R-Precision. Under the feature space of the pre-trained network in [14], given one motion sequence and 32 text descriptions (1 ground-truth and 31 randomly selected mismatched descriptions), motionretrieval precision calculates the text and motion Top 1/2/3 matching accuracy.

Frechet Inception Distance (FID). We measure the distribution distance between the generated and real motion using FID [17] on the extracted motion features [14]. The FID is calculated as:

where Σ is the covariance matrix. Tr denotes the trace of a matrix. µ gt and µ pred are the mean of ground-truth motion features and generated motion features.

Multimodal Distance (MM-Dist). Given N randomly generated samples, we calculate the average Euclidean distances between each text feature f t and the generated motion feature f m from that text. The multimodal distance is calculated as:

where f t,i and f m,i are the features of the i th text-motion pair.

Diversity. All generated motions are randomly sampled to two subsets ( { x 1 , x 2 , ..., x X d } and { x ′ 1 , x ′ 2 , ..., x ′ X d } ) of the same size X d . Then, we extract motion features [14] and compute the average Euclidean distances between the two subsets:

Multimodality (MModality). We randomly sample a set of text descriptions with size J m from all descriptions. For each text description, we generate 2 × X m motion sequences, forming X m pairs of motions. We extract motion features and calculate the average Euclidean distance between each pair. We report the average of all text descriptions. We define features of the j th pair of the i th text description as ( x j,i , x ′ j,i ). The multimodality is calculated as:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors identify two major issues of text-to-motion generation as overemphasis on action names and the coarseness of sentence-level representations.\nTo this end, hierarchical semantic graphs are adopted to factorize coarse sentences into fine-grained action concepts, thus refining the generated motion from coarse-to-fine.\nFurthermore, the hierarchical design allows flexible control of the generation procedure.\nExtensive experiments show the efficacy of the proposed method.\n\nSTRENGTHS:\nThe proposed three tiers of node as motions, actions and specifics are interesting. \n\nThe hierarchical diffusion is well-designed in disentangling different level of granularity. Also, the experimental results show reasonable improvements with more and levels.\n\nThe performance is impressive.\n\nThe motion refinement application is attractive and promising.\n\nThe analysis on distribution of diffusion steps might be interesting in identifying the performance bottleneck among different semantic levels.\n\nExtensive ablation experiments provide a nice view of the effectiveness of different components.\n\nWEAKNESSES:\nQualitative comparison of different hierarchies is missed. It would be helpful to visualize motion generated from different level of latent embeddings, since fundamentally human is the only gold standard to evaluate motion generation, given the rather close numeric metrics of different levels.\n\nThe ablation on the design of the semantic graph is missed.\n\nA more detailed description of the motion refining procedure, like a figure, an algorithm, or some equations might be added for better clarity.\n\nQUESTIONS:\nIs it possible to generate with Action level or Specific level only? It would be interesting to see the corresponding results, given there is a trend that the more diffusion steps for Specific level brings better performance.\n\nThe hierarchical nature of the method might be suitable for long sequence generation. Comparison on this would be an interesting thing to do.\n\nFor motion refinement, is it possible to modify the graph beyond changing edge weights, i.e., adding/deleting nodes? Or tuning some nodes' weight to zero. What would happen?\n\nLIMITATIONS:\nThe author described some limitations of the methods, while it would be more helpful to discuss whether the current top-down paradigm could be extended in a bottom-up manner.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work decomposes the motion description into three levels including motion, action and specifics, and proposes hierarchical semantic graphs to achieve fine-grained control of motion generation. \nExperiments with the proposed method on HumanML3D and KIT datasets demonstrate better motion generation and more sensitive to subtle differences in texts than existing techniques.\n\nSTRENGTHS:\nThe motivation is clear, and the proposed solutions are to the raised issues. \nThe results seem to be impressive and the experimental setup is somewhat reasonable.\nThe ability to continuously refine the generated motion is meaningful and helpful to the community.\n\nWEAKNESSES:\n1. The paper states several times that Transformer may overemphasize action names, but I think that the action level in the hierarchical semantic graph is also another kind of emphasis. Therefore, if possible, I hope the authors can experimentally demonstrate the \"overemphasis\" of the Transformer and prove that the hierarchical semantic graph does not have this problem.\n2. Why is the last one in the action level of Figure 1 resumes instead of walking? How does it handle \"squats to picks\" and distinguish \"walks and walking\"? Also, the hierarchical semantic graph is built on a valid semantic parsing tool, and I'm not sure whether the tool will greatly affect the overall robustness of the model.\n3. What are the results in the first row of Tables 3 and 4 in what configuration? By my understanding, these two rows should be the complete ablation of their respective parts, so they should be numerically different.\n4. Is the modification in Figure 5 implemented in other latest works as well? Does it only modify the weights of the edges and can it modify the nodes?\n\nQUESTIONS:\nSee the above weakness.\n\nLIMITATIONS:\nThe authors fully explain the social impact of the work in the appendix, but only elaborate on the shortcomings of the used models (e.g., diffusion model and VAE) in terms of limitations, and do not well explain their own limitations and failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a coarse-to-fine diffusion model coupled with a hierarchical semantic graphs to address the text-to-motion generation problem. To preserve the fine-grained control signals from captions, three-level textual features are extracted through GAT. Then, three diffusion models are adopted to recover the latent motion representations, which is decoded into motion sequences via VAE.\n\nSTRENGTHS:\n1. Semantic graphs are introduced to model the coarse-to-fine textual descriptions.\n2. Hierarchical diffusion model is technically sound to learn the latent motion distributions.\n3. The experimental results outperforms sota methods.\n4. Fine-grained controllability of the proposed model is interesting.\n\nWEAKNESSES:\n1. The motivation behind the imbalance problem needs to be more convincing. In line 31-33, deficience of the existing method boils down to imbalance and coarseness. I am curious about if there are some examples could prove that imbalance problem exists in the other methods.\n2. Many descriptions in this paper are misleading or incorrect, and it needs further explaination. In line 176, the term ``codebook size'' is ambiguous because it is generallly associated with VQVAE. In line 180-181, the purpose of diffusion process should be learning a mapping from gaussian noise sampled from $\\mathcal{N}(0,1)$ to motion latent representation $z^m$, and the condition is the global motion node $V^m$. In line 183, $\\beta_t$ is not step size, it is the noise schedule.\n\nQUESTIONS:\nQ1: Please present more examples to prove that imbalance problem exists in the other methods and how your method addresses it.\n\nQ2: According to Table D and E in your supplementary material. R-Precision of your proposed VAE models outperforms Real motion, does this mean that R-Precision is not suitable for evaluating these two motion datasets?\n\nLIMITATIONS:\nThe authors have discussed the potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel motion generation pipeline that utilizes a 3-level hierarchical semantic graph. The entire reverse process of the motion diffusion model is divided into three stages: overall motion, local actions, and action specifics. The semantic graph is extracted through semantic role parsing and further enhanced using a Graph Attention Network. The obtained node features are subsequently input into a single transformer based on their semantic level.\n\nSTRENGTHS:\n1. The proposed GraphMotion approach demonstrates excellent performance in terms of numerical metrics on two datasets, showing significant improvements compared to existing methods. Particularly noteworthy is the detailed comparison of GraphMotion at different diffusion steps with other state-of-the-art methods, which adds more persuasive evidence to the results.\n\n2. The motion refinement aspect is intriguing, as users are able to achieve a certain degree of motion editing by modifying the content of nodes. This adds an interesting and interactive element to the method.\n\n3. The motivation and objectives of the entire approach are well-explained, making it easy to understand the rationale behind the proposed method.\n\nWEAKNESSES:\n1. It is recommended that the authors conduct user studies to quantitatively compare their method with other existing works based on qualitative results. The current demo video lacks sufficient comparisons, and additional comparisons should be included to provide a comprehensive evaluation. Furthermore, it would be beneficial to showcase the qualitative results of motion refinement in the demo video to provide a more complete demonstration of the method's capabilities.\n\nQUESTIONS:\n1. Why is the video length of MotionDiffuse shorter than other methods in the demo video?\n\n2. Why is there a significant decrease in the FID metric on KIT-ML when the number of steps increases from 50 to 150?\n\n3. Can GraphMotion perform low-level motion editing or motion composition similar to MotionDiffuse and MDM?\n\nLIMITATIONS:\nLimitations have been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}