SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Comparisons to current state-of-the-art methods on the KIT test set. We repeat all the evaluations 20 times and report the average with a  $95\%$  confidence interval.
Methods	R-Precision ↑	FID ↓	MM-Dist ↓	Diversity →	MModality ↑
Top-1	Top-2	Top-3
Real motion	0.424±.005	0.649±.006	0.779±.006	0.031±.004	2.788±.012	11.08±.097	-
Seq2Seq [44]	0.103±.003	0.178±.005	0.241±.006	24.86±.348	7.960±.031	6.744±.106	-
Text2Gesture [5]	0.156±.004	0.255±.004	0.338±.005	12.12±.183	6.946±.029	9.334±.079	-
MDM [54]	0.164±.004	0.291±.004	0.396±.004	0.497±.021	9.191±.022	10.85±.109	1.907±.214
Language2Pose [2]	0.221±.005	0.373±.004	0.483±.005	6.545±.072	5.147±.030	9.073±.100	-
Hier [12]	0.255±.006	0.432±.007	0.531±.007	5.203±.107	4.986±.027	9.563±.072	2.090±.083
TM2T [15]	0.280±.005	0.463±.006	0.587±.005	3.599±.153	4.591±.026	9.473±.117	3.292±.081
TEMOS [42]	0.353±.006	0.561±.007	0.687±.005	3.717±.051	3.417±.019	10.84±.100	0.532±.034
T2M [14]	0.370±.005	0.569±.007	0.693±.007	2.770±.109	3.401±.008	10.91±.119	1.482±.065
MLD [8]	0.390±.008	0.609±.008	0.734±.007	0.404±.027	3.204±.027	10.80±.117	2.192±.071
T2M-GPT [61]	0.416±.006	0.627±.006	0.745±.006	0.514±.029	3.007±.023	10.92±.108	1.570±.039
MotionDiffuse [62]	0.417±.004	0.621±.004	0.739±.004	1.954±.062	2.958±.005	11.10±.143	0.730±.013
GraphMotion (step=50)	0.417±.008	0.635±.006	0.755±.004	0.262±.021	3.085±.031	11.21±.106	3.568±.132
GraphMotion (step=150)	0.429±.007	0.648±.006	0.769±.006	0.313±.013	3.076±.022	11.12±.135	3.627±.113
Finally, the training objective of the diffusion models can be defined as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {D I F}} = \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, 1), z \sim q (z | \mathcal {V}), t} \left[ \underbrace {\| \epsilon^ {m} - \phi_ {m} \left(z ^ {m} , t ^ {m} , \mathcal {V} ^ {m}\right) \| _ {2} ^ {2}} _ {\text {M o t i o n l e v e l}} + \underbrace {\| \epsilon^ {a} - \phi_ {a} \left(z ^ {a} , t ^ {a} , \left[ \mathcal {V} ^ {m} , \mathcal {V} ^ {a} , z ^ {m} \right]\right) \| _ {2} ^ {2}} _ {\text {A c t i o n l e v e l}} \right. \\ + \underbrace {\left\| \epsilon^ {s} - \phi_ {s} \left(z ^ {s} , t ^ {s} , \left[ \mathcal {V} ^ {m} , \mathcal {V} ^ {a} , \mathcal {V} ^ {s} , z ^ {a} \right]\right) \right\| _ {2} ^ {2}} _ {\text {S p e c i f i c l e v e l}} \Bigg ]. \tag {6} \\ \end{array}
$$
During the training stage, the motion variational autoencoders are frozen. During the inference stage, the fine-grained semantic layer generates results based on the results from the coarse-grained semantic layer. We take the output at the specific level as the final result and use the motion decoder to decode the latent representation into the motion sequence.
It is worth noting that our method is as efficient as the one-stage diffusion methods during the inference stage, even though we decompose the diffusion process into three parts. This is because we can control the total number  $T^{m} + T^{a} + T^{s}$  of iterations by restricting it to be the same as those of the one-stage diffusion methods, which we will discuss in experiments (see Tab. 5).
Datasets, Metrics and Implementation Details. Datasets. We compare the proposed method with other methods on two commonly used public benchmarks: HumanML3D [14] and KIT [43].
Figure 3: Qualitative comparison of the existing methods. We provide the motion results from three text prompts. The darker colors indicate the later in time. The generated results of our method better match the descriptions, while others have downgraded motions or improper semantics, demonstrating that our method achieves superior controllability compared to well-designed baseline models.
HumanML3D [14] is currently the largest 3D human motion dataset that originates from and textually reannotates the HumanAct12 [16] and AMASS [40] datasets. This dataset comprises 14,616 human motions and 44,970 text descriptions, with each motion accompanied by at least three precise descriptions. The lengths of these descriptions are around 12 words. KIT [43] contains 3,911 human motion sequences and 6,278 textual annotations. Each motion sequence is accompanied by one to four sentences, with an average description length of 8 words.
Metrics. Following previous works, we use the following five metrics to measure the performance of the model. (1) R-Precision. Under the feature space of the pre-trained network in [14], given one motion sequence and 32 text descriptions (1 ground-truth and 31 randomly selected mismatched descriptions), motion-retrieval precision calculates the text and motion Top 1/2/3 matching accuracy. (2) Frechet Inception Distance (FID). We measure the distribution distance between the generated and real motion using FID [17] on the extracted motion features [14]. (3) Multimodal Distance (MM-Dist). We calculate the average Euclidean distances between each text feature and the generated motion feature from that text. (4) Diversity. All generated motions are randomly sampled to two subsets of the same size. Then, we extract motion features [14] and compute the average Euclidean distances between the two subsets. (5) Multimodality (MModality). For each text description, we generate 20 motion sequences, forming 10 pairs of motions. We extract motion features and calculate the average Euclidean distance between each pair. We report the average of all text descriptions.
Implementation Details. For the motion variational autoencoder, motion encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  all consist of 9 layers and 4 heads with skip connection [46]. Following MLD [8], we utilize a frozen text encoder of the CLIP-ViT-L-14 [45] model for text representation. The dimension of node representation  $D$  is set to 768. The dimension of latent embedding  $D^{\prime}$  is set to 256. We set the token sizes  $C^{m}$  to 2,  $C^{a}$  to 4, and  $C^{s}$  to 8. We set  $\lambda$  to 1e-4. All our models are trained with the AdamW [26, 38] optimizer using a fixed learning rate of 1e-4. We use 4 Tesla V100 GPUs for the training, and there are 128 samples on each GPU, so the total batch size is 512. For the HumanML3D dataset, the model is trained for 6,000 epochs during the motion variational autoencoder stage and 3,000 epochs during the diffusion stage. The number of diffusion steps of each level is 1,000 during training, and the step sizes  $\beta_{t}$  are scaled linearly from  $8.5\times 1\mathrm{e - }4$  to 0.012. For runtime, training tasks 16 hours for motion variational autoencoder and 24 hours for denoiser on 4 Tesla V100 GPUs.
Comparisons to State-of-the-art. We compare the proposed GraphMotion with other methods on two benchmarks. In Tab. 1, we show the results on the HumanML3D test set. Tab. 2 shows the results
Table 3: Ablation study about each part of our method on the HumanML3D test set. “↑” denotes that higher is better. “↓” denotes that lower is better.
Semantic Graph	Graph Reasoning	Coarse-to-Fine Diffusion	R-Precision Top-1 ↑	FID ↓
0.485±.003	0.418±.013
✓	0.491±.003	0.281±.011
✓	✓	0.494±.003	0.212±.015
✓	✓	0.490±.004	0.196±.012
✓	✓	✓	0.504±.003	0.116±.007
Table 4: Ablation study of the coarse-to-fine motion diffusion model on the HumanML3D test set. “↑” denotes that higher is better. “↓” denotes that lower is better.
Motion level	Action level	Specific level	R-Precision Top-1 ↑	FID ↓
0.485±.003	0.418±.013
✓	0.488±.003	0.217±.009
✓	✓	0.494±.003	0.150±.011
✓	✓	✓	0.504±.003	0.116±.007
Table 5: Ablation study about the total number of diffusion steps on the HumanML3D test set. “↑” denotes that higher is better. “↓” denotes that lower is better. We repeat all the evaluations 20 times and report the average with a  $95\%$  confidence interval. “×” denotes that this method does not apply this parameter. To speed up the sampling process, we use DDIM in practice following MLD.
Methods	Diffusion Steps	R-Precision ↑	FID ↓
Motion Tm	Action Ta	Specific Ts	Top-1	Top-2	Top-3
The total number of diffusion steps is 1000 with DDPM [18]
MDM [54]	1000	×	×	0.320±.005	0.498±.004	0.611±.007	0.544±.044
MotionDiffuse [62]	1000	×	×	0.491±.001	0.681±.001	0.782±.001	0.630±.001
The total number of diffusion steps is 50 with DDIM [50]
MLD [8]	50	×	×	0.481±.003	0.673±.003	0.772±.002	0.473±.013
GraphMotion (Ours)	20	15	15	0.489±.003	0.676±.002	0.771±.002	0.131±.007
GraphMotion (Ours)	15	15	20	0.496±.003	0.686±.003	0.778±.002	0.118±.008
The total number of diffusion steps is 150 with DDIM [50]
MLD [8]	150	×	×	0.461±.002	0.649±.003	0.797±.002	0.457±.011
GraphMotion (Ours)	50	50	50	0.504±.003	0.699±.002	0.785±.002	0.116±.007
on the KIT test set. Our model consistently outperforms state-of-the-art methods on both benchmarks, which validates the effectiveness of our method. Moreover, we provide qualitative motion results in Fig. 3. Compared to other methods, our method generates motions that match the text descriptions better, indicating that our method is more sensitive to subtle differences in texts.
Figure 4: Performance at each level on the HumanML3D test set. We repeat all experiments 10 times and report the average with a  $95\%$  confidence interval. The performance of the specific level is the best, which confirms the effectiveness of the coarse-to-fine motion diffusion model.
Ablative Analysis. Effect of each part of our method. To explore the impact of each part of our method, we provide the ablation results in Tab. 3. "Semantic Graph" refers to the decomposition of the original motion description into nodes of three levels, namely motion, action, and specific. These nodes are then directly fed into the one-stage motion diffusion model to generate the final motion. "Graph Reasoning" utilizes a graph network to reason and update nodes in the hierarchical semantic graph. "Coarse-to-Fine Diffusion" decomposes the text-to-motion diffusion process into three semantic levels, from coarse to fine, which capture the overall motion, local actions, and action specifics, respectively. If not otherwise specified, all ablation experiments are performed using a diffusion step setting of 150. As shown in Tab. 3, we find that "Semantic Graph" can avoid the imbalanced learning of different textual components, thus significantly improving the semantic understanding ability of the model and the overall quality of motion generation. "Graph Reasoning" further enhances the semantic reasoning ability of the model. In addition, "Coarse-to-Fine Diffusion" can significantly improve the quality of motion generation. Our full model achieves the best performance, which demonstrates that the three parts are beneficial for motion generation.
Figure 5: Qualitative analysis of refining motion results. The darker colors indicate the later in time. The trajectory of the human body is indicated by an arrow. The trajectory associated with the modified edge is highlighted in red, and other parts are identified in blue.
Table 6: Quantitative experiment of the imbalance problem on the HumanML3D test set. “↑” denotes that higher is better. “↓” denotes that lower is better.
Methods	FID ↓	MM-Dist ↓	Diversity →	MModality ↑
MDM	5.622	7.163	8.713	3.578
MLD	3.492	5.632	8.874	3.596
GraphMotion	1.826	5.530	9.284	3.699
Table 7: User studies for quantitative comparison. We show the preference rate of GraphMotion over the compared model.
Methods Compared	Preference Rate
GraphMotion vs. MotionDiffuse	64.10%
GraphMotion vs. MLD	56.41%
GraphMotion vs. Ground Truth	48.72%
Analysis of the coarse-to-fine motion diffusion model. In Tab. 4, we provide the ablation study of the coarse-to-fine motion diffusion model on the HumanML3D test set. These results prove that coarse-to-fine generation is beneficial to motion generation. In addition, we show the performance at each level in Fig. 4. Among the three levels, the performance of the specific level is the best, which confirms the effectiveness of the coarse-to-fine motion diffusion model.
Effect of the diffusion steps. In Tab. 5, we show the ablation study of the total number of diffusion steps on the HumanML3D test set. Following MLD [8], we adopt the denoising diffusion implicit models [50] (DDIM) during interference. As shown in Tab. 5, our method consistently outperforms the existing state-of-the-art methods with the same total number of diffusion steps, which demonstrates the efficiency of our method. With the increase of the total diffusion steps, the performance of our method is further improved, while the performance of MLD saturates. We find that the number of diffusion steps at the higher level (e.g., specific level) has a greater impact on the result. Therefore, in scenarios requiring high efficiency, we recommend allocating more diffusion steps to the higher level.
Quantitative and Qualitative Discussion. Quantitative experiment of the imbalance problem. In this experiment, we mask the verbs and action names in the motion description to force the model to generate motion only from action specifics. For example, given the motion description "a person walks several steps forward in a straight line", we would mask "walks". Transformer extracts text features automatically and implicitly. However, it may encourage the model to take shortcuts, such as overemphasizing the action name "walks" at the expense of other important properties. Therefore, when the verbs and action names are masked, the other models, which directly use the transformer
Figure 6: Additional qualitative analysis of refining motion results. The qualitative results demonstrate that our approach provides a novel method of refining generated motions.
to extract text features, fail to generate motion well. By contrast, the hierarchical semantic graph explicitly extracts the action specifics. The explicit factorization of the language embedding space facilitates a comprehensive understanding of motion description. It allows the model to infer from action specifics such as "several steps forward" and "in a straight line" that the overall motion is "walking forward". As shown in Tab. 6, our method can synthesize motion by relying only on action specifics, while other methods fail to generate motion well. These results indicate that our method avoids the imbalance problem of other methods.
Human evaluation. In our evaluation, we randomly selected 39 motion descriptions for the user study. As shown in Tab. 7, GraphMotion is preferred over the other models most of the time.
Qualitative analysis of refining motion results. To fine-tune the generated results for more fine-grained control, our method can continuously refine the generated motion by modifying the edge weights of the hierarchical semantic graph. As illustrated in Fig. 5, we can alter the action attributes by manipulating the weights of the edges of the action node and the specific node. For example, by increasing the weights of the edges of "zags" and "to left", the human body will move farther to the left. Moreover, by fine-tuning the weights of the edges of the global motion node and the action node, we can fine-tune the duration of the corresponding action in the whole motion. For example, by enhancing the weights of the edges of the global motion node and "walking", the length of the walk will be increased. In Fig. 6, we provide additional qualitative analysis of refining motion results. Specifically, we perform the additional operations on the hierarchical semantic graphs: (1) masking the node by replacing it with the MASK token; (2) modifying the node; (3) deleting nodes; (4) adding a new node. The qualitative results demonstrate that our approach provides a novel method of refining generated motions, which may have a far-reaching impact on the community.
In this paper, we focus on improving the controllability of text-driven human motion generation. To provide fine-grained control over motion details, we propose a novel control signal called the hierarchical semantic graph, which consists of three kinds of abstract nodes, namely motions, actions, and specifics. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments demonstrate that our method achieves better controllability than the existing state-of-the-art methods. More encouragingly, our method can continuously refine the generated motion by modifying the edge weights of hierarchical semantic graphs, which may have a far-reaching impact on the community.
Acknowledgements. This work was supported by the National Key R&D Program of China (2022ZD0118101), Nature Science Foundation of China (No.62202014), and Shenzhen Basic Research Program (No.JCYJ20220813151736001).
[60] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV, pages 670-685, 2018.
[61] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In CVPR, 2023.
[62] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.
[63] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. arXiv preprint arXiv:2304.01116, 2023.
Abstract This appendix provides additional discussions (Sec. A), implementation details (Sec. B), more qualitative results (Sec. C), several additional experiments (Sec. D), details of motion representations and metric definitions (Sec. E).
Code Code is available at https://github.com/jpthu17/GraphMotion. In this code, we provide the process of the training and evaluation of the proposed method, and the pre-trained weights.
A Additional Discussions
A.1 Potential Negative Societal Impacts
While our work effectively enhances the quality of human motion synthesis, there is a potential risk that it may be used for generating fake content, such as generating fake news, which can pose a threat to information security. Moreover, when factoring in energy consumption, there is a possibility that the widespread use of generative models for synthesizing human motions may contribute to increased carbon emissions and exacerbate global warming.
A.2 Limitations of our Work
Although our method makes some progress, there are still many limitations worth further study. (1) The proposed GraphMotion inherits the randomness of diffusion models. This property benefits diversity but may yield undesirable results sometimes. (2) The human motion synthesis capabilities of GraphMotion are limited by the performance of the pre-trained motion variational autoencoders, which we will discuss in experiments (Tab. D and Tab. E). This defect also exists in the existing state-of-the-art methods, such as MLD [8] and T2M-GPT [61], which also use motion variational autoencoder. (3) Though the proposed GraphMotion brings negligible extra cost on computations, it is still limited by the slow inference speed of existing diffusion models. We will discuss the inference time in experiments (Tab. C). This defect also exists in the existing state-of-the-art methods, such as MDM [54] and MLD [8], which also use diffusion models.
A.3 Future Work
In this paper, we focus on improving the controllability of text-driven human motion generation. Recently, large language models have made remarkable progress, making large language models a promising text extractor for human motion generation. However, despite their strengths in general reasoning and broad applicability, large language models may not be optimized for extracting subtle motion nuances. In future research, we will incorporate the features of large-scale languages into our model, using hierarchical semantic graphs to give large language models the ability to extract fine-grained motion description structures.
B.1 Details of Hierarchical Semantic Graphs
To obtain actions, attributes of action as well as the semantic role of each attribute to the corresponding action, we implement a semantic parser of motion descriptions based on a semantic role parsing toolkit [48, 7]. Specifically, given the motion description, the parser extracts verbs that appeared in the sentence and attribute phrases corresponding verb, and the semantic role of each attribute phrase. The overall sentence is treated as the global motion node in the hierarchical graph. The verbs are considered as action nodes and connected to the motion node with direct edges, allowing for implicit learning of the temporal relationships among various actions during graph reasoning. The attribute phrases are specific nodes that are connected with action nodes. The edge type between action and specific nodes is determined by the semantic role of the specifics in relation to the action.
Table A: Node types and edge types in the parsed hierarchical semantic graph. Each edge type corresponds to a type of semantic role.
Node type	Description
Motion	global motion description
Action	verb
Specific	attribute of action
Edge type	Description
ARG0	agent
ARG1	patient
ARG2	instrument, benefactive
ARG3	start point
ARG4	end point
ARGM-LOC	location (where)
ARGM-MNR	manner (how)
ARGM-TMP	time (when)
ARGM-DIR	direction (where to/from)
ARGM-ADV	miscellaneous
ARGM-MA	motion-action dependencies
OTHERS	other argument types, e.g., action
Real	Ours	MDM (ICLR23)	MLD (CVPR23)	MotionDiffuse (arXiv22)
“Using both hands, a person bends over to pat the ground.”
A person sitting on the floor scratches their head and then leans back on their right hand.”
“A person gets on all fours and crawls forward, then turns around and crawls back.”
“Person kicks left legs back and forth.”
Figure A: Qualitative comparison of the existing methods. The darker colors indicate the later in time. The generated results of our method better match the descriptions, while others have downgraded motions or improper semantics, demonstrating that our method achieves superior controllability compared to well-designed baseline models. We have provided a supplemental video in our supplementary material. In the supplemental video, we show comparisons of text-driven motion generation. We suggest the reader watch this video for dynamic motion results.
As shown in Tab. A, we extract three types (motions, actions, and specifics) of nodes and twelve types of edges to represent various associations among the nodes.
B.2 Classifier-free Diffusion Guidance
Following MLD [8], our denoiser network is learned with classifier-free diffusion guidance [19]. The classifier-free diffusion guidance improves the quality of samples by reducing diversity in conditional diffusion models. Concretely, it learns both the conditioned and the unconditioned distribution (10% dropout [51]) of the samples. Finally, we perform a linear combination in the following manner, which is formulated as:
$$
\widehat {\epsilon^ {m}} _ {s c a l e} = \alpha^ {'} \phi_ {m} (z ^ {m}, t ^ {m}, \mathcal {V} ^ {m}) + (1 - \alpha^ {'}) \phi_ {m} (z ^ {m}, t ^ {m}, \emptyset),
$$
$$
\widehat {\epsilon} _ {s c a l e} ^ {a} = \alpha^ {\prime} \phi_ {a} \left(z ^ {a}, t ^ {a}, \left[ \mathcal {V} ^ {m}, \mathcal {V} ^ {a}, z ^ {m} \right]\right) + \left(1 - \alpha^ {\prime}\right) \phi_ {a} \left(z ^ {a}, t ^ {a}, \varnothing\right), \tag {A}
$$
$$
\widehat {\epsilon} _ {s c a l e} ^ {s} = \alpha^ {\prime} \phi_ {s} \left(z ^ {s}, t ^ {s}, \left[ \mathcal {V} ^ {m}, \mathcal {V} ^ {a}, \mathcal {V} ^ {s}, z ^ {a} \right]\right) + \left(1 - \alpha^ {\prime}\right) \phi_ {s} \left(z ^ {s}, t ^ {s}, \varnothing\right),
$$
A person is doing jumping jacks.	A person takes a jump sideways to their right.	A person is doing a dance.
A person hops up and down in place.	A person looks to the right then kicks something with their left foot.	The person is clapping.	Dancing randomly like an indian traditional dance and mix of western culture.
A person bend the knee and leand to the floor and start walked.	A person walking forward then walks back.	A person steps forward and reaches down to grab or place something with their right hand.