SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peng Jin $^{1,4}$  Yang Wu $^{3*}$  Yanbo Fan $^{3}$  Zhongqian Sun $^{3}$  Yang Wei $^{3}$  Li Yuan $^{1,2,4*}$
$^{1}$  School of Electronic and Computer Engineering, Peking University, Shenzhen, China
2 Peng Cheng Laboratory, Shenzhen, China 3 Tencent AI Lab, China
$^{4}$ AI for Science (AI4S)-Preferred Program, Peking University Shenzhen Graduate School, China
jp21@stu.pku.edu.cn dylan.yangwu@qq.com yuanli-ece@pku.edu.cn
Most text-driven human motion generation methods employ sequential modeling approaches, e.g., transformer, to extract sentence-level text representations automatically and implicitly for human motion synthesis. However, these compact text representations may overemphasize the action names at the expense of other important properties and lack fine-grained details to guide the synthesis of subtly distinct motion. In this paper, we propose hierarchical semantic graphs for fine-grained control over motion generation. Specifically, we disentangle motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Such global-to-local structures facilitate a comprehensive understanding of motion description and fine-grained control of motion generation. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments on two benchmark human motion datasets, including HumanML3D and KIT, with superior performances, justify the efficacy of our method. More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact on the community. Code and pre-trained weights are available at https://github.com/jpthu17/GraphMotion.
Human motion generation is a fundamental task in computer animation [54, 4] and has many practical applications across various industries including gaming, film production, virtual reality, and robotics [61, 8]. With the progress made in recent years, text-driven human motion generation has allowed for the synthesis of a variety of human motion sequences based on natural language descriptions. Therefore, there is a growing interest in generating manipulable, plausible, diverse, and realistic sequences of human motion from flexible natural language descriptions.
Existing text-to-motion generation methods [42, 61, 8, 1, 63] mainly rely on sentence-level representations of texts and directly learn the mapping from the high-level language space to the motion sequences. Recently, some works [54, 8, 62] propose conditional diffusion models for human motion synthesis and further improve the synthesized quality and diversity. Although these methods have made encouraging progress, they are still deficient in the following two aspects. (i) Imbalance. The model, which directly uses the transformers [55] to extract text features automatically and implicitly, may overemphasize the action names at the expense of other important properties like direction and
*Corresponding author: Yang Wu, Li Yuan.
Figure 1: We propose hierarchical semantic graphs, a fine-grained control signal, for text-to-motion generation and factorize text-to-motion generation into hierarchical levels including motions, actions, and specifics to form a coarse-to-fine structure. This approach enhances the fine-grained correspondence between textual data and motion sequences and achieves better controllability conditioning on hierarchical semantic graphs than carefully designed baselines.
intensity. As a typical consequence of this unbalanced learning, the network is insensitive to the subtle changes in the input text and lacks fine-grained controllability. (ii) Coarseness. On the one hand, motion descriptions frequently refer to multiple actions and attributes. However, the compact sentence-level representations extracted by current works usually fail to convey the clarity and detail needed to fully understand the text, leading to a lack of fine-grained details to guide the synthesis of subtly distinct motion. On the other hand, mapping directly of existing works from the high-level language space to motion sequences further hinders the generation of fine-grained details. Therefore, we argue that it is time to seek a more precise and detailed text-driven human motion generation method to ensure an accurate synthesis of complex human motions.
To this end, we propose a more fine-grained control signal, hierarchical semantic graphs, to represent different intentions for controllable motion generation and design a coarse-to-fine motion diffusion model, called GraphMotion. As shown in Fig. 1, motion descriptions inherently possess hierarchical structures and can be represented as hierarchical graphs composed of three types of abstract nodes, namely motions, actions, and specifics. Concretely, the overall sentence describes the global motion involving multiple actions, e.g., "walk", "pick", and "stand" in Fig. 1, which occur in sequential order. Each action consists of different specifics that act as its attributes, such as the agent and patient of the action. Such global-to-local structures contribute to a reliable and comprehensive understanding of motion descriptions. Correspondingly, to take full advantage of this fine-grained control signal, we decompose the text-to-motion diffusion process into three semantic levels from coarse to fine, which are responsible for capturing the overall motion, local actions, and action specifics, respectively.
The proposed GraphMotion has three compelling advantages: First, the explicit factorization of the language embedding space enables us to build a fine-grained correspondence between textual data and motion sequences, which avoids the imbalanced learning of different textual components and coarse-grained control signal representation. Second, the hierarchical refinement property of GraphMotion allows the model to progressively enhance the generated results from coarse to fine, which avoids the coarse-grained generated results. Third, to further fine-tune the generated results for more fine-grained control, our method can continuously refine the generated motion by modifying the edge weights of the hierarchical semantic graph. Experimental results on two benchmark datasets for text-to-motion generation, including HumanML3D [14] and KIT [43], demonstrate the advantages of GraphMotion. The main contributions of this work are as follows:
- To the best of our knowledge, we are the first to propose hierarchical semantic graphs, a fine-grained control signal, for text-to-motion generation. It decomposes motion descriptions into global-to-local three types of abstract nodes, namely motions, actions, and specifics.
- Correspondingly, we decompose the text-to-motion diffusion process into three semantic levels. This allows the model to gradually refine results from coarse to fine. Experiments show that our method achieves new state-of-the-art results on two text-to-motion datasets.
- More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact.
Text-driven Human Motion Generation. Text-driven human motion generation aims to generate 3D human motion based on text descriptions. Due to the user-friendliness and convenience of natural language [22], text-driven human motion generation is gaining significant attention and has many applications. Recently, two categories of motion generation methods have emerged: joint-latent models [2, 42] and diffusion models [8, 62, 47]. Joint-latent models, e.g., TEMOS [42], typically learn a motion variational autoencoder and a text variational autoencoder. These models then constrain the text and motion encoders into a shared latent space using the Kullback-Leibler divergences [29] loss. The latter category of methods, e.g., MDM [54], proposes a conditional diffusion model for human motion generation to learn a powerful probabilistic mapping from the textual descriptors to human motion sequences. Although these methods have made encouraging progress, they still suffer from two major deficiencies: unbalanced text learning and coarse-grained generated results. In this paper, we propose to leverage the inherent structure of language [6, 58] for motion generation. Specifically, we introduce hierarchical semantic graphs as a more effective control signal for representing different intentions and design a coarse-to-fine motion diffusion model using this signal.
Diffusion Generative Models. Diffusion generative models [49, 18, 11, 21, 50] are a type of neural generative model that uses the stochastic diffusion process, which is based on thermodynamics. The process involves gradually adding noise to a sample from the data distribution, and then training a neural network to reverse this process by gradually removing the noise. In recent years, diffusion models have shown promise in a variety of tasks such as image generation [18, 50, 11, 19, 57], natural language generation [3, 36, 13], as well as visual tasks [9]. Some other works [25, 32] have attempted to adapt diffusion models for cross-modal retrieval [33, 34, 35, 23, 24]. Inspired by the success of diffusion generative models, some works [62, 54, 8] have applied diffusion models to human motion generation. However, these methods typically learn a one-stage mapping from the high-level language space to motion sequences, which hinders the generation of fine-grained details. In this paper, we decompose the text-to-motion diffusion process into three semantic levels from coarse to fine. The resultant levels are responsible for capturing overall motion, local actions, and action specifics, which enhances the generated results progressively from coarse to fine.
Graph-based Reasoning. The graph convolutional network [28] is originally proposed to recognize graph data. It uses convolution on the neighborhoods of each node to produce outputs. Graph attention networks [56] further enhance graph-based reasoning by dynamically attending to the features of neighborhoods. The graph-based reasoning has shown great potential in many tasks, such as scene graph generation [60], visual question answering [20, 31, 30], natural language generation [6], and cross-modal retrieval [7]. In this paper, we focus on reasoning over hierarchical semantic graphs on motion descriptions for fine-grained control of human motion generation.
In this paper, we tackle the tasks of text-driven human motion generation. Concretely, given an arbitrary motion description, our goal is to synthesize a human motion  $x^{1:L} = \{x^i\}_{i=1}^L$  of length  $L$ . The overview of the proposed GraphMotion is shown in Fig. 2.
3.1 Hierarchical Semantic Graph Modeling
Existing works for text-driven human motion generation typically directly use the transformer [55] to extract text features automatically and implicitly. However, motion descriptions inherently possess hierarchical structures that can be divided into three sorts of abstract nodes, including motions, actions, and specifics. Compared with the sequential structure, such global-to-local structure contributes to a reliable and comprehensive understanding of the semantic meaning of motion descriptions and is a promising fine-grained control signal for text-to-motion generation.
Semantic Role Parsing. To obtain actions, attributes of action as well as the semantic role of each attribute to the corresponding action, we implement a semantic parser of motion descriptions based on a semantic role parsing toolkit [48, 7]. We extract three types (motions, actions, and specifics) of nodes and twelve types of edges to represent various associations among the nodes. For details about the types of nodes and edges, please refer to our supplementary material.
Figure 2: The overview of the proposed GraphMotion for text-driven human motion generation. We factorize motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Correspondingly, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics.
Specifically, given the motion description, the parser extracts verbs that appeared in the sentence and attribute phrases corresponding verb, and the semantic role of each attribute phrase. The overall sentence is treated as the global motion node in the hierarchical graph. The verbs are considered as action nodes and connected to the motion node with direct edges, allowing for implicit learning of the temporal relationships among various actions during graph reasoning. The attribute phrases are specific nodes that are connected with action nodes. The edge type between action and specific nodes is determined by the semantic role of the specifics in relation to the action.
Graph Node Representation. Given the motion description, we follow previous works [53, 54, 61, 8] and leverage the text encoder of CLIP [45] to extract the text embedding. For the global motion node  $v^{m}$ , we utilize the [CLS] token to summarize the salient event described in the sentence. For the action node  $v^{a}$ , we use the token of the corresponding verb as the action node representation. For the specific node  $v^{s}$ , we apply mean-pooling over tokens of each word in the attribute phrase.
Action-aware Graph Reasoning. The interactions across different levels in the constructed graph not only explain the properties of local actions and how local actions compose the global motion, but also reduce ambiguity at each node. For example, the verb "pick" in Fig. 1 can represent different actions without context, but the context "with both hands" constrains its semantics, so that it represents the action of "pick up with both hands" rather than "pick up with one hand." Therefore, we propose to reason over interactions in the graph to obtain hierarchical textual representations.
We utilize graph attention networks [56] (GAT) to model interactions in a graph. Specifically, given the initialized node embeddings  $v = \{v^m, v^a, v^s\}$ , we first transform the input node embeddings into higher-level embeddings  $h = \{h^m, h^a, h^s\}$  by:
$$
h ^ {m} = \boldsymbol {W} v ^ {m}, \quad h ^ {a} = \boldsymbol {W} v ^ {a}, \quad h ^ {s} = \boldsymbol {W} v ^ {s}, \tag {1}
$$
where  $\mathbf{W} \in \mathbb{R}^{D \times D}$  is a shared linear transformation and  $D$  is the dimension of node representation. For each pair  $\{h_i, h_j\}$  of connected nodes, we concatenate the node  $h_i \in \mathbb{R}^D$  with its neighbor node  $h_j \in \mathbb{R}^D$ , generating the input data  $\tilde{h}_{ij} = [h_i, h_j] \in \mathbb{R}^{2D}$  of the graph attention module.
However, in a graph with multiple types of edges, the vanilla graph networks need to learn separate transformation matrices for each edge type. This can be inefficient when learning from limited motion data, and prone to over-fitting on rare edge types.
To this end, we propose to factorize multi-relational weights into two parts: a common transformation matrix  $M \in \mathbb{R}^{2D \times 1}$  that is shared for all edge types and a relationship embedding matrix  $M_{r} \in \mathbb{R}^{2D \times N}$  that is specific for different edges, where  $N$  is the number of edge types. Following GAT [56], we apply LeakyReLU [39] in the calculation of attention coefficients and set the negative input slope to 0.2. The attention coefficient  $\tilde{e}_{ij}$  is formulated as:
$$
e _ {i j} = \operatorname {L e a k y R e L U} \left(\boldsymbol {M} ^ {\top} \tilde {h} _ {i j}\right) + \operatorname {L e a k y R e L U} \left(R _ {i j} \boldsymbol {M} _ {\boldsymbol {r}} ^ {\top} \tilde {h} _ {i j}\right), \quad \tilde {e} _ {i j} = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k \in \mathbb {N} _ {i}} \exp \left(e _ {i k}\right)}, \tag {2}
$$
where  $R_{ij} \in \mathbb{R}^{1 \times N}$  is a one-hot vector denoting the type of edge between node  $i$  and  $j$ .  $\mathbb{N}_i$  is the set of neighborhood nodes of node  $i$ . To alleviate over-smoothing [59] in graph networks, we apply skip connection when calculating output embeddings. The output embeddings  $\mathcal{V}$  are formulated as:
$$
\mathcal {V} _ {i} = \sigma \left(\sum_ {j \in \mathbb {N} _ {i}} \tilde {e} _ {i j} h _ {j}\right) + v _ {i}, \tag {3}
$$
where  $\sigma$  is a nonlinear function. Following GAT [56], we use ELU [10] as the nonlinear function  $\sigma$ .
3.2 Coarse-to-Fine Motion Diffusion Model for Graphs
To leverage the coarse-to-fine topology of hierarchical semantic graphs during generation, we also decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. During the reverse denoising process, the fine-grained semantic layer generates results based on the results from the coarse-grained semantic layer. This allows for a detailed and plausible representation of the intended motion.
Motion Representation. Following previous works [42, 8, 61], we first encode the motion into the latent space with a motion variational autoencoder [27] and then use diffusion models to learn the mapping from hierarchical semantic graphs to the motion latent space.
Specifically, we build the motion encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  based on the transformer [55, 41]. For the motion encoder  $\mathcal{E}$ , we take  $C$  learnable query tokens and motion sequence  $x^{1:L} = \{x^i\}_{i=1}^L$  as inputs to generate motion latent embeddings  $z \in \mathbb{R}^{C \times D'}$ , where  $D'$  is the dimension of latent representation. For the motion decoder  $\mathcal{D}$ , we take the latent embeddings  $z \in \mathbb{R}^{C \times D'}$  and the motion query tokens as the input to generate a human motion sequence  $x^{1:L} = \{x^i\}_{i=1}^L$  with  $L$  frames.
The loss  $\mathcal{L}_{\mathrm{VAE}}$  of the motion variational autoencoder can be divided into two parts. First, we use the mean squared error (MSE) to reconstruct the original input. Second, we use the Kullback-Leibler divergences (KL) loss [29] to narrow the distance between the distribution of latent space  $q(z|x)$  and the standard Gaussian distribution  $\mathcal{N}(0,1)$ . The full loss  $\mathcal{L}_{\mathrm{VAE}}$  is formulated as:
$$
\mathcal {L} _ {\mathrm {V A E}} = \mathbb {E} _ {x \sim q (x)} \left[ \underbrace {\| x - \mathcal {D} (\mathcal {E} (x)) \| _ {2} ^ {2}} _ {\text {M S E}} + \lambda \underbrace {\operatorname {K L} \left(\mathcal {N} (0 , \mathrm {I}) \| q (z | x)\right)} _ {\text {K L}} \right], \tag {4}
$$
where  $\lambda$  is the trade-off hyper-parameter.  $q(z|x) = \mathcal{N}(\mu_z,\Sigma_z)$  is obtained by sampling based on the mean  $\mu_z$  and variance  $\Sigma_z$  estimated by the model. To generate motion from coarse to fine step by step, we encode motion independently into three latent representation spaces  $z^{m}\in \mathbb{R}^{C^{m}\times D^{\prime}}$ ,  $z^{a}\in \mathbb{R}^{C^{a}\times D^{\prime}}$  and  $z^{s}\in \mathbb{R}^{C^{s}\times D^{\prime}}$ , where the number of tokens gradually increases, i.e.,  $C^m\leq C^a\leq C^s$ .
Hierarchical Graph-to-Motion Diffusion. Corresponding to the three-level structure of the hierarchical semantic graphs, we decompose the diffusion process into three semantic levels and build three transformer-based denoising models, which correspond to motions, actions, and specifics.
For the motion level model  $\phi_{m}$ , our goal is to learn the diffusion process from global motion node  $\mathcal{V}^{m}$  to motion latent representation  $z^{m}$ . In a forward diffusion process  $q(z_{t}^{m}|z_{t - 1}^{m})$ , noised sampled from Gaussian distribution is added to a ground truth data distribution  $z_0^m$  at every noise level  $t$ :
$$
q \left(z _ {t} ^ {m} \mid z _ {t - 1} ^ {m}\right) = \mathcal {N} \left(z _ {t} ^ {m}; \sqrt {1 - \beta_ {t}} z _ {t - 1} ^ {m}, \beta_ {t} \mathrm {I}\right), \quad q \left(z _ {1: T} ^ {m} \mid z _ {0} ^ {m}\right) = \prod_ {t = 1} ^ {T} q \left(z _ {t} ^ {m} \mid z _ {t - 1} ^ {m}\right), \tag {5}
$$
where  $\beta_{t}$  is the step size which gradually increases.  $T$  is the length of the Markov chain. We sample  $z_{t}^{m}$  by  $z_{t}^{m} = \sqrt{\bar{\alpha}_{t}} z_{0}^{m} + \sqrt{1 - \bar{\alpha}_{t}}\epsilon^{m}$ , where  $\bar{\alpha}_{t} = \prod_{i=1}^{t}(1 - \beta_{i})$ .  $\epsilon^{m}$  is a noise sampled from  $\mathcal{N}(0,1)$ . We follow previous works [18, 8] and predict the noise component  $\epsilon^{m}$ , i.e.,  $\widehat{\epsilon^{m}} = \phi_{m}(z^{m}, t^{m}, \mathcal{V}^{m})$ .
For the action level model  $\phi_{a}$ , to leverage the results generated by the motion level, we concatenate the action node  $\mathcal{V}^a$ , the motion node  $\mathcal{V}^m$ , and the result  $z^{m}$  generated by the motion level together as the input of the action level denoising network, i.e.,  $\widehat{\epsilon^a} = \phi_a(z^a,t^a,[\mathcal{V}^m,\mathcal{V}^a,z^m])$
For the specific level model  $\phi_s$ , we leverage the results generated by the action level and nodes at all semantic levels to predict the noise component, i.e.,  $\widehat{\epsilon^s} = \phi_s(z^s,t^s,[\mathcal{V}^m,\mathcal{V}^a,\mathcal{V}^s,z^a])$ .
Table 1: Comparisons to current state-of-the-art methods on the HumanML3D test set. “↑” denotes that higher is better. “↓” denotes that lower is better. “→” denotes that results are better if the metric is closer to the real motion. We repeat all the evaluations 20 times and report the average with a  $95\%$  confidence interval. Bold and underlined indicate the best and second-best results, respectively. For fair comparisons, we report results with total diffusion steps  $T^{m} + T^{a} + T^{s}$  of 50 and 150.
Methods	R-Precision ↑	FID ↓	MM-Dist ↓	Diversity →	MModality ↑
Top-1	Top-2	Top-3
Real motion	0.511±.003	0.703±.003	0.797±.002	0.002±.000	2.974±.008	9.503±.065	-
Text2Gesture [5]	0.165±.001	0.267±.002	0.345±.002	7.664±.030	6.030±.008	6.409±.071	-
Seq2Seq [44]	0.180±.002	0.300±.002	0.396±.002	11.75±.035	5.529±.007	6.223±.061	-
Language2Pose [2]	0.246±.001	0.387±.002	0.486±.002	11.02±.046	5.296±.008	7.676±.058	-
Hier [12]	0.301±.002	0.425±.002	0.552±.004	6.532±.024	5.012±.018	8.332±.042	-
MDM [54]	0.320±.005	0.498±.004	0.611±.007	0.544±.044	5.566±.027	9.559±.086	2.799±.072
TEMOS [42]	0.424±.002	0.612±.002	0.722±.002	3.734±.028	3.703±.008	8.973±.071	0.368±.018
TM2T [15]	0.424±.003	0.618±.003	0.729±.002	1.501±.017	3.467±.011	8.589±.076	2.424±.093
T2M [14]	0.457±.002	0.639±.003	0.740±.003	1.067±.002	3.340±.008	9.188±.002	2.090±.083
MLD [8]	0.481±.003	0.673±.003	0.772±.002	0.473±.013	3.196±.010	9.724±.082	2.413±.079
MotionDiffuse [62]	0.491±.001	0.681±.001	0.782±.001	0.630±.001	3.113±.001	9.410±.049	1.553±.042
T2M-GPT [61]	0.492±.003	0.679±.002	0.775±.002	0.141±.005	3.121±.009	9.722±.082	1.831±.048
GraphMotion (step=50)	0.496±.003	0.686±.003	0.778±.002	0.118±.008	3.143±.009	9.796±.069	2.603±.095
GraphMotion (step=150)	0.504±.003	0.699±.002	0.785±.002	0.116±.007	3.070±.008	9.692±.067	2.766±.096