SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Spatial Independence. In conventional motion generative models [20, 11, 18], the raw motion vectors are directly inputted into a fully-connected layer to generate a latent representation:  $\mathbf{F} = W^{F}(\Theta)$ , where  $\Theta$  signifies the raw data of a single motion sequence.  $\mathbf{F}$  represents the latent representation following the linear projection performed by  $W^{F}$ . This operation, while straightforward and effective, often overlooks the distinct interpretations of the motion representation. It also merges data from various body parts, which hampers body part-aware modeling. To overcome this, we manually divide the raw representation into  $N_{S}$  groups, each representing a commonly observed body part. In our work, we consider seven body parts: head, spine, left arm, right arm, left leg, right leg, and trajectory of the pelvis joint. We perform linear projection on each part independently and concatenate them into the latent representation  $\mathbf{F}' \in \mathbb{R}^{N_m \times N_S \times L_p}$ , where  $L_p$  represents the dimensionality of each part. Given our aim for each global template group to correspond exactly to one body part, we set  $N_S = N_g$ . Additionally, we utilize the Feed Forward Network (FFN) module to process each body part independently.
In addition, we enforce that features from each body part can only contribute to the corresponding group of global templates. In situations where our models are trained without detailed descriptions for each body part, text features from the overall sentence will contribute to each group. However, in the context of the Fine-grained Spatio-temporal Generation task, where we have multiple texts
describing different parts, each text feature sequence will only influence the corresponding body part. This group division approach ensures the preservation of spatial independence.
As a supplement, we introduce the spatial feature refinement here to allow the communication between different body parts in SAMI:
$$
\mathbf {Y} _ {k, i} ^ {S} = \sum_ {j = 1} ^ {N _ {S}} \mathbf {B} _ {k, j} \cdot \omega_ {i, j}, \tag {5}
$$
where  $\mathbf{B}_{k,j} \in \mathbb{R}^{L_g}$  is the projected feature from  $j$ th body part of  $k$ th element.  $\omega_{k,j} \in \mathbb{R}^{N_S \times N_S}$  is a learnable parameter to reflect the relative significance. The value of  $\omega$  is shared to calculate  $\mathbf{Y}_{k,i}^S$  for all  $k$  and  $i$ . It can be explained as a common knowledge to show the correlation between different body parts.
Sparsely-Activated Mixture-of-Expert The design of spatio-temporal independence decouples various components of the entire sequence, compelling the model to focus more on local consistency. This introduces certain complexities to the model's learning process. To address this, we attempt to broaden the overall network structure to enhance learning capability. Thus, we employ the Sparsely-activated Mixture-of-Expert technique, which significantly boosts learning effectiveness at the cost of marginal inference time increment. It can adaptively extract informative features from the motion sequence. Specifically, the linear projections we used for calculating  $\mathbf{B}_{k,j},\mathbf{Q}_{i,j},\mathbf{K}_{i,j}$  from motion features are empowered by MoE. The formulation can be written as:
$$
f _ {\mathrm {M o E}} (x) = \sum_ {i = 1} ^ {N _ {e}} \operatorname {T O P} _ {k} \left(\operatorname {S o f t m a x} \left(W _ {1} x\right)\right) \cdot W _ {2} \phi \left(W _ {3} x\right), \tag {6}
$$
where  $W_{1}, W_{2}, W_{3}$  are learnable parameters.  $\mathrm{TOP}_k(\cdot)$  is a one-hot vector to set all elements to zero except for the largest  $k$  ones. Since text features are unchanged during the denoising process, adaptive expression is unnecessary for this part.
Module Output. SAMI combines above-mentioned two refined features. The results can be formulated as  $\mathbf{Y}_{k,i} = \mathbf{Y}_{k,i}^{T} + \mathbf{Y}_{k,i}^{S}$ .
3.4 Zero-shot Motion Editing
In Section 3.2, we defined a fine-grained description format to precisely control the actions each body part should perform at each time interval. Based on this, we can interactively edit with users through a pretrained Large Language Model (LLM). We use this intermediate format as the current state corresponding to the action, and the user can input natural instructions. The LLM will modify this intermediate format according to the input content to get a new fine-grained description as shown in Figure 2. The motion generation model FineMoGen will then generate a new motion sequence from scratch based on the modified description, serving as the result of the editing. Such a zero-shot pipeline, using the fine-grained description and LLM as a medium, translates the user's natural instructions into content that the model can understand and accept, thereby achieving the effect of editing.
The teaser figure in Figure 1 provides an example of this procedure. It begins with the user providing a detailed description for a two-stage animation. Subsequently, an editing command, such as "I want the person to wave higher and for a longer duration," is issued to modify the motion description for both arms in the second stage. Consequently, the motion in the second stage differs from the previous version. Additionally, the motion in the first stage may also differ from the previous one due to the re-generation process.
In line with the motion diffusion model literature [21, 18], we randomly mask  $10\%$  of the text conditions to approximate  $p(\mathbf{x}_0)$ . Our training objective is to reduce the mean square error between the predicted initial sequence and the actual ground truth. During the training phase, we generally employ a diffusion process comprising 1000 steps, while a 50-step denoising process is applied at the inference stage.
Figure 4: Examples in HuMMan-MoGen dataset. A motion sequence will be divided into multiple stages. As for each stage, we provide detailed annotation for different body parts.
4.1 HuMMan-MoGen Datasets
In addition to commonly used dataset, we re-label the HuMMan [3] dataset, serving as a large-scale benchmark to evaluate Fine-grained Spatio-Temporal Motion Generation quantitatively. The HuMMan [3] dataset includes a total of 500 types of fitness movements, covering the majority of actions we perform in our daily lives across various body parts. We selected 2,968 videos from the 160 types of actions in the HuMMan dataset to be annotated in detail. For each type of action, we first define several standard action phases based on the specifications of that action. We provide detailed descriptions of the movements for each body part during each phase. Specifically, we annotated an overall description, and seven more detailed annotations to describe the head, torso, left arm, right arm, left leg, right leg, and trajectory of the pelvis joint. When annotating each video, we divide the original video into multiple standard action phases. For each phase, we apply the pre-defined fine-grained textual descriptions to accurately describe the spatial and temporal aspects of the captured motion data. Figure 4 show an example of our annotated motion sequence. The total number of annotations reached 102,336. This fine-grained spatio-temporal description provided by our data is not available in other datasets. The broad range of actions covered by this dataset makes it more challenging and valuable as a reference.
4.2 Standard Benchmarks and Evaluation Metrics
Standard Benchmarks. Apart from our collected HuMMan-MoGen dataset, we also evaluate our proposed pipeline on several leading benchmarks in text-driven motion generation tasks, the KIT-ML dataset[12], the HumanML3D dataset[7] and the Babel dataset[13]. We use these three benchmarks for more comparison.
Evaluation Metrics. We employ the same quantitative evaluation measures as used in the literature, including Frechet Inception Distance (FID), R Precision, Diversity, Multimodality, and Multi-Modal Distance. The details of these metrics are discussed in supplementary materials.
As for all four datasets, we choose almost the same hyperparameters. Regarding the motion encoder, we utilize a 4-layer transformer, with a latent dimension of  $7 * 64$ . Here, 7 corresponds to the number of body parts, and 64 represents the dimensionality for each part. For the text encoder, we build and apply a frozen text encoder found in the CLIP ViT-B/32, supplemented with two additional transformer encoder layers. In terms of the diffusion model, the variances, denoted as  $\beta_{t}$ , are predefined to linearly spread from 0.0001 to 0.02, with the total number of noising steps set as  $T = 1000$ . We use the Adam optimizer to train the model, initially setting the learning rate to 0.0002.
Table 1: Quantitative results on the HumanML3D test set.  $\uparrow$  (' $\downarrow$ ) indicates that the values are better if the metric is larger (smaller). We run all the evaluations 20 times and report the average metric and  $95\%$  confidence interval is. The best result are in bold and the second best result are underlined.
Methods	Top 1	R Precision↑ Top 2	Top 3	FID↓	MM Dist↓	Diversity↑	MultiModality↑
Real motions	0.511±.003	0.703±.003	0.797±.002	0.002±.000	2.974±.008	9.503±.065	-
Guo et al. [6]	0.457±.002	0.639±.003	0.740±.003	1.067±.002	3.340±.008	9.188±.002	2.090±.083
T2M-GPT [19]	0.491±.003	0.680±.003	0.775±.002	0.116±.004	3.118±.011	9.761±.081	1.856±.011
MDM [18]	-	-	0.611±.007	0.544±.044	5.566±.027	9.559±.086	2.799±.072
MotionDiffuse [20]	0.491±.001	0.681±.001	0.782±.001	0.630±.001	3.113±.001	9.410±.049	1.553±.042
ReMoDiffuse [21]	0.510±.005	0.698±.006	0.795±.004	0.103±.004	2.974±.016	9.018±.075	1.795±.043
Ours	0.504±.002	0.690±.002	0.784±.002	0.151±.008	2.998±.008	9.263±.094	2.696±.079
Table 2: Quantitative results on the KIT-ML test set.
Methods	Top 1	R Precision↑ Top 2	Top 3	FID↓	MM Dist↓	Diversity↑	MultiModality↑
Real motions	0.424±.005	0.649±.006	0.779±.006	0.031±.004	2.788±.012	11.08±.097	-
Guo et al. [6]	0.370±.005	0.569±.007	0.693±.007	2.770±.109	3.401±.008	10.91±.119	1.482±.065
T2M-GPT [19]	0.416±.006	0.627±.006	0.745±.006	0.514±.029	3.007±.023	10.921±.108	1.570±.039
MDM [18]	-	-	0.396±.004	0.497±.021	9.191±.022	10.847±.109	1.907±.214
MotionDiffuse [20]	0.417±.004	0.621±.004	0.739±.004	1.954±.062	2.958±.005	11.10±.143	0.730±.013
ReMoDiffuse [21]	0.427±.014	0.641±.004	0.765±.055	0.155±.006	2.814±.012	10.80±.105	1.239±.028
Ours	0.432±.006	0.649±.005	0.772±.006	0.178±.007	2.869±.014	10.85±.115	1.877±.093
This learning rate will gradually decay to 0.00002 in accordance with a cosine learning rate scheduler. Training is performed using one Tesla V100, with the batch size on a single GPU set at 128.
As for the HumanML3D and KIT-ML datasets, we follow Guo et al [6], where pose states mainly contain seven different parts:  $(r^{va}, r^{vx}, r^{vz}, r^{h}, \mathbf{j}^p, \mathbf{j}^v, \mathbf{j}^r)$ . Here Y-axis is perpendicular to the ground.  $r^{va}, r^{vx}, r^{vz} \in \mathbb{R}$  denotes the root joint's angular velocity along Y-axis, linear velocity along X-axis and Z-axis, respectively.  $r^h \in \mathbb{R}$  is the height of the root joint.  $\mathbf{j}^p, \mathbf{j}^v \in \mathbb{R}^{J \times 3}$  are the position and linear velocity of each joint, where  $J$  is the number of joints.  $\mathbf{j}^r \in \mathbb{R}^{J \times 6}$  is the 6D rotation of each joint. Specifically,  $J$  is 22 in HumanML3D and 21 in KIT-ML. Tuning to the BABEL dataset and our proposed HuMMan-MoGen, we follow TEACH [2] and converts SMPL parameters to a 6D rotation representation together with root translation.
4.4 Quantitative Results
We begin by assessing our proposed method on the standard task of text-driven motion generation. As depicted in Table 1 and Table 2, FineMoGen yields results that are competitive with those from existing methods. It's worth noting that while FineMoGen isn't specifically designed for this task, it still achieves satisfactory performance in comparison to other methods.
Table 3 displays the performance of temporal composition on both the BABEL dataset and the HuMMan-MoGen dataset. Our FineMoGen significantly surpasses existing methods in terms of accuracy. This underscores FineMoGen's ability to synthesize motion sequences that are not only more natural but also more aligned with the provided text conditions.
Please kindly refer to the supplementary material for qualitative comparison and more demos about both fine-grained motion generation and editing. From the demo video, we can observe the following: 1) Our method demonstrates a better understanding of the provided textual context compared to other approaches. Whether it's single-action generation or continuous action generation, our method generates motion sequences that are more consistent with the text. 2) Due to mathematically coherent temporal modeling, in continuous action generation, the transitions in our generated results are natural, with no apparent artifacts.
Table 3: Quantitative results of temporal composition on the BABEL test set and HuMMan-MoGen test set.
Methods	R Precision↑	FID↓	BABEL Diversity → MultiModality↑	R Precision↑	HuMMan-MoGen FID↓ Diversity → MultiModality↑
Real motions	0.62	0.004	8.51	3.57	0.49	0.007	7.28	2.15
TEACH [2]	0.46	1.12	8.28	7.14	0.38	1.59	7.04	6.37
PriorMDM [15]	0.43	1.04	8.14	7.39	0.31	1.43	6.89	6.45
Ours (zero-shot)	0.51	0.84	8.01	7.28	0.41	1.34	6.87	6.41
Ours (fully-supervised)	0.56	0.73	7.94	7.17	0.43	1.21	6.71	6.28
Table 4: Ablation study on HuMMan-MoGen test set. All methods use zero-shot setting, it means that they are not trained on the temporal composition data.
Methods	Spatial Dependence	Temporal Dependence	MoE	R Precision↑	FID↓	Diversity →	MultiModality↑
Real Motions	-	-	-	0.49	0.007	7.28	2.15
Baselines	-	-	-	0.24	3.79	7.10	6.95
✓	-	-	0.21	4.51	7.29	7.03
-	✓	-	0.34	2.48	7.05	6.72
-	-	✓	0.27	3.09	6.97	6.84
FineMoGen	✓	✓	✓	0.41	1.34	6.87	6.41
Table 4 presents the ablation study conducted on the HuMMan-MoGen dataset. For consistency, we consider only temporal composition in alignment with the results shown in Table 3. The outcomes suggest that both Temporal Dependence and MoE components contribute positively to the overall performance. Conversely, spatial dependence poses challenges for fitting the distribution. We concede that in a temporal composition setting, this module may not be necessary. We also provide ablation analysis on spatial combination in the supplementary material, which indicate consistent conclusions.
In this work, we introduce the task of Fine-grained Spatio-temporal Motion Generation, where more detailed textual descriptions allow users to control and edit the desired actions with greater flexibility. To address this task, we propose a new framework, FineMoGen, which decouples the influence of body parts and time on the feature refinement process in the attention mechanism. This results in superior performance in both fully-supervised and zero-shot scenarios compared to previous works. Given the lack of text-motion datasets specifically designed for fine-grained descriptions, we have compiled a dataset based on the HuMMan dataset. This dataset includes a wide range of actions and an abundance of descriptions segmented by time and body part. These data enable us to analyze and test the effects of different algorithms more effectively.
Limitation. While this study proposes a method for spatial decoupling, it is still heavily reliant on a specific fine-grained format, which to a certain extent limits its application scenarios. On the other hand, the efficacy of the motion editing component is strongly tied to the capability of the pre-trained Large Language Model (LLM). In practical use, it sometimes struggles to interpret user requirements accurately. Instruction tuning should be incorporated into the framework in the future, enabling the LLM to better understand instructions related to motion editing.
Boarder Impact. The ability to generate fine-grained human motion not only enhances the productivity of the animation industry, but also has potential benefits for helping medical research, improving training athletes and dancers, and providing more realistic interactive experiences in virtual reality. However, it may also be used for illegal activities such as generating fighting motions to create violent videos, or fraud and dissemination of false information.
This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
[17] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference on Computer Vision, pages 358–374. Springer, 2022.
[18] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2022.
[19] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[20] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.
[21] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. arXiv preprint arXiv:2304.01116, 2023.