SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Summary. Based on the above annotations, we bulid Motion-X, which has 81.1K clips with 15.6M SMPL-X poses and the corresponding pose and semantic text labels.
In this section, we first validate the accuracy of our motion annotation pipeline on the 2D keypoints and 3D SMPL-X datasets. Then, we build a text-driven whole-body motion generation benchmark on Motion-X. Finally, we show the effectiveness of Motion-X in whole-body human mesh recovery.
Input Image
(a) 2D Whole-body Keypoints Annotation
MediaPipe
(b) 3D Whole-body Mesh Annotation
Figure 7: Qualitative comparisons of (a) 2D keypoints annotation with widely used methods [65, 66] and (b) the 3D mesh annotation with the popular fitting method [25] with ours.
Method	hand ↑	face ↑	whole-body ↑
AP	AR	AP	AR	AP	AR
OpenPose [65]	38.6	43.3	76.5	84.0	44.2	52.3
HRNet [67]	50.3	60.2	73.7	80.9	58.2	67.1
ViTPose [18]	47.4	59.4	59.8	70.7	57.7	69.4
RTMPose-1 [68]	52.3	60.0	84.4	87.6	63.2	69.4
Ours	64.9	74.0	91.6	94.4	73.5↑16.3%	80.3↑15.7%
(a) Evaluation result on COCO-Wholebody [51] dataset.
Method	PA-MPJPE ↓	PA-MPVPE ↓	MPVPE ↓
Hand4Whole [17]	58.9	50.3	79.2
OSX [15]	55.6	48.7	70.8
PyMAF-X [69]	52.8	50.2	64.9
SMPLify-X [25]	62.6	52.9	-
Ours	33.5	31.8	44.7↓30.1%
Ours w/GT 3Dkpt	23.9	19.7	30.7 ↓52.7%
(b) Reconstruction error on EHF [25] dataset.
Table 3: Evaluation of motion annotation pipeline on (a) 2D keypoints and (b) 3D SMPL-X datasets.
5.1 Evaluation of the Motion Annotation Pipeline
2D Keypoints Annotation. We evaluate the proposed 2D keypoint annotation method on the COCO-WholeBody [51] dataset, and compare the evaluation result with four SOTA keypoints estimation methods [65, 67, 18, 68]. We use the same input image size of  $256 \times 192$  for all the methods to ensure a fair comparison. From Tab. 3(a), our annotation pipeline significantly surpasses existing methods by over  $15\%$  average precision. Additionally, we provide qualitative comparisons in Fig. 7(a), illustrating the robust and superior performance of our method, especially in challenging and occluded scenarios.
3D SMPL-X Annotation. We evaluate our learning-based fitting method on the EHF [25] dataset and compare it with four open-sourced human mesh recovery methods. Following previous works, we employ mean per-vertex error (MPVPE), Procrusters aligned mean per-vertex error (PA-MPVPE), and Procrusters aligned mean per-joint error (PA-MPJPE) as evaluation metrics (in mm). Results in Tab. 3(b) demonstrate the superiority of our progressive fitting method (over  $30\%$  error reduction). Specifically, PA-MPVPE is only  $19.71 \mathrm{~mm}$  when using ground-truth 3D keypoints as supervision. Fig. 7(b) shows the annotated mesh from front and side view, indicating reliable 3D SMPL-X annotations with reduced depth ambiguity. More results are presented in Appendix due to page limits.
Methods	Top 1	R Precision ↑ Top 2	Top 3	FID↓	MM Dist↓	Diversity→	MModality
Real	0.573±.005	0.765±.003	0.850±.005	0.001±.001	2.476±.002	13.174±.227	-
MDM [14]	0.290±.011	0.459±.010	0.577±.008	2.094±.230	6.221±.115	11.895±.354	2.624±.083
MLD [2]	0.440±.002	0.624±.004	0.733±.003	0.914±.056	3.407±.020	13.001±.245	2.558±.084
T2M-GPT [48]	0.502±.004	0.697±.005	0.791±.007	0.699±.012	3.192±.035	13.132±.127	2.510±.027
MotionDiffuse [9]	0.559±.001	0.748±.004	0.842±.003	0.457±.007	2.542±.018	13.576±.161	1.620±.152
Table 4: Benchmark of text-driven motion generation on Motion-X test set.  $\rightarrow$  means results are better if the metric is closer to the real motions and  $\pm$  indicates the  $95\%$  confidence interval. The best results are in bold.
Train Set	HumanML3D (Test)	Motion-X (Test)
R-Precision↑	FID↓	Diversity→	MModality	R-Precision↑	FID↓	Diversity→	MModality
Real (GT)	0.749±.002	0.002±.001	9.837±.084	-	0.850±.005	0.001±.001	13.174±.227	-
HumanML3D	0.657±.004	1.579±.050	10.098±.052	2.701±.143	0.570±.003	12.309±.127	9.529±.165	2.960±.066
Motion-X	0.695±.005	0.999±.042	9.871±.099	2.827±.138	0.733±.003	0.914±.056	13.001±.245	2.558±.084
Table 5: Cross-dataset comparisons of HumanML3D and Motion-X. We train MLD on the training set of HumanML3D and Motion-X, respectively, then evaluate it on their test sets.
5.2 Impact on Text-driven Whole-body Motion Generation
Experiment Setup. We randomly split Motion-X into the train  $(80\%)$ , val  $(5\%)$ , and test  $(15\%)$  sets. SMPL-X is adopted as the motion representation for expressive motion generation.
Evaluation metrics. We adopt the same evaluation metrics as [4], including Frechet Inception Distance (FID), Multimodality, Diversity, R-Precision, and Multimodal Distance. Due to the page limit, we leave more details about experimental setups and evaluation metrics in the appendix.
Benchmarking Motion-X. We train and evaluate four diffusion-based motion generation methods, including MDM [14], MLD [2], MotionDiffuse [9] and T2M-GPT [48] on our dataset. Since previous datasets only have sequence-level motion descriptions, we keep similar settings for minimal model adaptation and take the semantic label as text input. The evaluation is conducted with 20 runs (except for Multimodality with 5 runs) under a  $95\%$  confidence interval. From Tab. 4, MotionDiffuse demonstrates a superior performance across most metrics. However, it scores the lowest in Multimodality, indicating that it generates less varied motion. Notably, T2M-GPT achieves comparable performance on our dataset while maintaining high diversity, indicating our large-scale dataset's promising prospects to enhance the GPT-based method's efficacy. MDM gets the highest Multimodality score with the lowest precision, indicating the generation of noisy and jittery motions. The highest Top-1 precision is  $55.9\%$ , showing the challenges of Motion-X. MLD adopts the latent space design, making it fast while maintaining competent results. Therefore, we use MLD to conduct the following experiments to compare Motion-X with the existing largest motion dataset HumanML3D and ablation studies.
Comparison with HumanML3D. To validate the richness, expressiveness, and effectiveness of our dataset, we conduct a comparative analysis between Motion-X and HumanML3D, which is the largest existing dataset with text-motion labels. We replace the original vector-format poses of HumanML3D with the corresponding SMPL-X parameters from AMASS [11], and randomly extract facial expressions from BAUM [31] to fill in the face parameters. We train MLD separately on the training sets of Motion-X and HumanML3D, then evaluate both models on the two test sets. The results in Tab. 5 reveal some valuable insights. Firstly, Motion-X exhibits greater diversity (13.174) than HumanML3D (9.837), as evidenced by the real (GT) row. This indicates a wider range of motion types captured by Motion-X. Secondly, the model pretrained on Motion-X and then finetuned on HumanML3D subset performs well on
Figure 8: Visual comparisons of motions generated by MLD [2] trained on HumanML3D (in purple) or Motion-X (in blue). Please zoom in for a detailed comparison. The model trained with Motion-X can generate more accurate and semantic-corresponded motions.
Semantic Label	Pose Discription	FID↓
face text	body text	hand text
✓	0.914±.056
✓	✓	0.784±.032
✓	✓	✓	0.671±.016
✓	✓	✓	✓	0.565±.036
Table 6: Ablation study of text inputs.
Method	EHF [25] ↓	AGORA [70] ↓
all	hand	face	all	hand	face
w/o Motion-X	79.2	43.2	25.0	185.6	73.7	82.0
w/ Motion-X	73.0	41.0	22.6	184.1	73.3	81.4
Table 7: Mesh recovery errors of Hand4Whole [17] using different training datasets. MPVPE (mm) is reported.
the HumanML3D test set, even better than the intra-data training. These superior performances stem from the fact that Motion-X encompasses diverse motion types from massive outdoor and indoor scenes. For a more intuitive comparison, we provide the visual results of the generated motion in Fig. 8, where we can clearly see that the model trained on Motion-X excels at synthesizing semantically corresponding motions given text inputs. These results prove the significant advantages of Motion-X in enhancing expressive, diverse, and natural motion generation.
Ablation study of text labels. In addition to sequence-level semantic labels, the text labels in Motion-X also include frame-level pose descriptions, which is an important characteristic of our dataset. To assess the effectiveness of pose description, we conducted an ablation study on the text labels. The baseline model solely utilizes the semantic label as the text input. Since there is no method to use these labels, we simply sample a single sentence from the pose descriptions randomly, concatenate it with the semantic label, and feed the combined input into the CLIP text encoder. Interestingly, from Tab. 6, adding additional face and body pose texts brings consistent improvements, and combining whole-body pose descriptions results in a noteworthy  $38\%$  reduction in FID. These results validate that the proposed whole-body pose description contributes to generating more accurate and realistic human motions. More effective methods to utilize these labels can be explored in the future.
5.3 Impact on Whole-body Human Mesh Recovery
As discovered in this benchmark [21], the performance of mesh recovery methods can be significantly improved by utilizing high-quality pseudo-SMPL labels. Motion-X provides a large volume of RGB images and well-annotated SMPL-X labels. To verify its usefulness in the 3D whole-body mesh recovery task, we take Hand4Whole [17] as an example and evaluate MPVPE on the widely-used AGORA val [71] and EHF [25] datasets. For the baseline model, we train it on the commonly used COCO [51], Human3.6M [72], and MPII [73] datasets. We then train another model by incorporating an additional  $10\%$  of the single-view data sampled from Motion-X while keeping the other setting the same. As shown in Tab. 7, the model trained with Motion-X shows a significant decrease of  $7.8\%$  in MPVPE on EHF and AGORA compared to the baseline model. The gains come from the increase in diverse appearances and poses in Motion-X, indicating the effectiveness and accuracy of the motion annotations in Motion-X and its ability to benefit the 3D reconstruction task.
In this paper, we present Motion-X, a comprehensive and large-scale 3D expressive whole-body human motion dataset. It addresses the limitations of existing mocap datasets, which primarily focus on indoor body-only motions with limited action types. The dataset consists of 144.2 hours of whole-body motions and corresponding text labels. To build the dataset, we develop a systematic annotation pipeline to annotate 81.1K 3D whole-body motions, sequence-level motion semantic labels, and 15.6M frame-level whole-body pose descriptions. Comprehensive experiments demonstrate the accuracy of the motion annotation pipeline and the significant benefit of Motion-X in enhancing expressive, diverse, and natural motion generation, as well as 3D whole-body human mesh recovery.
Limitation and future work. There are two main limitations of our work. Firstly, the motion quality of our markless motion annotation pipeline is inevitably inferior to the multi-view marker-based motion capture system. Secondly, during our experiment, we found out that existing evaluation metrics are not always consistent with visual results. Thus, there is a need for further development of the motion generation models and evaluation metrics. As a large-scale dataset with multiple modalities, e.g., motion, text, video, and audio, Motion-X holds great potential for advancing downstream tasks, such as motion prior learning, understanding, and multi-modality pre-training. Besides, our large-scale dataset and scalable annotation pipeline open up possibilities for combining this task with a large language model (LLM) to achieve an exciting motion generation result in the future. With Motion-X, we hope to benefit and facilitate further research in relevant fields.
This research is partially funded through National Key Research and Development Program of China (Project No. 2022YFB36066), in part by the Shenzhen Science and Technology Project under Grant (JCYJ20220818101001004, JSGG20210802153150005), National Natural Science Foundation of China (62271414), the Young Scientists Fund of the National Natural Science Foundation of China under grant No.62106154, the Natural Science Foundation of Guangdong Province, China (General Program) under grant No.2022A1515011524, and Shenzhen Science and Technology Program JCYJ20220818103001002 and Shenzhen Science and Technology Program ZDSYS20211021111415025.
Thanks to everyone who participated in shooting the IDEA400 dataset and discussions. Thanks to Linghao Chen for helping to design the icons and website pages.
[1] C. Ahuja and L.-P. Morency, "Language2pose: Natural language grounded pose forecasting," in 3DV, 2019.
[2] X. Chen, B. Jiang, W. Liu, Z. Huang, B. Fu, T. Chen, J. Yu, and G. Yu, "Executing your commands via motion diffusion in latent space," in CVPR, 2023.
[3] G. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer, and G. Rogez, “Posescript: 3d human poses from natural language,” in ECCV, 2022.
[4] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, "Generating diverse and natural 3d human motions from text," in CVPR, 2022.
[5] M. Petrovich, M. J. Black, and G. Varol, “Temos: Generating diverse human motions from textual descriptions,” in ECCV, 2022.
[6] M. Plappert, C. Mandery, and T. Asfour, "The kit motion-language dataset," Big data, 2016.
[7] M. Plappert, C. Mandery, and T.Asfour, "Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks," Robotics and Autonomous Systems, 2018.
[8] A. R. Punnakkal, A. Chandrasekaran, N. Athanasiou, A. Quiros-Ramirez, and M. J. Black, “Babel: bodies, action and behavior with english labels,” in CVPR, 2021.
[9] M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu, "Motiondiffuse: Text-driven human motion generation with diffusion model," arXiv preprint arXiv:2208.15001, 2022.
[10] M. Zhao, M. Liu, B. Ren, S. Dai, and N. Sebe, "Modiff: Action-conditioned 3d motion generation with denoising diffusion probabilistic models," arXiv preprint arXiv:2301.03949, 2023.
[11] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, "Amass: Archive of motion capture as surface shapes," in ICCV, 2019.
[12] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020.
[13] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in ICLR, 2020.
[14] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano, "Human motion diffusion model," in ICLR, 2023.
[15] J. Lin, A. Zeng, H. Wang, L. Zhang, and Y. Li, "One-stage 3d whole-body mesh recovery with component aware transformer," in CVPR, 2023.
[16] R. Li, S. Yang, D. A. Ross, and A. Kanazawa, "Ai choreographer: Music conditioned 3d dance generation with aist++, in ICCV, 2021.
[59] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, "Expressive body capture: 3d hands, face, and body from a single image," in CVPR, 2019.
[60] S. Shimada, V. Golyanik, W. Xu, and C. Theobalt, "Physcap: Physically plausible monocular 3d motion capture in real time," ACM ToG, 2020.
[61] U. Iqbal, K. Xie, Y. Guo, J. Kautz, and P. Molchanov, “Kama: 3d keypoint aware body mesh articulation,” in 3DV, 2021.
[62] H. Zhang, X. Li, and L. Bing, "Video-llama: An instruction-finetuned visual language model for video understanding," 2023. [Online]. Available: https://github.com/DAMO-NLP-SG/Video-LLaMA
[63] A. Dutta and A. Zisserman, “The VIA annotation software for images, audio and video,” in ACM MM, 2019.
[64] A. Mollahosseini, B. Hasani, and M. H. Mahoor, "Affectnet: A database for facial expression, valence, and arousal computing in the wild," IEEE Transactions on Affective Computing, 2017.
[65] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, "Realtime multi-person 2d pose estimation using part affinity fields," in CVPR, 2017.
[66] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, "Mediapipe hands: On-device real-time hand tracking," arXiv preprint arXiv:2006.10214, 2020.
[67] K. Sun, B. Xiao, D. Liu, and J. Wang, "Deep high-resolution representation learning for human pose estimation," in CVPR, 2019.
[68] J. Zhang, D. Zhang, X. Xu, F. Jia, Y. Liu, X. Liu, J. Ren, and Y. Zhang, "Mobipose: Real-time multi-person pose estimation on mobile devices," in SenSys, 2020.
[69] H. Zhang, Y. Tian, Y. Zhang, M. Li, L. An, Z. Sun, and Y. Liu, "Pymaf-x: Towards well-aligned full-body model regression from monocular images," in TPAMI, 2023.
[70] C. Guo, X. Zuo, S. Wang, X. Liu, S. Zou, M. Gong, and L. Cheng, “Action2video: Generating videos of human 3d actions,” IJCV, 2022.
[71] P. Patel, C.-H. P. Huang, J. Tesch, D. T. Hoffmann, S. Tripathi, and M. J. Black, “AGORA: Avatars in geography optimized for regression analysis,” in CVPR, 2021.
[72] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, "Human3.6M: Large scale datasets and predictive methods for 3d human sensing in natural environments," in TPAMI, 2014.
[73] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2d human pose estimation: New benchmark and state of the art analysis,” in CVPR, 2014.