Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: 3D Human Motion;  Motion Generation; Mesh Recovery; Motion Capture
TL;DR: We propose a large-scale 3D expressive whole-body human motion dataset

Abstract: In this paper, we present Motion-X, a large-scale 3D expressive whole-body motion dataset. Existing motion datasets predominantly contain body-only poses, lacking facial expressions, hand gestures, and fine-grained pose descriptions. Moreover, they are primarily collected from limited laboratory scenes with textual descriptions manually labeled, which greatly limits their scalability. To overcome these limitations, we develop a whole-body motion and text annotation pipeline, which can automatically annotate motion from either single- or multi-view videos and provide comprehensive semantic labels for each video and fine-grained whole-body pose descriptions for each frame. This pipeline is of high precision, cost-effective, and scalable for further research. Based on it, we construct Motion-X, which comprises 15.6M precise 3D whole-body pose annotations (i.e., SMPL-X) covering 81.1K motion sequences from massive scenes. Besides, Motion-X provides 15.6M frame-level whole-body pose descriptions and 81.1K sequence-level semantic labels. Comprehensive experiments demonstrate the accuracy of the annotation pipeline and the significant benefit of Motion-X in enhancing expressive, diverse, and natural motion generation, as well as 3D whole-body human mesh recovery.

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset
Jing Lin 1i Ailing Zengl*t, Shunlin 1.31,
Yuanhao Cai Ruimao Zhang", Haoqian Wang", Lei Zhang!
'International Digital Economy Academy (IDEA)
Tsinghua University, 'The Chinese University of Hong Kong, Shenzhen
https://motion menotion--ddataeee github.io
Abstract
In this paper, we present Motion-X, a large-scale 3D expressive whole-body motion dataset. Existing motion datasets predominantly contain body-only poses, lacking facial expressions, hand gestures, and fine-grained pose descriptions. Moreover, they are primarily collected from limited laboratory scenes with textual descrip- tions manually labeled, which greatly limits their scalability. To overcome these limitations, we develop whole-body motion and text annotation pipeline, which can automatically annotate motion from either single- or multi-view videos and provide comprehensive semantic labels for each video and fine-grained whole-body pose descriptions for each frame. This pipeline is of high precision, cost-effective, and scalable for further research. Based on it, we construct Motion-X, which com- prises 15.6M precise 3D whole-body pose annotations (i.e., SMPL-X) covering 81.1K motion sequences from massive scenes. Besides, Motion-X provides 15.6M frame-level whole-body pose descriptions and 81. sequence- semantic labels. Comprehensive experiments demonstrate the accuracy of the annotation pipeline and the significant benefit of Motion-X in enhancing expressive, diverse, and natural motion generation, as well as 3D whole-body human mesh recovery.
Introduction
Human motion generation aims automatically synthesize natural human movements. It has wide applications in robotics, animation, games, and generative creation. Given a text description or audio command, motion generation can be controllable to obtain the desired human motion sequence. Text-conditioned motion generation has garnered increasing attention in recent years since it behaves in more natural interactive way [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].
Although existing text-motion datasets [4, 11, 6, 8] have greatly facilitated the development of motion generation [2. 12, 13, 14, 9], their scale, diversity, and expressive capability remain unsatisfactory. Imagine generating "a man is playing the piano happily", as depicted in Fig. l(a), the motion from existing dataset [4] only includes the body movements, without finger movements or facial expressions. The missing hand gestures and facial expressions severely hinder the high level of expressiveness and realism of the motion. Additionally, certain specialized motions, such as high- level skiing, aerial work, and riding are challenging to be captured in indoor scenes. To sum up, existing datasets suffer from four main limitations: 1) body-only motions without facial expressions and hand poses; 2) insufficient diversity and quantity, only covering indoor scenes; 3) lacking diverse and long-term motion sequences; and 4) manual text labels that are unscalable, unprofessional and labor-intensive. These limitations hinder existing generation methods synthesize expressive whole-bool body motion with diverse action types. Therefore, how to collect large-scall whole-body motion and text annotations from multi-scenarios are critical in addressing the data scarcity issue.
"Equal Contribution, Work done during an internship at IDEA Cooresponding author
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Someone standing playing the piana.
person sitting and playing piano happily.
(8) HumanML3D
(b) Motion-X (Ours)
Figure 1: Different from (a) previous motion dataset [4, 8], (b) our dataset captures body, facial expressions, and hand gestures. We highlight the comparisons of facial expressions and hand gestures.
Compared to indoor marker-based mocap systems, markerless vision-based motion capture meth- ods [15. 16, 17, 18, 19, 20] become promising to capture large-scale motions from massive videos. Mcanwhile, human motion be regarded automatically translated into pose seripts using rule-based techniques [3]. More importantly, although markerless capture (e.g., pseudo labels) is not as precise as marker- based methods, collecting massive and informative motions, especially local motions, could still be beneficial [21, 15, 22, 23, 24]. Be- sides, text-driven motion generation task requires semantically corresponding motion labels instead of vertex-correspondiig mesh labels, and thus have a higher tolerance of motion capture error. Bearing these considerations in mind, we design a scalable and systematic pipeline for motion and text annota- tion in both multi-view and single-view videos. Firstly, we gather and filter massive video recordings from variety of scenes with challenging, high-quality, multi-style motions and sequence-lever semantic labels. Subsequently, we estimate and optimize the parameters of the SMPL-X model [25] for the whole-body motion annotation. Due the depth ambiguity and various challenges different scenes, existing monocular estimation models typically fail to yield satisfactory results. To address this issue, we systematically design high- gesh-perffrmncce framework incorporating several innovative techniques, including a hierarchical approach for whole-body keypoint estimation, a score- adaptive temporal smoothing and optimization scheme, and a searning-bsser 3D human model fitting process. By integrating these techniques, we can accurately and efficiently capture the ultimate 3D motions. Finally, we design an automatic algorithm to caption frame-level descriptions whole-body poses. We obtain the body and hand scripts by calculating spatial relations among body parts and hand fingers based on the SMPL-XX parameters and extract the facial expressions with an emotion classifier. We then aggregate the low-level pose information and translate it into textual pose descriptions.
sequence of kinematic structures, which can be
score-guided guided
Based on the pipeline, we collect a large-scale whole-body expressive motion dataset named Motion-X, which includes 15.6M frames and 81.1K sequences with precise 3D whole-body motion annotations, pose descriptions, and semantic labels. To compile this dataset, we collect massive videos from the Internet, with a particular focus on game and animation motions, professional performance, and diverse outdoor actions. Additionally, we incorporated data from eight existing action datasets [26, 27, 28, 11, 29, 30, 32]. Using Motion-X, we build a benchmark for evaluating several state-of- the-art (SOTA) motion generation methods. Comprehensive experiments demonstrate the benefits of Motion-X X for diverse, expressive, and realistic motion generation (shown in Fig. I (b)). Furthermore, we validate the versatility and quality of Motion-X on the whole-body mesh recovery task.
Our contributions can be summarized as follows:
We propose a large-scale expressive motion dataset with precise 3D whole-body motions and corresponding sequence- leveel and frame-level text descriptions.
We elaborately design automatic motion and text annotation pipeline, enabling efficient capture of high-quality human text-motion data at scale.
Comprehensive experiments demonstrate the accuracy of the motion annotation pipeline and the benefits of Motion-X in 3D whole-body motion generation and mesh recovery tasks.
2 Preliminary and Related Work
In this section, we focus on introducing existing datasets for human motion generation. For more details about the motion generation methods, please refer to the appendix.

---

Motion Annotaton Whole-body?
Text Annotation
Scane
Clip
Source
Motion
Pose
Whole-body?
Indocr
KIT-M1216 [6]
3911
11.2
Marker-based MoCap
6278
x
AMASS [11]
11265
40.0
X
Marker-based McCapp
D
0
x
x
x
BARFI:21 [S]
13220
43.5
Marker-husod MoCapp
91408
0
Posescript
Marker-based MoCap
0
120k
HomanML3D [4]
14616
Markor-based MoCap
44970
Motion X (Ours)
81084
144.2
Psendo GT & MoCap
81084
15.0M
/
Table I:
Comparisons between Motion-X and existing fext-motion datascts. The first column shows the name and public year of datasets. Motion-X X provides both indoor and outdoor whole-body motion and text annotations.
Benchmarks annotated with sequential human motion and text mainly collected for three tasks:
action recognition [33,, 27, 34, 28, 35, 36), human object interaction [37, 38, 39, 29, 32, 40), and motion generation [4, 41, 11, 6, 8, 24]. Specifically, KIT Motion-Language Dataset [6] is the first public dataset with human motion and language descriptions, enabling multi-modality motion generation [1, 5]. Although several indoor human motion capture (mocap) datasets have been developed [42, 43, 44, 45], they are scattered. AMASS [11] is noteworthy as it collects and unifies 15 different optical marker-based mocap datasets to build large-scale motion dataset through common framework and parameterization via SMPL [46]. This great milestone benefits motion modeling and its downstream tasks. Additionally, BABEL [8] and HumanML3D [4] contribute the language labels through crowdsourced data collection. BABEL proposes either sequence labels or sub- sequence labels for sequential motion, while HumanML33 collects three text descriptions for each motion clip from different workers. Thanks to these text-motion datasets, various motion generation methods have rapidly developed and shown advantages in diverse, realistic, and fine- gre-graieed motion generation [2. 14, 47, 48, 9, 10].
However, existing text-motion datasets have several limitations, including the absence of facial expressions and hand gestures, insufficient data quantity, limited diversity of motions and scenes, gaasee-rrainedeeee and ambiguous descriptions, and the lack of long sequence motions. To bridge these gaps, we develop a large-scale whole-body expressive motion dataset with comprehensive sequence- and frame-level text labels. We aim address these limitations and open up new possibilities for future research. We provide quantitative comparisons of Motion-X X and existing datasets in Tab. 1.
3 Motion-X Dataset
3.1 Overview
As shown in Tab. 2, we collect Motion-X from eight datasets and online videos and provide the following motion and text annotations: 15.6M 3D whole-body SMPL-X annotation, 81.1 sequence- level semantic descriptions (e.g., walking with waving hand and laughing), and frame-level whole- body pose descriptions. Notably, original sub-datasets lack either whole-body motion or text labels and we unify them with our annotation pipeline. All annotations are manually checked to guarantee quality. In Fig. 2, we show the averaged temporal standard deviation of body, hand, and face keypoints of each sub-dataset, highlighting the diversity of hand movements and facial expressions, which fills in the gaps of previous body-only motion data.
Faco (H20)
Hand (18)
Body
Data
Clip
Frame
GT Mation
P-GT Motion
Text
Per Joins Standard Deviation
400
AMASS
26.3K
5.4M
B
H,
HAA500 (27)
5.2K
0.3M
B. H. F
300
AIST [30]
1.KK
0.3M
B, H. F
HuMMan [26]
0.7K
0.IM
B, B,H.F
200
GRAB [29]
1.3K
0.4M
B.H
F
EgoBody [32)
[31]
LOK
0.4M
0.2M
B.H
F
100
IDEA
12.5K
2.6M
H.
Online Videos
32.5K
6.0M
H.
AMASS
BAUM
AAT
Uniige
Intranunt
Fittees
Denor
Animation
Gaste
Porform
Kouafa
Motion-X X
81.1K
15.6M
B.
B.HLF
S.P
Table 2: Statistics of sub-datasets. B. H, F are body, hand, and face. S and P are semantic and pose texts. P-GT is pscudo ground truth. * denotes videos are collected by us.
Figure 2: Diversity statistics of the face, hand, and body motions in each subdatasets.
3

---

"Please list all instruments"
"guitar, piono, violin, erhu
OCR, Title Extractor, Video Caption
Text Prompt
Motion Annotation
Motion Label
Pose Caption
Text Label
LLM
Search
Video
Figure 3: Illustration of the overall data collection and annotation pipeline.
(8 Laboor Tabberrnoiin modiias (a) Fare oxpiesion
diagust
drigry
surprise
intenest ng
hoppy
Arn unostling
Play clmmnat
May Seahu
Ply Frngh
Homp
Suhar
Play Suph
bnush har
Wear
Do Outieer modimns
6PR
Bmx ridin
Hlyingg drum
handattand
Aeril work
Balance beam
Tel
Afrad of heghh
skate forward
Bull
snowmeen
(4) Menion seQuences
Figure 4: Overview of Motion-X. It includes: (a) diverse facial expressions extracted from BAUM [31]. (b) indoor motion with expressive face and hand motions, (c) outdoor motion with diverse and challenging poses, and (d) several motion sequences. Purple SMPL-X is the observed frame, and the others are neighboring poses.
3.2 Data Collection
As illustrated in Fig. 3, the overall data collection pipeline involves six key steps: 1) designing and sourcing motion text prompts via large language model (LLM) [49], 2) collecting videos, 3) preprocessing candidate videos through human detection and video transition detection, 4) capturing whole-body motion (Sec. 4.1), 5) captioning sequence-level semantic label and frame-level whole- body pose description(Sec (Sec. 4.2), and 6) performing the manual inspection.
We gather 37K motion sequences from existing datasets using our proposed unified annotation framework, including the multi-view datasets (AIST [30]). suman--cene-interaction datasets (EgoB- ody [32] and GRAB [29]), single-view action recognition datasets (HAA500 [27]. HuMMan [26]), and body-only motion capture dataset (AMASS [Ii]). For these datasets, steps I and are skipped. Notably, only EgoBody and GRAB datasets provide SMPL-X labels with body and hand pose, thus we annotate the SMPL-X label for the other motions. For AMASS, which contains the body and roughly static hand motions, we skip step 4 and fill in the facial expression with data augmentation mechanism. The facial expressions are collected from a facial datasets BAUM [31] via face capture and animation model EMOCA [50]. To enrich the expressive whole-body motions, we record an dataset IDEA400, which provides 13K motion sequences covering 400 diverse actions. Details about the processing of each sub-dataset and IDEA400 are available in the appendix.
To improve the appearance and motion diversity, we collect 32.5K monocular videos from online sources, covering various real-life scenes as depicted in Fig. 4. Since human motions and actions are contexl-dependent and vary with the scenario, we design action categories motion prompts based on the scenario and function of the action via LLM. To ensure comprehensive coverage of human actions, our dataset includes both general and domain- specifcc scenes. The general scenes encompass daily actions (e.g., brushing hair, wearing glasses, and applying creams), sports activities (e.g., high knee, kick legs, push-ups), various musical instrument playing, and outdoor scenes (e.g., BMX riding, CPR, building snowman). The inclusion of general scenes helps bridge the gap between existing data and real-life scenarios. addition, we incorporate domain-specific scenes that require high professional skills, such as dance, Kung Fu, Tai Chi, performing arts, Olympic events, entertainment

---

20 XPT
20 KPT
Esimation
SNPLX
Optimiatiinn
Giecal Transiation Optimzationn
SMPL-X Motion Fitting
Estimation
KPT
KPT
Bog
ViT ViT-llooy
Update
Crand Boy
Vit-Hand
Camera
InEN
KPT
Estimation
Musimmott
Camora
VIT WholeBody
The automatic pipeline for the whole-body motion capture from massive videos, including and 3D whole-body keypoints estimation, local pose optimization, and global tramslation optimization. This pipeline supports both single- and multi-view inputs. Dashed lines represent the handling of multi-view data exclusively.
Figure 5:
shows, games, and animation motions. Based on the prompts describing the above scenes, we run the collection pipeline to gather the data from online sources for our dataset.
4 Automatic Annotation Pipeline
4.1 Universal Whole-body Motion Annotation
Overview. To efficiently capture a large volume of potential motions from massive videos, we propose an annotation pipeline for high-quality whole-body motion capture with three novel techniques: (i) hierarchical whole-body keypoint estimation; (ii) score-guided adaptive temporal smoothing for jitter motion refinement; and (iii) learning-based 3D human model fitting for accurate motion capture.
2D Keypoint Estimation. 2D Whole-body keypoint estimation poses a challenge due to the small size of the hands and face regions. Although recent approaches have utilized separate networks to decode features different body parts [51, 52], they often struggle with hand aund-missing detection and prone errors due to occlusion or interaction. To overcome these limitations, a novel hierarchical keypoint annotation method, depicted in the blue box of Fig. 5. We train a VIT- WiT-Wholeoody based ViT-based model [18] the
customize
dataset [51] to estimate initial whole-body keypoints 222 E IR 133x2 with confidence scores. Leveraging the ViT model's ability to model semantic relations between full-body parts, enhance hand and face detection robustness even under severe occlusion. Subsequently, we obtain the hand and face bounding boxes based on the keypoints, and refine the boxes using the BodyHands detector [53] through an IoU matching operation. Finally, we feed the cropped body, hand, and face regions into three separately pre-trained ViT networks to estimate body, hand and face keypoints, which are used update K2D
Score-guided Adaptive Smoothing. To address the jitter resulting from per-frame pose estimation in challenging scenarios such as heavy occlusion, truncation, and motion blur, while preserving motion details, we introduce a novel score-guided adaptive smoothing technique into the traditional Savitzky-Golay filter [54]. The filter is applied sequence of 2D keypoints of a motion:
R.DD Σ
C3
K
(I)
where K, is the original keypoints of the inh frame, K. is the smoothed keypoints, w corresponds to half-width of filter window size, and ej are the filter coefficients. Different from existing smoothing methods with a fixed window size [55, 56, 54], we leverage the confidence scores of the keypoints to adaptively adjust the window size to balance between smoothness and motion details. Using a larger window size for keypoints with lower confidence scores mitigate the impact of outliers.
3D Keypoint Annotation. Precise 3D keypoint can boost the estimation of SMPL-X. We utilize novel information from large-scale pre-trained models. Accordingly, for single-view videos, we adopt a pretrained model [57], which trained on massive 3D datasets, to estimate precise 3D keypoints. For multi-view videos, utilize bundle adjustment calibrate and refine the camera parameters, and
5

---

then triangulate the 3D keypoints K based on the multi-view 2D keypoints. To enhance stability, we adopt temporal smoothing and enforce 3D bone length constraints during triangulation.
Local Pose Optimization. After obtaining the keypoints, we perform local pose optimization to register each frame's whole-body model SMPL--X [25]. Traditional optimization-basel methods [58, 25] are often time-consuming and may yield unsatisfactory results as they ignore image clues and motion prior. We propose a progressive learning-basse human mesh fitting method to address these limitations. Initially, we predict the SMPL-X parameter e with the SOTA whole-body mesh recovery method OSX [15] and face reconstruction model EMOCA [50]. And then, through iterative optimization of the network parameters, we fit the human model parameters Ô to the target 2D and 3D joint positions by minimizing the following functions, achieving an improved alignment accuracy:
Jjoint
K 3D
K-D
||
||K2
K2D + ||6 +|1
Here, KDD represents the predicted 3D joint positions obtained by applying a linear regressor to 3D mesh generated by the SMPL--X model. K2D derived by performing a perspective projection of the 3D keypoints. The last term of the loss function provides explicit supervision based on the initial parameter, serving as a 3D motion prior. To alleviate potential biophysical artifacts, such as interpenetration and foot skating, we incorporate set of physical optimization constraints:
L Join + Ammooth smooth + Apen + >phy
Here, À are weighting factors of each loss function and Tmmonth is first-order smoothness term:
Lymooth E1622 G + Σ
K 3D
KP--1ll.
(4)
where Ô, and K2D0 represent the SMPL--X parameters and joints of the i-th frame, respectively. To alleviate mesh interpenetration, we utilize a collision penalizer [59], denoted as Lpon Additionally, we incorporate the physical loss Lphy based on PhysCap [60] to prevent implausible poses.
Global Motion Optimization. To improve the consistency and realism of the estimated global trajectory, we perform global motion optimization based on GLAMR [19] to simultaneously refine the global motions and camera poses to align with video evidence, such as 2D keypoints:
Lg L2D + Araj Lraj + Acam Leam + Areg Ireg.
(5)
where LeD represents the 2D keypoint distance loss, Lunj quantifies the difference between the optimized global trajectory and the trajectory estimated by Kama [61]. Legg enforces regularization on the global trajectory, and Leam applies a smoothness constraint on the camera parameters.
Human Verification. To ensure quality, we manually checked the annotation by removing the motions that do not align with the video evidence or exhibit obvious biophysical artifacts.
4.2 Obtaining Whole-body Motion Descriptions
Sequence motion labels. The videos in Motion-X were collected from online sources and existing datasets. For action-related datasets [26, 27, 16, 28, 11, 29], we use the action labels as one of the sequence semantic labels. Meanwhile, we input the videos into Video-LLaMA [62] and filter the human action descriptions as supplemental texts. When videos contain semantic subtitles, EasyOCR automatically extracts semantic information. For online videos, we also use the search queries generated from LLM [49] as semantic labels. Videos without available semantic information, such as EgoBody [32], are manually labeled using the VGG Image Annotator (VIA) [63]. For the face database BAUM [31], we use the facial expression labels provided by the original creator.
Whole body pose labels:
Face: Concentrating
PoseScript
Body
Hand; Both hands make fist; All
Description
Whole-Body Body Desccripiion
completely bent.
Body; This person leans to the right;
Emotion Recognition
Emotion
Both knees are bent bir, and the
almost kicked straight The left fnor wide apart from righr with the feet extended back
Hand
HandScript
Description
Sequence Shaolin Kung
(b) Example of the Annotation Result Figure 6: Illustration of (a) annotation of the whole- whole-body pose description, and (b) an example of the text labels.
(a) Whole-body Pose Caption

---

Whole-body pose descriptions. The generation of fine- grained pose descriptions for each pose involves three distinct parts: face, body, and hand, as shown Fig. 6(a). Facial expression labeling uses the emotion recognition model EMOCA [50] pretrained on AffectNet [64] to classify the emotion. Body-specific descriptions utilizes the captioning process from PoseScript [3], which generates synthetic low-level descriptions in natural language based on given 3D keypoints. The unit of this information is called posecodes, such as 'the knees are completely bent'. A set of generic rules based on fine-grained categorical relations of the different body parts are used to select and aggregate the low-level pose information. The aggregated posecodes are then used to produce textual descriptions in natural language using linguistic aggregation principles. Hand gesture descriptions extends the pre-defined posecodes from body parts to fine- fine-grained hand gestures. We define six elementary finger poses via finger curvature degrees and distances between fingers to generate descriptions, such as 'bent' and 'speead apart'. We calculate the angle of each finger joint based on the 3D hand keypoints and determine the corresponding margins. For instance, if the angle between V (KWiist (ingetip) and V ingertip:: K Kingerooa) falls between 120 and 160 degrees, the finger posture is labeled as 'slightly bent'. We show an example of the annotated text labels in Fig. 6(b).
Summary. Based on the above annotations, we bulid Motion-X, X,, which has 81.1K clips with 15.6M SMPI.-X poses and the corresponding pose and semantic text labels.
5 Experiment
In this section, we first validate the accuracy of our motion annotation pipeline on the 2D keypoints and 3D SMPL-X datasets. Then, build a text-driven whole-body motion generation benchmark on Motion-X. Finally, we show the effectiveness of Motion-X in whole-body human mesh recovery.
Offere
Front View
Front Veew
Input Imagee
OpeniPose
MediaPipe
Ours
SMPUlf-X
Input image
(a) 2D Whole-body Keypoints Annotation
(b) 3D Whole-body Mesh Annotation
Figure 7: Qualitative comparisons of (a) 2D keypoints annotation with widely used methods [65, 66) and (b) the 3D mesh annotation with the popular fitting method [25] with ours.
Method
hand
Face AP
whole whole-body
Method
PA-MPIPE
PA-MPVPE
MPVPE 1
AP
AR
Hand4Whole [17]
58.9
50.3
79.2
OpenPose [65]
38.6
43.3
76.5
84,0
44.2
52.3
OSX (15)
55.6
48.7
70.8
HRNer [67]
50.3
60.2
73.7
80.9
58.2
67.1
PyMAR- [60]
52.8
50.2
64.9
ViTPose [18]
59.4
69.4
SMPLity-X (25)
62.6
52.9
RTMPose-1 [68]
52.3
60.0
84.4
87.6
63.2
69.4
Ours
33,5
31.8
14.7
Ours
64.9
74.0
91.6
94.4
73.5
80.3 T1mm
Ours WGT 3Dkpt
23,9
(a) Evaluation result on COCO- CiCO--Wolebodyy [51] dataset.
(b) Reconstruction error on EHF [25] dataset.
Table 3: Evaluation of motion annotation pipeline on (a) 2D keypoints and (b) 3D SMPL-X datasets.
5.1 Evaluation of the Motion Annotation Pipeline
2D Keypoints Annotation. We evaluate the proposed 2D keypoint annotation method on the COCO- WholeBody [51] dataset, and compare the evaluation result with four SOTA keypoints estimation methods [65, 67, 18, 68). We use the same input image size of 256 x 192 for all the methods to ensure a fair comparison. From Tab. 3(a), annotation pipeline significantly surpasses existing methods by over 15% average precision. Additionally, we provide qualitative comparisons in Fig. 7(a), illustrating the robust and superior performance of our method, especially in challenging and occluded scenarios.
3D SMPL-X Annotation. We evaluate our learning- based fitting method on the EHF [25] dataset and compare it with four open-sourced human mesh recovery methods. Following previous works, we employ mean per-vertex сиror (MPVPE), Procrusters aligned mean ver-vertex error (PA- (PA-MP)PE), and Procrusters aligned mean per-joint eиror (PA-MPJPE) as evaluation metrics (in mm). Results in Tab. 3(b) demonstrate the superiority of our progressive fitting method (over 30% error reduction). Specifically, PA-MPVPE is only 19.71 mm when using ground-truth 3D keypoints as supervision. Fig. 7(b) shows the annotated mesh from front and side view, indicating reliable 3D SMPL-X annotations with reduced depth ambiguity. More results are presented in Appendix due to page limits.
7

---

R Precision 1
Methods
FID
MM Dist.
Diversity+
MModality
Top 1
Top 2
Top 3
Real
0.573
0.705
0.850
0.001
2.476=00zz
13.174
MDM [14]
0. 0.299t-001 0011
0.459 000
0.577#.00 008
2. 2.094 280
6.221
11.895 354
2.621
ML.D [2]
0.440
0.624
0.733
0.914
3.407=.020 -20
13.001
2.558
T2M-GPT [48]
0.5024.000 004
0.6974.000
0.791
0.699=.122
3.192=
13.132
2.510
MifionDiffuse [9]
0.559
0.748
0.842
0.457
2.542
13.576
1.620
Table 4: Benchmark of text-driven motion generation on Motion-X test set. '->" means results are better if the
metric is closer to the real motions and = indicates the 95% confidence interval. The best results are in bold.
HumanML.3D (Test)
Motion-X (Test)
Train
Precisiont
FIDI
Diversity +
MModality
Precisiont
FIDL
Diversity
MModality
Real 0.749 on2
0.002 0n
9.8374 080
0,850 oos
0.001 2010
13. 13.1744-22
HumanML33
0.657
1.579-
10.098
2.701
0.57y
22.309-.1177
9.529
2,960
Motion-X
0.695
0.999 042
9.871
2.00
2.827 mns
0. 0.733 0.7 t.a
0.91 0
13 13.001 4.24
2.558 use
Table 5: Cross-datase comparisons HLmD and Motion-X. X. We train MLD the training of HumanML33D and Motion-X, X, respectively, then evaluate it on their test sets.
5.2 Impact on Text-driven Whole-body Motion Generation
Experiment Setup. We randomly split Motion-X into the train (80%), val (5%), and test (15%) sets. SMPL-X is adopted as the motion representation for expressive motion generation.
Evaluation metrics. We adopt the same evaluation metrics as [4], including Frechet Inception Distance (FID), Multimodality, Diversity, R- R-Precision, and Multimodal Distance. Due to the page limit, we leave more details about experimental setups and evaluation metrics in the appendix.
Benchmarking Motion-X. We train and evaluate four diffusion based motion generation methods, including MDM [14], MLD [2], MotionDiffuse [9] and T2M-GPT [48] on our dataset. Since previous datasets only have sequence-level motion descriptions, we keep similar settings for minimal model adaptation and take the semantic label as text input. The evaluation conducted with 20 runs (except for Multimodality with 5 runs) under 95% confidence interval. From Tab. 4, MotionDiffuse demonstrates a superior performance across most metrics. However, scores the lowest in Multimodality, indicating that it generates less varied motion. Notably, T2M-GPT achieves comparable performance on our dataset while maintaining high diversity, indicating our large-scale dataset's promising prospects to enhance the GPT-based method's ' efficacy. MDM gets the highest Multimodality score with the lowest precision, indicating the generation of noisy and jittery motions. The highest Top-1 precision is 55.9%, showing the challenges of Motion-X. MLD adopts the latent space design, making it fast while maintaining competent results. Therefore, we use MLD to conduct the following experiments compare Motion-X with the existing largest motion dataset HumanML33D and ablation studies.
Comparison with HumanML3D. To validate the richness, expressiveness, and effectiveness of our dataset, conduct comparative analy- sis between Motion-X and HumanML33D, which is the largest existing dataset with text-motion labels. We replace the original vector-format poses of HumanML3D with the correspond- ing SMPL-X parameters from AMASS [11], and randomly extract facial expressions from BAUM [31] to fill in the face parameters. We train MLD separately on the training sets of Motion-X and HumanML3D, then evaluate both models on the two sets. The results Tab. reveal some valuable insights. Firstly, Motion- X exhibits greater diversity (13.174) than Hu- manML3D (9,837), as evidenced by the real (GT) row. This indicates wider range of mo- tion types captured by Motion-X. Secondly, the model pretrained on Motion-X and then fine- tuned on HumanML3D subset performs well on
performing ballet happly.
playing grain practiong longfs. Smilling interesninly
Figure 8: Visual comparisons of motions generated by MLD [2] trained on HumanML33 (in purple) or Motion- in for detailed comparison.
X (in blue). Please The model trained with Motion-X can generate more accurate and semantic- correipoccrreepodeee motions.

---

Semantic
Label
Iace text
Pose Discription
body text
baad text
FID!
Method
all
EHF [25]
hand
face
all
AGORA [70]
hand
face
0.914
0,784
w/o Motion-X
79.2
43.2
25.0
185.6
73.7
82.0
0.671 t.ese
w/ Motion-X
73.0
41.0
22.6
184.1
73.3
81.4
0.565
Table 7: Mesh recovery errors of Hand4Whole [17] using different training datasets. MPVPE (mm) reported.
Table 6: Ablation study text inputs.
the HumanML3D test set, even better than the intra-data training. These superior performances stem from the fact that Motion-X encompasses diverse motion types from massive outdoor and indoor scenes. For a more intuitive comparison, we provide the visual results of the generated motion in clearly see that the model trained on Motion-X excels synthesizing seman-
Fig. 8, where tically corresponding motions given text inputs. These results prove the significant advantages of Motion-X in enhancing expressive, diverse, and natural motion generation.
Ablation study of text labels. In addition to sequence-level semantic labels, the text labels in Motion- X also include frame-level pose descriptions, which is an important characteristic of our dataset. To assess the effectiveness of pose description, we conducted an ablation study on the text labels. The baseline model solely utilizes the semantic label as the text input. Since there is method to use these labels, we simply sample a single sentence from the pose descriptions randomly, concatenate with the semantic label, and feed the combined input into the CLIP text encoder. Interestingly, from Tab. 6, adding additional face and body pose texts brings consistent improvements, and combining whole-body pose descriptions results in noteworthy 38% reduction in FID. These results validate that the proposed whole-body pose description contributes to generating more accurate and realistic human motions. More effective methods to utilize these labels can be explored in the future.
5.3 Impact on Whole-body Human Mesh Recovery
As discovered in this benchmark [21], the performance of mesh recovery methods can be significantly improved by utilizing high-quality pseudo-SMPL. labels. Motion-X provides large volume of RGB images and well-annotated SMPL-X labels. To verify its usefulness in the 3D whole- body mesh recovery task, we take Hand4Whole [17] as an example and evaluate MPVPE on the widely-used AGORA val [71] and EHF [25] datasets. For the baseline model, we train it on the commonly used COCO [51], Human3. 3.6 [72], and MPII [73] datasets. We then train another model by incorporating an additional 10% of the single-view data sampled from Motion-X X while keeping the other setting the same. As shown Tab. 7, the model trained with Motion-X X shows significant decrease of 7.8% in MPVPE on EHF and AGORA compared to the baseline model. The gains come from the increase in diverse appearances and poses Motion-X, indicating the effectiveness and accuracy of the motion annotations in Motion-X and its ability to benefit the 3D reconstruction task.
6 Conclusion
In this paper, we present Motion-X. a comprehensive and large-scale 3D expressive whole-body human motion dataset. It addresses the limitations of existing mocap datasets, which primarily focus on indoor body-only motions with limited action types. The dataset consists of 144.2 hours of whole- body motions and corresponding text labels. To build the dataset, we develop systematic annotation pipeline to annotate 81.1K 3D whole-body motions, sequence-level motion semantic labels, and 15.6M frame-level whole-body pose descriptions. Comprehensive experiments demonstrate the accuracy of the motion annotation pipeline and the significant benefit of Motion-X enhancing expressive, diverse, and natural motion generation, well as 3D whole-body human mesh recovery.
Limitation and future work. There are two main limitations our work. Firstly, the motion quality of our markless motion annotation pipeline is inevitably inferior to the multi-view marker-based motion capture system. Secondly, during our experiment, we found out that existing evaluation metrics are not always consistent with visual results. Thus, there is a need for further development of the motion generation models and evaluation metrics. As a large-scale dataset with multiple modalities, e.g., motion, text, video, and audio, Motion-X holds great potential for advancing downstream tasks, such as motion prior learning, understanding, and multi-modality pre-training. Besides, our large-scale dataset and scalable annotation pipeline open up possibilities for combining this task with a large language model (L.LM) to achieve an exciting motion generation result in the future. With M we hope to benefit and facilitate further research in relevant fields.

---

Acknowledgements
This research partially funded through National Key Research and Development Program of China (Project No. 2022YFB36066,, in part by the Shenzhen Science and Technology Project under Grant 1CC110020881000010004 JSGG202 1000200311000005), National Natural Science Foundation of China (62271414), the Young Scientists Fund of the National Natural Science Foundation of China under grant 16.62106154, the Natural Science Foundation of Guangdong Province, China (General Program) under grant No.2022AI1 202221115500 1524, and Shenzhen Science and Technology Program 01112002001110300010 7DSYS2021 1021100111111 1415025.
and Shenzhen Science and Technology Program
Thanks to everyone who participated in shooting the IDEA400 dataset and discussions. Thanks to Linghao Chen for helping to design the icons and website pages.
References
[1] C. Ahuja and L.-P. Morency, "Language2pose: Natural language grounded pose forecasting," in 3DV, 2019.
[2] X. Chen, B. Jiang, W. Liu, Z. Huang. B. Fu, T. Chen, J. Yu, and G. Yu, "Executing your commands via motion diffusion latent space," in CVPR, 2023.
[3] G. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer, and G. Rogez, "Posescript: 3d human poses from natural language," in ECCV, 2022.
[4] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, "Generating diverse and natural 3d human motions from text," in CVPR, 2022.
[5] M. Petrovich, M. J. Black, and G. Varol, "Temos: Generating diverse human motions from textual descriptions," ECCV, 2022.
[6] M. Plappert, C. Mandery, and T. Asfour, "The kit motion-language dataset," Big data, 2016.
[7] M. Plappert, C. Mandery, and T.Asfour, "Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks," Robotics and Autonomous Systems, 2018
[8] A. R. Punnakkal, A. Chandrasckaran, N. Athanasiou, A. Quiros-Ramires and M. J. Black, "Babel: bodies, action and behavior with english labels," in CVPR, 2021.
[9] M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu, "Motiondiffuse: Text-driven human motion generation with diffusion model," arXiv preprint arXiv:2208. /5001, 2022.
[10] M. Zhao, M. Liu, B. Ren, S. Dai, and N. Sebe, "Modiff: Action-conditioned 3d motion generation with denoising difffusion probabilistic models," arXiv preprint arXiv:2301.03949, 2023.
[11] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, "Amass: Archive of motion capture as surface shapes," in ICCV, 2019.
[12] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," NeurIPS, 2020.
[13] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," in ICLR, 2020.
[14] G. Tevet, Raab, Gordon, Shafir, Cohen-Or, and A. H. Bermano, "Human motion diffusion model," in ICLR, 2023.
[15] Lin,, A. Zeng, H. Wang, Zhang, and Li, "One-stage 3d whole-body mesh recovery with component aware transformer," in CVPR, 2023.
[16] R. Li, S. Yang, Ross, and Kanazawa, "Ai choreographer: Music conditioned 3d dance generation with aist++," in ICCV, 2021.
10

---

[17] G. Moon, H. Choi, and K. M. Lee, "Accurate 3d hand pose estimation for whole-body 3d human mesh estimation," CVPRW, 2020.
[18] Y. Xu, J. Zhang, Q Zhang, and D. Tao. "Vitpose: Simple vision transformer baselines for human pose estimation," in NeurIPS, 2022.
[19] Y. Yuan, U. lqbal, P. Molchanov, K. Kitani, and J. Kautz, "Glamr: Global occlusion-aware human mesh recovery with dynamic cameras," in CVPR, 2022.
[20] J. Yang, A. Zeng, S. Liu, Li, R. Zhang, and L. Zhang, "Explicit box detection unifies end-to-end multi-person pose estimation," in ICLR, 2023.
[21] H. E. Pang, Z. Cai, L. Yang, T. Zhang, and Z. Liu, "Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms," in NeurIPS Datasets and Benchmarks Track, 2022.
[22] G. Moon, H. Choi, and K. M. Lee, "Neuralannot: Neural annotator for 3d human mesh training sets," in CVPR, 2022.
[23] G. Moon, H. Choi, S. Chun, J. Lee, and S. Yun, "Three recipes for better 3d pseudo-gts of 3d human mesh estimation in the wild," CVPR, 2023.
[24] H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. J. Black, "Generating holistic 3d human motion from speech," in CVPR, 2023.
[25] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, "Expressive body capture: 3d hands, face, and body from single image," in CVPR, 2019.
[26] Z. Cai, D. Ren, A. Zeng, Z. Lin, T. Yu, W. Wang. X. Fan, Y. Gao, Y. Yu, L. Pan, Hong, M. Zhang, C. C. Loy, L. Yang, and Z. Liu, "Humman: Multi-modal 4d human dataset for versatile sensing and modeling," in ECCV, 2022.
[27] J. Chung, C.-h. Wuu, H.-r. Yang, Y.-W. Tai, and C.-K. Tang, "Haa500: Human-centric atomic action dataset with curated videos," in ICCV, 2021.
[28] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot, "Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding." in TPAMI, 2019.
[29] O. Taheri, N. Ghorbani, M. Black, and D. Tzionas, "Grab: A dataset of whole-body human grasping of objects," in ECCV, 2020.
[30] S. Tsuchida, S. Fukayama, M. Hamasaki, and M. Goto, "Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing." in ISMIR, 2019.
[31] S. Zhalehpour, O. Onder, Z. Akhtar, and C. E. Erdem, "Baum-1: 1: A spontancous audio-visual face database of affective and mental states," IEEE Transactions on Affective Computing, 2016.
[32] S. Zhang, Q. Ma, Y. Zhang, Z. Qian, T. Kwon, M. Pollefeys, F. Bogo, and S. Tang, "Fgobody: Human body shape and motion of interacting people from moun-mo devices," in ECCV, 2022.
[33] Carreira, E. Noland, C. Hillier, and A. Zisserman, "A short note the kinetics-700 human action dataset," arXiv preprint 1900.0:/07..66600 2019.
[34] C. Gu, C. Sun, D. Ross, C. Vondrick, C. Pantofaru, Y. Li. S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar et al., "Ava: video dataset of spatio-temporally localized atomic visual actions," in CVPR, 2018.
[35] A. Shahroudy, Liu, T.-T. Ng, and G. Wang, "Ntu rgb+ A large scale dataset for 3d human activity analysis," in CVPR, 2016.
[36] N. Trivedi, A. Thatipelli, and R. Sarvadevabhatla, "Ntu-x: an enhanced large-scale dataset for improving pose-based recognition of subtle human actions," in ICVGIP, 2021.

---

[37] M. Hassan, D. Ceylan, R. Villegas. J. Saito, J. Yang, Y. Zhou, and M. J. Black, "Stochastic scene-aware motion prediction." " in ICCV, 2021.
[38] M. Hassan, V. Choutas, Tzionas, and M. J. Black, "Resolving 3d human pose ambiguities with 3d scene constraints," in ICCV, 2019.
[39] Y.-L. Li, X. Liu, X. Wu, Li, Z. Qiu, L. Xu, Y. Xu, H.-S. Fang, and C. Lu, "Hake: a knowledge engine foundation for human activity understanding," in TPAMI, 2022.
[40] Y. Zheng, Y. Yang. K. Mo, Li, Yu, Y. Liu, C. K. Liu, and L. Guibas, "Gimo: Gaze- informed human motion prediction in context," in ECCV, 2022.
[41] C. Guo, X. Zuo, S. Wang, S. Zou, Q Sun, A. Deng, M. Gong, and L. Cheng, "Action2motion: Conditioned generation of 3d human motions," in ACM MM, 2020.
[42] R. Gross and J. Shi, "The emu motion of body (mobo) database," 2001.
[43] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, "Human3.6M Large scale datasets and predictive methods for 3d human sensing in natural environments," in TPAMI, 2014.
[44] L. Sigal, A. O. Balan, and M. J. Black, "Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion," IJVV, 2010.
[45] M. Trumble, A. Gilhert, C. Malleson, A. Hilton, and J. P. Collomosse, "Total capture: 3d human pose estimation fusing video and inertial sensors," in BMVC, 2017.
[46] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, "Smpl: A skinned multi- person linear model," ACM TOG, 2015.
[47] Y. Yuan, J. Song, U. Igbal, A. Vahdat, and J. Kautz, "Physdiff: Physics-guides human motion diffusion model," in ICCV, 2023.
[48] J. Zhang, Y. Zhang, X. Cun, S. Huang, Y. Zhang, H. Zhao, H. Lu, and X. Shen, "T2m-gpt: Generating human motion from textual descriptions with discrete representations," in CVPR, 2023.
[49] W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu, "Is chatgpt a good translator? a preliminary study," arXiv preprint arXiv:2301. 200002300.088745 2023.
[50] R. Daněček, M. J. Black, and T. Bolkart, "Emoca: Emotion driven monocular face capture and animation," in CVPR, 2022.
[51] S. Jin, L. Xu, J. Xu, C. Wang. W. Liu, C. Qian, W. Ouyang, and P. Luo, "Wholc-body human pose estimation in the wild," in ECCV, 2020.
[52] L. Xu, S. Jin, W. Liu, C. Qian, W. Ouyang, P. Luo, and X. Wang. "Zoomnas: searching for whole-body human pose estimation in the wild," " TPAMI, 2022.
[53] S. Narasinhaswamy, Nguyen, M. Huang, and M. Hoai, "Whose hands are these? hand detection and hand-body association in the wild," in CVPR, 2022.
[54] A. Savitzky and M. J. Golay, "Smoothing and differentiation of data by simplified least squares procedures." " Analytical chemistry, 1964.
[55] A. Zeng, X. Ju, L. Yang, R. Gao, X. Zhu, B. Dai, and Q. Xu, "Deciwatch: simple baseline for 10x efficient 2d and 3d pose estimation." in ECCV,
[56] A. Zeng, L. Yang, X. Ju, J. Li, J. Wang, and Q Xu, "Smoothnet: A plug-and-play network for refining human poses in videos," ECCV, 2022.
[57] Sárándi, Hermans, and B. Leibe, "Learning 3d human pose estimation from dozens of datasets using geometry-awan autoencoder to bridge between skeleton formats," in WACV, 2023.
[58] Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. Black, "Keep smpl: Automatic estimation of 3d human pose and shape from a single image." in ECCV, 2016.
12

---

[59] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, "Expressive body capture: 3d hands, face, and body from single image." in CVPR, 2019.
[60] Shimada, V. Golyanik, W. Xu, and C. Theobalt, "Physcap: Physically plausible monocular 3d motion capture in real time," ACM ToG, 2020.
[61] U. Iqbal, K. Xie, Y. Guo, J. Kautz, and P. Molchanov, "Kama: 3d keypoint aware body mesh articulation," in 3DV, 2021.
[62] H. Zhang, X. Li, and I.. Bing, "Video-llama: An instructioonfiieeuneed visual language model for video understanding," 2023. [Online]. Available: http://ggihub..oooDAMM... Video-LLaMA
[63] A. Dutta and A. Zisserman, "The VIA annotation software for images, audio and video," in ACM MM, 2019.
[64] A. Mollahosseini, B. Hasani, and M. H. Mahoor, "Affectnet: A database for facial expression, valence, and arousal computing in the wild," IEEE Transactions Affective Computing, 2017.
[65] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, "Realtime multi-person 2d pose estimation using part affinity fields," " in CVPR, 2017.
[66] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, "Mediapipe hands: On-device real-time hand tracking," arXiv preprint arXiv:2006. 10214, 2020.
[67] K. Sun, B. Xiao, D. Liu, and Wang, "Deep high- resouutoooltion representation learning for human pose estimation," in CVPR, 2019.
[68] J. Zhang, D. Zhang, X. Xu, F. Jia, Y. Liu, X. Liu, J. Ren, and Y. Zhang, "Mobipose: Real-time multi-person pose estimation on mobile devices," " in SenSys, 2020.
[69] H. Zhang, Y. Tian, Y. Zhang, M. Li, L. An, Z. Sun, and Y. Liu, "Pymaf-x: Towards well-aligned full-body model regression from monocular images," in TPAMI, 2023.
[70] C. Guo, X. Zuo, S. Wang, X. Liu, Zou, M. Gong, and L. Cheng, "Action2video: Generating videos of human 3d actions," IJCV, 2022.
[71] P. Patel, C.-H. P. Huang, J. Tesch, D. T. Hoffmman,, S. Tripathi, and M. J. Black, "AGORA: Avatars in geography optimized for regression analysis," in CVPR, 2021.
[72] C. lonescu, D. Papava, Olaru, and C. Sminchisescu, "Human3.66 6M: Large scale datasets and predictive methods for 3d human sensing in natural environments," in TPAMI, 2014.
[73] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, "2d human pose estimation: New benchmark and state of the art analysis," CVPR, 2014.
13


Reviews:
[{"review_id": "nlMqQjgyMh", "reviewer": "Reviewer_HxL5", "summary": "", "questions": "", "limitations": "This is a good paper which summarize all the available datasets and make them more powerful by including more annotations through automatic pipeline. Please see my question on the \"opportunity for improvement\" and \"ethic\" part.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.Motion-X is a large-scale 3D expressive whole-body motion dataset that overcomes the limitations of existing datasets by 1) including facial expressions, hand gestures, and fine-grained pose descriptions. 2) combining different dataset and online videos for sufficient diversity and quantity 3) achieving automatic text annotation to enable large-scale dataset\n\n2.The authors design a scalable and systematic pipeline for motion and text annotation, ensuring high precision and cost-effectiveness. The pipeline can automatically annotate motion from both multi-view and single-view videos, making it versatile for different scenarios. The experiments shows the advantages of the automatic pipeline for annotation compare with STOA methods\n\n3.Motion-X's strength lies in its ability to enhance diverse, expressive, and natural motion generation. The dataset serves as a benchmark for evaluating state-of-the-art motion generation methods and proves valuable for the 3D whole-body human mesh recovery task, showcasing its versatility and quality for various research applications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cYxzzsh9ID", "reviewer": "Reviewer_XGnA", "summary": "", "questions": "", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The contribution of the Motion-X dataset is valuable to the 3D human motion generation and recovery community.\n\n2. The automatic motion and description annotation pipeline could generate human text-motion data for online videos and is scalable. Quantitative experiments evaluate the accuracy.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ltR2R9pwHM", "reviewer": "Reviewer_asP3", "summary": "", "questions": "", "limitations": "Yes, the authors adequately addressed the limitations of motion quality of the annotation pipeline and the SMPL-X representations, and also the experimental setups.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed Motion-X dataset with 96K motion sequences is currently the largest text-motion datasets, which will definitely promote the research works on human motion generation;\n2. The automatic pipeline for whole-body motion and text annotation is interesting and will serve as a step towards automatic text-motion generation tool.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sA0ecl11Tm", "reviewer": "Reviewer_Kpcx", "summary": "", "questions": "", "limitations": "Limitations are well discussed.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Human motion modeling has been an interesting research topic. The proposed dataset combines multiple existing datasets on human motion, and augments them with consistent annotation. It can benefit the research community on human-motion-related topics.\n2. An automatic pipeline is proposed for the data processing and annotation purpose. Experimental results show that the proposed pipeline outperforms other methods on 2D keypoint and 3D SMPL-X annotation.\n3. The authors show by experiments that the proposed dataset can help data-drive methods on body motion generation and human mesh recovery tasks.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "G208mPQTLm", "reviewer": "Reviewer_4k8C", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "As the authors mentioned, most existing datasets have body-only annotations. Hence, the newly introduced dataset will be greatly useful as it additionally provides 3D hand poses, 3D facial expression, and text descriptions.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This submission introduces a large-scale 3D whole-body motion dataset, which consists of GT from motion capture datasets (e.g., AMASS) and pseudo-GT from in-the-wild videos. One of the major bottlenecks for 3D whole-body human motion estimation is a lack of dataset, which makes this work highly useful for the community. There were some unclear and misleading descriptions in the original submission, for example, do not clearly discriminate GT from motion capture datasets and pseudo-GT from in-the-wild videos. However, in the rebuttal, the authors clearly addressed this concern and promised to revise the manuscript to clearly discriminate against them. Given positive reviews from all reviewers, AC recommends acceptance of this paper.
Decision: Accept (Poster)