SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jing Lin $^{1,2*‡}$ , Ailing Zeng $^{1*†}$ , Shunlin Lu $^{1,3*‡}$ , Yuanhao Cai $^{2}$ , Ruimao Zhang $^{3}$ , Haoqian Wang $^{2}$ , Lei Zhang $^{1}$ $^{1}$ International Digital Economy Academy (IDEA)  
 $^{2}$ Tsinghua University,  $^{3}$ The Chinese University of Hong Kong, Shenzhen https://motion-x-dataset.github.io
In this paper, we present Motion-X, a large-scale 3D expressive whole-body motion dataset. Existing motion datasets predominantly contain body-only poses, lacking facial expressions, hand gestures, and fine-grained pose descriptions. Moreover, they are primarily collected from limited laboratory scenes with textual descriptions manually labeled, which greatly limits their scalability. To overcome these limitations, we develop a whole-body motion and text annotation pipeline, which can automatically annotate motion from either single- or multi-view videos and provide comprehensive semantic labels for each video and fine-grained whole-body pose descriptions for each frame. This pipeline is of high precision, cost-effective, and scalable for further research. Based on it, we construct Motion-X, which comprises 15.6M precise 3D whole-body pose annotations (i.e., SMPL-X) covering 81.1K motion sequences from massive scenes. Besides, Motion-X provides 15.6M frame-level whole-body pose descriptions and 81.1K sequence-level semantic labels. Comprehensive experiments demonstrate the accuracy of the annotation pipeline and the significant benefit of Motion-X in enhancing expressive, diverse, and natural motion generation, as well as 3D whole-body human mesh recovery.
Human motion generation aims to automatically synthesize natural human movements. It has wide applications in robotics, animation, games, and generative creation. Given a text description or audio command, motion generation can be controllable to obtain the desired human motion sequence. Text-conditioned motion generation has garnered increasing attention in recent years since it behaves in a more natural interactive way [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].
Although existing text-motion datasets [4, 11, 6, 8] have greatly facilitated the development of motion generation [2, 12, 13, 14, 9], their scale, diversity, and expressive capability remain unsatisfactory. Imagine generating "a man is playing the piano happily", as depicted in Fig. 1(a), the motion from existing dataset [4] only includes the body movements, without finger movements or facial expressions. The missing hand gestures and facial expressions severely hinder the high level of expressiveness and realism of the motion. Additionally, certain specialized motions, such as high-level skiing, aerial work, and riding are challenging to be captured in indoor scenes. To sum up, existing datasets suffer from four main limitations: 1) body-only motions without facial expressions and hand poses; 2) insufficient diversity and quantity, only covering indoor scenes; 3) lacking diverse and long-term motion sequences; and 4) manual text labels that are unscalable, unprofessional and labor-intensive. These limitations hinder existing generation methods to synthesize expressive whole-body motion with diverse action types. Therefore, how to collect large-scale whole-body motion and text annotations from multi-scenarios are critical in addressing the data scarcity issue.
*Equal Contribution, ‡ Work done during an internship at IDEA
Someone is standing and playing the piano.
(a) HumanML3D
A person is sitting and playing the piano happily.
(b) Motion-X (Ours)
Figure 1: Different from (a) previous motion dataset [4, 8], (b) our dataset captures body, facial expressions, and hand gestures. We highlight the comparisons of facial expressions and hand gestures.
Compared to indoor marker-based mocap systems, markerless vision-based motion capture methods [15, 16, 17, 18, 19, 20] become promising to capture large-scale motions from massive videos. Meanwhile, human motion can be regarded as a sequence of kinematic structures, which can be automatically translated into pose scripts using rule-based techniques [3]. More importantly, although markerless capture (e.g., pseudo labels) is not as precise as marker-based methods, collecting massive and informative motions, especially local motions, could still be beneficial [21, 15, 22, 23, 24]. Besides, text-driven motion generation task requires semantically corresponding motion labels instead of vertex-corresponding mesh labels, and thus have a higher tolerance of motion capture error. Bearing these considerations in mind, we design a scalable and systematic pipeline for motion and text annotation in both multi-view and single-view videos. Firstly, we gather and filter massive video recordings from a variety of scenes with challenging, high-quality, multi-style motions and sequence-level semantic labels. Subsequently, we estimate and optimize the parameters of the SMPL-X model [25] for the whole-body motion annotation. Due to the depth ambiguity and various challenges in different scenes, existing monocular estimation models typically fail to yield satisfactory results. To address this issue, we systematically design a high-performance framework incorporating several innovative techniques, including a hierarchical approach for whole-body keypoint estimation, a score-guided adaptive temporal smoothing and optimization scheme, and a learning-based 3D human model fitting process. By integrating these techniques, we can accurately and efficiently capture the ultimate 3D motions. Finally, we design an automatic algorithm to caption frame-level descriptions of whole-body poses. We obtain the body and hand scripts by calculating spatial relations among body parts and hand fingers based on the SMPL-X parameters and extract the facial expressions with an emotion classifier. We then aggregate the low-level pose information and translate it into textual pose descriptions.
Based on the pipeline, we collect a large-scale whole-body expressive motion dataset named Motion-X, which includes 15.6M frames and 81.1K sequences with precise 3D whole-body motion annotations, pose descriptions, and semantic labels. To compile this dataset, we collect massive videos from the Internet, with a particular focus on game and animation motions, professional performance, and diverse outdoor actions. Additionally, we incorporated data from eight existing action datasets [26, 27, 28, 11, 29, 30, 31, 32]. Using Motion-X, we build a benchmark for evaluating several state-of-the-art (SOTA) motion generation methods. Comprehensive experiments demonstrate the benefits of Motion-X for diverse, expressive, and realistic motion generation (shown in Fig. 1 (b)). Furthermore, we validate the versatility and quality of Motion-X on the whole-body mesh recovery task.
- We propose a large-scale expressive motion dataset with precise 3D whole-body motions and corresponding sequence-level and frame-level text descriptions.
- We elaborately design a automatic motion and text annotation pipeline, enabling efficient capture of high-quality human text-motion data at scale.
- Comprehensive experiments demonstrate the accuracy of the motion annotation pipeline and the benefits of Motion-X in 3D whole-body motion generation and mesh recovery tasks.
In this section, we focus on introducing existing datasets for human motion generation. For more details about the motion generation methods, please refer to the appendix.
Dataset	Motion Annotation	Text Annotation	Scene
Clip	Hour	Whole-body?	Source	Motion	Pose	Whole-body?	Indoor	Outdoor	RGB
KIT-ML'16 [6]	3911	11.2	X	Marker-based MoCap	6278	0	X	✓	X	X
AMASS'19 [11]	11265	40.0	X	Marker-based MoCap	0	0	X	✓	X	X
BABEL'21 [8]	13220	43.5	X	Marker-based MoCap	91408	0	X	✓	X	X
Posescript'22 [3]	-	-	X	Marker-based MoCap	0	120k	X	✓	X	X
HumanML3D'22 [4]	14616	28.6	X	Marker-based MoCap	44970	0	X	✓	X	X
Motion-X (Ours)	81084	144.2	✓	Pseudo GT & MoCap	81084	15.6M	✓	✓	✓	✓
Benchmarks annotated with sequential human motion and text are mainly collected for three tasks: action recognition [33, 27, 34, 28, 35, 36], human object interaction [37, 38, 39, 29, 32, 40], and motion generation [4, 41, 11, 6, 8, 24]. Specifically, KIT Motion-Language Dataset [6] is the first public dataset with human motion and language descriptions, enabling multi-modality motion generation [1, 5]. Although several indoor human motion capture (mocap) datasets have been developed [42, 43, 44, 45], they are scattered. AMASS [11] is noteworthy as it collects and unifies 15 different optical marker-based mocap datasets to build a large-scale motion dataset through a common framework and parameterization via SMPL [46]. This great milestone benefits motion modeling and its downstream tasks. Additionally, BABEL [8] and HumanML3D [4] contribute to the language labels through crowdsourced data collection. BABEL proposes either sequence labels or subsequence labels for a sequential motion, while HumanML3D collects three text descriptions for each motion clip from different workers. Thanks to these text-motion datasets, various motion generation methods have rapidly developed and shown advantages in diverse, realistic, and fine-grained motion generation [2, 14, 47, 48, 9, 10].
However, existing text-motion datasets have several limitations, including the absence of facial expressions and hand gestures, insufficient data quantity, limited diversity of motions and scenes, coarse-grained and ambiguous descriptions, and the lack of long sequence motions. To bridge these gaps, we develop a large-scale whole-body expressive motion dataset with comprehensive sequence-and frame-level text labels. We aim to address these limitations and open up new possibilities for future research. We provide quantitative comparisons of Motion-X and existing datasets in Tab. 1.
3 Motion-X Dataset
As shown in Tab. 2, we collect Motion-X from eight datasets and online videos and provide the following motion and text annotations: 15.6M 3D whole-body SMPL-X annotation, 81.1K sequence-level semantic descriptions (e.g., walking with waving hand and laughing), and frame-level whole-body pose descriptions. Notably, original sub-datasets lack either whole-body motion or text labels and we unify them with our annotation pipeline. All annotations are manually checked to guarantee quality. In Fig. 2, we show the averaged temporal standard deviation of body, hand, and face keypoints of each sub-dataset, highlighting the diversity of hand movements and facial expressions, which fills in the gaps of previous body-only motion data.
Table 1: Comparisons between Motion-X and existing text-motion datasets. The first column shows the name and public year of datasets. Motion-X provides both indoor and outdoor whole-body motion and text annotations.
Data	Clip	Frame	GT Motion	P-GT Motion	Text
AMASS [4]	26.3K	5.4M	B	H, F	S
HAA500 [27]	5.2K	0.3M	-	B, H, F	S
AIST [30]	1.4K	0.3M	-	B, H, F	S
HuMMan [26]	0.7K	0.1M	-	B, H, F	S
GRAB [29]	1.3K	0.4M	B,H	F	S
EgoBody [32]	1.0K	0.4M	B,H	F	-
BAUM [31]	1.4K	0.2M	-	F	S
IDEA400*	12.5K	2.6M	-	B, H, F	-
Online Videos*	32.5K	6.0M	-	B, H, F	-
Motion-X	81.1K	15.6M	B, H	B,H,F	S,P
Table 2: Statistics of sub-datasets. B, H, F are body, hand, and face. S and P are semantic and pose texts. P-GT is pseudo ground truth. * denotes videos are collected by us.
Figure 2: Diversity statistics of the face, hand, and body motions in each subdatasets.
Figure 3: Illustration of the overall data collection and annotation pipeline.
Figure 4: Overview of Motion-X. It includes: (a) diverse facial expressions extracted from BAUM [31], (b) indoor motion with expressive face and hand motions, (c) outdoor motion with diverse and challenging poses, and (d) several motion sequences. Purple SMPL-X is the observed frame, and the others are neighboring poses.
As illustrated in Fig. 3, the overall data collection pipeline involves six key steps: 1) designing and sourcing motion text prompts via large language model (LLM) [49], 2) collecting videos, 3) preprocessing candidate videos through human detection and video transition detection, 4) capturing whole-body motion (Sec. 4.1), 5) captioning sequence-level semantic label and frame-level whole-body pose description(Sec. 4.2), and 6) performing the manual inspection.
We gather 37K motion sequences from existing datasets using our proposed unified annotation framework, including the multi-view datasets (AIST [30]), human-scene-interaction datasets (EgoBody [32] and GRAB [29]), single-view action recognition datasets (HAA500 [27], HuMMan [26]), and body-only motion capture dataset (AMASS [11]). For these datasets, steps 1 and 2 are skipped. Notably, only EgoBody and GRAB datasets provide SMPL-X labels with body and hand pose, thus we annotate the SMPL-X label for the other motions. For AMASS, which contains the body and roughly static hand motions, we skip step 4 and fill in the facial expression with a data augmentation mechanism. The facial expressions are collected from a facial datasets BAUM [31] via a face capture and animation model EMOCA [50]. To enrich the expressive whole-body motions, we record an dataset IDEA400, which provides 13K motion sequences covering 400 diverse actions. Details about the processing of each sub-dataset and IDEA400 are available in the appendix.
To improve the appearance and motion diversity, we collect 32.5K monocular videos from online sources, covering various real-life scenes as depicted in Fig. 4. Since human motions and actions are context-dependent and vary with the scenario, we design action categories as motion prompts based on the scenario and function of the action via LLM. To ensure comprehensive coverage of human actions, our dataset includes both general and domain-specific scenes. The general scenes encompass daily actions (e.g., brushing hair, wearing glasses, and applying creams), sports activities (e.g., high knee, kick legs, push-ups), various musical instrument playing, and outdoor scenes (e.g., BMX riding, CPR, building snowman). The inclusion of general scenes helps bridge the gap between existing data and real-life scenarios. In addition, we incorporate domain-specific scenes that require high professional skills, such as dance, Kung Fu, Tai Chi, performing arts, Olympic events, entertainment
Figure 5: The automatic pipeline for the whole-body motion capture from massive videos, including 2D and 3D whole-body keypoints estimation, local pose optimization, and global translation optimization. This pipeline supports both single- and multi-view inputs. Dashed lines represent the handling of multi-view data exclusively.
shows, games, and animation motions. Based on the prompts describing the above scenes, we run the collection pipeline to gather the data from online sources for our dataset.
4 Automatic Annotation Pipeline
4.1 Universal Whole-body Motion Annotation
Overview. To efficiently capture a large volume of potential motions from massive videos, we propose an annotation pipeline for high-quality whole-body motion capture with three novel techniques: (i) hierarchical whole-body keypoint estimation; (ii) score-guided adaptive temporal smoothing for jitter motion refinement; and (iii) learning-based 3D human model fitting for accurate motion capture.
2D Keypoint Estimation. 2D Whole-body keypoint estimation poses a challenge due to the small size of the hands and face regions. Although recent approaches have utilized separate networks to decode features of different body parts [51, 52], they often struggle with hand-missing detection and are prone to errors due to occlusion or interaction. To overcome these limitations, we customize a novel hierarchical keypoint annotation method, depicted in the blue box of Fig. 5. We train a ViT-WholeBody based on a ViT-based model [18] on the COCO-Wholebody dataset [51] to estimate initial whole-body keypoints  $\mathbf{K}^{2\mathrm{D}} \in \mathbb{R}^{133 \times 2}$  with confidence scores. Leveraging the ViT model's ability to model semantic relations between full-body parts, we enhance hand and face detection robustness even under severe occlusion. Subsequently, we obtain the hand and face bounding boxes based on the keypoints, and refine the boxes using the BodyHands detector [53] through an IoU matching operation. Finally, we feed the cropped body, hand, and face regions into three separately pre-trained ViT networks to estimate body, hand and face keypoints, which are used to update  $\mathbf{K}^{2\mathrm{D}}$ .
Score-guided Adaptive Smoothing. To address the jitter resulting from per-frame pose estimation in challenging scenarios such as heavy occlusion, truncation, and motion blur, while preserving motion details, we introduce a novel score-guided adaptive smoothing technique into the traditional Savitzky-Golay filter [54]. The filter is applied to a sequence of 2D keypoints of a motion:
$$
\bar {\mathbf {K}} _ {i} ^ {\mathrm {2 D}} = \sum_ {j = - w} ^ {w} c _ {j} \mathbf {K} _ {i + j} ^ {\mathrm {2 D}}, \tag {1}
$$
where  $\mathbf{K}_i^{2\mathrm{D}}$  is the original keypoints of the  $i_{\mathrm{th}}$  frame,  $\bar{\mathbf{K}}_i^{2\mathrm{D}}$  is the smoothed keypoints,  $w$  corresponds to half-width of filter window size, and  $c_{j}$  are the filter coefficients. Different from existing smoothing methods with a fixed window size [55, 56, 54], we leverage the confidence scores of the keypoints to adaptively adjust the window size to balance between smoothness and motion details. Using a larger window size for keypoints with lower confidence scores can mitigate the impact of outliers.
3D Keypoint Annotation. Precise 3D keypoint can boost the estimation of SMPL-X. We utilize novel information from large-scale pre-trained models. Accordingly, for single-view videos, we adopt a pretrained model [57], which is trained on massive 3D datasets, to estimate precise 3D keypoints. For multi-view videos, we utilize bundle adjustment to calibrate and refine the camera parameters, and
then triangulate the 3D keypoints  $\bar{\mathbf{K}}^{3\mathrm{D}}$  based on the multi-view 2D keypoints. To enhance stability, we adopt temporal smoothing and enforce 3D bone length constraints during triangulation.
Local Pose Optimization. After obtaining the keypoints, we perform local pose optimization to register each frame's whole-body model SMPL-X [25]. Traditional optimization-based methods [58, 25] are often time-consuming and may yield unsatisfactory results as they ignore image clues and motion prior. We propose a progressive learning-based human mesh fitting method to address these limitations. Initially, we predict the SMPL-X parameter  $\Theta$  with the SOTA whole-body mesh recovery method OSX [15] and face reconstruction model EMOCA [50]. And then, through iterative optimization of the network parameters, we fit the human model parameters  $\hat{\Theta}$  to the target 2D and 3D joint positions by minimizing the following functions, achieving an improved alignment accuracy:
$$
L _ {\text {j o i n t}} = \left\| \hat {\mathbf {K}} ^ {3 \mathrm {D}} - \bar {\mathbf {K}} ^ {3 \mathrm {D}} \right\| _ {1} + \left\| \hat {\mathbf {K}} ^ {2 \mathrm {D}} - \bar {\mathbf {K}} ^ {2 \mathrm {D}} \right\| _ {1} + \| \hat {\boldsymbol {\Theta}} - \boldsymbol {\Theta} \| _ {1}. \tag {2}
$$
Here,  $\hat{\mathbf{K}}^{3\mathrm{D}}$  represents the predicted 3D joint positions obtained by applying a linear regressor to a 3D mesh generated by the SMPL-X model.  $\hat{\mathbf{K}}^{2\mathrm{D}}$  is derived by performing a perspective projection of the 3D keypoints. The last term of the loss function provides explicit supervision based on the initial parameter, serving as a 3D motion prior. To alleviate potential biophysical artifacts, such as interpenetration and foot skating, we incorporate a set of physical optimization constraints:
$$
L = \lambda_ {\text {j o i n t}} L _ {\text {j o i n t}} + \lambda_ {\text {s m o o t h}} L _ {\text {s m o o t h}} + \lambda_ {\text {p e n}} L _ {\text {p e n}} + \lambda_ {\text {p h y}} L _ {\text {p h y}}. \tag {3}
$$
Here,  $\lambda$  are weighting factors of each loss function and  $L_{\mathrm{smooth}}$  is a first-order smoothness term:
$$
L _ {\text {s m o o t h}} = \sum_ {t} \| \hat {\Theta} _ {2: t} - \hat {\Theta} _ {1: t - 1} \| _ {1} + \sum_ {t} \| \hat {\mathbf {K}} _ {2: t} ^ {\mathrm {3 D}} - \hat {\mathbf {K}} _ {1: t - 1} ^ {\mathrm {3 D}} \| _ {1}, \tag {4}
$$
where  $\hat{\Theta}_i$  and  $\hat{\mathbf{K}}_i^{3\mathrm{D}}$  represent the SMPL-X parameters and joints of the  $i$ -th frame, respectively. To alleviate mesh interpenetration, we utilize a collision penalizer [59], denoted as  $L_{\mathrm{pen}}$ . Additionally, we incorporate the physical loss  $L_{\mathrm{phy}}$  based on PhysCap [60] to prevent implausible poses.
Global Motion Optimization. To improve the consistency and realism of the estimated global trajectory, we perform a global motion optimization based on GLAMR [19] to simultaneously refine the global motions and camera poses to align with video evidence, such as 2D keypoints:
$$
L _ {g} = \lambda_ {2 \mathrm {D}} L _ {2 \mathrm {D}} + \lambda_ {\text {t r a j}} L _ {\text {t r a j}} + \lambda_ {\text {c a m}} L _ {\text {c a m}} + \lambda_ {\text {r e g}} L _ {\text {r e g}}, \tag {5}
$$
where  $L_{2\mathrm{D}}$  represents the 2D keypoint distance loss,  $L_{\mathrm{traj}}$  quantifies the difference between the optimized global trajectory and the trajectory estimated by Kama [61].  $L_{\mathrm{reg}}$  enforces regularization on the global trajectory, and  $L_{\mathrm{cam}}$  applies a smoothness constraint on the camera parameters.
Human Verification. To ensure quality, we manually checked the annotation by removing the motions that do not align with the video evidence or exhibit obvious biophysical artifacts.
4.2 Obtaining Whole-body Motion Descriptions
Sequence motion labels. The videos in Motion-X were collected from online sources and existing datasets. For action-related datasets [26, 27, 16, 28, 11, 29], we use the action labels as one of the sequence semantic labels. Meanwhile, we input the videos into Video-LLaMA [62] and filter the human action descriptions as supplemental texts. When videos contain semantic subtitles, EasyOCR automatically extracts semantic information. For online videos, we also use the search queries generated from LLM [49] as semantic labels. Videos without available semantic information, such as EgoBody [32], are manually labeled using the VGG Image Annotator (VIA) [63]. For the face database BAUM [31], we use the facial expression labels provided by the original creator.
(a) Whole-body Pose Caption
(b) Example of the Annotation Result
Figure 6: Illustration of (a) annotation of the whole-body pose description, and (b) an example of the text labels.
Whole-body pose descriptions. The generation of fine-grained pose descriptions for each pose involves three distinct parts: face, body, and hand, as shown in Fig. 6(a). Facial expression labeling uses the emotion recognition model EMOCA [50] pretrained on AffectNet [64] to classify the emotion. Body-specific descriptions utilizes the captioning process from PoseScript [3], which generates synthetic low-level descriptions in natural language based on given 3D keypoints. The unit of this information is called posecodes, such as 'the knees are completely bent'. A set of generic rules based on fine-grained categorical relations of the different body parts are used to select and aggregate the low-level pose information. The aggregated posecodes are then used to produce textual descriptions in natural language using linguistic aggregation principles. Hand gesture descriptions extends the pre-defined posecodes from body parts to fine-grained hand gestures. We define six elementary finger poses via finger curvature degrees and distances between fingers to generate descriptions, such as 'bent' and 'spread apart'. We calculate the angle of each finger joint based on the 3D hand keypoints and determine the corresponding margins. For instance, if the angle between  $\vec{\mathbf{V}} (\mathbf{K}_{\mathrm{wrist}},\mathbf{K}_{\mathrm{fingertip}})$  and  $\vec{\mathbf{V}} (\mathbf{K}_{\mathrm{fingertip}},\mathbf{K}_{\mathrm{fingeroot}})$  falls between 120 and 160 degrees, the finger posture is labeled as 'slightly bent'. We show an example of the annotated text labels in Fig. 6(b).