SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongang Cai\*,1,2,3, Wanqi Yin\*,2,4, Ailing Zeng\*, Chen Wei\*, Qingping Sun\*, Yanjun Wang\*, Hui En Pang\*, Haiyi Mei\*, Mingyuan Zhang\*, Lei Zhang\*, Chen Change Loy\*, Lei Yang\*, Ziwei Liu\*,1
<sup>1</sup> S-Lab, Nanyang Technological University, <sup>2</sup> SenseTime Research, <sup>3</sup> Shanghai AI Laboratory, <sup>4</sup> The University of Tokyo, <sup>5</sup> International Digital Economy Academy (IDEA)
Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current state-of-the-art methods still depend largely on a confined set of training datasets. In this work, we investigate scaling up EHPS towards the first generalist foundation model (dubbed SMPLer-X), with up to ViT-Huge as the backbone and training with up to 4.5M instances from diverse data sources. With big data and the large model, SMPLer-X exhibits strong performance across diverse test benchmarks and excellent transferability to even unseen environments. For the data scaling, we perform a systematic investigation on 32 EHPS datasets, including a wide range of scenarios that a model trained on any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, we optimize our training scheme and select datasets that lead to a significant leap in EHPS capabilities. For the model scaling, we take advantage of vision transformers to study the scaling law of model sizes in EHPS. Moreover, our finetuning strategy turn SMPLer-X into specialist models, allowing them to achieve further performance boosts. Notably, our foundation model SMPLer-X consistently delivers state-of-the-art results on seven benchmarks such as AGORA (107.2 mm NMVE), UBody (57.4 mm PVE), EgoBody (63.6 mm PVE), and EHF (62.3 mm PVE without finetuning).
The recent progress in expressive human pose and shape estimation (EHPS) from monocular images or videos offers transformative applications for the animation, gaming, and fashion industries. This task typically employs parametric human models (e.g., SMPL-X [49]) to adeptly represent the highly complicated human body, face, and hands. In recent years, a large number of diverse datasets have entered the field [4, 6, 6, 61, 66, 37, 3, 12, 14, 14, 62, 7], providing the community new opportunities to study various aspects such as capture environment, pose distribution, body visibility, and camera views. Yet, the state-of-the-art methods remain tethered to a limited selection of these datasets, creating a bottleneck in performance across varied scenarios and hindering the ability to generalize to unseen situations.
Our mission in this study is to explore existing data resources comprehensively, providing key insights crucial for establishing robust, universally applicable models for EHPS. Accordingly, we establish the first systematic benchmark for EHPS, utilizing 32 datasets and evaluating their performance
*Equal contributions. †Co-corresponding authors.
$^{2}$  Homepage: https://caizhongang.github.io/projects/SMPLer-X/.
across five major benchmarks. We find that there are significant inconsistencies among benchmarks, revealing the overall complicated landscape of EHPS, and calling for data scaling to combat the domain gaps between scenarios. This detailed examination emphasizes the need to reassess the utilization of available datasets for EHPS, advocating for a shift towards more competitive alternatives that offer superior generalization capabilities, and highlights the importance of harnessing a large number of datasets to capitalize on their complementary nature.
Moreover, we systematically investigate the contributing factors that determine the transferability of these datasets. Our investigation yields useful tips for future dataset collection: 1) the more is not necessarily, the merrier: datasets do not have to be very large to be useful as long as they exceed approximately 100K instances based on our observation. 2) Varying indoor scenes is a good alternative if an in-the-wild (including outdoor) collection is not viable. 3) synthetic datasets, despite having traceable domain gaps, are becoming increasingly potent to a surprising extent. 4) Pseudo-SMPL-X labels are useful when ground truth SMPL-X annotations are unavailable.
Equipped with the knowledge procured from the benchmark, we exhibit the strength of massive data with SMPLer-X, a generalist foundation model that is trained using a diverse range of datasets and achieves exceptionally balanced results across various scenarios. To decouple from algorithmic research works, we design SMPLer-X with a minimalist mindset: SMPLer-X has a very simple architecture with only the most essential components for EHPS. We hope SMPLer-X could facilitate massive data and parameter scaling and serve as a baseline for future explorations in the field instead of a stringent investigation into the algorithmic aspect. Experiments with various data combinations and model sizes lead us to a well-rounded model that excels across all benchmarks that contests the community norm of limited-dataset training. Specifically, our foundation models demonstrate significant performance boost through both data scaling and model size scaling, reducing the mean primary errors on five major benchmarks (AGORA [48], UBody [37], EgoB-
Figure 1: Scaling up EHPS. Both data and model scaling are effective in reducing mean errors on primary metrics across key benchmarks: AGORA [48], UBody [37], EgoBody [66], 3DPW [56] and EHF [49]. OSX [37] and H4W [44] are SOTA methods. Area of the circle indicates model size, with ViT variants as the reference (top right).
ody [66], 3DPW [56], and EHF [49]) from over  $110\mathrm{mm}$  to below  $70~\mathrm{mm}$  (demonstrated in Fig. 1), and showcases impressive generalization capabilities by effectively transferring to new scenarios, such as DNA-Rendering [7] and ARCTIC [12].
Furthermore, we validate the efficacy of finetuning our generalist foundation models to evolve into domain-specific specialists, delivering outstanding performance on all benchmarks. Specifically, we follow the same data selection strategy that empowers our specialist models to set new records on the AGORA leaderboard by being the first model to hit  $107.2\mathrm{mm}$  in NMVE (an  $11.0\%$  improvement) and achieving SOTA performance on EgoBody, UBody, and EHF.
Our contributions are three-fold. 1) We build the first systematic and comprehensive benchmark on EHPS datasets, which provides critical guidance for scaling up the training data toward robust and transferable EHPS. 2) We explore both data and model scaling in building the generalist foundation model that delivers balanced results across various scenarios and extends successfully to unseen datasets. 3) We extend the data selection strategy to finetune the foundation model into potent specialists, catering to various benchmark scenarios.
Expressive Human Pose and Shape Estimation (EHPS). Due to the erupting 3D virtual human research applications [64, 65, 19, 18, 5] and the parametric models (e.g., SMPL [40] and SMPL-X [49]), capturing the human pose and shape (HPS) [26, 31, 28, 29, 36, 57, 58], and additionally hands and face (EHPS) [49, 59, 8, 51, 68, 13, 54, 63] from images and videos have attracted increasing
d) Appearance (OSX)
Figure 2: Dataset attribute distributions. a) and d) are image feature extracted by HumanBench [55] and OSX [37] pretrained ViT-L backbone. b) Global orientation (represented by rotation matrix) distribution. c) Body pose (represented by 3D skeleton joints) distribution. Both e) scenes and f) Real/Synthetic are drawn on the same distribution as d). All: all datasets. UMAP [41] dimension reduction is used with the x and y-axis as the dimensions of the embedded space (no unit).
e) Scenes
f) Real/Synthetic
attention. Optimization-based methods (e.g., SMPLify-X [49]) detect 2D features corresponding to the whole body and fit the SMPL-X model. However, they suffer from slow speed and are ultimately limited by the quality of the 2D keypoint detectors. Hence, learning-based models are proposed. One of the key challenges of EHPS is the low resolution of hands and face compared with the body-only estimation, making the articulated hand pose estimation and high-quality expression capture difficult. Accordingly, mainstream whole-body models first detect and crop the hands and face image patches, then resize them to higher resolutions and feed them into specific hand and face networks to estimate the corresponding parameters [8, 51, 68, 13, 54, 44, 63, 33]. Due to the highly complex multi-stage pipelines, they inevitably cause inconsistent and unnatural articulation of the mesh and implausible 3D wrist rotations, especially in occluded, truncated, and blurry scenes. Recently, OSX [37] proposes the first one-stage framework based on ViT-based backbone [11] to relieve the issues in previous multi-stage pipelines. This method provides a promising and concise way to scale up the model. However, they only use confined training datasets for a fair comparison and do not explore the combination of more data toward generalizable and precise EHPS.
Multi-dataset Training for Human-centric Vision. Recent efforts have been using multiple datasets in pretraining a general model for a wide range of downstream human-centric tasks. For example, HumanBench [55] leverages 37 datasets, whereas UniHCP [9] utilizes 33 datasets for tasks such as ReID, pedestrian detection, and 2D pose estimation. However, these works have only evaluated the efficacy of 2D tasks. Sárándi et al. [52] take advantage of 28 datasets in training a strong model for 3D keypoint detection, which recovers only the skeleton of subjects without estimating body shapes and meshes. Pang et al. [47] analyze 31 datasets for human pose and shape estimation (i.e., SMPL estimation). However, hands and face estimation is not included, and only fewer than ten datasets are used concurrently in the most diverse training. This paper targets to scale training data and model size for EHPS, that simultaneously recovers the expressive pose and shape of the human body, hands, and face.
3 Benchmarking EHPS Datasets
SMPL-X. We study expressive human pose and shape estimation via 3D parametric human model SMPL-X [49], which models the human body, hands, and face geometries with parameters. Specifically, our goal is to estimate pose parameters  $\theta \in \mathbb{R}^{55\times 3}$  that include body, hands, eyes, and jaw poses; joint body, hands and face shape  $\beta \in \mathbb{R}^{10}$ , and facial expression  $\psi \in \mathbb{R}^{10}$ . The joint regressor  $\mathcal{J}$  is used to obtain 3D keypoints from parameters via  $R_{\theta}(\mathcal{J}(\beta))$  where  $R_{\theta}$  is a transformation function along the kinematic tree.
Table 1: Benchmarking EHPS datasets. For each dataset, we train a model on its training set and evaluate its performance on the val set of AGORA and testing sets of UBody, EgoBody (EgoSet), 3DPW, and EHF. Datasets are then ranked by mean primary error (MPE). Top-1 values are bolded, and the rest of Top-5 are underlined. #Inst.: number of instances used in training. ITW: in-the-wild. EFT [25], NeuralAnnot (NeA) [45] and UP3D [32] produce pseudo labels.
Dataset	#Inst.	Scene	Real/Synthetic	SMPL	SMPL-X	AGORA [48]	UBody [37]	EgoBody [66]	3DPW [56]	EHF [49]
PVE↓	★	PVE↓	★	PVE↓	★	MPJPE↓	★	PVE↓	★
BEDLAM [4]	951.1K	ITW	Syn	-	Yes	164.7	4	132.5	8	109.1	2	98.1	1	81.1	1
SynBody [61]	633.5K	ITW	Syn	-	Yes	166.7	5	144.6	11	136.6	4	106.5	5	112.9	5
InstaVariety [27]	2184.8K	ITW	Real	NeA	-	195.0	9	125.4	4	140.1	9	100.6	3	110.8	4
GTA-Human II [6]	1802.2K	ITW	Syn	-	Yes	161.9	3	143.7	10	139.2	8	103.4	4	126.0	12
MSCOCO [38]	149.8K	ITW	Real	EFT	NeA	191.6	8	107.2	2	139.0	7	121.2	10	116.3	7
EgoBody-MVSet [66]	845.9K	Indoor	Real	Yes	Yes	190.9	7	191.4	18	127.0	3	99.2	2	101.8	2
AGORA [48]	106.7K	ITW	Syn	Yes	Yes	124.8	1	128.4	6	138.4	6	131.1	12	164.6	24
Egobody-EgoSet [66]	90.1K	Indoor	Real	Yes	Yes	207.1	15	126.8	5	103.1	1	134.4	18	121.4	10
RICH [20]	243.4K	ITW	Real	-	Yes	195.6	10	168.1	15	137.9	5	115.5	8	127.5	13
MPII [2]	28.9K	ITW	Real	EFT	NeA	202.1	11	123.9	3	155.5	15	131.9	14	140.8	16
MuCo-3DHP [43]	465.3K	ITW	Real	Yes	-	187.7	6	185.4	17	146.4	12	119.4	9	134.7	15
PROX [17]	88.5K	Indoor	Real	-	Yes	204.1	13	180.3	16	151.8	13	132.5	17	122.5	11
UBody [37]	683.3K	ITW	Real	-	Yes	207.0	14	78.7	1	145.6	11	149.4	23	132.1	14
SPEC [30]	72.0K	ITW	Syn	Yes	-	161.5	2	146.1	12	154.8	14	139.7	21	197.8	27
CrowdPose [34]	28.5K	ITW	Real	NeA	-	207.1	16	129.8	7	156.9	16	156.3	25	154.5	22
MPI-Inf-3DHP [42]	939.8K	ITW	Real	NeA	NeA	221.5	20	166.7	14	142.7	10	131.6	13	155.5	23
HumanSC3D [15]	288.4K	Studio	Real	-	Yes	215.2	18	237.8	22	167.3	17	113.0	7	107.1	3
PoseTrack [1]	28.5K	ITW	Real	EFT	-	218.1	19	161.0	13	180.8	21	150.2	24	149.9	21
BEHAV[3]	44.4K	Indoor	Real	Yes	-	208.3	17	205.8	20	175.8	19	132.0	15	145.0	18
CH3D [14]	252.4K	Studio	Real	-	Yes	203.3	12	264.7	25	175.7	18	122.6	11	121.0	9
Human3.6M [21]	312.2K	Studio	Real	Yes	NeA	226.0	21	276.1	26	200.6	24	112.3	6	120.8	8
DNA-R-HiRes [7]	998.1K	Studio	Real	-	Yes	230.0	22	278.2	27	179.2	20	134.5	19	149.7	20
3DPW [56]	22.7K	ITW	Real	Yes	NeA	234.0	23	259.3	23	192.6	23	140.6	22	142.9	17
ARCTIC [12]	1539.1K	Studio	Real	-	Yes	308.5	29	200.7	19	186.4	22	202.5	26	182.5	25
DNA-R [7]	3992.0K	Studio	Real	-	Yes	274.7	26	341.5	30	214.4	27	138.4	20	115.5	6
UP3D [32]	7.1K	ITW	Real	UP3D	-	257.5	24	224.1	21	216.6	28	211.5	27	194.8	26
Talkshow [62]	3326.9K	Indoor	Real	-	Yes	286.4	27	133.2	9	203.6	25	291.3	29	201.9	28
FIT3D [16]	1779.3K	Studio	Real	-	Yes	329.7	30	404.0	31	213.8	26	132.1	16	148.1	19
MTP [46]	3.2K	ITW	Real	Yes	Yes	272.7	25	284.9	28	273.2	29	265.2	28	244.6	29
OCHuman [67]	2.5K	ITW	Real	EFT	-	307.1	28	263.3	24	279.3	30	293.4	30	281.7	30
LSPET [23]	2.9K	ITW	Real	EFT	-	365.7	31	292.6	29	340.1	31	339.8	31	316.3	31
SSP3D [53]	311	ITW	Real	Yes	-	549.8	32	522.4	32	548.1	32	439.0	32	539.5	32
Evaluation Metrics. We use standard metrics for EHPS. PVE (per-vertex error) and MPJPE (mean per-joint position error) measure the mean L2 error for vertices and regressed joints, respectively. The "PA" prefix indicates Procrutes Alignment is conducted before error computation. AGORA Leaderboard [48] introduces NMVE (normalized mean vertex error) and NMJE (normalized mean joint error) that take detection performance F1 score into consideration. Moreover, we propose MPE (mean primary error) that takes the mean of multiple primary metrics (MPJPE for 3DPW [56] test, and PVE for AGORA, UBody, EgoBody, and EHF) to gauge generalizability. All errors are reported in millimeters (mm).
3.2 Overview of Data Sources
In this work, we study three major types of datasets. 1) motion capture datasets that leverage optical [21, 14, 12, 15, 16, 42] or vision-based [66, 17, 7, 5] multi-view motion capture systems, are typically collected in a studio environment. However, it is possible to include an outdoor setup, or utilize additional sensors such as IMUs [56]. These datasets generally provide high-quality 3D annotations but are less flexible due to physical constraints, especially those built with immobile capture systems that require accurate sensor calibrations. 2) pseudo-annotated datasets [38, 1, 34, 37, 43, 27, 67, 2, 23, 46, 62, 53] that re-annotation existing image datasets with parametric human annotations [24, 45, 37]. These datasets take advantage of the diversity of 2D datasets, and the pseudo-3D annotations, albeit typically not as high-quality, have been proven effective [47, 31, 24]. 3) synthetic datasets [4, 6, 30, 48, 61] that are produced with renderings engines (e.g., Unreal Engine). These datasets produce the most accurate 3D annotations and can easily scale up with high diversity. However, the synthetic-real gap is not fully addressed. Key attributes of the datasets are included in Table 1.
To evaluate the EHPS capability across diverse scenarios, we select multiple key datasets to form a comprehensive benchmark. They should possess the desirable traits such as 1) having accurate SMPL or SMPL-X annotations, 2) being representative of certain aspects of real-life scenarios, 3) being widely used, but this requirement is relaxed for the new datasets which are released within two years, and 4) has a clearly defined test set. To this end, five datasets (AGORA [48], UBody [37], EgoBody [66], 3DPW [56], and EHF [49]) representing different aspects are selected as the evaluation datasets. We briefly introduce these five datasets and the rest in the Supplementary Material. AGORA
is the most widely-used benchmark for SMPL-X evaluation. It is a synthetic dataset featuring diverse subject appearances, poses, and environments with high-quality annotation. We evaluate on both validation and test set (leaderboard) as the latter has a monthly limit of submissions. UBody is the latest large-scale dataset with pseudo-SMPL-X annotations that covers fifteen real-life scenarios, such as talk shows, video conferences, and vlogs, which primarily consist of the upper body in images. We follow the intra-scene protocol in training and testing, where all scenarios are seen. EgoBody captures human motions in social interactions in 3D scenes with pseudo-SMPL-X annotations. It comprises a first-person egocentric set (EgoSet) and a third-person multi-camera set (MVSet). We test on the EgoSet with heavy truncation and invisibility. 3DPW is the most popular in-the-wild dataset with SMPL annotations. Since SMPL-X annotation is not available, we map SMPL-X keypoints and test on 14 LSP [22] keypoints following the conventional protocol [26, 31]. EHF is a classic dataset with 100 curated frames of one subject in an indoor studio setup, with diverse body poses and especially hand poses annotated in SMPL-X vertices. It has a test set but no training or validation sets. Hence, it is only used to evaluate cross-dataset performance.
Besides being popular or the latest evaluation sets for EHPS, we further analyze if these five datasets collectively provide wide coverage of existing datasets. In Fig. 3, we randomly downsample all datasets to equal length (1K examples) and employ UMAP [41] to visualize several key aspects. We use pretrained ViT-L from HumanBench [55] and OSX [37] to process patch tokens flattened as feature vectors from images cropped by bounding boxes. HumanBench is trained for various human-centric tasks (e.g., Re-ID, part segmentation, and 2D pose estimation), whereas OSX is an expert model on EHPS. As for global orientation, it is closely associated with camera pose as we convert all data into the camera coordinate frame; we plot its distribution by using flattened rotation matrix representations. Moreover, we follow [50, 6, 47] to represent poses as 3D keypoints regressed from the parametric model. Specifically, we flatten 21 SMPL-X body keypoints, and 15 hand keypoints from each hand, regressed with zero parameters except for the body pose and hand poses. It is shown that 1) the five benchmark datasets have varied distribution, which is expected due to their different designated purposes, and 2) collectively, the five datasets provide a wide, near-complete coverage of the entire dataset pool.
3.3 Benchmarking on Individual Datasets
In this section, we aim to benchmark datasets and find those that do well in various scenarios. To gauge the performance of each dataset, we train a SMPLer-X model with the training set of that dataset and evaluate the model on the val/testing sets of five evaluation datasets: AGORA, UBody, EgoBody, 3DPW, and EHF. Here, the benchmarking model is standardized to use ViT-S as the backbone, trained on 4 V100 GPUs for 5 epochs with a total batch size of 128 and a learning rate of  $1 \times 10^{-5}$ . The dataset preprocessing details are included in the Supplementary Material.
In Table 1, we report the primary metrics (Sec. 3.1) and ranking of the 32 datasets. The complete results in the Supplementary Material. We also compute the mean primary error (MPE) to facilitate easy comparison between individual datasets. Note that for AGORA, UBody, EgoBody, and 3DPW, their performances on their own test set are excluded from computing MPE. This is because indomain evaluation results are typically much better than cross-domain ones, leading to significant error drops. In addition, note that there are datasets designed for specific purposes (e.g., Talkshow [62] for gesture generation, DNA-Rendering [7] for human NeRF reconstruction), being ranked lower on our benchmark, which focuses on EHPS (a perception task) does not reduce their unique values and contributions to the computer vision community.
From the benchmark, we observe models trained on a single dataset tend to perform well on the same domain but often cannot do well on other domains. For example, the model trained on AGORA is ranked  $1^{st}$  on AGORA (val), but  $6^{th}$  on UBody,  $6^{th}$  on EgoBody,  $12^{th}$  on 3DPW, and  $24^{th}$  on EHF. This observation indicates that 1) the test scenarios are diverse, showcasing the challenging landscape of EHPS, and 2) data scaling is essential for training a robust and transferable model for EHPS due to significant gaps between different domains.
3.4 Analyses on Dataset Attributes
In this section, we study attributes that contribute to generalizability. However, it is important to acknowledge that such analyses are not a straightforward task: the attributes often exhibit coupled
Figure 3: Analysis on dataset attributes. We study the impact of a) the number of training instances, b) scenes, c) real or synthetic appearance, and d) annotation type, on dataset ranking in Table 1.
effects. Consequently, counter-examples are inevitable (e.g., we observe that InstaVariety, an in-the-wild dataset, demonstrates strong performance, whereas LSPET, another in-the-wild dataset, does not perform as well). Despite the challenges in pinpointing the exact factors that determine the success of an individual dataset, we adopt a collective perspective and aim to identify general trends with several key factors [47, 39, 6] in Fig. 3, and discussed below.
First, Fig. 3a) shows that the performance of a dataset (in terms of ranking) is not strongly associated with the number of training instances once the instance number exceeds approximately  $100\mathrm{K}$ . Although a very small amount of training data is insufficient to train a strong model, having an exceedingly large amount of data does not guarantee good performance either. For example, MSCOCO only comprises  $149.8\mathrm{K}$  training instances but achieves a higher ranking compared to datasets with  $10\times$  larger scales. This may be attributed to the diverse appearance and complex scenes present in the MSCOCO dataset. Hence, it would be more cost-effective to channel resources to improve diversity and quality, when the dataset has become adequately large.
Second, we categorize datasets into 1) in-the-wild, which contains data from diverse environments; 2) indoor with several scenes; 3) studio, which has a fixed multi-view setup. Particularly, Fig. 3b) shows that the top 10 are mostly in-the-wild datasets, indoor datasets concentrate in the top 20 and the studio dataset tends to be ranked lower in the benchmark. Moreover, Fig. 2e) illustrates that in-the-wild datasets exhibit the most diverse distribution, covering both indoor and studio datasets. Indoor datasets display a reasonable spread, and studio datasets have the least diversity. Our findings validate previous studies that suggest an indoor-outdoor domain gap [25]. Differing from Pang et al. [47], which does not differentiate between indoor and studio datasets, we argue that categorizing all datasets collected indoors into a single class oversimplifies the analysis. For example, consider EgoBody [66] and Human3.6M [21]. Both datasets does not have outdoor data; however, EgoBody consists of a wide variety of indoor scenes, whereas Human3.6M consists of only one scene, which may contribute to the better ranking of EgoBody compared to Human3.6M. Hence, this suggests that in-the-wild data collection is the most ideal, but diversifying indoor scenes is the best alternative.