SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Third, most of the five contemporary synthetic datasets [4, 61, 48, 30, 6] demonstrate surprising strength and are ranked highly in Fig. 3c). It is worth noting that four (UBody, EgoBody, 3DPW, and EHF) of the five evaluation benchmarks used are real datasets, indicating that knowledge learned from synthetic data is transferable to real scenarios. To explain this observation, we take a close look at Fig. 2f): although real and synthetic datasets do not have extensive overlap, synthetic data possesses two ideal characteristics. First, there is a high overlap between real and synthetic data at the rightmost cluster. Referring to Fig. 2e), which is drawn from the same distribution, we find that this cluster primarily represents in-the-wild data. Therefore, synthetic data includes a substantial number of in-the-wild images that closely resemble real in-the-wild scenarios. Second, synthetic data also have scatters of image features on other clusters, indicating that synthetic data provides coverage to some extent for various real-world scenarios.
Fourth, Fig. 3d) reveals that a dataset can be valuable with accurate or pseudo-SMPL-X annotations, as they constitute the most of the top 10 datasets. A prominent example is InstaVariety [27], which has only pseudo-SMPL-X annotation produced by NeuralAnnot [45], yet, is ranked third in our benchmark. However, due to the differences in parameter spaces, SMPL annotations are less effective: it is observed that datasets with SMPL annotations tend to cluster in the lower bracket of the benchmark, especially those with pseudo-SMPL annotations. This observation suggests that SMPL-X
Table 2: Foundation Models. We study the scaling law of the amount of data and the model sizes. The metrics are MPJPE for 3DPW, and PVE for other evaluation benchmarks. Foundation models are named "SMPLer-X-MN", where M indicates the size of ViT backbone (S, B, L, H), N is the number of datasets used in the training. FPS: inference speed (frames per second) on a V100 GPU. MPE: mean primary error. AGORA uses the validation set, and EgoBody uses the EgoSet.
#Datasets	#Inst.	Model	#Param.	FPS	AGORA [48]	EgoBody [66]	UBody [37]	3DPW [56]	EHF [49]	MPE
5	0.75M	SMPLer-X-S5	32M	36.2	119.0	114.2	110.1	110.2	100.5	110.8
10	1.5M	SMPLer-X-S10	32M	36.2	116.0	88.6	107.7	97.4	89.9	99.9
20	3.0M	SMPLer-X-S20	32M	36.2	109.2	84.3	70.7	87.5	86.6	87.7
32	4.5M	SMPLer-X-S32	32M	36.2	105.2	82.5	68.1	83.2	74.1	82.6
5	0.75M	SMPLer-X-B5	103M	33.1	102.7	108.1	105.8	104.8	96.1	103.5
10	1.5M	SMPLer-X-B10	103M	33.1	97.8	76.4	107.3	89.9	74.7	89.2
20	3.0M	SMPLer-X-B20	103M	33.1	95.6	75.5	65.3	83.5	73.0	78.6
32	4.5M	SMPLer-X-B32	103M	33.1	88.0	72.7	63.3	80.3	67.3	74.3
5	0.75M	SMPLer-X-L5	327M	24.4	88.3	98.7	110.8	97.8	89.5	97.0
10	1.5M	SMPLer-X-L10	327M	24.4	82.6	69.7	104.0	82.5	64.0	80.6
20	3.0M	SMPLer-X-L20	327M	24.4	80.7	66.6	61.5	78.3	65.4	70.5
32	4.5M	SMPLer-X-L32	327M	24.4	74.2	62.2	57.3	75.2	62.4	66.2
5	0.75M	SMPLer-X-H5	662M	17.5	89.0	87.4	102.1	88.3	68.3	87.0
10	1.5M	SMPLer-X-H10	662M	17.5	81.4	65.7	100.7	78.7	56.6	76.6
20	3.0M	SMPLer-X-H20	662M	17.5	77.5	63.5	59.9	74.4	59.4	67.0
32	4.5M	SMPLer-X-H32	662M	17.5	69.5	59.5	54.5	75.0	56.8	63.1
annotations are critical to EHPS; fitting pseudo labels is a useful strategy even if they could be noisy. Moreover, using SMPL labels effectively for SMPL-X estimation remains a challenge.
4 Scaling up EHPS
4.1 Model Architectures
Catering to our investigation, we design a minimalistic framework (dubbed SMPLer-X) that only retains the most essential parts for two reasons. First, it must be scalable and efficient as we train with a large amount of data. Second, we aim to create a framework that is decoupled from specific algorithm designs, providing a clean foundation for future research. To this end, SMPLer-X consists of three parts: a backbone extracts image features, which we employ Vision Transformer [11] for its scalability; a neck that predicts bounding boxes and crop regions of interest from the feature map for hands and face; regres
sion heads that estimate parameters for each part. Note that SMPLer-X does not require third-party detectors [51], cross-part feature interaction modules [8, 13], projection of coarse SMPL-X estimations [63], or a heavy decoder [37]. As the design of SMPLer-X is not the focus of our investigation, more details are included in the Supplementary Material.
Figure 4: Architecture of SMPLer-X, which upholds the idea that "simplicity is beauty". SMPLer-X contains a backbone that allows for easy investigation on model scaling, a neck for hand and face feature cropping, and heads for different body parts. Note that we wish to show in this work that model and data scaling are effective, even with a straightforward architecture.
4.2 Training the Generalist Foundation Models
The SOTA methods [37, 44] usually train with only a few (e.g., MSCOCO, MPII, and Human3.6M) datasets, whereas we investigate training with many more datasets. However, we highlight that the dataset benchmark in Table 1 cannot be used: selecting datasets based on their performance on the test sets of the evaluation benchmarks leaks information about the test sets. Hence, we construct another dataset benchmark in the Supplementary Material, that ranks individual datasets on the training set of the major EHPS benchmarks. We use four data amounts: 5, 10, 20, and 32 datasets as the training set, with a total length of 0.75M, 1.5M, 3.0M, and 4.5M instances. We always prioritize higher-ranked
Table 3: AGORA test set. † denotes the methods that are finetuned on the AGORA training set. *denotes the methods that are trained on AGORA training set only.
Method	NMVE↓ (mm)	NMJE↓ (mm)	MVE↓ (mm)	MPJPE↓ (mm)
All	Body	All	Body	All	Body	Face	LHand	RHand	All	Body	Face	LHand	RHand
BEDLAM [4]	179.5	132.2	177.5	131.4	131.0	96.5	25.8	38.8	39.0	129.6	95.9	27.8	36.6	36.7
Hand4Whole [44]†	144.1	96.0	141.1	92.7	135.5	90.2	41.6	46.3	48.1	132.6	87.1	46.1	44.3	46.2
BEDLAM [4]†	142.2	102.1	141.0	101.8	103.8	74.5	23.1	31.7	33.2	102.9	74.3	24.7	29.9	31.3
PyMaF-X [63]†	141.2	94.4	140.0	93.5	125.7	84.0	35.0	44.6	45.6	124.6	83.2	37.9	42.5	43.7
OSX [37]*	130.6	85.3	127.6	83.3	122.8	80.2	36.2	45.4	46.1	119.9	78.3	37.9	43.0	43.9
HybrIK-X [33]	120.5	73.7	115.7	72.3	112.1	68.5	37.0	46.7	47.0	107.6	67.2	38.5	41.2	41.4
SMPLer-X-L20	133.1	88.1	128.9	84.6	123.8	81.9	37.4	43.6	44.8	119.9	78.7	39.5	41.4	44.8
SMPLer-X-L32	122.8	80.3	119.1	77.6	114.2	74.7	35.1	41.3	42.2	110.8	72.2	36.7	39.1	40.1
SMPLer-X-L20†	107.2	68.3	104.1	66.3	99.7	63.5	29.9	39.1	39.5	96.8	61.7	31.4	36.7	37.2
Table 4: AGORA Val set. † and * are finetuned on the AGORA training set, and trained on the AGORA training set only, respectively.
Method	PA-PVE↓ (mm)	PVE↓ (mm)
All	Hands	Face	All	Hands	Face
Hand4Whole [44]†	73.2	9.7	4.7	183.9	72.8	81.6
OSX [37]	69.4	11.5	4.8	168.6	70.6	77.2
OSX [37]*	45.0	8.5	3.9	79.6	48.2	37.9
SMPLer-X-B1*	48.9	8.6	4.0	86.1	51.5	41.2
SMPLer-X-L20	48.6	8.9	4.0	80.7	51.0	41.3
SMPLer-X-L32	45.1	8.7	3.8	74.2	47.8	38.7
SMPLer-X-L20†	39.1	9.3	3.8	62.5	42.3	32.8
Table 5: EHF. As EHF does not have a training set to benchmark datasets, we do not perform finetuning. Moreover, EHF is not seen in our training and can be used to validate our foundation models' transferability.
Method	PA-PVE↓ (mm)	PVE↓ (mm)
All	Hands	Face	All	Hands	Face
Hand4Whole [44]	50.3	10.8	5.8	76.8	39.8	26.1
OSX [37]	48.7	15.9	6.0	70.8	53.7	26.4
SMPLer-X-L20	37.8	15.0	5.1	65.4	49.4	17.4
SMPLer-X-L32	37.1	14.1	5.0	62.4	47.1	17.0
datasets. To prevent larger datasets from shadowing smaller datasets, we adopt a balanced sampling strategy. Specifically, all selected datasets are uniformly upsampled or downsampled to the same length and add up to the designated total length. To facilitate training, we follow OSX [37] to use AGORA, UBody, MPII, 3DPW, Human3.6M in COCO-format [38], and standardize all other datasets into the HumanData [10] format. We also study four ViT backbones of different sizes (ViT-Small, Base, Large and Huge), pretrained by ViTPose [60]. The training is conducted on 16 V100 GPUs, with a total batch size of 512 (256 for ViT-Huge) for 10 epochs. More training details such as adapting SMPL or gendered SMPL-X in the training are included in the Supplementary Material.
In Table 2, we show experimental results with a various number of datasets and foundation model sizes. Foundation models are named "SMPLER-X-MN", where M can be S, B, L, H that indicates the size of the ViT backbone, and N indicates the number of datasets used in the training. For example, SMPLER-X-L10 means the foundation model takes ViT-L as the backbone, and is trained with Top 10 datasets (ranked according to the individual dataset performance on the training sets of the key evaluation benchmarks). It is observed that 1) more training data (data scaling) leads to better performance in terms of MPE. The model performance improves gradually as the number of training datasets increases. However, besides the increment in training instances, more datasets provide a richer collection of diverse scenarios, which we argue is also a key contributor to the performance gain across evaluation benchmarks. 2) A larger foundation model (model scaling) performs better at any given amount of data. However, the marginal benefits of scaling up decrease beyond model
size L. Specifically, ViT-H has more than twice the parameters than ViT-L, but the performance gain is not prominent. 3) The foundation model always performs better than in-domain training on a single training set. For example, SMPLer-X-B20, performs better on the validation set of AGORA, and test sets of UBody, EgoBody, and 3DPW, than models trained specifically on the corresponding training set in Table 1. This is useful for real-life applications: instead of training a model for each of the user cases, a generalist foundation model contains rich knowledge to be a one-size-fits-all alternative.
Table 10: 3DPW.  $\ddagger$  denotes the methods that use a head for SMPL regression.  $\dagger$  and  $*$  are finetuned on the 3DPW training set and trained on 3DPW training set only, respectively. Unit: mm.
Method	MPJPE	PA-MPJPE
Body-only (SMPL) Methods
OSX-SMPL [37]‡*	74.7	45.1
HybrIK [35]	71.6	41.8
CLIFF [36]	68.0	43.0
Whole-Body (SMPL-X) Methods
Hand4Whole [44]	86.6	54.4
ExPose [8]	93.4	60.7
OSX [37]†	86.2	60.6
SMPLer-X-B1*	95.6	67.6
SMPLer-X-L20	78.3	52.1
SMPLer-X-L32	75.2	50.5
SMPLer-X-L20†	76.8	51.5
Table 6: UBody. † denotes the methods that are finetuned on the UBody training set. * denotes the methods that are trained on UBody training set only.
Method	PA-PVE↓ (mm)	PVE↓ (mm)
All	Hands	Face	All	Hands	Face
PIXIE [13]	61.7	12.2	4.2	168.4	55.6	45.2
Hand4Whole [44]	44.8	8.9	2.8	104.1	45.7	27.0
OSX [37]	42.4	10.8	2.4	92.4	47.7	24.9
OSX [37]†	42.2	8.6	2.0	81.9	41.5	21.2
SMPLeer-X-B1*	38.5	10.8	3.0	64.8	45.4	22.3
SMPLeer-X-L20	33.2	10.6	2.8	61.5	43.3	23.1
SMPLeer-X-L32	30.9	10.2	2.7	57.3	39.2	21.6
SMPLeer-X-L-20†	31.9	10.3	2.8	57.4	40.2	21.6
Table 8: ARCTIC. † and * denote the methods that are finetuned on the ARCTIC training set and trained on the ARCTIC training set only, respectively.
Method	PA-PVE↓ (mm)	PVE↓ (mm)
All	Hands	Face	All	Hands	Face
Hand4Whole [44]	63.4	18.1	4.0	136.8	54.8	59.2
OSX [37]	56.9	17.5	3.9	102.6	56.5	44.6
OSX [37]†	33.0	18.8	3.3	58.4	39.4	30.4
SMPler-X-B1*	45.2	18.9	3.4	66.6	42.5	34.0
SMPler-X-L10	46.9	18.1	2.3	76.9	50.8	33.2
SMPler-X-L32	29.4	18.9	2.7	48.6	38.8	26.8
SMPler-X-L10†	33.1	19.0	2.7	54.9	40.1	27.3
Table 7: EgoBody-EgoSet. † denotes the methods that are finetuned on the EgoBody-EgoSet training set. * denotes the methods that are trained on EgoBody-EgoSet training set only.
Method	PA-PVE↓ (mm)	PVE↓ (mm)
All	Hands	Face	All	Hands	Face
Hand4Whole [44]	58.8	9.7	3.7	121.9	50.0	42.5
OSX [37]	54.6	11.6	3.7	115.7	50.6	41.1
OSX [37]†	45.3	10.0	3.0	82.3	46.8	35.2
SMPLer-X-B1*	56.1	10.7	3.5	87.2	49.4	34.9
SMPLer-X-L20	38.9	9.9	3.0	66.6	42.7	31.8
SMPLer-X-L32	36.3	9.8	2.9	62.2	41.4	30.7
SMPLer-X-L20†	37.8	9.9	2.9	63.6	42.5	30.8
Table 9: DNA-Rendering-HiRes. † and * are finetuned on the DNA-Rendering-HiRes training set and trained on the DNA-Rendering-HiRes training set only, respectively.
Method	PA-PVE↓ (mm)	PVE↓ (mm)
All	Hands	Face	All	Hands	Face
Hand4Whole [44]	62.8	11.0	4.2	111.4	56.4	52.6
OSX [37]	59.9	10.6	4.3	105.7	55.0	52.5
OSX [37]†	43.5	7.5	3.5	67.1	43.3	38.2
SMPLer-X-B1*	45.6	7.5	3.4	63.2	40.7	34.2
SMPLer-X-L20	44.4	11.1	4.5	77.7	47.5	43.2
SMPLer-X-L32	35.8	7.2	3.2	54.4	36.7	34.0
SMPLer-X-L20†	37.9	7.3	3.4	56.5	38.4	34.9
Besides the errors on key benchmarks, we also report the inference speed (in terms of FPS, or frames per second) of the SMPLer-X model family in Table 2. The testing is conducted on a single V100 GPU with batch size 1, excluding data loading. SMPLer-X family is faster than OSX (12.2 FPS on a single V100 GPU) using the same test setting, and the smaller versions such as SMPLer-X-S and SMPLer-X-B can achieve real-time performance, with SMPLer-X-L on the verge of achieving real-time speeds. The high inference speed is attributed to the minimalistic architecture of SMPLer-X, which only retains the most essential components for EHPS.
Moreover, we show detailed by-part results of body, hands, and face on main benchmarks such as AGORA test set (Table 3), AGORA validation set (Table 4), UBody (Table 6), EgoBody-EgoSet (Table 7) and EHF (Table 5). We also compare our results with whole-body methods on 3DPW (Table 10). We highlight that the foundation models show strong and balanced performances on all benchmarks.
Furthermore, we evaluate the transferability of our foundation models on two more benchmarks: ARCTIC (Table 8) and DNA-Rendering (Table 8). ARCTIC features complicated hand-object interaction with whole-body annotations, and DNA-Rendering includes diverse subjects, motions, and garments. Note that ARCTIC is not seen by foundation models trained on Top 10 datasets, and DNA-Rendering is not seen by foundation models trained on Top 20 datasets. The foundation models, however, achieve much better performance than SOTAs with conventional data sampling strategies.
In addition, we compare our foundation model with SOTA methods, such as Hand4Whole [44] and OSX [37] in various scenarios in Fig. 5. These scenarios feature challenging aspects such as heavy truncation (from only half of the body visible to only the arms visible), difficult body poses in diverse backgrounds, and rare camera angles (extremely high or low elevation angles). SMPLer-X demonstrates the strength of massive training data and consistently produces robust estimations.
4.3 Finetuning the Specialists
Training the foundation model with a large number of data is expensive. For example, SMPLer-X-H20 takes more than 400 GPU hours to train. Hence, it is critical to investigate finetuning strategies that allow for low-cost adaptation of the foundation model to specific scenarios. We reiterate that in real-life applications, the test set is inaccessible. Hence, we use our benchmarking strategy and select five high-ranking datasets on the target training set to finetune the model for 5 epochs. We perform
Figure 5: Visualization. We compare SMPLER-X-L32 with OSX [37] and Hand4Whole [44] (trained with the MSCOCO, MPII, and Human3.6M) in various scenarios such as those with heavy truncation, hard poses, and rare camera angles.
finetune experiments on ViT-L to match the backbone of current SOTA [37]. The results are shown in the same tables as the foundation models (Table 3, 4, 5, 6, 7, and 10), where finetuning always lead to substantial performance enhancement on the foundation models.
In this work, we benchmark datasets for EHPS that provide us insights for training and finetuning a foundation model. Our work is useful in three ways. First, our pretrained model (especially the backbone) can be a plug-and-play component of a larger system for EHPS and beyond. Second, our benchmark serves to gauge the performances of future generalization studies. Third, our benchmarking-finetuning paradigm can be useful for the rapid adaptation of any foundation model to specific scenarios. Specifically, users may collect a training set, evaluate pretrained models of various other datasets on it, and select the most relevant datasets to finetune a foundation model.
Limitations. First, although we use five comprehensive benchmark datasets to gauge the generalization capability, they may still be insufficient to represent the real-world distribution. Second, our experiments do not fully investigate the impact of various model architectures due to the prohibitive cost of training the foundation model.
Potential negative societal impact. As we study training strong EHPS models and release the pretrained models, they may be used for unwarranted surveillance or privacy violation.
This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). The project is also supported by NTU NAP and Singapore MOE AcRF Tier 2 (MOET2EP20221-0012).
[61] Zhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu, Zhaoxi Chen, Weiye Xiao, Yukun Wei, Zhongfei Qing, Chen Wei, Bo Dai, et al. Synbody: Synthetic dataset with layered human models for 3d human perception and modeling. arXiv preprint arXiv:2303.17368, 2023.
[62] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 469-480, June 2023.
[63] Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[64] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.
[65] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. arXiv preprint arXiv:2304.01116, 2023.
[66] Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 180-200. Springer, 2022.
[67] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu. Pose2seg: Detection free human instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 889-898, 2019.
[68] Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu. Monocular real-time full body capture with inter-part correlations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4811-4822, 2021.