SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation
Authors: Zhongang Cai
Wanqi Yin
Ailing Zeng
CHEN WEI
Qingping SUN
Yanjun Wang
Hui En Pang
Haiyi Mei
Mingyuan Zhang
Lei Zhang
Chen Change Loy
Lei Yang
Ziwei Liu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Expressive Human Pose and Shape Estimation
Benchmarks
Datasets
Foundation Models
TL;DR: Scaling up training data and model size for robust and generalizable expressive human pose and shape estimation

Abstract: Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current state-of-the-art methods still depend largely on a confined set of training datasets. In this work, we investigate scaling up EHPS towards the first generalist foundation model (dubbed SMPLer-X), with up to ViT-Huge as the backbone and training with up to 4.5M instances from diverse data sources. With big data and the large model, SMPLer-X exhibits strong performance across diverse test benchmarks and excellent transferability to even unseen environments. 1) For the data scaling, we perform a systematic investigation on 32 EHPS datasets, including a wide range of scenarios that a model trained on any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, we optimize our training scheme and select datasets that lead to a significant leap in EHPS capabilities. 2) For the model scaling, we take advantage of vision transformers to study the scaling law of model sizes in EHPS. Moreover, our finetuning strategy turn SMPLer-X into specialist models, allowing them to achieve further performance boosts. Notably, our foundation model SMPLer-X consistently delivers state-of-the-art results on seven benchmarks such as AGORA (107.2 mm NMVE), UBody (57.4 mm PVE), EgoBody (63.6 mm PVE), and EHF (62.3 mm PVE without finetuning).

SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation
Zhongang Cai+1.2.3, Wanqi Yin 224 Ailing Zeng", Chen Wei', Qingping Sun?, Yanjun Wang", Hui En Pang 222 Haiyi Mei², Mingyuan Zhang', Lei Zhang Chen Change Loy', Lei Yangi.2.. Ziwei Liu
S S-Lab, Nanyang Technological University, SenseTTme Research, 3 Shanghai AI Laboratory, The University of Tokyo, International Digital Economy Academy (IDEA)
Abstract
Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current cate-offthheart methods still depend largely on a confined datasets. In this work, we investigate scaling up EHPS towards the first generalist foundation model (dubbed SMPLer-X), with up to ViT-Huge as the backbone and training with up to 4.5M instances from diverse data sources. With big data and the large model, SMPLer-X exhibits strong performance across diverse test benchmarks and excellent transferability to even unseen environments. I) For the data scaling, we perform a systematic investigation on 32 EHPS datasets, including wide range of scenarios that model trained any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, optimize training scheme and select datasets that lead to a significant leap in EHPS capabilities. 2) For the model scaling, we take advantage of vision transformers to study the scaling law of model sizes in EHPS. Moreover, our finetuning strategy turn SMPLer-X into specialist models, foundation
training
allowing them achieve further performance boosts. Notably, model SMPLer-X consistently delivers state-of-the- results on seven benchmarks such as AGORA (107.2 NMVE), UBody (57.4 mm PVE), EgoBody (63.6 mm PVE), and EHF (62.3 mm PVE without finetuning).
Introduction
The recent progress expressive human pose and shape estimation (EHPS) from monocular images or videos offers transformative applications for the animation, gaming, and fashion industries. This task typieally employs parametric human models (e.g., SMPL-X
to adeptly represent the highly
complicated human body, face, and hands. In recent years, a large number of diverse datasets have entered the field
B 12 14 62 Z]. providing the community new opportunities to study various aspects such as capture environment, pose distribution, body visibility, and camera views. Yet, the state-of-the-ar methods remain tethered to a limited selection these datasets, creating a bottleneck in performance across varied scenarios and hindering the ability to generalize to unseen situations.
61 66 37
Our mission in this study is to explore existing data resources comprehensively, providing key insights crucial for establishing robust, universally applicable models for EHPS. Accordingly, we establish the first systematic benchmark for EHPS, utilizing 32 datasets and evaluating their performance
"Equal contributions. 'Co-corresponding authors.
"Homepage; https::/ hagtigggnnnggggggbbggeee io/projjettsSMMPer-
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

across five major benchmarks. We find that there are significant inconsistencies among benchmarks, revealing the overall complicated landscape of EHPS, and calling for data scaling combat the domain gaps between scenarios. This detailed examination emphasizes the need to reassess the utilization of available datasets for EHPS, advocating for a shift towards more competitive alternatives that offer superior generalization capabilities, and highlights the importance of harnessing large number of datasets to capitalize on their complementary nature.
Moreover, we systematically investigate the contributing factors that determine the transferability of these datasets. Our investigation yields useful tips for future dataset collection: 1) the more is not necessarily, the merrier: datasets do not have to be very large to be useful as long as they exceed approximately 100K instances based on our observation. 2) Varying indoor scenes is good alternative if an in-the-wild (including outdoor) collection is not viable. 3) synthetic datasets, despite having traceable domain gaps, are becoming increasingly potent to surprising extent. 4) Pseudo-SMPL-X labels are useful when ground truth SMPL-X annotations are unavailable.
Equipped with the knowledge procured from the benchmark, we exhibit the strength of massive data with SMPLer-X, a generalist foundation model that is trained using a diverse range of datasets and achieves exceptionally balanced results across various scenarios. To decouple from algorithmic research works, we design SMPLer-X with minimalist mindset: SMPLer- has very simple architecture with only the most essential components for EHPS. We hope
HHW ic (CV9RW23)
(SX1CV19))
Mcan Primary Eeror (mm)
110-
VIE- WTA WI-I
90
SMPlla----8
80-
SMPL
SMPLer-X could facilitate massive data and pa- rameter scaling and serve as a baseline for future explorations in the field instead of a stringent investigation into the algorithmic aspect. Ex- periments with various data combinations and model sizes lead to well-rounded model that excels across all benchmarks that contests the community norm of limited-dataset train- ing. Specifically, our foundation models demon- strate significant performance boost through both data scaling and model size scaling, re- ducing the mean primary errors on five major benchmarks (AGORA 488 UBody (37]. EgoB-
SXK-a---11
Number of Training Instances (Millions)
Figure 1: Scaling up EHPS. Both data and model scaling are effective in reducing mean errors primary metrics across key bench- marks; AGORA [48], UBody [37], EgoBody (66), 3DPW [56] and EHF [4]]. OSX [37 and
H4W 44) are SOTA methods. Area of the cir-
cle indicates model size, with ViT variants as the reference (top right).
ody (66). 3DP (66, and EHF 49) from over 110 mm to below 70 mm (demonstrated in Fig. [ and showcases impressive generalization capabilities by effectively transferring to new scenarios, such as DNA ENA-Rendering 7 and ARCTIC [12].
Furthermore, we validate the efficacy of finetuning our generalist foundation models to evolve into dopain- specialists, delivering outstanding performance all benchmarks. Specifically, follow the same data selection strategy that empowers our specialist models to set new records on the AGORA leaderboard by being the first model to hit 107. 17.2mm in NMVE (an 11.0% improvement) and achieving SOTA performance on EgoBody, UBody, and EHF.
Our contributions are three-fold. 1) We build the first systematic and comprehensive benchmark on EHPS datasets, which provides critical guidance for scaling up the training data toward robust and transferable EHPS. 2) We explore both data and model scaling in building the generalist foundation model that delivers balanced results across various scenarios and extends successfully to unseen datasets. 3) We extend the data selection strategy to finetune the foundation model into potent specialists, catering to various benchmark scenarios.
Related Work
Expressive Human Pose and Shape Estimation (EHPS). Due to the erupting 3D virtual human
research applications 64
65
19
18
5 and the parametric models (e.g.. SMPL 40 and SMPL.
[4))), capturing the human pose and shape (HPS) 26
RI 28 36
57 58 and additionally
hands and face (EHPS) 49 59
8
5
68
1
54
63 from images and videos have attracted increasing

---

a) Appearance (HumanBench)
b) Global Orient
Body Pose
Tedialy
330M
Appearance (OSX)
Scenes
f) Real/Symthetic
Figure 2: Dataset attribute distributions. a) and d) are image feature extracted by HumanBench [55] and OSX (37] pretrained ViT-L backbone. b) Global orientation (represented by rotation matrix) distribution. c) Body pose (represented by 3D skeleton joints) distribution. Both e) scenes and Real/Synthetic are drawn on the same distribution as d). All: all datasets. UMAP [41] dimension reduction is used with the and y-axis the dimensions of the embedded space (no unit).
attention. Optimization- bassad methods (e.g., SMPLify-X X (49) detect 2D features corresponding to the whole body and fit the SMPL-X model. However, they suffer from slow speed and are ultimately limited by the quality of the 2D keypoint detectors. Hence, learning-based models are proposed. One of the key challenges of EHPS the low resolution of hands and face compared with the body-only estimation, making the articulated hand pose estimation and high-quality expression capture difficult. Accordingly, mainstream whole-body models first detect and crop the hands and face image patches, then resize them to higher resolutions and feed them into specific hand and face networks to estimate the corresponding parameters 88 5I
54 63 33) Due to the highly complex multi-stage pipelines, they inevitably cause inconsistent and unnatural articulation of the mesh and implausible 3D wrist rotations, especially in occluded, truncated, and blurry scenes. Recently, OSX 377 proposes the first one-stage framework based on ViT-based backbone [I] to relieve the issues in previous multi-stage pipelines. This method provides a promising and concise way to scale up the model. However, they only use confined training datasets for a fair comparison and do not explore the combination of more data toward generalizable and precise EHPS.
Multi-dataset Training for Human-centria Vision. Recent efforts have been using multiple datasets in pretraining a general model for a wide range of downstream human-centric centric tasks. For example, HumanBench (55) leverages 37 datasets, whereas UniHCP (9] utilizes 33 datasets for tasks such as ReID, pedestrian detection, and 2D pose estimation. However, these works have only evaluated the efficacy of 2D tasks. Sárándi et al. [52] take advantage of 28 datasets in training strong model for 3D keypoint detection, which recovers only the skeleton of subjects without estimating body shapes and meshes. Pang et al. [47] analyze 31 datasets for human pose and shape estimation (i.e., SMPL estimation). However, hands and face estimation is not included, and only fewer than ten datasets are used concurrently in the most diverse training. This paper targets to scale training data and model size for EHPS, that simultaneously recovers the expressive pose and shape of the human body, hands, and face.
3 Benchmarking EHPS Datasets
3.1 Preliminaries
SMPL-X. We study expressive human pose and shape estimation via 3D parametric human model SMPL-X 49) which models the human body, hands, and face geometries with parameters. Specif- ically, our goal is to estimate pose parameters E 55x3 that include body, hands, eyes, and jaw poses; joint body, hands and face shape B E RR10 and facial expression & E R100 The joint regressor J used to obtain 3D keypoints from parameters via (B)) where Rs is a transformation function along the kinematic tree.
3

---

Table I: Benchmarking EHPS datasets. For each dataset, we train a model on its training set and evaluate its performance on the val set of AGORA and resting sets of UBody, EgoBody (EgoSet), 3DPW, and EHF. Datasets are then ranked by mean primary error (MPE). Top-1 values are bolded, and the rest of T are underlined. #lnst.: number of instances used in training. ITW: mn-the-wild.
EFT 225] Nnno (NeA) 45 and UP3D 32
produce pseudo labels.
AGORA
[8]
UBidly
57
EgeBody
66
3DPW 56
EHF [9)
Dutasct
floot.
Suere
RealiSyuthctic
SMPL
SMPL-X
PVEL
*
PVEL
*
PVE]
MPIPEL
PVE.
MPE]
REDLAM [
USL.1K
ITW
Syn
Yos
164.2
109.1
81.1
117.1
SynBody
633.5K
ITW
Syn
165.
144.6
136.5
106.5
112.9
133.5
2154.8K
ITW
Real
NoA
125.4
148.1
10016
110.
134.3
1802.3X
ITW
Sym
Yea
143.7
139.3
103.4
126.11
134.8
MSCOCO
ITW
Real
HFT
NoA
121.2
116.3
EpeBody MVSet
Indoor
Real
Yes
191.4
122.0
101.88
AGORA
ITW
Sym
128.
164.6
Real
121.4
147.5
RICH
ITW
MPI
ITW
Real
RFT
155.5
150.
MrCh-anHP
ITW
Real
Yis
185.4
154.7
PROX
85.5K
Indoer
Real
180.
17
122.5
158.2
12
653.3K
ITW
Real
207.0
71.7
145.6
182.1
158.5
SPEC
72.0K
1TW
Sya
Y:s
146.1
154.
197.8
1600
CrowdPox
34
25.5K
1TW
Real
NoA
207.1
156.9
156.3
154.5
160.9
JNF 3D13P [
939.8K
1TW
Real
NoA
142.7
131.6
155.5
163.6
HumanSCDD
15
253.4K
Studio
Real
Yes
167.3
113.0
168.1
PoseTrack
[]
23.5K
1TW
Real
EFT
2181
180.
150.2
149.
172.0
DEHAVE
64.4K
Indoor
Real
Yes
208.5
205.6
20
1758
132.0
15
145.0
173.
CHIND [1]
252.4K
Stutio
Real
Yes
203.5
264
25
175.7
122.
11
121.0
127.5
Hamand.nk 3.M [1]
312.2K
Stilio
Real
NeA
226.0
21
376
36
200.6
24
112.3
6
1201
187.2
998.1K
Stulio
Real
Vis
250.0
22
278
27
179.3
20
134.5
19
149.7
20
194.3
3DPW [55
22.7K
ITW
Real
Yo
NeA
234.0
23
250.3
23
192.6
23
140.6
22
142.9
17
207.2
ARCTIC 1B
L530.1K 1K
Stulio
Real
Yes
308.5
29
210.7
19
196.4
22
202.5
26
182.5
25
216.1
DNA-R.1 H
3992,0K
Studio
Real
Yes
274.7
25
341.5
30
214.4
27
138.4
20
115.5
216.9
UP3D 52
7.1K
ITW
Real
UP3D
257.5
24
224.1
21
216.6
28
211.5
27
194.8
26
220.9
Talkshow [9
3326.9K
Indoor
Real
Yes
286-4
27
133.2
203.6
25
291.3
29
201.9
28
225.3
FTE3D [b
1779 3X
Studio
Real
Yes
329.7 7
30
404.0
31
213.8
26
132.1
16
148.1
19
245.5
MIP [m]
3.2K
LTW
Real
Yes
Yes
2727
25
264.9
28
2732
29
265 2
28
244.6
29
268
OCIlimann
2.5K
ITW
Real
EFT
307.1
25
263.3
24
2793
30
293.
30
281.7
30
285.0
LSPET [3]
2.9K
ITW
Real
EFT
365 7
51
292.6
20
$40.1
31
3.9.
31
116.3
51
330.
SSP113 13
311
ITW
Real
Via
5498
82
522.4
32
$48.1
92
439.0
32
599.5
82
5198
Evaluation Metries. We use standard metrics for EHPS. PVE (per-vertex error) and MPJPE (mean per-joint position error) measure the mean 1.2 error for vertices and regressed joints, respectively. The "PA" prefix indicates Procrutes Alignment is conducted before error computation. AGORA Leaderboard (48) introduces NMVE (normalized mean vertex error) and NMJE (normalized mean joint error) that take detection performance Fl score into consideration. Moreover, (mean primary error) that takes the mean of multiple primary metrics (MPJPE for 3DPW [56] test, and PVE for AGORA, UBody, EgoBody, and EHF) to gauge generalizability. All errors are reported in millimeters (mm).
propose MPE
3.2 Overview of Data Sources
In this work. we study three major types of datasets. 1) motion capture datasets that leverage optical [4]] [2 15 16 (2) or vision- [66 17 5 multi-view motion capture systems, are typically collected in a studio environment. However, is possible to include an outdoor setup, or utilize additional sensors such as IMUs (56). These datasets generally provide high-quality 3D annotations but are less flexible due to physical constraints, especially those built with immobile capture systems that require accurate sensor calibrations. 2) aecudo-annotate datasets [38 [ (34]
37 4
27
67
2
53 that e--anotaae existing image datasets with parametric human These datasets take advantage of the diversity of 2D datasets, and the
annotations 24
45 37
pseudo-3D annotations, albeit typically not as high-quality, have been proven effective [4 31 24
30 48 61 that are produced with renderings engines (e.g.. Unreal Engine). These datasets produce the most accurate 3D annotations and can casily scale up with high diversity. However, the synthetic-real gap is not fully addressed. Key attributes of the datasets are included in Table
3) synthetic datasets 4 6
To evaluate the EHPS capability across diverse scenarios, we select multiple key datasets to form a comprehensive benchmark. They should possess the desirable traits such as 1) having accurate SMPL or SMPL-X annotations, 2) being representative of certain aspects of real-life scenarios, 3) being widely used, but this requirement is relaxed for the new datasets which are released within two years, and 4) has a clearly defined test set. To this end, five datasets (AGORA 4% UBody M7 EgoBody [66], 3DPW (56). and EHF (49) representing different aspects are selected as the evaluation datasets. We briefly introduce these five datasets and the rest in the Supplementary Material. AGORA

---

is the most uidely-used benchmark for SMPL-X evaluation. It is a synthetic dataset featuring diverse subject appearances, poses, and environments with high-quality annotation. We evaluate on both validation and test set (leaderboard) as the latter has monthly limit of submissions. UBody is the latest large-scale dataset with pseudo-SMPI-X annotations that covers fifteen real-life scenarios, such as talk shows, video conferences, and vlogs, which primarily consist of the upper body in images. We follow the intra-scene protocol in training and testing, where all scenarios are seen. EgoBody captures human motions in social interactions in 3D scenes with Secudo-SMPL-X annotations. It comprises a first-person egocentric set (EgoSet) and a third-person multi-camera set (MVSet). We test on the EgoSet with heavy truncation and invisibility. 3DPW is the most popular in mo-the-wild dataset with SMPL annotations. Since SMPL--X annotation is not available, we map SMPL-X keypoints and test on 14 LSP 22 keypoints following the conventional protocol (26 31. EHF is a classic dataset with 100 curated frames of one subject in an indoor studio setup, with diverse body poses and especially hand poses annotated in SMPL-X vertices. It has test set but training or validation sets. Hence, it is only used to evaluate cross-dataset performance.
Besides being popular or the latest evaluation sets for EHPS, we further analyze if these five datasets collectively provide wide coverage of existing datasets. In Fig. 3 we randomly downsample all datasets to equal length (IK examples) and employ UMAP 41 to Visualize several key aspects. We use pretrained ViT-L from HumanBench (55) and OSX 37 to process patch tokens flattened as feature vectors from images cropped by bounding boxes. HumanBench is trained for various human- centric tasks (e.g., Re-ID, part segmentation, and 2D pose estimation), whereas OSX is an expert model on EHPS. As for global orientation, it is closely associated with camera pose as we convert all data into the camera coordinate frame; we plot its distribution by using flattened rotation matrix representations. Moreover, we follow 50 6 47 to represent poses as 3D keypoints regressed from the parametric model. Specifically, we flatten 21 SMPL-X body keypoints, and 15 hand keypoints from each hand, regressed with zero parameters except for the body pose and hand poses. It is shown that 1) the five benchmark datasets have varied distribution, which is expected due to their different designated purposes, and 2) collectively, the five datasets provide a wide, near-complete coverage of the entire dataset pool.
3.3 Benchmarking on Individual Datasets
In this section, we aim to benchmark datasets and find those that do well in various scenarios. To gauge the performance of each dataset, train SMPLer-X model with the training set of that dataset and evaluate the model on the val/testing sets of five evaluation datasets: AGORA, UBody, EgoBody, 3DPW, and EHF. Here, the benchmarking model is standardized to use ViT-S as the backbone, trained on 4 V100 GPUs for 5 epochs with a total batch size of 128 and a learning rate of 1 10-5 The dataset preprocessing details are included in the Supplementary Material.
In Table ! we report the primary metrics (Sec. (.Ic.B.I} and ranking of the 32 datasets. The complete results in the Supplementary Material. We also compute the mean primary error (MPE) to facilitate easy comparison between individual datasets. Note that for AGORA, UBody, EgoBody, and 3DPW, their performances on their own test set are excluded from computing MPE. This is because in- domain evaluation results are typically much better than cross-domale ones, leading to significant error drops. In addition, note that there datasets designed for specific purposes (e.g., Talkshow [62] for gesture generation, PONA-Renderng 7 for human NeRF reconstruction). being ranked lower on our benchmark, which focuses on EHPS (a perception task) does reduce their unique values and contributions to the computer vision community.
From the benchmark, we observe models trained single dataset tend to perform well on the same domain but often cannot do well on other domains. For example, the model trained on AGORA is ranked 1st on AGORA (val), but 6th on UBody, 6th on EgoBody, 12th on 3DPW, and 24th on EHF, This observation indicates that 1) the test scenarios are diverse, showcasing the challenging landscape of EHPS, and 2) data scaling is essential for training a robust and transferable model for EHPS due to significant gaps between different domains.
3.4 Analyses on Dataset Attributes
In this section, we study attributes that contribute to generalizability. However, it is important to acknowledge that such analyses are not straightforward task: the attributes often exhibit coupled
5

---

25
Benchmark Ranking
23-
Count
Count
Count
bot
2-
2-
Rall Sosdaci
10'
Top 12
Next 10
Betori to
Top
Nec. 10
Betton 32
Top Nos 12 Budom 10
Number Training Instances (Log Seale)
b) Scenes
c) ReallSyythetie
4) Annotation
Figure 3: Analysis on dataset attributes. We study the impact of a) the number of training instances, b) scenes, c) real or synthetic appearance, and d) annotation type, on dataset ranking in Table
effects. Consequently, counter-examplae are inevitable (e.g., we observe that Insta Variety, an in-the- wild dataset, demonstrates strong performance, whereas LSPET, another in-the-wild dataset, does not perform as well). Despite the challenges in pinpointing the exact factors that determine the success of an individual dataset, we adopt collective perspective and aim to identify general trends with several key factors [47 39 6 in Fig. 3. and discussed below.
First, Fig. 3R) shows that the performance of a dataset (in terms of ranking) is not strongly as- sociated with the number of training instances once the instance number exceeds approximately 100K. Although a very small amount of training data is insufficient to train strong model, having an exceedingly large amount of data does not guarantee good performance either. For example, MSCOCO only comprises 149.8K training instances but achieves higher ranking compared to datasets with 10x larger scales. This may be attributed to the diverse appearance and complex scenes present in the MSCÓCO dataset. Hence, would be more cost- cost-effective to channel resources to improve diversity and quality, when the dataset has become adequately large.
Second, we categorize datasets into 1) in-the-wild, which contains data from diverse environments; 2) indoor with several scenes; 3) studio, which has fixed multi-view setup. Particularly, Fig. (3)) shows that the top 10 are mostly in-the-wild datasets, indoor datasets concentrate in the top 20 and the studio dataset tends to be ranked lower in the benchmark. Moreover, Fig. 2 illustrates that in-the-wild datasets exhibit the most diverse distribution, covering both indoor and studio datasets. Indoor datasets display reasonable spread, and studio datasets have the least diversity. Our findings validate previous studies that suggest an indoor-outdon domain gap 25). Differing from Pang er al. [47], which does not differentiate between indoor and studio datasets, we argue that categorizing all datasets collected indoors into single class oversimplifies the analysis. For example, consider EgoBody (66] and Human3.6M, [21). Both datasets does not have outdoor data; however, EgoBody consists of a wide variety of indoor scenes, whereas Human3.6M consists of only one scene, which may contribute the better ranking of EgoBody compared to Human3. 6M. Hence, this suggests that in-the-wild data collection is the most idcal, but diversifying indoor scenes is the best alternative.
Third, most of the five contemporary synthetic datasets 4 strength and are ranked highly in Fig. 33). It worth noting that four (UBody, EgoBody, 3DPW, and EHF) of the five evaluation benchmarks used are real datasets, indicating that knowledge learned from synthetic data is transferable to real scenarios. To explain this observation, we take a close look at Fig. 2) although real and synthetic datasets do not have extensive overlap, synthetic data possesses two ideal characteristics. First, there is a high overlap between real and synthetic data at the rightmost cluster. Referring to Fig. 2 which is drawn from the same distribution, we find that this cluster primarily represents in-the-wild data. Therefore, synthetic data includes a substantial number of in-the-will wild images that closely resemble real wi-the-wild scenarios. Second, synthetic data also have seatters image features on other clusters, indicating that synthetic data provides coverage to some extent for various real-world scenarios.
61 48
30 6 demonstrate surprising
Fourth, Fig. 3d) reveals that dataset can be valuable with accurate or Aseudo-SMPLE annotations, as they constitute the most of the top 10 datasets. A prominent example is Insta Insta Variety (27]. which has only pseudo-SMPL-X annotation produced by NeuralAnnot [45, yet, is ranked third in our benchmark. However, due to the differences in parameter spaces, SMPI. annotations are less effective:
observed that datasets with SMPL. annotations tend to cluster in the lower bracket of the benchmark, especially those with pseudo-SMPL annotations. This observation suggests that SMPL-X

---

Table 2: Foundation Models, We study the scaling law of the amount of data and the model sizes. The metrics are MPIPE for 3DPW, and PVE for other evaluation benchmarks. Foundation models are named "SMPLer- X-MN", where M indicates the size of VIT backbone (S, B, L., H), N is the number of datasets used the training. FPS: inference speed (frames per second) on a V100 GPU. MPE: mean primary emror AGORA uses the validation set, and EgoBody uses the EgoSet.
#aattaset
Afnst.
Model
#Param
FPS
AGORA [8]
EgoBBdd (6)
UBody
37
3DPW 56
EHF 49)
MPF
0.75M
SMPLer-X-S5
32M
36.2
119.0
114.2
110.1
110.2
100.5
110.8
10
1.5M
SMPLer-X-S10
32M
36.2
116.0
88.6
107.7
97.4
89.9
99.9
SMPLer-X-S20
32
4.5M
SMPLor-X-S32
32M
36.2
105.2
82.5
68.1
83.2
74.1
82.6
SMPLer
103.5
10
1.5M
SMP11:X--B10
103M
33.1
97.8
76.4
107.3
89.9
74.7
89.2
3.0M
SMPLerX-B2I
105M
33.1
95.6
75.5
65.3
83.5
73.0
78.6
4.5M
SMPLe----B32
103M
33.1
88.0
72.7
63.3
80.3
67.3
74.3
0.75M
SMPLer-X-LS
327M
24.4
88.3
98.7
110.8
97.8
89.5
97.0
1.5M
SMPLer-X-L10
527M
24.4
82.6
69.7
104.0
82.5
64.0
20
3.0M
SMP
327M
24.4
80.7
66.6
61.55
78.3
65.4
70.5
SMPLcc-X-L32
327M
62.2
62.4
0.75M
AMPLe:X-115
662M
17.5
89.0
87.4
102.1
88.3
68.3
87.0
SMPLer---H10
3.0M
SMPLer X-1100
662M
17.5
77.5
63.5
59.9
74.4
59.4
67.0
4.5M
SMPLer-X-H52
662M
17.5
69.5
59.5
54.5
75.0
56.8
63.1
annotations are critical to EHPS; fitting pseudo labels is a useful strategy even if they could be noisy. Moreover, using SMPL labels effectively for SMPL-X estimation remains a challenge.
Scaling up EHPS
4.1 Model Architectures
Catering to our investigation, we design a minimalistic framework (dubbed SMPLer-X) that only retains the most essential parts for two rea- sons. First, it must be scalable and ef- ficient as we train with large amount of data. Second, we aim to create framework that decoupled from spe- cific algorithm designs, providing clean foundation for future research. To this end, SMPLer-X consists of three parts: a backbone extracts im- age features, which employ Vision Transformer [[T] for its scalability; a neck that predicts bounding boxes and crop regions of interest from the fea- ture map for hands and face; regres- sion heads that estimate parameters for each part. Note that SMPLer-X does not require third-party detectors [51), cross-part feature interaction modules 13 projection coarse SMPL-X estima- tions (63] or heavy decoder (37). As the design of SMPLer-X is not the focus of our investigation, more details are included in the Supplementary Material.
Ting
Timo
Hand Hizd
ROI
VII
Face Head
Proainal
Head
Taan
Toot
Bacibone
Neck
Figure 4: Architecture of SMPLer-X, which upholds the idea that "simplicity is beauty". SMPLer-X contains a back- bone that allows for easy investigation model scaling, neck for hand and face feature cropping, and heads for differ- body parts. Note that we wish show in this work that model and data scaling are effective, even with a straightfor- ward architecture.
4.2 Training the Generalist Foundation Models
The SOTA methods 37 usually train with only a few (e.g., MSCOCO, MPII, and Human3. 6M) datasets, whereas we investigate training with many more datasets. However, highlight that the dataset benchmark in Table cannot be used: selecting datasets based on their performance on the test sets the evaluation benchmarks leaks information about test sets. Hence, we construct another dataset benchmark in the Supplementary Material, that ranks individual datasets on the training set of the major EHPS benchmarks. We use four data amounts: 5, 10, 20, and 32 datasets as the training set, with total length of 0.75M, 1.5M, 3.0M, and 4.5M instances. We always prioritize higher-ranked

---

Table 3: AGORA test set. i denotes the methods that are finetuned on the AGORA training set. *denotes the methods that are trained on AGORA training set only.
NMVEL (nom)
NMJEL (mmn)
MVE. (mmn)
MPJPE, (mum)
Method
All
Body
All
Body
All
Body
Face
LHHand
RHandd
All
Body
Faze
LHHand
RHHhand
BEDLAM
179.5
177.5
131.0
25.8
38.8
95.9
Hand
144.
96.0
141.1
92.7
135.5
90.2
41.6
46.3
48.1
132.6
87.1
46.1
44.3
46.2
BEDLAM
4
142.2
102.1
141.0
101.8
103.8
74.5
23.1
31.7
33.2
102.9
74.3
24.7
29.9
31.3
141.2
140.0
125.7
37.9
OSX
37
130.6
85.3
127.6
83.3
122.8
80.2
36.2
45.4
46.1
119.9
78.3
37.9
43.0
43.9
HybrIK-X 33
120.5
73.7
115.7
72.3
112.1
68.5
37.0
46.7
47.0
107.6
67.2
38.5
41.2
41.4
SMPLer-X-L20
133.1
88.
128.9
84.6
123.8
81.9
37.4
43.6
44.8
119.
78.7
39.5
41.4
44.8
SMPLer-X-L32
122.8
80.3
119.1
77.6
114.2
74.7
35.1
41.3
42.2
110.8
72.2
36.7
39.1
40.1
SMP1.er---1.00
107.2
68.3
104.1
66.3
99.7
63.5
29.9
39.1
39.5
96.8
61.7
31.4
36.7
37.2
Table 4: AGORA Val set. f and are finetuned on the AGORA training set, and trained on the AGORA training set only, respectively.
Table 5: EHF. As EHF does not have a training set to benchmark datasets, we do not perform finetuming. Moreover, EHF is not seen in our training and can be used to validate our founda- tion models' transferability.
PA-PYE] (amm)
Method
All
Hunds
Face
All
Hands
Face
HandeWhole H+1
73.2
9.7
4.7
183.9
72.8
81.6
PA-PVEL (mon)
PVE. (mm))
OSX
45.0
11.5
8.5
4.8
3.9
168.6
79.6
48.2
37.9
Method
Hands
Face
Face
SMP er--11--11
48.9
8.6
4.0
86.1
$1.5
41.2
WanddWhole 44
50.3
10.8
5.8
76.8
39.8
26.1
SMPLa--X-120
OSX
48.7
26.4
SMPLer-X-152
45.1
8.7
3.8
74.2
47.8
38.7
SMPLe-X-La
X--20
37.8
15.0
5.1
65.4
49.4
17.4
SMPLer-X-L20f
39.1
9.3
3.8
62.5
42.3
32.8
SMPLer-
37.1
14.1
530
62.4
47.1
17.0
datasets. To prevent larger datasets from shadowing smaller datasets, we adopt a balanced sampling strategy. Specifically, all selected datasets are uniformly upsampled or downsampled to the same length and add up to the designated total length. To facilitate training, we follow OSX (37 to use AGORA, UBody, MPII, 3DPW, Human3.6N in COCO-format [38), and standardize all other datasets into the HumanData [0] format. We also study four ViT backbones of different sizes Vill-mmll,, Base, Large and Huge), pretrained by ViTPose (60]. The training is conducted 16 V 100 GPUs, with total batch size of 512 (256 for ViT-Huge) for 10 epochs. More training details such as adapting SMPL or gendered SMPL-X in the training are included in the Supplementary Material.
In Table 2, we show experimental results with a various number of datasets and foundation model sizes. Founda- tion models are named "SMPLer-X-MN", where M can be S, B, L, H that indicates the size of the ViT backbone, and N indicates the number of datasets used the train- ing. For example, EMPLer-X-L10 means the foundation model takes ViT-L as the backbone, and is trained with Top 10 datasets (ranked according to the individual dataset performance on the training sets of the key evaluation benchmarks). It is observed that 1) more training data (data scaling) leads to better performance in terms of MPE, The model performance improves gradually the number of training datasets increases. However, besides the incre- ment in training instances, more datasets provide a richer collection of diverse scenarios, which we argue also a key contributor to the performance gain across evaluation benchmarks. 2) A larger foundation model (model scaling) performs better at any given amount of data. However, the marginal benefits of scaling up decrease beyond model size L. Specifically, ViT-H has more than twice the parameters than VIT-L, but the performance gain is not prominent. 3) The foundation model always performs better than in-domain training on single training set. For example, SMPLer-XX --20., performs better on the validation set of AGORA, and test sets of UBody, EgoBody, and 3DPW, than models trained specifically on the corresponding training set in Table This useful for real-life applications: instead of training a model for each the user cases, generalist foundation model contains rich knowledge to be one-size-fits alternative.
Table 10: 3DPW. & denotes the methods that use a head for SMPL regression. and are finetuned on the 3DPW train- ing set and trained on 3DPW training set only, respectively. Unit: mm.
Method
MPJPE PA-MPIPE
Body-only (SMPL) Methods
OSX-SMPL t
74.7
45.1
HybrIK 35
71.6
41.8
CL.FFF 36
68.0
43.0
Whole-Body (SMPL-X) Methods
Hand4Whole 44
86.6
54.4
ExPose H
93.4
60.7
OSX [37]
86.2
60.6
SMPLer-X-B1+
95.6
67.6
SMPLer-X-L20
78.3
52.1
SMPl.rrXX-132
75.2
50.5
SMPLer-X-L200
76.8
51.5
8

---

Table 6: UBody. denotes the methods that are finetuned on the UBody training set. denotes the methods that are trained on UBody training set only.
Table 7: EgoBody-EguSet. i denotes the meth- ods that are finetuned on the EgoBody-EgoSen training set. denotes the methods that trained on EgoBody-EgoSet training set only.
PA-PVE] (mmn)
PVEL (mom)
PA PVEP (mm)
PVEL (mme)
Metbod
All
Hands
Face
All
Hands
Face
Method
All
Hands
Face
All
Hands
Face
Hand4Whole
OSX
PIXIE
[7
13
44
42.4
44.8
10.8
3.9
2.4
104.1
92.4
47.7
45.7
24.9
27.0
OSX B
Hand4Whole
OSX 37
51.6
45.3
58.8
11.6
10.0
9.7
3.0
3.7
3.7
121.9
115.7
82.3
50.6
46.8
50.0
41.1
35.2
42.5
SMPLer-X-LL2
SMPLerX- 2-1
SMPLer.XX...2 -.r.x--X1..0 1.20
SMPLenXX-D X-Bl+
33.2
38.5
31.9
10.3
10.8
10.6
2,8
3.0
288
61.5
64.8
57.4
45.4
43.3
40.2
22.3
23.1
21.6
SMPL.arX-1.20f
SMPLe-X-L220
SMPLexX-L33
e---Ier-X-B1+
36.3
38.9
37.8
10.7
9.8
9.9
2.9
3.0
66.6
62.2
41.4
42.7
34,9
31.8
30.7
Table 8: ARCTIC. i and * denote the methods that are finetuned on the ARCTIC training set and trained on the ARCTIC training set only, respectively.
Table 9: Ree-RReendi--iiieee finetuned on the DNA-Rende -eee--eenndddniieeeeeee training set and trained the DNA- SeaA-Rendering-Hiiee training set only, respectively.
and
PA-PVEL (mm)
PVEL (mmm)
PA-PVEL (mm)
PVEL (mmm)
Method
All
Hands
Face
All
Hands
Face
Method
All
Hands
Face
All
Hands
Face
HanddWhole
11.0
OSX
37
56.9
17.5
3.9
102.6
56.5
44.6
OSX 37
59.9
10.6
4.3
105.7
55.0
52.5
OSX
3
33.0
18.8
3.3
58.4
39.4
30.4
OSX
43.5
7.5
3.5
67.1
43,3
38.2
SMPLer-X-Bl
SMPLer
SMPL.ar-X-L10
46.9
18.1
2.3
76.9
50.8
33.2
SMPLLar-X-L.0
44.4
11.1
4.5
77.7
47.5
43.2
SMPLerXX-L32
29.4
18.9
2.7
48.6
38.8
26.8
SMPLerX-L32
35.8
7.2
3.2
54.4
36.7
34,0
SMPLerX
33.1
27.3
SMPLexXX-L20f
37.9
2.3
3.4
56.5
38.4
Besides the errors on key benchmarks, we also report the inference speed (in terms FPS, or frames per second) of the SMPLer-X X model family in Table The testing is conducted on single V100 GPU with batch size 1, excluding data loading. SMPLer-X family is faster than OSX (12.2 FPS on single V100 GPU) using the same test setting, and the smaller versions such as SMPLer-X-S and SMPLer-X-a can achieve real-time performance, with SMPLer-X-L on the verge of achieving real-time speeds. The high inference speed is attributed to the minimalistic architecture of SMPLer-X, which only retains the most essential components for EHPS.
Moreover, we show detailed by-part results of body. hands, and face on main benchmarks such as AGORA test set (Table 3) AGORA validation set (Table 4), UBody (Table (6) Ego (Table 7) and EHF (Table 5). We also compare our results with whole-body methods on 3DPW (Table 10p. We highlight that the foundation models show strong and balanced performances on all benchmarks.
Furthermore, evaluate the transferability of foundation models two more benchmarks: ARCTIC (Table R) and DNA-Rendering (Table 8). ARCTIC features complicated hand-object interaction with whole-body annotations, and DNA-Rendering includes diverse subjects, motions, and garments. Note that ARCTIC is not seen by foundation models trained on Top 10 datasets, and DNA-Rendering is not seen by foundation models trained on Top 20 datasets. The foundation models, however, achieve much better performance than SOTAs with conventional data sampling strategies.
In addition, we compare foundation model with SOTA methods, such as Hand4Whole 44 and OSX (37 in various scenarios in Fig. 5 These scenarios feature challenging aspects such as heavy truncation (from only half of the body visible to only the arms visible), difficult body poses in diverse backgrounds, and rare camera angles (extremely high or low elevation angles). SMPLer-X demonstrates the strength of massive training data and consistently produces robust estimations.
4.3 Finetuning the Specialists
Training the foundation model with large number of data is expensive. For example, SMPLer-X- H20 takes more than 400 GPU hours to train. Hence, is critical to investigate finetuning strategies that allow for low-cost adaptation of the foundation model to specific scenarios. We reiterate that in real-life applications, the test set inaccessible. Hence, we use our benchmarking strategy and select five high-ranking datasets on the target training set to finetune the model for 5 epochs. We perform

---

Hand4Whold
OSX
SMPLer-X-L32
Whode
OSX
SMPLer-X- X-L3322
Figure 5: Visualization. We compare SMPLer-X-L32 with OSX 37 and Whnd4 44 (trained with the MSCOCO, MPII, and Human3.6M) in various scenarios such as those with heavy truncation, hard poses, and rare camera angles.
finetune experiments on ViT-L to match the backbone of current SOTA 37.. The results are shown in the same tables as the foundation models (Table 6 7 and TO), where finetuning always lead to substantial performance enhancement on the foundation models.
5 Conclusion
In this work, we benchmark datasets for EHPS that provide us insights for training and finetuning foundation model. Our work is useful in three ways. First, our pretrained model (especially the backbone) can be a plug--nn--laa component of larger system for EHPS and beyond. Sec- ond, our benchmark serves to gauge the performances of future generalization studies. Third, our cettimaarkinn-fintuuiii
paradigm can be useful for the rapid adaptation of any foundation model to specific scenarios. Specifically, users may collect training set, evaluate pretrained models of various other datasets on it, and select the most relevant datasets to finetune a foundation model.
five comprehensive benchmark datasets gauge the general- ization capability, they may still be insufficient to represent the real-world distribution. Second, our experiments do not fully investigate the impact of various model architectures due to the prohibitive cost of training the foundation model.
Limitations. First, although
Potential negative societal impact. As we study training strong EHPS models and release the pretrained models, they may be used for unwarranted surveillance privacy violation.
10

---

Acknowledgement
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). The project also supported by NTU NAP and Singapore MOE AcRF Tier 2 (MOET2EP20221- 0012).
References
[1] Mykhaylo Andriluka, Umar lqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5167-5176, 2018.
[2] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
[3] Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Poll-Moll. Behave: Dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15935-15946, 2022.
[4] Michael J Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8726-8737, 2023.
[5] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, et al. Humman: Multi-modal 4d human dataset for versatile sensing and modeling. In European Conference Computer Vision, pages 557-577. Springer, 2022.
[6] Zhongang Cai, Mingyuan Zhang, Jiawei Ren, Chen Wei, Daxuan Ren, Zhengyu Lin, Haiyu Zhao, Lei Yang, Chen Change Loy, and Ziwei Liu. Playing for 3d human recovery. arXiv preprint 10.05:21 2021.
[7] Wei Cheng, Ruixiang Chen, Siming Fan, Wanqi Yin, Keyu Chen, Zhongang Cai, Jingbo Wang. Yang Gao, Zhengming Yu, Zhengyu Lin, et al. Dna-rendering: A diverse neural actor repository for high-fidelity human-centric rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19982-19993, 2023.
[8] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Monocular expressive body regression through body-driven attention. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pages 20-40. Springer, 2020.
[9] Yuanzheng Ci, Yizhou Wang. Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihcp: A unified model for human-centric perceptions. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 17840-17852, 2023.
[10] MMHuman3D Contributors. Opemmmlab 3d human parametric model toolbox and benchmark. 2021.
https://githu/: www.://pe..mmmma//mmmmmeeeeeeeee
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[12] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctie: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12943-12954, 2023.
11

---

[13] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collabo- rative regression of expressive bodies using moderation. In 2021 International Conference on 3D Vision (3DV), pages 792-804. IEER, 2021.
[14] Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu. Three-dimensional reconstruction human interactions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7214-7223, 2020.
[15] Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-lonut Popa, Vlad Olaru, and Cristian Sminchisescu. Learning complex 3d human self contaccn In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1343- 1351, 2021.
[16] Mihai Fieraru, Mihai Zanfir, Silviu Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu. Aifit: Automatic 3d human- iitmaa-niterrretabll feedback models for fitness training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9919-9928, 2021.
[17] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2282-2292, 2019.
[18] Fangzhou Hong, Liang Pan, Zhongang Cai, and Ziwei Liu. Garment4d: Garment reconstruction from point cloud sequences. Advances in Neural Information Processing Systems, 34:27940- 27951, 2021.
[19] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang. and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Transactions on Graphics (TOG), -1(4):1-19, 2022.
[20] Chun-Hao P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13274-13285, 2022.
[21] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchiseseu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Iransactions on pattern analysis and machine intelligence, 36(7: 2013.
[22] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, pages 1-11. British Machine Vision Association, 2010.
[23] Sam Johnson and Mark Everingham. Learning effective human pose estimation from inaccurate annotation. In CVPR 20/1, pages 1475- IEEE, 2011.
[24] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation. In 202 International Conference on 3D Vision (3DV), pages 42-52. IEEE, 2021.
[25] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar fine tin-ttnnig for 3d human model fitting towards in wn-the-wild 3d human pose estimation. 3DV, 2022.
[26] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7122-7131, 2018
[27] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5614-5623, 2019.
[28] Muhammed Kocabas, Nikos Athanasiou, and Michael Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9253-5263, 2020.
[29] Muhammed Kocabas, Chun-Har Huang, Otmar Hilliges, and Michael J Black. Pare: Part attention regressor for 3d human body estimation. arXiv preprint arXiv:2104.08527,0 2021.
12

---

[30] Muhammed Kocabas, Chun-Hao P Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael Black. Spec: Seeing people in the wild with an estimated camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11035-11044 2021.
[31] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2252-2261, 2019.
[32] Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J Black, and Peter V Gehler. Unite the people: Closing the loop between 3d and 2d human representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6050- 6059, 2017.
[33] Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, and Cewu Lu. Hybrik-x: Hybrid analytical-neura inverse kinematics for whole-body mesh recovery. arXiv preprint M3KYv:2304.06990 2023.
[34] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Effi- cient crowded scenes pose estimation and new benchmark. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10863-10872, 2019.
[35] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In CVPR, pages 1383-3393. Computer Vision Foundation IEEE, 2021.
[36] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In European Conference Computer Vision, pages 590 900-606.600 Springer, 2022.
[37] Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1115-221166 2023.
[38] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[39] Qihao Liu, Adam Kortylewski, and Alan L. Yuille. Poseexaminer: Automated testing of out-of- distribution robustness in human pose and shape estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 672-681, 2023.
[40] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[41] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: #0Xiv:/802.03426, 2018. 8.
[42] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko. Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 2017 international conference on 3D vision (3DV), pages 506-516. IEEE, 2017.
[43] Dushyant Mehta, Oleksandr Sotnychenko, Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and C. Theobalt. Single-shot multi-person 3d pose estimation from monocular rgh. 2018 International Conference on 3D) Vision (3DV), pages 120-130, 2018.
[44] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Accurate 3d hand pose estimation for whole-body 3d human mesh estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2308-2317, 2022.
[45] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Neuralannot: Neural annotator for 3d human mesh training sets. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2299-2307, 2022.
13

---

[46] Lea Muller, Ahmed AA Osman, Siyu Tang, Chun-Hao P Huang, and Michael J Black. On self-contact and human pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9990-9999, 2021.
[47] Hui En Pang, Zhongang Cai, Lei Yang, Tianwei Zhang. and Ziwei Liu. Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[48] Priyanka Patel, Chun-Hao P Huang, Joachim Tesch, David T Hoffmann, Shashank Tripathi, and Michael Black. AGORA: Avatars geography optimized for regression analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13468 13478, 2021.
[49] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[50] Yu Rong, Ziwei Liu, Cheng Li, Kaidi Cao, and Chen Change Loy. Delving deep into hybrid annotations for 3d human recovery in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5340-5348, 2019.
[51] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749-1759, 2021.
[52] István Sárándi, Alexander Hermans, and Bastian Leibe. Learning 3d human pose estimation from dozens of datasets using a geometry-aware autoencoder bridge between skeleton formats. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2956-2966, 2023.
[53] Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Synthetic training for accurate 3d human pose and shape estimation in the wild. In British Machine Vision Conference (BMVC), September 2020.
[54] Yu Sun, Tianyu Huang, Qian Bao, Wu Liu, Wenpeng Gao, and Yili Fu. Learning monocular mesh recovery of multiple body parts via synthesis. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2022.
IEEE,
[55] Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang, Li Yi, et al. Humanbench: Towards general human-centric perception with projector assisted pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21970-21982, 2023.
[56] Timo von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons- Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European Conference on Computer Vision (ECCV), pages 601-617, 2018.
[57] Wenjia Wang, Yongtao Ge, Haiyi Mei, Zhongang Cai, Qingping Sun, Yanjun Wang, Chunhua Shen, Lei Yang, and Taku Komura. Zolly: Zoom focal length correctly for perspectiie-ddisorree human mesh reconstruction. arXiv preprint 2303. 2023.
[58] Yanjun Wang, Qingping Sun, Wenjia Wang, Jun Ling, Zhongang Cai, Rong Xie, and Li Song. Learning dense uv completion for human mesh recovery. arXiv preprint arXiv:2307.1 1007, 2023.
[59] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965-10974, 2019.
[60] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 15:38571-38584, 2022.
14

---

[61] Zhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu, Zhaoxi Chen, Weiye Xiao, Yukun Wei, Zhongfei Qing, Chen Wei. Bo Dai, et al. Synbody: Synthetic dataset with layered human models for 3d human perception and modeling. arXiv preprint M.Niv:2303.17300 2023.
[62] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael Black. Generating holistic 3d human motion from speech. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 469-4800 40094400,400 June 2023.
[63] Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[64] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
[65] Mingyuan Zhang. Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong. Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmentee motion diffusion model. arXiv preprint arXiv:2304.01116. 2023.
[66] Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Tacin Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 20-2000.55 Springer, 2022.
[67] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang. Haozhi Huang, and Shi-Min Hu. Pose2seg: Detection free human instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2019.
[68] Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu. Monocular real-time full body capture with inter-part correlations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4811-4822. 2021.
15


Reviews:
[{"review_id": "w9TsSHBYud", "reviewer": "Reviewer_er9Y", "summary": "", "questions": "", "limitations": "The authors adequately describe the limitations of their work. 1) The comprehensive benchmark datasets is insufficient to represent the real-world distribution. 2) Finetuning harms the generalization capability of foundation models.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "To achieve a general and accurate EHPS, this paper explores more data combinations and model scaling methods. The proposed pretrained model is generalist foundation model across various scenarios and can be finetuned for any specific scenarios. The motivation for this paper is well justified, and implemented, and the overall organization of the paper flows well.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HA9CvSRXeO", "reviewer": "Reviewer_nrrt", "summary": "", "questions": "", "limitations": "The potential negative societal impact is well discussed in the paper", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The investigated problem in this paper is intriguing and important.\n\n2. The authors' efforts in data and model scaling are commendable. The presented approach surpasses previous work with significant performance improvements, highlighting the strength of their methodology.\n\n3. The paper is well-written, and the authors have made their code openly available The accompanying document is also comprehensive, providing sufficient information.", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FISz19XZbq", "reviewer": "Reviewer_6mpM", "summary": "", "questions": "", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Comprehensive analyses of existing 3D human mesh datasets, which provide the community with valuable tips for future dataset collection.\n\n2. By benchmarking existing datasets, this paper provides several insights for training and finetuning a foundation model.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "e7t1AZWuSv", "reviewer": "Reviewer_uVnR", "summary": "", "questions": "", "limitations": "The paper proposed the dataset and it seemingly leads to better performance compared to others. However, there is no much visualization for the samples and it is hard to judge there exists diverse combination of body, hands and face motions.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset is quite large.\nThe performance obtained is clearly better than others.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "lky4SV7BOu", "reviewer": "Reviewer_5yuA", "summary": "", "questions": "", "limitations": "The authors have discussed this briefly in their paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The author's design of SMPLer-X demonstrates the immense potential of large models and highlights the significance of combining large models with large datasets.\n2. The author extensively researched various datasets and conducted detailed analyses of their attributes.\n3. The abundant experimental evidence presented in the supplementary materials validates the comprehensiveness of the experiments.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper got primarily positive ratings from reviewers, with scores of 7, 7, 7, 6, and 4. Reviewers who gave positive ratings appreciated the paper's exploration of the existing 32 datasets for Expressive Human Pose and Shape (EHPS) estimation. Reviewers viewed it as a valuable contribution, especially from the perspective of a foundation model, as it demonstrated robust performance in related benchmarks.

A reviewer who gave a negative rating expressed concerns about the paper's lack of presenting new data. In the authors' rebuttal, the authors defended their position, addressing that the paper still aligns with the NeurIPS Dataset and Benchmark track's goals.

Upon reviewing the D&B track's motivation, the AC concurred with the authors' perspective. The AC believes that the current paper effectively fulfills the track's objectives by offering a fresh viewpoint on consolidating and exploring the extensive array of existing datasets, thereby creating new value for the community, especially in the direction of building a foundation model for EHPS. The advantage of the proposed direction is well-supported by the experiments.

Considering the recognition of the paper's contributions by other reviewers, the AC concludes that the current version of the paper meets the bar of NeurIPS Datasets and Benchmarks Track. The authors are encouraged to carefully review the feedback provided by the reviewers and take steps to enhance the paper's quality for the camera-ready version.
Decision: Accept (Poster)