SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Second, we investigate ten data augmentations, and classify them into three categories (see Figure 7 for the Body subnetwork, and Figure 29 in Appendix for Hand subnetwork): (1) image-variant augmentations such as color jittering, blur, occlusion and background swapping; (2) location-variant augmentations involving translation and scaling; (3) pose-variant augmentations including rotation and horizontal flipping. Our ablation study in Table 10 shows that augmentations with varied global orientation are detrimental to the model performance. Consequently, we exclude such modifications when constructing positive pairs. Instead, each positive sample is constructed utilizing a random combination of location-variant and color-variant augmentations.
Formally, for a batch of  $N$  images, we construct another  $N$  images by applying augmentation to each sample. For each anchor  $i$ , let  $j$  be the corresponding augmented sample. Then  $i$  is contrasted against  $2N - 1$  terms (1 positive and  $2N - 2$  negatives). The loss takes the following form:
$$
\mathcal {L} _ {\text {c o n}} = \sum_ {i = 1} ^ {N} \left(\tau_ {\text {p o s}} \left(| \mathrm {d} \left(\boldsymbol {p} _ {i}, \boldsymbol {p} _ {j}\right) - \mathrm {d} \left(\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}\right) |\right) + \tau_ {\text {n e g}} \sum_ {k = 1} ^ {2 N} \mathbb {1} _ {[ k \neq i, j ]} \left(| \mathrm {d} \left(\boldsymbol {p} _ {i}, \boldsymbol {p} _ {k}\right) - \mathrm {d} \left(\boldsymbol {z} _ {i}, \boldsymbol {z} _ {k}\right) |\right)\right) \tag {2}
$$
where  $\mathbf{z}_i, \mathbf{z}_j$  and  $\mathbf{z}_k$  denote the predicted pose representations, and  $\mathbf{p}_i, \mathbf{p}_j$  and  $\mathbf{p}_k$  denote the ground-truth pose representations for the anchor, positive and negative samples in the batch. The objective of this loss function is to minimize the distance between the positive pairs and maximize the distance between the negative pairs, in alignment with the pose similarity. Note that unlike traditional approaches where the distance is the same for all negative samples, the pairwise distance  $\mathrm{d}(\mathbf{p}_i, \mathbf{p}_k)$  varies depending on the pose similarity.
4.4 Pixel Alignment Module
This module employs differentiable rendering to ensure that the projected mesh aligns precisely at the pixel level. The alignment is supervised by the projected mask loss. Attaining a proper alignment between the ground-truth part segmentation and rendered mesh requires the accurate prediction of pose, shape, and camera parameters, which subsequently leads to a more precise estimation process.
Datasets. For whole-body training, we employ Human3.6M (H36M) [13], COCO-Wholebody [14] (the whole-body version of MSCOCO [29]) and MPII [1]. The 3D pseudo-ground truths for training are acquired using NeuralAnnot [36]. For hand-specific training, we use FreiHAND [62], Interhand [34] and COCO-Wholebody Hands [14]. For face-specific training, we use FFHQ [18], BUPT [52] and AffectNet [32]. For evaluations specific to 3D body, 3D hand, and 3D face, we utilize 3DPW [50], FreiHAND [62], and Stirling [11], respectively. For the 3D whole-body evaluation, we use EHF [41] and AGORA [39]. Additionally, we present qualitative results on the MSCOCO validation set.
Metrics. Mean Per Joint Position Error (MPJPE) and Mean Per-Vertex Position Error (MPVPE) are employed to evaluate the positions of 3D joint and mesh vertices, respectively. Each metric calculates the average 3D joint distance (in mm) and 3D mesh vertex distance (in mm) between the predicted and ground-truth values after aligning the root joint translation. The pelvis serves as the root joint for whole-body and body, whereas the wrists and neck are utilized as root joints for hands and face. Procrustes Aligned (PA) variants of these metrics, PA-MPJPE and PA-MPVPE, further align with rotation and scale. We report the average errors for the left and right hands as the 3D hand error.
Table 1: Evaluation of the Hand subnetwork.
Method	PA-PVE ↓	PA-MPJPE ↓	F-Scores ↑
* Hand-only
FreiHAND [62]	10.7	-	0.529/0.935
Pose2Mesh [4]	7.8	0.674/0.969
I2L-MeshNet [33]	7.6	0.681/0.973
METRO (HR64) [28]	6.7	0.717/0.981
* Whole-body
ExPose [41]	11.8	0.484/0.918
Zhou et al. [61]	-	-/-
FrankMocap [44]	11.6	0.553/0.951
PIXIE [11]	12.1	0.468/0.919
Hand4Whole † [35]	7.7	0.664/0.971
HMR (Baseline) [17]	8.6	0.605/0.963
PyMAF [58]	8.1	0.638/0.969
PyMAF † [58]	7.5	0.671/0.974
RoboSMPLX	7.3	0.683/0.976
RoboSMPLX †	7.1	0.688/0.978
RoboSMPLX (HR64)	6.7	0.715/0.981
Table 2: Evaluation of the Body subnetwork.
Method	PA-MPJPE ↓	MPJPE ↓	PVE ↓
HMR (Res50) [17]	76.7	130	-
GraphCMR (Res50) [24]	70.2	-	-
SPIN (Res50) [22]	59.2	96.9	116.4
HMR-EFT (Res50) [16]	54.3	-	-
ROMP (Res50)	53.5	89.3	105.6
PARE (Res50) [20]	52.3	82.9	99.7
PARE (HR32) [20]	50.9	82	97.9
PyMAF (Res50)[58]	49.0	79.7	94.4
PyMAF (HR48) [58]	47.1	78.0	91.3
Baseline (Res50)	52.4	85.2	103.6
RoboSMPLX (Res50)	49.8	80.8	96.7
Baseline (HR48)	50.3	84.5	101.5
RoboSMPLX (HR48)	48.5	80.1	95.2
Table 3: Evaluation of the Face subnetwork.
Method	LQ Mean(mm) ↓	HQ Mean(mm) ↓
ExPose [6]	2.27	2.42
ExPose †	2.46	2.38
HMR	2.18	2.11
HMR †	2.31	2.27
HMR *	2.02	2.04
PyMAF *	1.97	1.92
RoboSMPLX	2.12	2.08
RoboSMPLX †	2.12	2.10
Table 4: PA-PVE/PVE errors of the Hand subnetwork under different positional augmentations.
Normal	Transx +0.2x	Transx -0.2x	Transy +0.2y	Transy -0.2y	Scale 1.3x	Scale 0.7x
Hand4Whole [35]	7.47/ 15.70	8.51/ 21.58	8.38/ 20.36	8.74/ 22.51	8.48/ 19.85	7.73/ 16.44	7.78/ 17.00
RoboSMPLX	7.24/ 15.23	7.27/ 15.62	7.36/ 15.59	7.28/ 15.50	7.34/ 15.50	7.49/ 15.90	7.45/ 16.51
Hand Subnetwork. Table 1 compares the performance of the Hand subnetwork with different hand-only and whole-body methods. Our method outperforms that of our whole-body counterparts when trained with only the FreiHAND dataset (i.e._PIXE, Hand4Whole, PyMAF) or under mixed datasets (i.e. Hand4Whole  $\dagger$ , PyMAF  $\dagger$ ) using an identical backbone. Prior research [33, 48] demonstrated that whole-body methods generally employ a parametric representation of the hand mesh, and are numerically inferior to the non-parametric representation used in recent hand-only methods [33, 28]. Despite such reported gap, RoboSMPLX manages to outperform mesh-based techniques, and achieve comparable results as the state-of-the-art METRO when using the same backbone (HRNet-64). Table 4 compares the estimation errors of the Hand subnetwork in Hand4Whole (current whole-body method with SOTA on hands) and RoboSMPLX under different positional augmentations on the FreiHAND test set. It is clear that RoboSMPLX exhibits much better robustness than Hand4Whole. More visualizations are provided in Figure 20 in Appendix.
Body Subnetwork. Table 2 compares the performance of the Body subnetwork across different methods on the 3DPW test set. We observe the competitiveness of RoboSMPLX in relation to other SMPL-based approaches. Besides, since the performance of various methods may significantly differ based on their backbone initialization, datasets and training strategies [38], we establish a baseline to evaluate the effectiveness of our added modules in Table 12 in Appendix. RoboSMPLX achieves a substantial improvement compared to the baseline.
Face Subnetwork. Table 3 compares the performance of the Face subnetwork for different methods on the Stirling3D test set. When training with the same dataset, RoboSMPLX outperforms ExPose. The performance of ExPose declines when training on multiple datasets, while RoboSMPLX can still keep low and consistent errors. Figure 8 in Appendix shows some qualitative results for the in-the-wild scenarios, which demonstrates the high generalization of RoboSMPLX. Table 5 compares the robustness of ExPose and RoboSMPLX under different positional augmentations. We also observe that RoboSMPLX has lower errors with different translation and scaling operations. More visualizations are provided in Figure 21 in Appendix.
Whole-body Network. We further provide results of the whole-body network on two benchmarks: EHF val set and AGORA test set in Table 6. On EHF, RoboSMPLX outperforms other full-body approaches, particularly in hand and face performance evaluations, and under different positional augmentations (Table 7). It gives subpar performance on AGORA as the predominant source of
${}^{1} \dagger$  denotes training with extra datasets in the following evaluation and tables.
Table 5: 3DRMSE errors of the Face subnetwork under different positional augmentations.
Normal	Transx +0.2x	Transx -0.2x	Transy +0.2y	Transy -0.2y	Scale 1.3x	Scale 0.7x
ExPose [6]	2.27	2.38	2.29	2.46	2.30	2.46	2.27
RoboSMLPX	2.12	2.20	2.17	2.13	2.18	2.24	2.10
Table 6: Evaluation of wholebody network on EHF and AGORA test set.
Method	EHF	AGORA
PVE ↓	PA-PVE ↓	PVE ↓	N-PVE ↓
WB	H	F	WB	H	F	WB	B	F	LH/RH	WB	B
ExPose [6]	77.1	51.6	35	54.5	12.8	5.8	217.3	151.5	51.1	74.9/71.3	265	184.8
PIXIE [11]	89.2	42.8	32.7	55	11.1	4.6	191.8	142.2	50.2	49.5/49.0	233.9	173.4
Hand4Whole [35]	76.8	39.8	26.1	50.3	10.8	5.8	135.5	90.2	41.6	46.3/48.1	144.1	96.0
OSX (ViT-L)	70.8	53.7	26.4	48.7	15.9	6.0	122.8	80.2	36.2	45.4/46.1	130.6	85.3
PyMAF-X (HR48)	64.9	29.7	19.7	50.2	10.2	5.5	125.7	84	35	44.6/45.6	141.2	94.4
Ours	73.7	34.9	17.8	49.7	10.0	4.6	132.3	85	39.4	45.3/46.1	138.2	91.5
error is the misidentification of individuals under intense person-person occlusion. We give detailed investigation in Appendix D.
Contrastive loss. We validate prior contrastive SSL methods [63, 46, 5] are not particularly adept at learning useful embeddings for human pose and shape estimation. Figures 9 - 12 in Appendix visualize the retrieved images based on the top-5 embedding similarity. They show that without labels, the model primarily extracts features based on background information instead of pose information. Table 8 shows the estimation errors of top-1 retrieved pose (COCO-train) and query pose (COCO-test) with different methods and contrastive loss functions. We observe that SimCLR has higher mean errors than the supervised training method HMR. These results are aligned with [5] that the representations learned through SSL are not transferable for human pose and shape estimation tasks. RoboSMPLX incorporates contrastive loss and positive samples ("HMR +  $L_{con}$ , +ve"), which can produce similar representations under varied augmentations, enhancing its robustness.
Table 9 shows the estimation errors when applying contrastive loss with different representations in Section 4.3: "pose" (a concatenated form of global orientation and rotational pose), "go+pose" (global orientation and rotational pose as separate entities), "keypoint" (3D joints regressed from the body model). We observe that regressed 3D joints are the most effective representation, as they encode both shape and pose information in a normalized space. In contrast, the representation of pose as relative rotation has a detrimental impact on the model performance. Incorporating positive samples ("pose, +ve" and "keypoint, +ve") bolsters contrastive learning, encouraging the model to generate similar representations under varied augmentations. Table 10 compares the model performance with different augmentations. Prior methods [63, 46] employed pose-variant augmentations (e.g., rotation and flipping), which can adversely affect the learning by altering the global orientation, and lead to increased errors ("pose") compared to "baseline". Conversely, color-variant, location-variant and their combination provide an improvement over the baseline, showing these augmentations are helpful.
Location features. Table 11 shows the ablation of different modules on the Hand subnetwork (ablation for the Body subnetwork is in Table 12 in Appendix). The baseline model is trained that randomly augments images with a scale factor of 0.2 and bounding box jitter of 0.2. We observe that training using strongaug with a larger scale and jitter factor harms the baseline performance. This is likely due to a domain shift. Hand4Whole [35] employs sampled features from positional pose-guided pooling (PPP) to predict
pose parameters while shape and camera parameters only utilize backbone features. Our method focuses on explicitly learning the location and part silhouettes, utilizing sparse and dense supervision methods. This proves advantageous as the location information ("LF") improve the performance of pose and shape estimations, with the reduced joint and vertex errors of the regressed mesh. Moreover,
Table 11: Ablation of different modules on Hand subnetwork. Results are trained and evaluated on FreiHAND.
Supervision	PA-↓	MPJPE↓	PA-↓	PVE↓
Base (R50)	8.06	16.78	7.85	16.71
Base (R50) + Strongaug	8.47	17.01	8.11	16.17
Base (DR54)	7.8	15.57	7.67	15.72
Base (DR54)	LKSS	7.68	15.8	7.62	16.29
PPP [35]	LKSS	7.65	15.93	7.56	16.37
LF	LKSS	7.52	15.84	7.56	16.15
joints	LKSS	7.86	15.92	7.75	16.24
LF (all)	LKSS	7.49	15.51	7.46	15.59
LF (all) + Lcon	LKSS	7.48	15.01	7.32	15.29
LF (all) + Lcon, +ve	LKSS	7.42	14.88	7.16	14.57
LF (all)	LKSS,Lsegm	7.44	14.92	7.58	15.30
LF (all)	LKSS,Lsegm,Lproj	7.36	14.38	7.53	15.05
LF (all)+ Lcon, +ve	LKSS,Lsegm,Lproj	7.33	14.59	7.02	14.11
Table 7: Wholebody, Hand and Face PA-PVE errors under different positional augmentations.
Method	Normal	Transx +0.2x	Transx -0.2x	Transy +0.2y	Transy -0.2y	Scale 1.3x	Scale 0.7x
Hands	ExPose [6]	14.39	17.36	17.86	14.93	17.21	14.15	14.56
PIXIE [11]	14.68	15.05	16.11	15.32	15.85	14.52	14.79
Hand4Whole [35]	10.83	11.15	11.34	10.50	13.70	10.77	11.25
OSX [27]	15.97	16.42	16.55	16.94	17.86	15.91	17.24
RoboSMPLX	10.00	10.37	10.21	10.16	12.49	9.98	10.19
Face	ExPose [6]	6.34	10.28	6.71	8.17	6.43	6.24	6.24
PIXIE [11]	5.63	6.67	6.94	6.53	6.94	5.84	5.84
Hand4Whole [35]	5.81	5.88	5.91	5.74	5.93	5.76	5.76
OSX [27]	6.09	6.03	6.09	5.83	5.96	5.92	5.92
RoboSMPLX	4.65	5.10	5.38	4.75	5.30	4.77	5.22
Wholebody	ExPose [6]	54.82	61.64	65.98	65.03	65.98	54.03	59.23
PIXIE [11]	54.85	66.16	69.26	64.83	69.26	56.28	60.31
Hand4Whole [35]	50.37	59.10	67.85	64.64	67.85	48.10	55.28
OSX [27]	48.79	51.09	55.96	95.97	55.96	47.35	50.89
RoboSMPLX	49.79	52.46	53.62	61.65	63.99	47.90	51.39
Table 8: Ablation of contrastive learning methods and loss.
Scale factor	Mean ↓	Std ↓
SimCLR	0.227	0.0915
SimCLR (+ pose-variant aug.)	0.230	0.0911
SimCLR (+ background aug.)	0.222	0.0959
SimCLR (+ Lcon)	0.164	0.0772
HMR	0.140	0.0823
HMR (+ Lcon)	0.124	0.0624
HMR (+ Lcon, +ve samples)	0.119	0.0679
Table 9: Ablation of different representation for contrastive loss.
Representation	PA-↓	MPIJE↓	PA ↓	PVE↓
baseline	7.49	15.51	7.46	15.59
pose	8.11	15.81	7.67	16.08
go + pose	7.71	14.98	7.54	14.91
keypoint	7.48	15.01	7.32	15.29
pose, +ve	7.45	14.94	7.20	14.77
keypoint, +ve	7.31	14.62	7.18	15.01
Table 10: Ablation of augmentation +ve samples, using pose rotation as representation.
Augmentation	PA-↓	MPJPE↓	PA-↓	PVE↓
baseline (no +ve)	8.11	15.81	7.67	16.08
color	7.42	15.01	7.18	14.94
pose	8.59	16.96	8.15	17.21
location	7.80	15.98	7.46	15.56
color + location	7.45	14.94	7.20	14.77
we find that using location features "LF (all)" for predicting shape and camera parameters is also beneficial.
Pixel alignment. Tables 11 also show that incorporating differential rendering and using projected segmentation loss  $(L_{proj})$  for the mesh in RoboSMPLX helps to achieve lower PVE and MPJPE errors. It facilitates the learning of more precise body model and camera parameters to improve the alignment between the rendered 3D model and 2D image. Notably, metrics such as PVE and MPJPE errors is calculated after root alignment and may not sufficiently reflect the quality of mesh projection onto the image space. To offer a more precise analysis, we evaluate the discrepancies between the projected 2D vertices of the ground-truth and projected meshes. More quantitative and qualitative comparisons can be found in Appendix F.
In this paper, we introduce a new framework RoboSMPLX to advance the field of whole-body pose and shape estimation. It enhances the whole-body pipeline by learning more precise localization for part crops while ensuring that part subnetworks are robust enough to handle suboptimal part crops and produce reliable outputs. It achieves this goal with three innovations: accurate subject localization by explicitly learning both sparse and dense predictions of the subject, robust feature extraction with supervised contrastive learning, and accurate pixel alignment of outputs with differentiable rendering. Nevertheless, it is important to acknowledge that there are instances in which our framework exhibits limitations, such as (1) inaccurate beta estimation due to out-of-distribution data (children), (2) challenges posed by severe object-occlusion, (3) difficulties arising from person-person occlusion, and (4) the potential for prediction errors in multi-person scenarios, as exemplified by the cases detailed in Appendix G. These challenges represent important avenues for future refinement of our approach.
There are several potential avenues for future research. First, the current approach does not deliberately select negative samples during training. Future work could explore if hard mining by intentionally selecting similar poses in a batch could enhance learning. Second, the careful selection of augmentations is essential. While augmentations that modify the global orientation, such as flipping and rotation, have proven detrimental and are not employed, the effects of individual augmentations and their combinations are not examined. Future research could explore the potential for automatically determining the optimal selection of augmentations to achieve improved performance. Additionally, simplifying the complex framework without sacrificing performance is a beneficial direction for future work. Lastly, considering that videos are a prevalent input format, the integration of video-based estimation can contribute to bolstering model robustness can enhance model robustness, alleviate depth ambiguity, and improve temporal consistency.
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG3-PhD-2023-08-049T). This study is also supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). We sincerely thank the anonymous reviewers for their valuable comments on this paper.
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2D human pose estimation: New benchmark and state of the art analysis. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3686-3693, 2014. ISSN 10636919. doi: 10.1109/CVPR.2014.471.
[2] Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. CoKe: Contrastive Learning for Robust Keypoint Detection. Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023, pp. 65-74, 2023. doi: 10.1109/WACV56688.2023.00015.
[3] Adnane Boukhayma, Rodrigo De Bem, and Philip H.S. Torr. 3D hand shape and pose from images in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:10835-10844, 2019. ISSN 10636919. doi: 10.1109/CVPR.2019.01110.
[4] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12352 LNCS:769-787, 2020. ISSN 16113349. doi: 10.1007/978-3-030-58571-6_45.
[5] Hongsuk Choi, Hyeongjin Nam, Taeryung Lee, Gyeongsik Moon, and Kyoung Mu Lee. Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimation. International Conference on Learning Representations (ICLR), pp. 1-18, 2023. URL http://arxiv.org/abs/2303.05370.
[6] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body-driven attention. In European Conference on Computer Vision (ECCV), 2020. URL https://expose.is.tue.mpg.de.
[7] MMHuman3D Contributors. Openmmlab 3d human parametric model toolbox and benchmark. https://github.com/open-mmlab/mmhuman3d, 2021.
[8] Radek Danecek, Michael Black, and Timo Bolkart. EMOCA: Emotion Driven Monocular Face Capture and Animation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June:20279-20290, 2022. ISSN 10636919. doi: 10.1109/CVPR52688.2022.01967.
[9] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2019-June:285-295, 2019. ISSN 21607516. doi: 10.1109/CVPRW.2019.00038.
[10] Sai Kumar Dwivedi, Nikos Athanasiou, Muhammed Kocabas, and Michael J. Black. Learning to Regress Bodies from Images using Differentiable Semantic Rendering. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11230-11239, 2021. ISBN 9781665428125. doi: 10.1109/iccv48922.2021.01106.
[11] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collaborative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV), 2021.