Towards Robust and Expressive Whole-body Human Pose and Shape Estimation
Authors: Hui En Pang
Zhongang Cai
Lei Yang
Qingyi Tao
Zhonghua Wu
Tianwei Zhang
Ziwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Whole-body
SMPLX Model
Human Pose and Shape Estimation
Human Mesh Recovery

Abstract: Whole-body pose and shape estimation aims to jointly predict different behaviors (e.g., pose, hand gesture, facial expression) of the entire human body from a monocular image. Existing methods often exhibit suboptimal performance due to the complexity of in-the-wild scenarios. We argue that the prediction accuracy of these models is significantly affected by the quality of the _bounding box_, e.g., scale, alignment. The natural discrepancy between the ideal bounding box annotations and model detection results is particularly detrimental to the performance of whole-body pose and shape estimation.
In this paper, we propose a novel framework to enhance the robustness of whole-body pose and shape estimation. Our framework incorporates three new modules to address the above challenges from three perspectives: (1) a **Localization Module** enhances the model's awareness of the subject's location and semantics within the image space; (2) a **Contrastive Feature Extraction Module** encourages the model to be invariant to robust augmentations by incorporating a contrastive loss and positive samples; (3) a **Pixel Alignment Module** ensures the reprojected mesh from the predicted camera and body model parameters are more accurate and pixel-aligned. We perform comprehensive experiments to demonstrate the effectiveness of our proposed framework on body, hands, face and whole-body benchmarks.

Towards Robust and Expressive Whole-body Human Pose and Shape Estimation
Hui En Pang',
Zhongang Cail., Lei Yang", Qingyi Tao², Zhonghua Wu²,
Tianwei Zhang! Ziwei Liu'
L-bLa, Nanyang Technological University TSeeseTime Research (huien001, tianwei. zhang, ziwei. riwei.liu)@ntu. hwm.l@ntt..duu.aee.. {caizhongang, yanglei, taoqingyi, muzhonghua)@senseettno
Abstract
Whole-body pose and shape estimation aims to jointly predict different behav- iors (c.g.. pose, hand gesture, facial expression) of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We argue that the accuracy and reliability these models are significantly affected by the quality of the predicted bounding box, e.g., the scale and alignment of body parts. The natural discrepancy between the ideal bounding box annotations and model detection results is particularly detrimental to the performance of whole-body pose and shape estimation. In this paper, we propose a novel framework RoboSMPLX enhance the robustness of whole-body pose and shape estimation. RoboSMPLX incorporates three new mod- ules address the above challenges from three perspectives: 1) Localization Module enhances the model's 's awareness of the subject's s location and semantics within the image space. 2) Contrastive Feature Extraction Module encourages the model to be invariant to robust augmentations by incorporating contrastive loss with dedicated positive samples. 3) Pixel Alignment Module ensures the repro- jected mesh from the predicted camera and body model parameters are accurate and pixel-aligned. We perform comprehensive experiments demonstrate the effectiveness of RoboSMPLX on body, hands, face and whole-body benchmarks. Codebase available at https: / //gs //github com/ wwwpooopswooooooooroooooeeeee/ee
Introduction
Human pose and shape estimation tries to build human body models from monocular RGB images or videos. has gained widespread attention owing to its extensive applications in various fields, including robotics, computer graphics, and augmented/virtuus reality. Early works various statistical models (e.g., SMPL [31], MANO [43), FLAME [26]) to individually reconstruct different parts, including human body [17, 22, 4, 16, 24, 10, 21, 20], face [9, 8, 12], and hand [28, 63). Recently, there a growing interest in whole-body estimation [11, 6, 61, 44, 58). which jointly estimates the pose, hand gestures and facial expressions of the entire human body from the input. Commonly these methods first employ separate sub-networks to extract the features of body, hands and face. These features are then used to predict whole-body 3D joint rotations and other parameters (e.g., body shape, facial expression), which are further combined to generate the whole-body 3D mesh. This crucial step towards modeling human behaviors in an efficient and practical manner.
However, achieving accurate and robust whole-body estimation is particularly challenging as requires precise estimation of each body part and the correct connectivity between them. In partieular, due to the smaller sizes of hand and face images, they are typically localized, cropped and resized to higher resolutions before being processed by the relevant sub-network. To tackle the absence
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of ground truth bounding boxes in the real-worke scenarios, existing whole-body methods utilize various detection techniques to obtain the crops. The accuracy of the whole-body estimation highly sensitive to the quality of input crops. Our experiment results in Section 3 show that even minor fluctuations in the scale and alignment of input crops can significantly affect the model performance, indicating a limited ability to localize and extract meaningful features about the subject in the image.
The lack of robustness in existing whole-body pose and shape estimation methods highlights three critical aspects that can be improved upon: 1) accurate localization of the subject and its parts, 2) accurate extraction of useful features, and 3) accurate pixel alignment of outputs. Inspired by these findings, we propose three novel modules, each specifically designed to address particular goal:
Localization Module. This module implements sparse and dense prediction branches to ensure the model is aware of the location and semantics of the subject's parts in the image. The learned location of the joint positions are helpful in recovering the relative rotations.
Contrastive Feature Extraction Module. This module incorporates a pose- and shape-aware contrastive loss, along with positive samples, to promote better feature extraction under robust augmentations. By minimizing the contrastive loss, the model can produce consistent representa- tions for the same subject, even when presented with different augmentations, making robust to various transformations and capable of extracting meaningful invariant features.
Pixel Alignment Module. This module applies differentiable rendering to ensure more precise pixel alignment of the projected mesh, and learn more accurate pose, shape and camera parameters.
By integrating these three modules, we build more robust and reliable whole-body pose and shape estimation framework, RoboSMPLX. Comprehensive evaluations demonstrate its effectiveness on body, face, hands and whole-bodt body benchmarks.
2 Related Works
Whole-body Mesh Recovery. Despite significant progress in 3D body-specific [23, 22, 4, 16, 24, 10, 21. 20], hand-specific [28, 3]. and face-specific [9] mesh recovery methods, there have been limited attempts to simultancously recover all those parts. Early studies on whole-body pose and shape estimation primarily fit a 3D human model to 2D or 3D evidence [15, 53, 41, 54]. which can be slow and susceptible to noise. Recent studies utilized neural networks to regress the SMPL-X parameters for a Mhole-body 3D human mesh. The model is composed of separate sub-networks to process body, hand and face, respectively. One-stage methods, e.g., OS-X [27], have the benefit of reduced computational costs and improved communication within part modules for more natural mesh articulation. However, the omission of hand and face experts makes difficult for the model to leverage the widely available part-specific datasets, thus decreasing the hand and face performance. Multi-stage methods, e.g., ExPose [6], Frank [44], PIXIE [11] and Hand4Whole [35], use different techniques to localize part crops.
Expose [41] and PIXIE [11] localize hand and part crops from the body mesh, making them dependent on the accuracy of body poses. Minor rotation errors accumulated along the kinematic chain may joint locations and thus inaccurate part crops. In contrast, Hand4Whole [35] predicts hand and face bounding boxes using a network leveraging image features and 3D joint heatmaps, but the resulting crops have low resolution. PyMAF-X [1 relies on an off tfi-the-shelf whole- body pose estimation model to obtain crops, which, while more accurate, incurs extra computation. More detailed comparison with PyMAF-X are in Appendix C.
result in deviations
Robustness in vision tasks. Efforts to tackle robustness in vision tasks have utilized di- verse strategies such as data augmentation, architectural innovations, and training methodologies [25, 56, 42, 30, 51, 60, 2]. AdvMix [51] employs adversarial augmentation and knowledge distillation, challenging models with corrupted images to foster learning from complex samples. Architectural modifications, such as novel heatmap regression [60], have been introduced to mitigate the impact of minor perturbations. HuMoR [42] utilizes conditional variational autoencoder to capture the dynamics of human movement, thereby achieving generalization across diverse motions and body shapes. Additionally, PoseExaminer [30] employs a multi-agent reinforcement learning system to uncover failure modes inherent in human pose estimation models, highlighting model limitations in real-world scenarios. Complementing these efforts, Robo3D [25] provides comprehensive bench- mark for assessing the robustness of 3D detectors and segmentors in distribbutii scenarios.

---

Vertical Transiation
Horizontal Transiation
Law Resolution
Scale
Rotation
Hue
Sharpness
Graymess
Brightness
Cantrast
Figure 1:
Wholebody PA-PVE errors under difffrrnt augmentations (sorted in descending order). The dashed line indicates baseline performance without augmentation.
Furthermore, [56] utilize confidence-guidee framework to improve the accuracies of propagated labels. Contrastive learning, as demonstrated by CoKe [2]. has also been employed to enhance robustness in keypoint detection, especially in occlusion-pron scenarios.
Contrastive Learning. Recently contrastive learning has demonstrated state-of -fe--o---hherrttt performance among self superviee learning (SSL) approaches. This strategy has been applied to 3D hand pose and shape estimation [46, 63]. Sanyal al. [45] incorporate a novel shape consistency loss for 3D face shape and pose estimation that encourages the face shape parameters to be similar when the identity is the same and different for different people. Choi et al. [5] were the first to apply contrastive learning for 3D human pose and shape estimation. They found that SSL is not useful for this task, as learned representations could be challenging to embed with high-level human-related information. Khosla et al. [19] proposed supervised contrastive learning for image classification tasks, which incorporates label information during training. Currently there not attempt to apply this strategy to human pose and shape estimation, where the definition of positive samples is unclear, and data lie in a continuous space. We are the first to overcome these challenges and integrate supervised contrastive learning with whole-body pose and shape estimation.
Many studies have been done learn the subject's location in an image. Some works implicitly supervise the location. They primarily utilize projected meshes by supervising 2D joints regressed from the mesh [17, 23, 22, 4, 16, 24, 10, 21, 20]. Further supervision, such as dense body landmarks, silhouettes, and body part segmentation, is also employed to better align the predictions with the image [55, 37, 40, 49. 59, 57, 10]. Some other works explicitly learn the subject's location. Moon et al. [35] explicitly predict the keypoint locations in the image. Semantic body part segmentation used as an explicit intermediate representation [41, 37]. PARE [41] employs renderer to project the ground-truth mesh to the image space, and supervise the predicted part silhouette mask. However, dense part segmentation and differentiable rendering have not been employed in whole-bodt pose and shape estimation, which will be achieved in our framework.
Pixel Alignment in Pose and Shape Estimation.
3 Motivation
As discussed in Section 1, existing whole-body pose and shape estimation approaches suffer from the robustness issue, due to the models' sensitivity the quality of input crops. To investigate the reasons and disclose the influence factors, we conduct comprehensive evaluation of four state-of- the-art methods: ExPose [6). PIXIE [11], Wand4Whole [35] and OS-X [27]. We opt for a set of ten commonly encountered augmentations and vary their scales within a realistic range (see Appendix for more details). The augmentations can be classed into three categories (1) image-variant augmentations: they affect the image without altering the objects' 3D poses positions, such as color jittering; (2) location-varite variant augmentations: they modify the subject's location without changing its pose, involving operations like translation and scaling: (3) pose-variant augmentations: they simultaneously alter both the 3D pose and location, including rotation.
Impact of subject localization. We first reveal that existing models demonstrate high sensitivity to the subject's position, indicating potential difficulties subject localization. Figure reports the PA-PVE errors of the whole body under different augmentations. We observe that image-variant augmentations (contrast, sharpness, brightness, hue and grayscale) lead an acceptable range of error rates (approximately in the 50s) and minimal fluctuation (around +2). In contrast, location-

---

Figure 3: Sensitivity of existing body and hand models to different alignments (left) and scales (right).
variant augmentations altering the subject's position within the frame, such as rotation, scaling, and horizontal or vertical translation, result in substantially higher error magnitudes. This demonstrates the heightened sensitivity of existing models to changes in the subject's position. Appendix, we provide the results of other metrics and benchmarks in Figures 22 23, and visualizations of whole-body estimation under different settings Figures 25 26.
Such position-altering augmentations are common in real-world scenarios, where the subject in the image is often localized using external detection models and control over the quality of crops less feasible, In practice, to guarantee the visibility of the subject, crops are often made broader, This can lead to significant performance degradation, as errors increase with smaller augmentation scale factors (<1.0) (Figure 1). Besides, horizontal and vertical translations, which correspond to scenarios where the subject is not perfectly centralized en- tirely visible within the frame, can further decrease the performance. Similarly, the alignment and scale of these crops also influence the pose and shape estimation systems targeting body, face and hands (Figure 3, more quantitative and qualitative evidence in Appendix M). Whole-body methods bear the additional responsibility of accurately localizing body parts such as hands and face. Inaccurate part crops (Figure 2) can adversely affect the performance of part subnetworks, and further the whole-body estimation.
Figure 2: Crops from (a) ExPose [6] (b) PIXIE
[11], (c) Hand4Whole [35] (d) RoboSMPLX.
Impact of feature extraction. The deterioration of performance the face of such variations suggests that the model struggles to extract meaningful features. Under alterations in translation or scale, the subject remains within the image frame, though the proportion of background content may vary. It is difficult for existing methods to effectively disregard irrelevant background elements and extract relevant features related to the subject of interest, To enhance the model's robustness, is critical to produce consistent features irrespective of various augmentations applied to the image.
Impact of output pixel alignment. Pixel alignment is a critical aspect of high model performance. In certain instances, despite having precise subject localization, the model fails to produce properly aligned results (Figure 25 in Appendix). This often caused by the suboptimal camera parameter estimation. To address this issue, we need to accurately estimate the camera parameters, ensuring the projected mesh is precisely aligned with the ground-truth the pixel level. Such precision would enhance the effectiveness of the model in producing accurate pose, shape and camera parameter predictions, improving the overall accuracy and reliability of the estimation process.
4 RoboSMPLX Framework
We design RoboSMPLX to enhance the robustness of whole-body pose and shape estimation. It provides three specialized modules to address each challenge in Section 3: 1) Localization Module (Section 4.2): explicitly learning the location information of the subject and incorporating it into model estimations for pose, shape and camera: 2) Contrastive Feature Extraction Module (Section 4.3): reliably extracting pertinent features under various augmentations, thereby improving the model's 's generalization ability and robustness to a broader range of real-world scenarios; 3) Pixel Alignment Module (Section 4.4): ensuring that the outputs are pixel aligned.
We start with the description of RoboSMPLX architecture with Body, Hand and Face subnetworks (Section 4.1). Each subnetwork is integrated with the Localization Module and Pixel Alignment

---

Sprul
Bubnetwors
Crop Module
Hand Subnetwors
Bopy 0i hand parameters
render
lvt
Outputs
SMPPX
whole-kody
Models
Subnetworrs
Outpur Mesh
Operations
Figure 4: Pipeline of our RoboSMPLX framework consisting of Body, Hand and Face subnetworks.
Figure 5: Examples of keypoint and part segmentation supervision for Body, Hand and Face subnetworks.
Module, and applies the Contrastive Feature Extraction Module for learning more robust features. Figure shows the Hand subnetwork architecture. The other two subnetworks have the same designs.
4.1 Architecture and Training Details
Figure 4 shows the overall pipeline RoboSMPLX for whole-body 3D human pose and mesh estimation. The Body subnetwork outputs 3D body joint rotations 0, g21x3 global orientation 0ng R shape parameters 3, R 10 camera parameters E R and whole-body joints K R 137x3 Joints corresponding to the hand and face are used to derive bounding boxes. Subsequently, hand and face images are cropped from high-resolution image preserve details. The Hand subnetwork predicts left and right hand 3D finger rotations 0, E TR 15x3 Simultaneously. the Face subnetwork generates 3D jaw rotation 0y E R$ and expression 4 IR When training Hand and Face subnetworks with part-specific datasets, additional parameters such as global orientation 0fg E 1R3 shape 3, R50, and camera Ty E IR3 are estimated. These branches are discarded during whole- body estimation and training. Additional information concerning each subnetwork can be found in Appendix B. Further details regarding the training and inference durations are elaborated upon in Appendix
Subnetworks are trained separately, then integrated in a multi-stage manner. Initial whole-body training runs for 20 epochs. The hand and face modules substituted with the trained Hand and Face subnetworks, followed by 20 epochs of fine-tuning to better unify the knowledge from the Hand and Face subnetworks into the whole-body understanding. Each subnetwork trained by minimizing the following loss function L:
= Aan + AanLan + ABM L.BM + Axpra Pproj + Asegm Tosegm + Acon Tecon
Here LBM the L1 distance between the predicted and ground-trune body model parameters. LyD denotes the L1 distance between 3D keypoints and joints regressed from the body model. Lan signifies the distance of the ground-truth 2D keypoints to predicted and projected 2D joints. The latter are obtained by projecting the regressed 3D coordinates from the 3D mesh to the image space using the perspective projection [17]. The part segmentation loss Langin the cross-entropy loss between Pr... after softmax and Pr,w averaged over HxW elements, following [20]. Lproj refers to the projected segmentation loss, which is the sigmoid loss between the projected mesh and the ground-trrr cround-truth segmentation map. Leon is the contrastive loss described in Section 4.3. For wholebody
training, Lhow is added to measure the L1 distance between the predicted and actual center and seale of the hands' and face's boxes.
4.2 Localization Module
This module focuses on subject localization by explicitly learning both sparse and dense predictions of the subject within the image. Figure shows an example of the supervision used for each subnetwork.

---

Localization Module
Plxel Alignment Module
Conn+ Upsamsla
Difteremtbble
Spreel
Teras
Conv
I.Fs Location
Sprat
RL
Hand Encoder
F
1.%%
Lineer
Pa.
MANO
Input Image
SELANG
Hand Subnetwork
Pa.
MANO
Ko
Aligmment
Hand Subretwork
Pa.
Aze
Hand Subnetwork
CONOONAAOOOO Agreement M.C. pose differecce
Operations
Badiground
Hand
9g.6%
Contractive Modele
Figure 6: Subnetwork Architecture with three modules. We use the Hand subnetwork an example. represents normalized JaD while p corresponds to the ground-trute couutd-truthh of Green and red dashed lines refers to contrastive loss positive and negative samples respectively.
Original
Blur
Occlusion
Background
Cut out
Color Jitter
Scale
Transi ate
Filp
Rotate
Figure 7: Augmentations for the Body subnetwork. Black, blue and red labels represent image-variant, location-variant and pose-variant augmentations, respectively.
In contrast to prior methods that directly output pose rotations from backbone features, this module aims make the model explicitly conscious of the subject's location and semantics while predicting pose, shape and camera parameters. can reduce the model's sensitivity to the variations of the subject's position, caused by minor shifts in the scale and aligmment of the bounding box.
As shown in Figure 6, given an image, a convolutional backbone utilized to extract its feature map F E R512x32x32 Following [35], a 1x convolutional layer is then used predict 3D feature maps LF 1832xx32x3 from F, where represents the number of predicted joints with a feature map depth of 32. LF contains valuable information about the mesh's position in the image and semantics of various parts. is concatenated with the backbone feature map to predict pose E R.P shape 3 R 10 and camera translation R3, where P is the number of body parts. Mcanwhile. LF is also used to obtain extra information with two branches: (1) 3D joint coordinates KE Ix3 are obtained from LF using the soft-argmax operation [47] in differentiable manner. (2) 2D part segmentation maps S E R PP4166xxe are extracted from LF with several convolution layers, which model P part segmentation and 1 background mask. Here, 64 represents the height and width of the feature volume, and each pixel (h, w) stores the likelihood of belonging to a body part P.
Note that learning part segmentation maps and 3D joint coordinates is complementary, 3D joint coordinates encode depth information that may inform part ordering segmentation maps. Additionally, joints often reside at the boundaries of part segmentation maps, serving as separators for distinct parts. The Body subnetwork utilizes 24 parts P and 137 joints J, the Hand subnetwork employs 16 parts P and 21 joints J, while the Face subnetwork employs 15 parts P and 73 joints J.
4.3 Contrastive Feature Extraction Module
This module incorporates pose- and shape-aware contrastive loss, along with positive samples. By minimizing this loss, the model can produce consistent representations for the same subject, even when presented with different augmentations, thus fostering the extraction of meaningful features.
Conventional contrastive learning methods based on SSL (e.g., SimCLR) face challenges unifying similar pose embeddings and distancing dissimilar ones in human pose and shape estimation tasks. Without labels for guidance, images with similar poses could misidentified as negative samples and contrasted away, complicating the self-organization of the embeddings in pose space. Figures 9 in Appendix show their ineffectiveness for the human pose and shape task [5] by visualizing the retrieved samples from the embeddings. The supervised contrastive learning approach by Khosla

---

et al. [19], though effective for image classification, might not extend well to human pose and shape dimensionn regression problem and poses exist continuous space
estimation, which rather than well-defined classes.
Our module overcomes the aforementioned issues with two innovations. First, experiment with three human pose representations & and the corresponding distance functions: (1) A concatenated form of the global orientation and rotational pose; (2) global orientation and rotational pose as separate entities (3) 3D root-aligned joints regressed from the body model, derived from pose and shape inputs. For (1) and (2), we explore relative rotations two forms: 6D vector and rotation matrix representation. For (3), L1, Smooth LI, and Mean Squared Error (MSE) was used (Table 9),
Second, investigate ten data augmentations, and classify them into three categories (see Figure 7 for the Body subnetwork, and Figure 29 in Appendix for Hand subnetwork): (1) image-variant augmentations such color jittering, blur, occlusion and background swapping: (2) location-variant augmentations involving translation and scaling; (3) pose-variant augmentations including rotation and horizontal flipping. Our ablation study in Table 10 shows that augmentations with varied global orientation are detrimental to the model performance. Consequently, we exclude such modifications when constructing positive pairs. Instead, each positive sample is constructed utilizing a random combination of location- and color-variant augmentations.
Formally, for batch of N images, we construct another N images by applying augmentation to each sample. For each anchor i, let j be the corresponding augmented sample. Then is contrasted against 2N terms positive and 2N negatives). The loss takes the following form:
2N
Leon
Tpos (ld (p...,) d (2), Treg Uuzis (ld (p) (zz, (21,z1)))
where 2i, 2j and 2y: denote the predicted pose representations, and P:: and Px denote the ground-truth pose representations for the anchor, positive and negative samples in the batch. The objective of this loss function is to minimize the distance between the positive pairs and maximize the distance between the negative pairs, in alignment with the pose similarity. Note that unlike traditional approaches where the distance is the same for all negative samples, the pairwise distance d(p,, Ps) varies depending the pose similarity.
4.4 Pixel Alignment Module
This module employs differentiable rendering to ensure that the projected mesh aligns precisely at the pixel level. The alignment is supervised by the projected mask loss. Attaining a proper alignment between the ground-truth part segmentation and rendered mesh requires the accurate prediction of pose, shape. and camera parameters, which subsequently leads to more precise estimation process.
5 Experiments
Datasets. For whole-body training. we employ Human3.6M, (H36M) [13], Shho-Wholybody [14]
(the whole-body version of MSCOCO [29]) and MPII [1]. The 3D pseudo-ground truths for training
are acquired using NeuralAnnot [36]. For hand-specific training, we use FreiHAND [62], Interhand
[34] and COCO-Wholebode Hands [14]. For face-specific training, we use FFHQ [18), BUPT [52] and AffectNet [32]. For evaluations specific to 3D body. 3D hand, and 3D face, we utilize 3DPW [50], FreiHAND [62]. and Stirling [1 I], respectively. For the 3D whole-bod, evaluation, we use EHF [41] and AGORA [39]. Additionally, we present qualitative results on the MSCOCO validation set.
Metrics. Mean Per Joint Position Error (MPJPE) and Mean Per- Vertex Position Error (MPVPE) are employed to evaluate the positions of 3D joint and mesh vertices, respectively. Each metric calculates the average 3D joint distance (in mm) and 3D mesh vertex distance (in mm) between the predicted and ground-truth values after aligning the root joint translation. The pelvis serves as the root joint for whole-body and body, whereas the wrists and neck are utilized as root joints for hands and face. Procrustes Aligned (PA) variants of these metrics, PA-MPJPE and PA-MPVPE, further align with rotation and scale. We report the average errors for the left and right hands the 3D hand error.
7

---

Table 2: Evaluation of the Body subnetwork.
Table 1: Evaluation of the Hand subnetwork.
PA-MPIPE:
MPJPE
PVE.
HMR (Rex50) [17]
76.7
130
Method
PA-PVE / PA-MPJPE
F-Scores
GraphCMR
[24]
SPIN (Res50) [22]
59.2
96.9
116.4
Hand-only
HMR-EFT (Res50) [16]
54.3
FreiHAND [62]
10.7
0.529/0,935
Pose2Mesh [4]
7,8
7.7
0.674/0,960 969
PARE (Res30) [20]
52.3
82.9
99.7
12L-MeshNet [33]
7.6
7.4
0.681/0.973
PARE (HR32) [20]
50,9
82
97.9
METRO (HR64) [28]
6.7
6.8
0.717/0.981
PyMAF (Res501[88
79.7
PyMAF (HR48) [58]
47.1
78.0
91.3
WWhle-body
Baseline (Res50)
52.4
85.2
103.6
ExPose [41]
11.8
12.2
0,484/0.918
RoboSHPLX (Res50)
49.8
80.8
96.7
Zhou et al. [61]
15.7
-f-
Daseline (HR48)
50.3
84.5
101.5
RoboSMPLX (HR48)
48.5
80.1
95.2
FrankMocap [44]
11.6
9.2
0,553/0.951
PIXIE [11]
12.1
12
0.468/0.919
Table 3: Evaluation
Face subnetwork.
Hand4WWole [35]
7.7
0.664/009771
Method
1.0 Mean(mm) L HQ Meantrm)
HMR (Baseline) [17]
8.6
8.9
0.605/0.963
ExPose [6]
2.27
PyMAF
8.1
8.4
0.638/0.969
ExPose f
2,46
2.38
PYMAF [58)
7.5
7.7
0.67 1/0,974
HMR
2.18
RoboSMPLX
7.3
0.683/0.976
HMR
2.31
2.27
RoboSMPLX
7.1
7.4
0. 1688009978
HMR
2,02
2.04
RoboSMPLX (IIR64)
6.7
6.9
0.715/0.981
PYMAF
1.97
1.92
RobaSMPLX
2.12
2,08
RoboSMPLX
2.12
2.10
Table 4: PA-PYE/PVE errors of the Hand subnetwork under different positional augmentations.
Normal
Transx +0.2x
Transx -0.2x
Transy +0.2y
Transy -0.2y
Scale 1.3x
Scale 0.7x
Wand4Whole [35]
7,47/ 15.70
8.51/21.58 7.27/ 15.62
8.38/ 20.36
8.74/ 22.51
8.4N/ 19.85
7.73/ 16.44
7.78/ 17.00
RoboSMPLX
7,24/ 15.23
7.36/ 15.59
7.28/ 15.50
7.34/ 15.50
7.49/ 15.90
7.45/ 16.51
5.1
Benchmarking Results
Hand Subnetwork. Table I compares the performance of the Hand subnetwork with different hand- only and whole-body methods. Our method outperforms that of our whole-body counterparts when trained with only the FreiHAND dataset (i.e. PIXIE, WhodeWhole, PyMAF) under mixed datasets (i.e. Hand4Whole f. PyMAF f) using an identical backbone. Prior research [33, 48] demonstrated that whole-body methods generally employ a parametric representation of the hand mesh, and numerically inferior to the non-parametria representation used in recent hand-only methods [33, 28). Despite such reported gap, RoboSMPLX manages outperform mesh-based techniques, and achieve comparable results as the state-of-the-art METRO when using the same backbone (HRNet-644, Table 4 compares the estimation errors of the Hand subnetwork in Hand4Whole (current whole-body method with SOTA on hands) and RoboSMPLX under different positional augmentations on the FreiHAND test set. It clear that RoboSMPLX exhibits much better robustness than Hand4Whole. More visualizations are provided in Figure 20 in Appendix.
Body Subnetwork. Table 2 compares the performance of the Body subnetwork across different methods on the 3DPW test set. We observe the competitiveness of RoboSMPLX relation to other SMPL-based approaches. Besides, since the performance of various methods may significantly differ based on their backbone initialization, datasets and training strategies [38], we establish a baseline evaluate the effectiveness of our added modules in Table 12 Appendix. RoboSMPLX achieves substantial improvement compared to the baseline.
Face Subnetwork. Table compares the performance of the Face subnetwork for different methods on the Stirling3D test set. When training with the same dataset, RoboSMPLX outperforms ExPose. The performance of ExPose declines when training multiple datasets, while RoboSMPLX keep low and consistent errors. Figure 8 in Appendix shows some qualitative results for the in-the- wild scenarios, which demonstrates the high generalization of RoboSMPLX. Table 5 compares the robustness of ExPose and RoboSMPLX under different positional augmentations. We also observe that RoboSMPLX has lower errors with different translation and scaling operations. More visualizations are provided in Figure 21 in Appendix.
still
Whole-body Network. We further provide results of the whole-body network on two benchmarks: EHF val set and AGORA test set in Table 6. On EHF, RoboSMPLX outperforms other full-body approaches, particularly in hand and face performance evaluations, and under different positional augmentations (Table 7). It gives subpar performance on AGORA as the predominant source of
' denotes training with extra datasets in the following evaluation and tables.

---

Table 5: 3DRMSE errors of the Face subnetwork under different positional augmentations.
Normal
Transx +0.2x
Transx .0.2x
Transy +0.2y
Transy -0.2y
Scale 1.3x
Scale 0.7x
ExPose [6]
2.27
2.38
2.29
2.46
2.30
2.46
2.27
RoboSMPLX
2.12
2.20
2.17
2.13
2.18
2.24
2.10
Table 6: Evaluation of wholebody network on EHF and AGORA test set.
EHF
AGORA
Method
PVE 4
PA-PVE
LVB
N-PVE L
WB
H
F
WB
H
WB
LH/RH
WB
B
ExPose [6]
77.1
51.6
35
54.5
12.8
5.8
217.3
151.5
51.1
74.9/71.3
265
184.8
PIXIE [11]
89.2
42.8
32.7
55
11.1
191.8
142.2
50.2
49.5/49.0
233.9
173.4
Wald [35]
76.8
39.8
26.1
50.3
10.8
5.8
135.5
90.2
41.6
46.3/48.1
144.1
96.0
OSX (VTT-L.
70.8
53.7
26.4
48.7
15.9
6.0
122.8
80.2
36.2
45.4/46.1
130.6
85.3
PyMAF-X (HR48)
64.9
29.7
19.7
50.2
10.2
5.5
125.7
84
35
44.6/45.6
141.2
94.4
Ours
73.7
34.9
17.8
49.7
10.0
4.6
132.3
85
39.4
45.3/46.1
138.2
91.5
error is the misidentification of individuals under intense person-person occlusion. We give detailed investigation in Appendix D.
5.2 Ablation Studies
Contrastive loss. We validate prior contrastive SSL methods [63, 46, 5] are not particularly adept at learning useful embeddings for human pose and shape estimation. Figures 9 - 12 in Appendix visualize the retrieved images based on the top-5 embedding similarity. They show that without labels, the model primarily extracts features based on background information instead of pose information. Table 8 shows the estimation errors of top-1 retrieved pose (COCO-train) and query pose (COCO- test) with different methods and contrastive loss functions. We observe that SimCLR has higher mean errors than the supervised training method HMR. These results are aligned with [5] that the representations learned through SSL are not transferable for human pose and shape estimation tasks. RoboSMPLX incorporates contrastive loss and positive samples ("HMR + Leon. +ve"), which can produce similar representations under varied augmentations, enhancing its robustness.
Table 9 shows the estimation errors when applying contrastive loss with different representations in Section 4.3: "pose" (a concatenated form of global orientation and rotational pose). "go+pose" (global orientation and rotational pose as separate entities). "keypoint" (3D joints regressed from the body model). We observe that regressed 3D joints are the most effective representation, they encode both shape and pose information in a normalized space. In contrast, the representation of pose as relative rotation has detrimental impact on the model performance. Incorporating positive samples ("pose, +ve" and "keypoint, +ve") bolsters contrastive learning, encouraging the model to generate similar representations under varied augmentations. Table 10 compares the model performance with different augmentations. Prior methods [63, 46) employed pose-variant augmentations (e.g.. rotation and flipping), which can adversely affect the learning by altering the global orientation, and lead to increased errors (" ("pose") compared to "baseline". Conversely, color-variant, location-varie and their combination provide an improvement over the baseline, showing these augmentations are helpful.
Location features. Table 11 shows the ablation of different modules on the Hand subnetwork (ablation for the Body sub- network is in Table 12 in Appendix). The baseline model is trained that randomly augments images with scale factor of 0.2 and bounding box jitter of 0.2. We ob- serve that training using strongaug with a larger scale and jitter factor harms the baseline performance. This is likely due to domain shift. Hand4Whole [35] em- ploys sampled features from positional pose- pase-guided guided pooling (PPP) to predict
Table 11: Ablation of different modules on Hand subnet- work, Results are trained and evaluated on FreiHAND.
Supervision
PA-1
MPJPET
PN-1
PVE.
Base (R50)
8.06
16.78
7.85
16.71
Base (R55)+ Stronguug
17.01
16.17
Base (DRS4)
7.8
15.57
7.67
15.72
Base (DR54)
Lxs
7.68
15.8
7.62
16.29
[P5
Lxs
LF
Lxs
7.52
15.84
7.56
16.15
joints
Lxs
7.86
15:92
7.75
16.24
(all)
Lxcs
15.51
15.59
LF tall) (all) J..m.
+ Loon
Lxs
7.48
15.01
7.32
15.29
7.42
14.88
7.16
14.57
LF LF tall) I.F tallje Tor...
Lxc. Laagre
7.44
14.92
7.58
15.30
LIc. Leseam Legres
7.36
14.38
7.53
15.05
Lacos breeues Fepen)
14.59
14.11
pose parameters while shape and camera parameters only utilize backbone features. Our method focuses on explicitly learning the location and part silhouettes, utilizing sparse and dense supervision methods. This proves advantageous as the location information ("LF") improve the performance of pose and shape estimations, with the reduced joint and vertex errors of the regressed mesh. Moreover,

---

Table 7: Wholebody, Hand and Face PA-PVE errors under different positional augmentations.
Method
Normal
Transx
Transy +0.2y
Trancy -0.2y
Scale
Scale
ExPose
14.39
17.86
17.21
14.56
PIXTE [1]]
14,68
15.05
16.11
15.32
15.85
14.52
14.79
Hands
Hand4Whole [35]
10.83
11.15
11.34
10.50
13.70
10.77
11.25
OSX [27]
15.97
16.42
16.55
16.94
17.86
15,91
17.24
RoboSMPLX
10.00
10.37
10.21
10.16
12.49
9.98
10.19
ExPose
8.17
6.24
PIXIE [11]
5.63
6.67
6.94
6.53
6.94
5.84
5.84
Face
Hand4Whole [35]
5.81
5.88
5.91
5.74
5.93
5,76
5.76
OSX [27]
6.09
6.03
6.09
5.83
5.96
5.92
5.92
RoboSMPLX
4.65
5.10
5.38
4.75
5.30
4.77
5.22
ExPose [6]
54.82
61.64
65.98
65.03
65.98
54.03
59.23
PIXIE [11]
54.85
66.16
69.26
64.83
69.26
56,28
60.31
Wholehody
Hand4Whole [35]
50.37
59.10
67.85
64.64
67.85
48.10
55.28
OSX [27]
48.79
51.09
55.96
95.97
55,96
47.35
50.89
RoboSMPLX
49.79
52.46
53.62
61.68
63.99
47.90
51.39
Table 8: Ablation of contrastive learning methods and loss.
Table 9: Ablation of different repre-
Table 10: Ablation of augmenta- tion +ve samples, using pose rota-
sentation for contrastive loss.
Scall
Moa
1
Std
Regresentation
PA-1
MPIPEL
PA
PVE.
tion as representation.
buseline
7.19
15.51
7.46
15.59
Augmentatioo
PA-]
MPIPEL
PA-
PVE]
SinCI posostariant sup
SinCLR buckgrouna aug.)
pose
beauline
8.11
15.81
7.67
16:08
SinCLR Luse)
HMR
keypuint
go pose
7.48
15.01
7.32
15.29
7.42
8.59
15.01
16.96
8.15
7.18
14.90
17:21
HMR L.ec)
post +ve
7.45
14.94
7.30
14.77
lecation
7.50
15.58
7,46
15.56
HMR Leees samples)
0,0679
keypoint +se
7.31
14.62
7.18
15.81
location
7.45
14.94
7.30
14.77
find that using location features "LF (all)" for predicting shape and camera parameters is also beneficial.
Pixel alignment. Tables 11 also shows that incorporating differential rendering and using projected segmentation loss (Eproo) for the mesh in RoboSMPLX helps to achieve lower PVE and MPJPE errors. It facilitates the learning of more precise body model and camera parameters to improve the alignment between the rendered 3D model and 2D image. Notably, metries such as PVE and MPJPE errors is calculated after root alignment and may not sufficiently reflect the quality of mesh projection onto the image space. To offer a more precise analysis, we evaluate the discrepancies between the projected 2D vertices of the ground-truth and projected meshes. More quantitative and qualitative comparisons can be found in Appendix F.
6 Conclusion
In this paper, we introduce a new framework RoboSMPLX to advance the field of whole-body pose and shape estimation. It enhances the whole-body pipeline by learning more precise localization for part crops while ensuring that part subnetworks are robust enough to handle suboptimal part crops and produce reliable outputs. It achieves this goal with three innovations: accurate subject localization by explicitly learning both sparse and dense predictions of the subject, robust feature extraction with supervised contrastive learning, and accurate pixel alignment of outputs with differentiable rendering. Nevertheless, it is important to acknowledge that there are instances in which our framework exhibits limitations, such as (1) inaccurate beta estimation due challenges posed by severe occluuio,,,, (3) difficulties arising from person-perye person occlusion, and (4) the potential for prediction errors in multi pelti- scenarios, as exemplified by the cases detailed in Appendix G. These challenges represent important avenues for future refinement of our approach.
dittiitiinttion data (children), (2)
There are several potential avenues for future research. First, the current approach does not de- liberately select negative samples during training. Future work could explore if hard mining by intentionally selecting similar poses in a batch could enhance learning. Second, the careful selection of augmentations essential. While augmentations that modify the global orientation, such as flipping and rotation, have proven detrimental and are not employed, the effects of individual aug- mentations and their combinations are not examined. Future research could explore the potential for automatically determining the optimal selection of augmentations to achieve improved performance. Additionally, simplifying the complex framework without sacrificing performance is beneficial di- rection for future work. Lastly, considering that videos are prevalent input format, the integration of video-based estimation can contribute to bolstering model robustness can enhance model robustness, alleviate depth ambiguity, and improve temporal consistency.
10

---

Acknowledgements
This research/project is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG3-PhD--200 2022-00-04990 This study is also supported by the Ministry of Education, Singapore, under its MOE AcRF Tier (MOE-T26P20221-000 NAP, and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). We sincerely thank the anonymous reviewers for their valuable comments this paper.
NTU
References
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2D human pose estimation: New benchmark and state of the art analysis. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 3686-3693, 2014. ISSN 10636919. doi: 10. 00.100//92.2220444477
[2] Yutong Bai. Angtian Wang. Adam Kortylewski, and Alan Yuille. CoKe: Contrastive Learning for Robust Keypoint Detection. Proceedings 2023 IEEE Winter Conference on Applications of Computer Vision WACV 2023, pp. 65-74, 2023, doi: 10.1 R..1109/WACV66633 2023.00015
[3] Adnane Boukhayma, Rodrigo De Bem, and Philip H.S. Torr. 3D hand shape and pose from images in the wild. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2019-June: 10835-10844, 2019. ISSN 10636919. doi: 10.1109/C
2019.01110.
[4] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from 2D Human Pose. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12352 LNCS:769-787, 2020. ISSN 16113349. doi: 10.1007/ 378-3-030-585711--22
[5] Hongsuk Choi, Hyeongjin Nam, Taeryung Lee, Gyeongsik Moon, and Kyoung Mu Lee. Re- thinking Self-Supervised Visual Representation Learning in Pre- Pre-training for 3D Human Pose and Shape Estimation. International Conference on Learning Representations (ICLR), pp. 1-18,
2023. URL http:://axxiiii org/ abs/ 05370.
[6] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body-driven attention. European Conference on Computer Vision (ECCV), 2020. URL
is tue . mpg........ de.
[7] MMHuman3D Contributors. Openmmlab 3d human parametric model toolbox and benchmark. https://gg gi thub. com/
www/opwooooo/e//////o//ee opeeo memen-mmabb mmmmmamnddm 2021.
[8] Radek Danecek, Michael Black, and Timo Bolkart. EMOCA: Emotion Driven Monocular Face Capture and Animation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-lune:20299-20099 2022. ISSN 10636919.
doi: 10. 10110C9R522288..222 202222619.202211966
[9] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3D face reconstruction with weakly-supervisee learning: From single image to image set. IEEE Computer Society Conference Computer Vision and Pattern Recognition Workshops, 2019- June:285-295, 2019. ISSN 21607516. doi: 10.1 11110//VP..2009000000
[10] Sai Kumar Dwivedi, Nikos Athanasiou, Muhammed Kocabas, and Michael J. Black. Learning to Regress Bodies from Images using Differentiable Semantic Rendering. In IEEE/CVF Interna- tional Conference Computer Vision (ICCV), pp. 11230-11239, 2021. ISBN 9781665428125. doi: 10.1109iccv4499222222..oo
[11] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collabo- rative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV), 1.

---

[12] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed
3D face model from in-the-wild images. In ACM Transactions on Graphics, (Proc. SIGGRAPH), volume 40, 2021. URL. https::t///.. //doi. org/ 10. 1145/3450626 3459936.
[13] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6M. leee Transactions on Pattern Analysis and Machine intelligence, pp. 1, 2014. ISSN 01628828. URL http://109. 101. 234. 42/ documente, / httpp/ublllaiiiinniieeeeiaaaaiit 82...1111110000 pdf.
[14] Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-Body Human Pose Estimation in the Wild. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12354 LNCS:196-214, 2020. ISSN 16113349 doi: -1.007/978-3-00--554444
[15] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 8320-8329, 2018. ISSN 10636919, doi: 10.1 (000/CVPR.2018.00666
[16] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild Wild 3D Human Pose Estimation. Proceedings 2021 Interna- rional Conference on 3D Vision, 3DV 2021, pp. 42-52, 2021. doi: 10. 1,111990553999222211 00015.
[17] Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-End Recovery of Human Shape and Pose. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 7122-7131, 2018. ISBN 978 9785538664209 doi: 10.1109/CVPR.2018.0000 0.0444...
[18] Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 43 (12):4217-4228, 2021. ISSN 19393539, doi: 10. 10.1199//AMM.2200229900 2977099919.
[19] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.1 1362, 2020.
[20] Muhammed Kocabas, Chun-Hao Huang, Otmar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In Proc. International Conference on Computer Vision (ICCV), pp- 11 11122-11133 October 2021.
[21] Muhammed Kocabas, Chun-Hao P. Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael Black. SPEC: Seeing people in the wild with estimated camera. In Proc. International Conference on Computer Vision (ICCV), pp. 11035-11045 October 2021.
[22] Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Daniilidis. Learning to reconstruct 3D human pose and shape via model-fitting the loop. Proceedings of the IEEE International Conference on Computer Vision, 0tt1900ctob22222222... 2019. ISSN 15505499. doi: 10.1 10111//CC..2221000333333
[23] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2019-Jume:4496 4505, 2019. ISSN 10636919. doi: 00.1099/CPPR.221900045
[24] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic Modeling for Human Mesh Recovery. In International Conference on Computer Vision (ICCV), pp. 11544, 2021. ISBN 978 9781665428125, doi: 10. 11.10//cc4499222221.0001 140.
[25] Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 19994-20006, 2023.
12

---

[26] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), (6:6))
2017. URI. htt.::tttt:/o/oo org/ 10. 1115/113300000 31308833.
[27] Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. arXiv preprint arXiv:2303.16160 2023.
[28] Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-End Human Pose and Mesh Reconstruction with Transformers. Proceedings of the IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 1954-1963, 2021. ISSN 10636919, doi: 10.1109/CVP444437. 2021.00199
[29] Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: Common objects in context. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 8693 LNCS(PART 5):740-755, 2014. ISSN 16113349. doi: 10.1007/978-3-4 1--319910002-11 48.
[30] Qihao Liu, Adam Kortylewski, and Alan L Yuille. Poseexaminer: Automated testing of out-of- distribution robustness in human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 672-681, 2023.
[31] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-2488 16, October 2015.
[32] Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. JEEE Transactions on Affective Computing, 10(1):18-31, 2019. ISSN 19493045. doi: 10. 10.1 109/TAAFFC.22111274022.
[33] Gyeongsik Moon and Kyoung Mu Lee. 12L-MeshNet: Image-to-Lixxt Prediction Network for Accurate 3D Human Pose and Mesh Estimation from Single RGB Image. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12352 L.CS:752-7688, 2020. ISSN 161 16113349, doi: 10.1007/ 978-3-030-58571-6644
[34] Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6mm A dataset and baseline for 3d interacting hand pose estimation from single rgh image. In European Conference on Computer Vision (ECCV), 2020.
[35] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Accurate 3d hand pose estimation for whole-body 3d human mesh estimation. In Computer Vision and Pattern Recognition Workshop (CVPRW), 2022.
[36] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets. IEEE Computer Society Conference Computer Vision and Pattern Recognition Workshops, 2022-June:2298 2306, 2022. ISSN 21607516. doi: 10.1 LLL19///VVPW553377222222000
[37] Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. Proceedings 2018 International Conference on 3D Vision, 3DV 2018, pp. 484-494, 2018. doi:
10.1109/DV2.00180000
[38] Hui En Pang. Zhongang Cai, Lei Yang, Tianwei Zhang, and Ziwei Liu. Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms. In NeurlPS, 2022.
[39] Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael Black. AGORA: Avatars in geography optimized for regression analysis. In Proceedings JEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), June 2021.
13

---

[40] Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. Learning to Estimate 3D Human Pose and Shape from Single Color Image. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4699468, 2018 ISSN 10636919. doi: 10.1 109/CV99 201.P..011.0005555
[41] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Osman, DImitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from single image. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, volume 2019-June, pp. 10967-10977, 2019. ISBN 9781728 9781728132938. doi: 10.1 10.1100/CVP..221..01111
[42] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11488-11499, 2021.
[43] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPHI Asia), 36(6), November 2017.
[44] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration. In Proceedings of the IEEE Inter- national Conference Computer Vision, volume 2021-Octob, pp. 1749-1759, 2021. ISBN 9781665401913. doi: 10. 20010000
[45] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black. Learning regress 3d face shape and expression from an image without 3d supervision. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2019.
[46] Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. Self-Supervised
3D Hand Pose Estimation from monocular RGB via Contrastive Learning. Proceedings of the IEEE International Conference on Computer Vision, pp. 1 11210-11219, 2021. ISSN 15505499. doi: N0.1100/CCCV44992222000
[47] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. Lecture Notes in Computer Science (including subseries Lecture Notes in Artifi- cial Intelligence and Lecture Notes in Bioinformatics), 11210 LNCS:536 553,, 2018. ISSN 161 13349. doi:
0000/99783-030001231.1 33.
[48] Yating Tian, Hongwen Zhang, Yebin Liu, and Limin Wang. Recovering 3D Human Mesh from Monocular Images: A Survey. arXiv preprint arXiv:2203.01923, pp. 1-20, 2022. URL
http://arxiv.. http://axxvvv..raaaaaa wttp:////////.////// 210000013...........
[49] Hsiao Yu Fish Tung, Hsiao Wei Tung, Ersin Yumer, and Katerina Fragkiadaki. Self-supervised learning of motion capture. Advances in Neural Information Processing Systems, 2017- December(Nips)55237-544 2017. ISSN 10495258.
[50] Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, and Gerard Pons- Moll. Recovering accurate 3D human pose in the wild using IMUs and moving camera. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11214 LNCS:614 631, 2018. ISSN 161 13349. doi:
[51] Jiahang Wang, Sheng Jin, Wentao Liu, Weizhong Liu, Chen Qian, and Ping Luo. When human pose estimation meets robustness: Adversarial algorithms and benchmarks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp-
11550-18859, 2021. ISSN 10636919. doi: 10. 10.100///244444..22....0.1
[52] Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness-aware reinforcement learning. Proceedings of the IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition, pp. 9319-9328, 2020. ISSN 10636919. doi:
02.109//VPP44260.222000000
14

---

[53] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 11119-June:0957-111900 2019. ISSN 10636919. doi: 10.1 109/ 2019.2019.01122
[54] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T. Freeman, Rahul Sukthankar, and Cristian Sminchiseseu. GHUM GHUML: Generative 3D human shape and articulated pose models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 6183-6192, 2020. ISSN 10636919. doi: 10. 00.10099CVP4422002222000000 00622..
[55] Yuanlu Xu, Song Chun Zhu, and Tony Tung. DenseRaC: Joint 3D pose and shape estimation by dense render-and-comparee Proceedings of the IEEE International Conference on Computer Vision, 2019-0ctob:7759-7766 2019. ISSN 15505499. doi: 10.1 109/1CCV.2019.000)
[56] Lei Yang, Qingqiu Huang, Huaiyi Huang. Linning Xu, and Dahua Lin. Learn to propagate reliably on noisy affinity graphs. In European Conference on Computer Vision, pp. 447-464. Springer, 2020.
[57] Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12351 LNCS:465 481, 2020.
ISSN 16113349. doi:
0111//7883-00-58555- 6_
[58] Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zheman Sun, and Yebin Liu. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
[59] Tianshu Zhang, Buzhen Huang, and Yangang Wang. Object- Human Shape and Pose Estimation from a Single Color Image. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 7374-7383, 2020. ISSN 10636919, doi: 10.1 11110///2442200000000000 202.0.000...0
[60] Yumeng Zhang. Li Chen, Yufeng Liu, Xiaoyan Guo, Wen Zheng, and Junhai Yong. Improving robustness for pose estimation via stable heatmap regression. Neurocomputing, 492:322-342,
2022. ISSN 0925-2312. doi: https://do..org/100 oor/1...10///ii......... wwt.://j.mmccomm222222.. 2022.04.444 04.6 URL https:
h/www/sciintte sci www.ssciieeeeeec.cccmm com/ stsssccinceeeaeeeee /tt.//ceeaatiice//p.iiii//////
[61] Yuxiao Zhou, Mare Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu. Monocular Real-time Full Body Capture with Inter-part Correlations. In Proceed- ings of the IEEE Computer Society Conference on Computer Vision and Patern Recogni- tion, number 61822111 11 in 61822111, pp. 4809-4820, 2021. ISBN 978 778666445092. doi: 10.1109/CVPR46437.00 20w.0444337222.0047....
[62] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single rgb images. Proceedings of the IEEE International Conference on Computer Vision, 2019-October: 813-822, 2019. ISSN 15505499. doi: 10. (09//CCV.2019000000
[63] Christian Zimmermann, Max Argus, and Thomas Brox. Contrastive Representation Learning for Hand Shape Estimation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13024 2665:250-264
2021. ISSN 161 13349. doi: 10. 1000/9797 3-030- 92659-5_16
15


Reviews:
[{"review_id": "Jj0XK8Vp3k", "reviewer": "Reviewer_p9Xq", "summary": "This submission proposes RoboSMPLX for whole-body pose and shape estimation. RoboSMPLX incorporates three modules, including a localization module, a contrastive feature extraction module, and a pixel alignment module. The localization module is aware of the location and semantics of body parts so that cropping could be more accurate. The contrastive feature extraction module incorporates a pose- and shape-aware contrastive loss, along with positive samples, for better feature extraction under robust augmentations. The pixel alignment  module applies differentiable rendering to inherence of the re-projection alignment of the mesh.", "questions": "How about the run time of the proposed method?", "limitations": "The limitations of this paper are mainly the novelty and the lack of comprehensive comparisons with recent state-of-the-art solutions. Given such clear defects in the experimental results, I rate this paper as the one below the acceptance bar.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation of the proposed method is clear: using a localization module to improve the robustness of the cropping. Overall, the proposed method technically makes sense and can be easily reproduced. The experiments also show comparable or better performances with previous methods.", "weaknesses": "The most severe weakness is the novelty of the proposed method and the lack of comparisons with recent state-of-the-art solutions.  \n\nTo be more specific, there are several major issues:\n\n- The proposed localization and pixel alignment modules have very limited contributions to the community, as these operations are commonly used in this field. For instance, similar localization strategies are used in [39,56], and differentiable modules are used in [i].\n[i] SK Dwivedi, N Athanasiou, M Kocabas, MJ Black, Learning to regress bodies from images using differentiable semantic rendering, ICCV 2021.\n\n- There is a lack of discussion and comparison with the recent state-of-the-art method PyMAF-X [54]. An in-depth discussion and comparison with [54] is necessary to support the claim of the proposed method. It is recommended to include [54] in the Related Work section, compare results with [54] in Tables 3,4,5, and Table 7, and show qualitative results of the proposed method and [54] for compressive comparisons.  \n\n- This paper claims robust performances of whole-body pose and shape estimation, but no video result is provided in the supplementary materials. To convincingly demonstrate the robustness of the proposed method, it is also recommended to include a side-by-side comparison video [54].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging the proposed method is clear. We will polish the paper, add the experiments and clarify below points in the revised version. \n\n**Q1: \"The proposed localization and pixel alignment modules have very limited contributions to the community, as these operations are commonly used in this field.\"**\n\n**A1:** Our method is significantly different from the mentioned works [39, 56, i]. Specifically, Expose [39] directly regresses pose and shape parameters from the image **without taking into account any location information.**\n\nFor [56], its IK modules (BodyIKNet and HandIKNet) focus **solely on keypoint data** for deriving pose parameters, and the FaceNet operates on a direct regression from the image. In contrast, our Localization module effectively captures **both sparse (through a 2.5 heatmap for keypoint location) and dense (through part segmentation) predictions for each human body part (body, face and hand).** The encoded location information is then used for prediction of pose, shape and camera parameters. The ablation study in Table 7 shows the superiority of our design over [56]. \n\nOur method is technically different from [i] in several primary aspects: \n1. [i] relies on an existing clothing segmentation model to retrieve a clothing segmentation mask, while we capitalize on the projected ground-truth mesh to source the part-segmentation mask, which is more cost-efficient as many SMPL-X datasets contain ground-truth wholebody parameters. \n2. We have found that a differentiable part segmentation map holds a higher efficacy compared to normal silhouette supervision. This encourages learning of correct prediction of body part and silhouette,  even in instances of object or self-occlusion.\n3. The module we introduce aims to bridge the disparity in pixel alignment observed in numerous SMPL-X models. Moreover, the integration of this module predominantly facilitates the accurate determination of camera parameters and enhances pixel alignment. Even marginal variations in camera parameters can induce significant alignment shifts. \n\n\n**Q2a: \"There is a lack of discussion and comparison with the recent state-of-the-art method PyMAF-X [54]\"**\n\n**A2a: 1)** Thanks for the suggestion. Below we provide detailed discussions and comparisons with PyMAF-X. We will also include [54] in the Related Work section.\n1. **Acquisition of part bounding boxes**: PyMAF-X relies on an off-the-shelf whole-body pose estimation model (OpenPifpaf) to obtain whole body 2D keypoints of the person in the image, from which part crops are derived. During the EHF evaluation, PyMAF-X employs ground-truth hand and face bounding boxes. In contrast, our method and other works (ExPose [5], PIXIE [9], Hand4Whole [32], OS-X [25]) encompass a self-integrated module designed to extract hand and face bounding boxes directly from the image. So it is unfair to directly compare these works with PyMAF-X. \n2. **Operational efficiency**: Openpifpaf imposes extra computation during inference, making PyMAF-X less efficient than our method. Please refer to Table 1 in General Comments 2.\n3. **Network architecture**: Due to the diverse backbone and dataset combinations utilized, it is challenging for us to make whole-body network comparisons. In our paper (Table 1), we focus on contrasting RoboSMPLX’s Hand subnetwork with PyMAF’s Hand subnetwork. Both networks are trained and evaluated on the same backbone and dataset, FreiHAND. In this context, our method surpasses PyMAF.\n4. **Performance**: On the EHF metrics, our performance lags behind PyMAF-X. This could potentially arise from variations in the training datasets employed. While the training pipeline of the body network for PyMAF-X has been disclosed, the training specifics for hands and face and the methodology to integrate hand, face, and body module PyMAF-X, remains undisclosed. We intend to replicate with similar training datasets in the future.\n\n**Q2b: \"recommended to compare results with [54] in Tables 3,4,5, and Table 7, and show qualitative results of the proposed method and [54] for compressive comparisons.\"\"**\n\nFor comparisons on Tables 3, 4, 5, PyMAF-X did not provide the [pre-trained hand and face models](https://cloud.tsinghua.edu.cn/d/3bc20811a93b488b99a9/?p=%2Fdata%2Fpretrained_model&mode=list) to be evaluated on the respective FreiHand and Stirling benchmarks. In addition, the face model of PyMAF-X was trained on [VGGFace2](https://www.robots.ox.ac.uk/~vgg/data/vgg_face2/) which is no longer publicly available. Therefore, we were unable to reproduce a result with similar training configurations. \n\nFor Table 7 when evaluating the whole body, the hand and face evaluations are affected by the accuracy of the part bounding box detected. In PyMAF-X’s evaluation on EHF dataset, the ground-truth part- bounding boxes are fed in for evaluation. Therefore this poses an unfair comparison. If we were to obtain the part-bounding box from OpenPifPaf, we will be indirectly evaluating OpenPifPaf, rather than PyMAF-X as they do not have any module for locating the part-bounding boxes. \n\n**Q3: \"include a side-by-side comparison video \"**\n\nWe plan to include additional visualizations here https://github.com/robosmplx/RoboSMPLX/.\n\n**Q4: \"run time of the proposed method\"**\n\nPlease refer to our response to General Concerns 2.\n\nPlease don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!\n\n[1] SK Dwivedi, N Athanasiou, M Kocabas, MJ Black, Learning to regress bodies from images using differentiable semantic rendering, ICCV 2021.\n\n"}, {"review_id": "62LG3htS3r", "reviewer": "Reviewer_go2c", "summary": "This paper proposes a method to improve the robustness of whole-body pose and shape estimation, which mainly contains three components: 1) localization module to give the network awareness of location and semantic part; 2) contrastive feature extraction module to predict consistent representations under different augmentations; 3) pixel-alignment module to ensure alignment between projected mesh and 2d evidence.\n", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) This paper is well-written and easy to understand\n\n2) The topic of robustness of pose and shape estimation is meaningful and  the proposed method is effective against augmentations\n\n3) The experiments are comprehensive and visualizations are very nice to help understand the method\n", "weaknesses": "1) Since the main topic is the robustness of whole body pose and shape estimation, the literature review of robustness in vision tasks, especially in pose estimation tasks should be included. For example, [1][2].\n\n[1] Bai, Yutong, et al. \"CoKe: Contrastive Learning for Robust Keypoint Detection.\"\n[2] Zhang, Yumeng, et al. \"Improving robustness for pose estimation via stable heatmap regression.\"\n\n\n2) I wonder why the target task is whole-body pose and shape estimation rather than body/hand pose estimation?\n3) Why does ‘baseline’ only appear in table2 but not in table1 and 3? Also, the definition of baseline is not clear. The definition of baseline is not clear either in table 9\n4) Where is the result of robustness of body subnetwork again augmentations?\n5) Where is the details of table 7?\n6) Eq,2 typo\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: \"literature review of robustness in vision tasks, especially in pose estimation tasks should be included\"**\n\n**A1:** Thank you for your suggestion of a comprehensive literature review. We would like to emphasize that **prior works on 2D pose estimation are different from our 3D pose and shape estimation tasks. They focus on 2D/3D joint locations while we target the prediction of pose, shape and camera parameters for image subjects.** Below is our detailed discussion, which will be added to the revision. \n\n“Tackling robustness in vision tasks have motivated extensive works. Within the domain of human pose estimation, strategies including data augmentation, architectural innovations, and diverse training strategies have been actively explored. Specifically, (1) \"AdvMix\" [3] stands out by enhancing the robustness of pose estimation models using data augmentation. This method combines adversarial augmentation with knowledge distillation, where a generative framework mixes various corrupted images to confuse a pose estimator. Through such adversarial training, the estimator is conditioned to learn from harder samples, making it more robust. (2) Architectural modifications have been explored as a means to increase robustness. For instance, the work in [2] delineates a unique heatmap regression approach, encompassing three core components: a row-column correlation layer, a highly differentiated heatmap regression, and a maximum stability training protocol. This strategy is devised to buffer the network against minute perturbations. (3) Contrastive learning has also been applied to improve robustness.  Bai et al. [1] introduced “CoKe”, a contrastive learning framework tailored for keypoint detection. By detecting each keypoint independently, the method demonstrates robustness, especially against occlusions, compared to more conventional approaches.\n\n**Q2: \"I wonder why the target task is whole-body pose and shape estimation rather than body/hand pose estimation?\"**\n\n**A2:** 3D pose and shape estimation and 2D/3D pose estimation are fundamentally two different tasks. **The former involves determining the \"pose\" and \"shape\" parameters of a statistical human body model, whereas the latter concentrates on predicting the locations of 2D/3D keypoints within an image.** We follow existing works [16, 21, 19, 23] in the task definition of human pose and shape estimation. 3D whole-body pose and shape estimation allows us to retrieve a human mesh from the predicted pose and shape parameters, opening a myriad of applications across diverse domains such as computer graphics and augmented/virtual reality. It has been a popular topic and attracts lots of attention [16, 21, 3, 15, 23, 8, 20, 19, 9, 5, 56, 42, 54] \n\n**Q3: \"Why does ‘baseline’ only appear in table2 but not in table1 and 3? Also, the definition of baseline is not clear. The definition of baseline is not clear either in table 9\"**\n\n**A3:** Sorry for the confusion. Baseline refers to training with the same datasets and backbone on HMR, but with the addition of the extra modules (Localization, Contrastive FE and Pixel Alignment Modules). For Table 1, the Baseline is HMR which is included. For Table 3, we have updated the baseline below. \n\nUpdated Table 3. Evaluation of the Face subnetwork.\n| Method      | LQ Mean(mm) &darr; | HQ Mean(mm) &darr; |\n| ----------- | ------------- | ------------- |\n| ExPose [5]  | 2.27          | 2.42          |\n| ExPose †    | 2.46          | 2.38          |\n| HMR         | 2.18          | 2.11          |\n| HMR†        | 2.31          | 2.27          |\n| RoboSMPLX   | 2.12          | 2.08          |\n| RoboSMPLX † | 2.12          | 2.1           |\n\n**Q4: Where is the result of robustness of body subnetwork again augmentations?\"**\n\n**A4:** \nThanks for the great suggestion. Please find the result in the table below. Similar to the findings in Table 11, we conclude that training with strong augmentations can cause domain shift. This was also found in prior works [15, 19, 35]. In Table 4 of PARE [19], and Table 4 of HMR-EFT [15], Table 8 of [35], they also showed that adding crop augmentation can harm performance on existing benchmarks.\n\nAblation of different modules on Body subnetwork. Results are trained on EFT-COCO and tested on 3DPW test set.\n|                            | PA-MPJPE &darr; | MPJPE &darr;|\n| -------------------------- | -------- | ----- |\n| Baseline (HMR)             | 60.8     | 96.2  |\n| Baseline (HMR) + strongaug | 63.2     | 101.5 |\n\n**Q5: Where is the details of table 7?\"**\n\n**A5:** Thanks for pointing this out. It was previously wrongly referenced to Table 18 in Line 288. We have fixed it, it is now referenced to Table 7 instead. We hope this helps to clarify.\n\n**Q6: Eq,2 typo\"**\n\n**A6:** We would like to clarify what does “Eq,2 typo” refer to?\n\nPlease don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!\n\n"}, {"review_id": "WNxH1oRQU2", "reviewer": "Reviewer_bkwo", "summary": "The paper introduces RoboSMPLX, a method for whole-body 3d human pose and shape estimation from a monocular image. Motivated by the poor robustness of existing methods, especially w.r.t. the quality of bounding boxes, three components are proposed: 1) a localization module, 2) a contrastive feature extraction module, and 3) a pixel alignment module. The robustness of existing methods is evaluated by applying different types of image, location and pose augmentations. By using these augmentations in a contrastive setting during training of RoboSMPLX, RoboSMPLX achieves higher robustness when evaluated on these augmentations. Additionally, the performance of Hand, Body, Face and Wholebody reconstructions are evaluated on typical benchmark datasets.", "questions": "The following requires clarification:\n- What is the estimation error reported in Table8?\n- Table 11: what does DR54 and KS stand for? What is the row named \"joints\"?\n- Regarding line 204: why should the model produce consistent representations for the same subject under different poses?\n- For how long is the model trained on what hardware? What's the inference time?\n- How are the joints for the body, hand and face determined? How does the 137 joints body skeleton look like?", "limitations": "More failure cases could be shown in the paper. Also in comparisons to the competitors.\n\nThe potential societal impact is not discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Increasing the robustness of whole-body human pose and shape estimation methods is an important topic. The problem is well motivated, and the evaluation of the performance of existing methods under different augmentations is interesting.", "weaknesses": "I have the following main concerns about the paper:\n\n1) Evaluation seems inconsistent and incomplete.\nIt is unclear why some competitors are omitted in different experiments. For example, the performance of PyMAF[54] is reported in Table 1 and 2, but not in Table 3 and 6, even though [54] reports the relevant numbers. OSX[25] is also missing in Table 2. The competitors outperform RoboSMPLX in the experiments where they are omitted. \n\n2) Robustness of RoboSMPLX. \nThe robustness evaluation seems not realistic. It is not surprising that RoboSMPLX shows better performance under the different augmentations the method is trained on. However, most relevant is the performance on in-the-wild scenarios. This is not properly evaluated for the whole-body task. On the contrary, RoboSMPLX performs worse on AGORA than OSX. Although AGORA is a synthetic dataset, it contains realistic and diverse scenes with multiple persons and occlusions. On the other hand, EHF only consists of 100 images of a single subject recorded in a mocap studio. It is therefore questionable, if RoboSMPLX really succeeds in being more robust than its competitors. To better assess the in-the-wild performance, the methods could for example be evaluated on RICH (CVPR'22) (https://rich.is.tue.mpg.de/) or on BEDLAM (CVPR'23) (https://bedlam.is.tue.mpg.de/). Additionally, to show the effectiveness of the localization module, the accuracy of the predicted bounding boxes could be evaluated.\n\nFuther concerns:\n- The contrastive module is not well motivated. Why use the contrastive setting with all its overhead, instead of simply using the different transformations as data augmentation? Especially since adding positive samples only has small influence.\n\n- Why are the ablation studies in Table 11 not conducted with the whole body model? Mean+std of multiple runs should be reported since the differences are so small.\n\n- Why is the pixel alignment not evaluated quantitatively? This would be more meaningful instead of only showing some examples e.g. in Figure 13.\n\n- The experiments section should be revised. It is difficult to read due to the many experiments and referrals to the appendix. \n\n- The novelty is limited since the augmentations and the pixel alignment module already have been used extensively in the literature for estimating human pose and shape, e.g. [35] and [45].\n\n- IIn Table 10 7.18 PA-PVE should be in bold, and 14.38 MPJPE in Table 11.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedback. We will polish the paper, add the experiments and make the clarifications in the revised version.\n\n**Q1: \"Evaluation seems inconsistent and incomplete. It is unclear why some competitors are omitted in different experiments.\"**\n\n**A1:** **Our evaluation methodology has been systematically designed based on the underlying models of the compared methods.** Specifically, the body network was trained on datasets tailored for SMPL and hence was benchmarked against other methods utilizing the SMPL framework, including PyMAF [54]. Conversely, OS-X, which is rooted in the SMPL-X framework, was compared solely with its SMPL-X counterparts, such as Expose, Hand4Whole and PIXIE in Table 6 and not Table 2. For comparisons with PyMAF-X, please refer to  **Response 1 A2**\n \n**Q2: \"Additionally, to show the effectiveness of the localization module, the accuracy of the predicted bounding boxes could be evaluated**\n\n**A2:** Thank you for the suggestion. We have evaluated the accuracy of predicted part bounding boxes on the EHF test set. We have employed the Intersection over Union (IoU) as our metric (Please refer to Figure and Table 1 in the Rebuttal.pdf). Our method obtains the highest IoU scores in Table 1.\n\n**Q3: \"Why are the ablation studies in Table 11 not conducted with the whole body model?\"**\n\n**A3:** **The primary objective of the ablation study was to demonstrate the efficacy of individual modules.** Given that wholebody network contains three subnetworks (body, hand, face), and each subnetwork encompasses all three modules, **it was deemed more methodologically sound to isolate and assess the impact of each module within a distinct subnetwork.** This approach ensures a clearer understanding of the contribution of each module without the potential confounding effects of evaluating them within the entirety of the whole body model.\n\n**Q4: The contrastive module is not well motivated. Why use the contrastive setting with all its overhead, instead of simply using the different transformations as data augmentation? \"**\n\n**A4:** Our use of the contrastive module is motivated by the need to constrain/maintain the same pose feature for different augmentations, to avoid domain shift caused by strong augmentation alone. The experiments show that the use of strong augmentation alone for training can lead to performance deterioration, while combining it with the contrastive loss consistently results in minimal errors (Table 2 in Rebuttal.pdf)).\n\nThe contrastive features extraction module is not used in the inference, and only used during training. Therefore, during inference time it will not pose any computation overhead. Contrastive learning is a successful strategy in other CV areas [43, 58], and we are the first to introduce it on pose for Human Pose and Shape Estimation.\n\nTo illustrate this further, we delved into a visualization of the pose similarity for augmented samples. The findings reveal that augmented samples are perceived as dissimilar in both Model 0 and Model 1 (Table 3 in Rebuttal.pdf)). Yet, when examining Model 2, a marked increase in embedding similarity is evident, underscoring the advantage of the contrastive approach. \n\n**Q5: Why is the pixel alignment not evaluated quantitatively? \"**\n\n**A5:** In assessing the pixel alignment, it's crucial to recognize that standard metrics like PVE and MPJPE errors are computed post root alignment. They also do not measure the accuracy of mesh projection within the image space. Presently, there's an absence of a metric tailored to gauge the degree of pixel alignment of a mesh in this context. As such, our study primarily offers a qualitative analysis. \n\nNevertheless, in an effort to address your concern and introduce some quantitative results, we measure the errors between the projected 2D vertices of ground-truth and projected mesh (Please refer to Figure 2 and Table 3 in Rebuttal.pdf). From Table 3, it is evident that omitting the pixel alignment module leads to suboptimal outcomes. In contrast, our pixel alignment strategy, leveraging rendered segmentation maps, showcases better performance than using vertex loss as supervision.\n\n**Q6: The experiments section should be revised. \"**\n\n**A6:** Thanks for the suggestion. If we have the opportunity for revision, we will include important experiments in the main text as much as possible, while maintaining the completeness of information without frequent reference to supplementary materials. \n\n**Q7: In Table 10 7.18 PA-PVE should be in bold, and 14.38 MPJPE in Table 11\"**\n\n**A7:**  Thanks for pointing that out, we will fix the bolding in Table 11. \n\n**Q8: What is the estimation error reported in Table8?\"**\n\n**A8:** The estimation error for Table 8 refers to the difference between the top-1 retrieved pose (COCO-train) and query pose (COCO-test). Thanks for pointing this out, we will edit the caption to make it clearer.\n\n**Q9: Regarding line 204: why should the model produce consistent representations for the same subject under different poses?**\n\n**A9:** L204:  “By minimizing this loss, the model can produce consistent representations for the same subject, even when presented with different augmentations”. To clarify, this means that the model should produce consistent representation for the same subject with the same pose under different augmentations. \n\n**Q10: For how long is the model trained on what hardware? What's the inference time?**\n\n**A10:** Our model was trained utilizing a cluster of 8xTesla V100-SXM2-32GB GPUs. Specific to the training duration, the hand models required approximately one day, whereas the body and face models necessitated two days. The joint training process was completed within a day. For details on inference time, please refer to the section labeled \"General Concerns 2.\"\n\nPlease don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!\n"}, {"review_id": "othgAZ251J", "reviewer": "Reviewer_4HYU", "summary": "This paper addresses the task of whole-body pose and shape estimation, including human mesh, hand gestures, and facial expressions, from monocular images. The author identifies the impact of predicted bounding box quality on the accuracy and reliability of existing methods. Based on this observation, a novel framework called RoboSMPLX is proposed to enhance robustness through three modules: a localization module, a contrastive feature extraction module, and a pixel alignment module. Comprehensive experiments are conducted to demonstrate the effectiveness of the framework.", "questions": "See weakness", "limitations": "1.  Although the proposed framework achieves impressive performance, the overall framework and training strategy is somewhat complex. Simplifying the framework without sacrificing performance would be beneficial. \n\n2. Currently, the proposed framework only allows for whole-body estimation from a single image. Future work could focus on video-based estimation to further improve robustness, alleviate depth ambiguity, and enhance temporal smoothness.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the paper is of high quality, with clear motivation and well-organized sections. It is easy to follow and presents novel modules that address existing limitations. The paper includes comprehensive experiments and provides sufficient visualizations of the generated mesh. The author's contributions are highly appreciated.\n\n2. The empirical study on the impact of subject localization, feature extraction, and pixel alignment, as stated in lines 91-100, is crucial in breaking through barriers. The paper provides extensive qualitative and quantitative results, both in the main paper and supplementary material, making it solid and convincing.\n\n3. The three modules work together effectively, aligning intermediate representations such as pose, dense landmarks, segmentation, and silhouette masks. This enhances robustness through proper data augmentation techniques.\n\n4. The proposed framework consistently achieves impressive results for various tasks across multiple datasets, demonstrating superior performance in whole-body pose and shape estimation.", "weaknesses": "1. Since the author does not mention code release, it would be beneficial to release the code to contribute to the research community.\n\n2. The paper lacks a discussion on computational complexity and inference speed, which are important evaluation metrics, especially for practical deployment. I suggest reporting total parameters, FLOPs, and fps with a detailed discussion to strengthen the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging that we present novel modules, and recognising that our extensive qualitative and quantitative results are solid and convincing. We will polish the paper and add the clarifications below in the revised version. \n\n\nBelow we would like to provide point-to-point responses to address all the raised questions: \n\n**Q1: \"it would be beneficial to release the code to contribute to the research community.\"**\n\n**A1:** Thanks for your suggestion. We plan to release code here at https://github.com/robosmplx/RoboSMPLX/ when the paper is accepted. \n\n**Q2: \"reporting total parameters, FLOPs, and fps with a detailed discussion to strengthen the paper\"**\n\n**A2:** Please refer to our response to General Concerns 2.\n\n**Q3: \"the overall framework and training strategy is somewhat complex. Simplifying the framework without sacrificing performance would be beneficial\"**\n\n**A3:** Thanks for this great suggestion. Our current goal is to enhance the model robustness. How to simplify the framework without performance drop will be our future work. \n\n**Q4: \" Future work could focus on video-based estimation to further improve robustness, alleviate depth ambiguity, and enhance temporal smoothness\"**\n\n**A4:**  Thanks for this great suggestion. This is indeed an interesting and important direction, and we will explore this in future work.\n\n\nPlease don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!\n"}, {"review_id": "Rlmo4s7bCP", "reviewer": "Reviewer_Uf3X", "summary": "The paper focuses on improving whole-body pose and shape estimation from monocular images, a task that often struggles with complex, real-world scenarios. The authors argue that the performance of these models is significantly impacted by the quality of the predicted bounding box, such as the scale and alignment of body parts. The difference between ideal bounding box annotations and model detection results poses a substantial challenge to whole-body pose and shape estimation. To address this, the authors introduce RoboSMPLX, a framework to enhance the robustness of whole-body pose and shape estimation. RoboSMPLX integrates three new modules: a Localization Module to improve model awareness of the subject's location and semantics within the image space, a Contrastive Feature Extraction Module that uses contrastive loss with dedicated positive samples to help the model be invariant to robust augmentations, and a Pixel Alignment Module to ensure the reprojected mesh from predicted camera and body model parameters are accurate and pixel-aligned. The effectiveness of the method is showcased through comprehensive experiments on body, hands, face, and whole-body benchmarks.", "questions": "The paper seems sound to me and I do not have much questions.", "limitations": "Although some limitations have been discussed it might not be adequate. Please add some visual limitations of the model during the inference time.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-structured and reader-friendly.\n\nIt begins with a compelling motivation that efficiently highlights the issue of the current state-of-the-art methods' low robustness against varying crops and misalignments. The authors also convincingly justify potential solutions to these problems.\n\nThe proposed method by the authors is sound even if it lacks complete novelty.\n\nImpressively, the authors have performed extensive experiments and provided an in-depth ablation study, substantiating their proposed approach effectively.\n\nAlthough their method doesn't top all the benchmarks, its efficacy is undeniable, particularly in its demonstrated robustness to variations in crop size and alignment - a challenge that other methods fail to meet. The authors provide a thorough justification with robust qualitative and quantitative results.", "weaknesses": "The claim made in lines 47-51 and 76-77, suggesting that contrastive learning is not used in the parametric estimation of human meshes, is inaccurate. Previous works, such as [1], have already incorporated a form of contrastive and triplet loss for 3D face shape and pose estimation from monocular face images. Thus, the application of contrastive loss for mesh recovery from monocular images isn't entirely novel. The authors should moderate their claims and include a thorough discussion concerning related work in this field.\n\nFurthermore, the pixel alignment loss achieved by projecting the mesh as a mask, a technique used by ICON [2] for improved SMPL pose parameter estimation, is not entirely new.\n\nThe authors should also include instances of the method's failure during testing within the main body of the paper.\n\n[1] Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision. CVPR 2019\n[2] ICON: Implicit Clothed humans Obtained from Normals. CVPR  2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our extensive experiments and in-depth ablation study, and recognising the efficacy of our methods. We will polish the paper and add the clarifications below in the revised version. \n\nBelow we would like to provide point-to-point responses to address all the raised questions: \n\n**Q1: \"application of contrastive loss for mesh recovery from monocular images isn't entirely novel. The authors should moderate their claims and include a thorough discussion concerning related work in this field.\"**\n\n**A1:** Thank you for your suggestion. We have added [1] in the discussion for related works in this field. \n\nAs stated in Sec.4.3, our novel design lies in: \n1.  **The scheme of supervised contrastive learning for high-dimensional regression.** Prior SSL methods [43, 58, 4] are not particularly adept at learning useful embeddings for human pose and shape estimation. Without labels, the model primarily extracts features based on background information instead of pose information. \n2. **The design of human pose representation.** Sanyal et al. [1] use shape as the representation to disentangle expression, head pose and camera parameters. To learn accurate poses, we find that using the normalized keypoints representation in 3D is more useful than using pose parameters (joint rotations). \n3. **The sampling strategy for selecting positive samples.** Prior works [43, 58] employed pose-variant augmentations (e.g., rotation and flipping), which can adversely affect the learning by altering the global orientation.\n\nBelow is the discussion and comparisons with existing methods, which will be added to the related works in the revision: \n\n“Contrastive Learning. Recently contrastive learning has demonstrated state-of-the-art performance among self-supervised learning (SSL) approaches. This strategy has been applied to 3D hand pose and shape estimation [43, 58]. **Sanyal et al [1] incorporates a novel shape consistency loss for 3D face shape and pose estimation that encourages the face shape parameters to be similar when the identity is the same and different for different people.** Choi et al. [4] were the first to apply contrastive learning for 3D human pose and shape estimation. They found that SSL is not useful for this task, as the learned representations could be challenging to embed with high-level human-related information. Khosla et al. [18] proposed supervised contrastive learning for image classification tasks, which incorporates label information during training. **Currently there is no attempt to apply this strategy to human pose and shape estimation, where the definition of positive samples is unclear, and data lie in a continuous space. We are the first to overcome these challenges and integrate supervised contrastive learning with whole-body pose and shape estimation.**”\n\n**Q2: \"pixel alignment loss achieved by projecting the mesh as a mask, a technique used by ICON [2] for improved SMPL pose parameter estimation, is not entirely new.\"**\n\n**A2:** We acknowledge the methodological parallels between our work and that of ICON. However, it's crucial to delineate the distinctions in implementation and objectives.\n\nICON [2] relies on an existing segmentation model to retrieve a silhouette mask of a clothed human. Pixel alignment loss is employed during inference time to refine the SMPL parameters by fitting the projected SMPL silhouette to the “ground-truth” clothed silhouette mask.  \n\nOur method differs from ICON is several ways:\n- Our methodology adopts a regression approach to directly predict pixel-aligned mesh. Pixel alignment loss is exclusively employed during the training phase to supervise the ground-truth and predicted part segmentation map. This offers a notable speed-up over ICON's optimization procedure. \n- Additionally, our approach circumvents the need for an external pretrained network dedicated to clothed silhouette mask prediction. The fitting results of ICON [2] depend on the accuracy of the segmentation model, which remains limited in occlusion scenarios. \n- We have found that a differentiable part segmentation map holds a higher efficacy compared to normal silhouette supervision. This encourages learning of correct prediction of body part and silhouette, even in instances of object or self-occlusion.\n\n\n**Q3: \"instances of the method's failure during testing\"**\n\n**A3:** Thank you for the suggestion. Please refer to Figure 3 in Rebuttal document.\n\n[1] Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision. CVPR 2019 \n[2] ICON: Implicit Clothed humans Obtained from Normals. CVPR 2022\n\nPlease don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!"}]

Author rebuttal:
We sincerely thank all the reviewers for your constructive feedback and recognitions of this work, especially for acknowledging that the problem is well motivated and meaningful [p9Xq, bkwo, go2c, 4HYU, Uf3X], the experiments are comprehensive [go2c, 4HYU, Uf3X], and the proposed modules are effective [Uf3X, 4HYU, go2c]. 

We will follow your suggestions to polish the paper, add the experiments and make the clarifications in the revised version. 

**General concerns:**

**1. Clarification about the novelty of the proposed method**

Below, we summarize the novelty of our approach and differences from previous works:

Our research aims to enhance the robustness of whole-body pose and shape estimation. Notably, many current methods face challenges in maintaining performance under the augmentations commonly observed in complex in-the-wild scenarios. We posit that the accuracy and reliability of such models are influenced by the quality of the predicted bounding box, especially concerning the scale and alignment of individual body components. In addressing these issues, we introduce three novel modules and empirically validate their efficacy across body, hand, face and whole body models.
1. **Localization Module**: This module incorporates both sparse and dense prediction branches, ensuring the model is aware of the location and semantics of the subject’s parts in the image. The learned location features that encode information about the sparse and dense representations are helpful in recovering relative rotations, shape and camera parameters. Our work is distinct from previous SMPL-X methods [5, 9] that do not use any location information. We also demonstrate that our method is more effective than previous methods that only use joint features [32, 25] or keypoints [56] for recovering pose, but neglect the effectiveness to recover shape and camera parameters. In the whole-body estimation pipeline, these location information play a pivotal role in localizing bounding boxes for hand and face subnetworks, which previous methods have a separate bounding box predictor for.
2. **Contrastive Module**: Merely leveraging strong data augmentation can introduce domain shifts. This was also found in prior works [15, 19, 35]. In Table 4 of PARE [19], and Table 4 of HMR-EFT [15], Table 8 of [35], it shows that adding crop augmentation can harm performance on existing benchmarks.  Our approach integrates supervised contrastive learning, utilizing the regressed keypoints from the mesh as the representation. This encourages the network to learn to produce similar embeddings for samples with the same pose under different augmentations.
3. **Pixel Alignment Module**: Minor deviations in scale or positional translations often result in visible misalignments in the projected mesh, indicative of errors in camera parameter estimations. While prior work relied on the supervision of projected keypoint for learning camera parameters, we introduce dense supervision of projected mesh using part-segmentation maps through differentiable rendering. This helps to ensure accurate pixel alignment of outputs. Notably, the combination of part-segmentation supervision and differentiable rendering has not been applied in whole-body pose and shape estimation.

**2. Run-time/ inference speed**

We measure the model size, computation complexity and inference time for different models including ours, as shown in the table below. Although our framework has sophisticated design, it has comparable inference speed as others, validating its efficacy. 

Table 1: These results are tested on RTX3090. FLOP refers to the total number of floating point operations required for a single forward pass. The higher the FLOPs, the slower the model and hence low throughput. Inference Time is obtained by averaging across 100 runs.
|                       | Total parameters (M) | GFLOPs         | Inference time (s) |
| --------------------- | -------------------- | -------------- | ------------------ |
| ExPose                | 26.06                | 21.04          | 0.1330 &pm; 0.0050      |
| PIXIE                 | 109.67               | 24.23          | 0.1670 &pm; 0.0065      |
| Hand4Whole            | 77.84                | 17.98          | 0.0709 &pm; 0.0022      |
| OSX                   | 422.52               | 83.77          | 0.1998 &pm; 0.0028      |
| PyMAF-X (gt H/F bbox) | 205.93               | 33.41          | 0.2194 &pm; 0.0027      |
| PyMAF-X + OpenPifpaf   | 205.93 + 115.0       | 33.41 + 120.52 | 0.2727 &pm; 0.0136      |
| RoboSMPLX             | 120.68               | 29.66          | 0.2008 &pm; 0.0220      |


Meta-review: This paper got mixed ratings (A, WA, BA, R, BR) from reviewers. 

The major strength recognized by reviewers is their high performance and practical efficacy on whole-body human pose and shape estimation, achieved by the proposed three modules.

The major concerns raised by the reviewers on the negative sides are regarding the level of novelty and some missing evaluations. 

The authors' rebuttal and answers mostly resolved other concerns of reviewers.  

The AC is inclined to the positive side. While the AC agreed that the technical innovation of each module would not be very strong, the contribution of each component is well supported by the experiments and their combinations demonstrate their practical advantages on the well-motivated research problem. The AC also believes the authors convincingly answered the questions and concerns raised by reviewers. Overall, the paper meets the bar of NeurIPS.  
Authors should very carefully check the reviewer’s feedback and improve the quality of the paper for the camera-ready version.
Decision: Accept (poster)