Title: Towards Robust and Expressive Whole-body Human Pose and Shape Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Whole-body

SMPLX Model

Human Pose and Shape Estimation

Human Mesh Recovery
Abstract: Whole-body pose and shape estimation aims to jointly predict different behaviors (e.g., pose, hand gesture, facial expression) of the entire human body from a monocular image. Existing methods often exhibit suboptimal performance due to the complexity of in-the-wild scenarios. We argue that the prediction accuracy of these models is significantly affected by the quality of the _bounding box_, e.g., scale, alignment. The natural discrepancy between the ideal bounding box annotations and model detection results is particularly detrimental to the performance of whole-body pose and shape estimation.
In this paper, we propose a novel framework to enhance the robustness of whole-body pose and shape estimation. Our framework incorporates three new modules to address the above challenges from three perspectives: (1) a **Localization Module** enhances the model's awareness of the subject's location and semantics within the image space; (2) a **Contrastive Feature Extraction Module** encourages the model to be invariant to robust augmentations by incorporating a contrastive loss and positive samples; (3) a **Pixel Alignment Module** ensures the reprojected mesh from the predicted camera and body model parameters are more accurate and pixel-aligned. We perform comprehensive experiments to demonstrate the effectiveness of our proposed framework on body, hands, face and whole-body benchmarks.

Towards Robust and Expressive Whole-body Human Pose and Shape Estimation

Abstract

Whole-body pose and shape estimation aims to jointly predict different behaviors (e.g., pose, hand gesture, facial expression) of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We argue that the accuracy and reliability of these models are significantly affected by the quality of the predicted bounding box , e.g., the scale and alignment of body parts. The natural discrepancy between the ideal bounding box annotations and model detection results is particularly detrimental to the performance of whole-body pose and shape estimation. In this paper, we propose a novel framework RoboSMPLX to enhance the robustness of whole-body pose and shape estimation. RoboSMPLX incorporates three new modules to address the above challenges from three perspectives: 1) Localization Module enhances the model's awareness of the subject's location and semantics within the image space. 2) Contrastive Feature Extraction Module encourages the model to be invariant to robust augmentations by incorporating contrastive loss with dedicated positive samples. 3) Pixel Alignment Module ensures the reprojected mesh from the predicted camera and body model parameters are accurate and pixel-aligned. We perform comprehensive experiments to demonstrate the effectiveness of RoboSMPLX on body, hands, face and whole-body benchmarks. Codebase is available at https://github.com/robosmplx/robosmplx .

1 Introduction

Human pose and shape estimation tries to build human body models from monocular RGB images or videos. It has gained widespread attention owing to its extensive applications in various fields, including robotics, computer graphics, and augmented/virtual reality. Early works use various statistical models (e.g., SMPL [31], MANO [43], FLAME [26]) to individually reconstruct different parts, including human body [17, 22, 4, 16, 24, 10, 21, 20], face [9, 8, 12], and hand [28, 3, 63]. Recently, there is a growing interest in whole-body estimation [11, 6, 61, 44, 58], which jointly estimates the pose, hand gestures and facial expressions of the entire human body from the input. Commonly these methods first employ separate sub-networks to extract the features of body, hands and face. These features are then used to predict whole-body 3D joint rotations and other parameters (e.g., body shape, facial expression), which are further combined to generate the whole-body 3D mesh. This is a crucial step towards modeling human behaviors in an efficient and practical manner.

However, achieving accurate and robust whole-body estimation is particularly challenging as it requires precise estimation of each body part and the correct connectivity between them. In particular, due to the smaller sizes of hand and face images, they are typically localized, cropped and resized to higher resolutions before being processed by the relevant sub-network. To tackle the absence

of ground-truth bounding boxes in the real-world scenarios, existing whole-body methods utilize various detection techniques to obtain the crops. The accuracy of the whole-body estimation is highly sensitive to the quality of input crops. Our experiment results in Section 3 show that even minor fluctuations in the scale and alignment of input crops can significantly affect the model performance, indicating a limited ability to localize and extract meaningful features about the subject in the image.

The lack of robustness in existing whole-body pose and shape estimation methods highlights three critical aspects that can be improved upon: 1) accurate localization of the subject and its parts, 2) accurate extraction of useful features, and 3) accurate pixel alignment of outputs. Inspired by these findings, we propose three novel modules, each specifically designed to address a particular goal:

Localization Module . This module implements sparse and dense prediction branches to ensure the model is aware of the location and semantics of the subject's parts in the image. The learned location of the joint positions are helpful in recovering the relative rotations.

Contrastive Feature Extraction Module . This module incorporates a pose- and shape-aware contrastive loss, along with positive samples, to promote better feature extraction under robust augmentations. By minimizing the contrastive loss, the model can produce consistent representations for the same subject, even when presented with different augmentations, making it robust to various transformations and capable of extracting meaningful invariant features.

Pixel Alignment Module . This module applies differentiable rendering to ensure a more precise pixel alignment of the projected mesh, and learn more accurate pose, shape and camera parameters.

By integrating these three modules, we build a more robust and reliable whole-body pose and shape estimation framework, RoboSMPLX . Comprehensive evaluations demonstrate its effectiveness on body, face, hands and whole-body benchmarks.

2 Related Works

Whole-body Mesh Recovery. Despite significant progress in 3D body-specific [23, 22, 4, 16, 24, 10, 21, 20], hand-specific [28, 3], and face-specific [9] mesh recovery methods, there have been limited attempts to simultaneously recover all those parts. Early studies on whole-body pose and shape estimation primarily fit a 3D human model to 2D or 3D evidence [15, 53, 41, 54], which can be slow and susceptible to noise. Recent studies utilized neural networks to regress the SMPL-X parameters for a whole-body 3D human mesh. The model is composed of separate sub-networks to process body, hand and face, respectively. One-stage methods, e.g., OS-X [27], have the benefit of reduced computational costs and improved communication within part modules for more natural mesh articulation. However, the omission of hand and face experts makes it difficult for the model to leverage the widely available part-specific datasets, thus decreasing the hand and face performance. Multi-stage methods, e.g., ExPose [6], FrankMocap [44], PIXIE [11] and Hand4Whole [35], use different techniques to localize part crops.

Expose [41] and PIXIE [11] localize hand and part crops from the body mesh, making them dependent on the accuracy of body poses. Minor rotation errors accumulated along the kinematic chain may result in deviations in joint locations and thus inaccurate part crops. In contrast, Hand4Whole [35] predicts hand and face bounding boxes using a network leveraging image features and 3D joint heatmaps, but the resulting crops have low resolution. PyMAF-X [11] relies on an off-the-shelf wholebody pose estimation model to obtain crops, which, while more accurate, incurs extra computation. More detailed comparison with PyMAF-X are in Appendix C.

Robustness in vision tasks. Efforts to tackle robustness in vision tasks have utilized diverse strategies such as data augmentation, architectural innovations, and training methodologies [25, 56, 42, 30, 51, 60, 2]. AdvMix [51] employs adversarial augmentation and knowledge distillation, challenging models with corrupted images to foster learning from complex samples. Architectural modifications, such as novel heatmap regression [60], have been introduced to mitigate the impact of minor perturbations. HuMoR [42] utilizes a conditional variational autoencoder to capture the dynamics of human movement, thereby achieving generalization across diverse motions and body shapes. Additionally, PoseExaminer [30] employs a multi-agent reinforcement learning system to uncover failure modes inherent in human pose estimation models, highlighting model limitations in real-world scenarios. Complementing these efforts, Robo3D [25] provides a comprehensive benchmark for assessing the robustness of 3D detectors and segmentors in out-of-distribution scenarios.

Vertical

Horizontal

Low

Translation

Translation

Scale

Resolution

Rotation

Hue

Sharpness

Grayness

Brightness

Contrast

130

125

120

115

110

105

PA-PVE

55

50

45

0000990

000000000000

ExPose

PIXIE

Hand4Whole

OS-X

Furthermore, [56] utilize a confidence-guided framework to improve the accuracies of propagated labels. Contrastive learning, as demonstrated by CoKe [2], has also been employed to enhance robustness in keypoint detection, especially in occlusion-prone scenarios.

Contrastive Learning. Recently contrastive learning has demonstrated state-of-the-art performance among self-supervised learning (SSL) approaches. This strategy has been applied to 3D hand pose and shape estimation [46, 63]. Sanyal et al. [45] incorporate a novel shape consistency loss for 3D face shape and pose estimation that encourages the face shape parameters to be similar when the identity is the same and different for different people. Choi et al. [5] were the first to apply contrastive learning for 3D human pose and shape estimation. They found that SSL is not useful for this task, as the learned representations could be challenging to embed with high-level human-related information. Khosla et al. [19] proposed supervised contrastive learning for image classification tasks, which incorporates label information during training. Currently there is not attempt to apply this strategy to human pose and shape estimation, where the definition of positive samples is unclear, and data lie in a continuous space. We are the first to overcome these challenges and integrate supervised contrastive learning with whole-body pose and shape estimation.

Pixel Alignment in Pose and Shape Estimation. Many studies have been done to learn the subject's location in an image. Some works implicitly supervise the location. They primarily utilize projected meshes by supervising 2D joints regressed from the mesh [17, 23, 22, 4, 16, 24, 10, 21, 20]. Further supervision, such as dense body landmarks, silhouettes, and body part segmentation, is also employed to better align the predictions with the image [55, 37, 40, 49, 59, 57, 10]. Some other works explicitly learn the subject's location. Moon et al. [35] explicitly predict the keypoint locations in the image. Semantic body part segmentation is used as an explicit intermediate representation [41, 37]. PARE [41] employs a renderer to project the ground-truth mesh to the image space, and supervise the predicted part silhouette mask. However, dense part segmentation and differentiable rendering have not been employed in whole-body pose and shape estimation, which will be achieved in our framework.

3 Motivation

As discussed in Section 1, existing whole-body pose and shape estimation approaches suffer from the robustness issue, due to the models' sensitivity to the quality of input crops. To investigate the reasons and disclose the influence factors, we conduct a comprehensive evaluation of four state-ofthe-art methods: ExPose [6], PIXIE [11], Hand4Whole [35] and OS-X [27]. We opt for a set of ten commonly encountered augmentations and vary their scales within a realistic range (see Appendix A for more details). The augmentations can be classed into three categories (1) image-variant augmentations: they affect the image without altering the objects' 3D poses or positions, such as color jittering; (2) location-variant augmentations: they modify the subject's location without changing its pose, involving operations like translation and scaling; (3) pose-variant augmentations: they simultaneously alter both the 3D pose and location, including rotation.

Impact of subject localization . We first reveal that existing models demonstrate high sensitivity to the subject's position, indicating potential difficulties in subject localization. Figure 1 reports the PA-PVE errors of the whole body under different augmentations. We observe that image-variant augmentations (contrast, sharpness, brightness, hue and grayscale) lead to an acceptable range of error rates (approximately in the 50s) and minimal fluctuation (around ± 2 ). In contrast, location-

variant augmentations altering the subject's position within the frame, such as rotation, scaling, and horizontal or vertical translation, result in substantially higher error magnitudes. This demonstrates the heightened sensitivity of existing models to changes in the subject's position. In Appendix, we provide the results of other metrics and benchmarks in Figures 22 - 23, and visualizations of whole-body estimation under different settings in Figures 25 - 26.

Such position-altering augmentations are common in real-world scenarios, where the subject in the image is often localized using external detection models and control over the quality of crops is less feasible. In practice, to guarantee the visibility of the subject, crops are often made broader, This can lead to significant performance degradation, as errors increase with smaller augmentation scale factors (<1.0) (Figure 1). Besides, horizontal and vertical translations, which correspond to scenarios where the subject is not perfectly centralized or entirely visible within the frame, can further decrease

the performance. Similarly, the alignment and scale of these crops also influence the pose and shape estimation systems targeting body, face and hands (Figure 3, more quantitative and qualitative evidence in Appendix M). Whole-body methods bear the additional responsibility of accurately localizing body parts such as hands and face. Inaccurate part crops (Figure 2) can adversely affect the performance of part subnetworks, and further the whole-body estimation.

Impact of feature extraction . The deterioration of performance in the face of such variations suggests that the model struggles to extract meaningful features. Under alterations in translation or scale, the subject remains within the image frame, though the proportion of background content may vary. It is difficult for existing methods to effectively disregard irrelevant background elements and extract relevant features related to the subject of interest. To enhance the model's robustness, it is critical to produce consistent features irrespective of various augmentations applied to the image.

Impact of output pixel alignment . Pixel alignment is a critical aspect of high model performance. In certain instances, despite having precise subject localization, the model fails to produce properly aligned results (Figure 25 in Appendix). This is often caused by the suboptimal camera parameter estimation. To address this issue, we need to accurately estimate the camera parameters, ensuring the projected mesh is precisely aligned with the ground-truth at the pixel level. Such precision would enhance the effectiveness of the model in producing accurate pose, shape and camera parameter predictions, improving the overall accuracy and reliability of the estimation process.

4 RoboSMPLX Framework

We design RoboSMPLX to enhance the robustness of whole-body pose and shape estimation. It provides three specialized modules to address each challenge in Section 3: 1) Localization Module (Section 4.2): explicitly learning the location information of the subject and incorporating it into model estimations for pose, shape and camera ; 2) Contrastive Feature Extraction Module (Section 4.3): reliably extracting pertinent features under various augmentations, thereby improving the model's generalization ability and robustness to a broader range of real-world scenarios; 3) Pixel Alignment Module (Section 4.4): ensuring that the outputs are pixel aligned.

We start with the description of RoboSMPLX architecture with Body, Hand and Face subnetworks (Section 4.1). Each subnetwork is integrated with the Localization Module and Pixel Alignment

Sproj

Face

0f.4f

Subnetwork

faceparameters

Crop

Module

Hand

Orh,Oth

Subnetwork

handparameters

render

Irh

Ib

K

SMPLX

whole-body3Djoints

Models

Body

Subnetwork

Ob,Bb,tc

Output Mesh

Operations

bodyparameters

Module , and applies the Contrastive Feature Extraction Module for learning more robust features. Figure 6 shows the Hand subnetwork architecture. The other two subnetworks have the same designs.

4.1 Architecture and Training Details

Figure 4 shows the overall pipeline of RoboSMPLX for whole-body 3D human pose and mesh estimation. The Body subnetwork outputs 3D body joint rotations θ b ∈ R 21 × 3 , global orientation θ bg ∈ R 3 , shape parameters β b ∈ R 10 , camera parameters π b ∈ R 3 , and whole-body joints K ∈ R 137 × 3 . Joints corresponding to the hand and face are used to derive bounding boxes. Subsequently, hand and face images are cropped from a high-resolution image to preserve details. The Hand subnetwork predicts left and right hand 3D finger rotations θ h ∈ R 15 × 3 . Simultaneously, the Face subnetwork generates 3D jaw rotation θ f ∈ R 3 and expression ψ f ∈ R 10 . When training Hand and Face subnetworks with part-specific datasets, additional parameters such as global orientation θ fg ∈ R 3 , shape β f ∈ R 50 , and camera π f ∈ R 3 are estimated. These branches are discarded during wholebody estimation and training. Additional information concerning each subnetwork can be found in Appendix B. Further details regarding the training and inference durations are elaborated upon in Appendix K.

Subnetworks are trained separately, then integrated in a multi-stage manner. Initial whole-body training runs for 20 epochs. The hand and face modules are substituted with the trained Hand and Face subnetworks, followed by 20 epochs of fine-tuning to better unify the knowledge from the Hand and Face subnetworks into the whole-body understanding. Each subnetwork is trained by minimizing the following loss function L :

Here L BM is the L1 distance between the predicted and ground-truth body model parameters. L 3 D denotes the L1 distance between 3D keypoints and joints regressed from the body model. L 2 D signifies the L1 distance of the ground-truth 2D keypoints to predicted and projected 2D joints. The latter are obtained by projecting the regressed 3D coordinates from the 3D mesh to the image space using the perspective projection [17]. The part segmentation loss L segm is the cross-entropy loss between P h,w after softmax and P h,w averaged over H × Welements, following [20]. L proj refers to the projected segmentation loss, which is the sigmoid loss between the projected mesh and the ground-truth segmentation map. L con is the contrastive loss described in Section 4.3. For wholebody training, L box is added to measure the L1 distance between the predicted and actual center and scale of the hands' and face's boxes.

4.2 Localization Module

This module focuses on subject localization by explicitly learning both sparse and dense predictions of the subject within the image. Figure 5 shows an example of the supervision used for each subnetwork.

Pixel AlignmentModule

LocalizationModule

Lproj

Lsegm

Upsample

Conv+

Spred

Differentiable

6s

Jgt2D

Conv

Location

LFn

Sproj

Rendering

Feature

SoftArgmax

3D

LKS

Occlusion

Encoder

Hand

Fh

Fh

LFn

Linear

MANO

T

IInputHandImage

LMANO

In

Scale

Subnetwork

Hand

0n,0n

MANO

Z

agreement

Maximize

Zi

J3D

J2D

L3D

L2D

Im

Alignment

>

Subnetwork

Hand

0n,0n

MANO

>

Zk1

Lcon

Minimize

agreementw.r.t.

Operations

Loss

Background

Subnetwork

Hand

0n,0n

MANO

Zk2

.........................

posedifference

Outputs

Models

Contrastive Module

Original

Blur

Occlusion

Background

Cutout

Colorjitter

Scale

Translate

Flip

Rotate

In contrast to prior methods that directly output pose rotations from backbone features, this module aims to make the model explicitly conscious of the subject's location and semantics while predicting pose, shape and camera parameters. It can reduce the model's sensitivity to the variations of the subject's position, caused by minor shifts in the scale and alignment of the bounding box.

As shown in Figure 6, given an image, a convolutional backbone is utilized to extract its feature map F ∈ R 512 × 32 × 32 . Following [35], a 1 × 1 convolutional layer is then used to predict 3D feature maps LF ∈ R 32 J × 32 × 32 from F , where J represents the number of predicted joints with a feature map depth of 32. LF contains valuable information about the mesh's position in the image and semantics of various parts. It is concatenated with the backbone feature map F to predict pose θ ∈ R P , shape β ∈ R 10 and camera translation π ∈ R 3 , where P is the number of body parts. Meanwhile, LF is also used to obtain extra information with two branches: (1) 3D joint coordinates K ∈ R J × 3 are obtained from LF using the soft-argmax operation [47] in a differentiable manner. (2) 2D part segmentation maps S ∈ R P +1 × 64 × 64 are extracted from LF with several convolution layers, which model P part segmentation and 1 background mask. Here, 64 represents the height and width of the feature volume, and each pixel ( h, w ) stores the likelihood of belonging to a body part P .

Note that learning part segmentation maps and 3D joint coordinates is complementary, as 3D joint coordinates encode depth information that may inform part ordering in segmentation maps. Additionally, joints often reside at the boundaries of part segmentation maps, serving as separators for distinct parts. The Body subnetwork utilizes 24 parts P and 137 joints J , the Hand subnetwork employs 16 parts P and 21 joints J , while the Face subnetwork employs 15 parts P and 73 joints J .

4.3 Contrastive Feature Extraction Module

This module incorporates a pose- and shape-aware contrastive loss, along with positive samples. By minimizing this loss, the model can produce consistent representations for the same subject, even when presented with different augmentations, thus fostering the extraction of meaningful features.

Conventional contrastive learning methods based on SSL (e.g., SimCLR) face challenges in unifying similar pose embeddings and distancing dissimilar ones in human pose and shape estimation tasks. Without labels for guidance, images with similar poses could be misidentified as negative samples and contrasted away, complicating the self-organization of the embeddings in pose space. Figures 9 to 12 in Appendix show their ineffectiveness for the 3D human pose and shape task [5] by visualizing the retrieved samples from the embeddings. The supervised contrastive learning approach by Khosla

et al. [19], though effective for image classification, might not extend well to human pose and shape estimation, which is a high-dimensional regression problem and poses exist in a continuous space rather than well-defined classes.

Our module overcomes the aforementioned issues with two innovations. First, we experiment with three human pose representations z and the corresponding distance functions: (1) A concatenated form of the global orientation and rotational pose; (2) global orientation and rotational pose as separate entities (3) 3D root-aligned joints regressed from the body model, derived from pose and shape inputs. For (1) and (2), we explore relative rotations in two forms: 6D vector and rotation matrix representation. For (3), L1, Smooth L1, and Mean Squared Error (MSE) was used (Table 9).

Second, we investigate ten data augmentations, and classify them into three categories (see Figure 7 for the Body subnetwork, and Figure 29 in Appendix for Hand subnetwork): (1) image-variant augmentations such as color jittering, blur, occlusion and background swapping; (2) location-variant augmentations involving translation and scaling; (3) pose-variant augmentations including rotation and horizontal flipping. Our ablation study in Table 10 shows that augmentations with varied global orientation are detrimental to the model performance. Consequently, we exclude such modifications when constructing positive pairs. Instead, each positive sample is constructed utilizing a random combination of location-variant and color-variant augmentations.

Formally, for a batch of N images, we construct another N images by applying augmentation to each sample. For each anchor i , let j be the corresponding augmented sample. Then i is contrasted against 2 N -1 terms (1 positive and 2 N -2 negatives). The loss takes the following form:

/negationslash

where z i , z j and z k denote the predicted pose representations, and p i , p j and p k denote the ground-truth pose representations for the anchor, positive and negative samples in the batch. The objective of this loss function is to minimize the distance between the positive pairs and maximize the distance between the negative pairs, in alignment with the pose similarity. Note that unlike traditional approaches where the distance is the same for all negative samples, the pairwise distance d( p i , p k ) varies depending on the pose similarity.

4.4 Pixel Alignment Module

This module employs differentiable rendering to ensure that the projected mesh aligns precisely at the pixel level. The alignment is supervised by the projected mask loss. Attaining a proper alignment between the ground-truth part segmentation and rendered mesh requires the accurate prediction of pose, shape, and camera parameters, which subsequently leads to a more precise estimation process.

5 Experiments

Datasets. For whole-body training, we employ Human3.6M (H36M) [13], COCO-Wholebody [14] (the whole-body version of MSCOCO [29]) and MPII [1]. The 3D pseudo-ground truths for training are acquired using NeuralAnnot [36]. For hand-specific training, we use FreiHAND [62], Interhand [34] and COCO-Wholebody Hands [14]. For face-specific training, we use FFHQ [18], BUPT [52] and AffectNet [32]. For evaluations specific to 3D body, 3D hand, and 3D face, we utilize 3DPW [50], FreiHAND [62], and Stirling [11], respectively. For the 3D whole-body evaluation, we use EHF [41] and AGORA [39]. Additionally, we present qualitative results on the MSCOCO validation set.

Metrics. Mean Per Joint Position Error (MPJPE) and Mean Per-Vertex Position Error (MPVPE) are employed to evaluate the positions of 3D joint and mesh vertices, respectively. Each metric calculates the average 3D joint distance (in mm ) and 3D mesh vertex distance (in mm ) between the predicted and ground-truth values after aligning the root joint translation. The pelvis serves as the root joint for whole-body and body, whereas the wrists and neck are utilized as root joints for hands and face. Procrustes Aligned (PA) variants of these metrics, PA-MPJPE and PA-MPVPE, further align with rotation and scale. We report the average errors for the left and right hands as the 3D hand error.

Table 3: Evaluation of the Face subnetwork .

5.1 Benchmarking Results

Hand Subnetwork . Table 1 compares the performance of the Hand subnetwork with different handonly and whole-body methods. Our method outperforms that of our whole-body counterparts when trained with only the FreiHAND dataset (i.e. PIXIE, Hand4Whole, PyMAF) or under mixed datasets (i.e. Hand4Whole † , PyMAF † ) 1 using an identical backbone. Prior research [33, 48] demonstrated that whole-body methods generally employ a parametric representation of the hand mesh, and are numerically inferior to the non-parametric representation used in recent hand-only methods [33, 28]. Despite such reported gap, RoboSMPLX manages to outperform mesh-based techniques, and achieve comparable results as the state-of-the-art METRO when using the same backbone (HRNet-64). Table 4 compares the estimation errors of the Hand subnetwork in Hand4Whole (current whole-body method with SOTA on hands) and RoboSMPLX under different positional augmentations on the FreiHAND test set. It is clear that RoboSMPLX exhibits much better robustness than Hand4Whole. More visualizations are provided in Figure 20 in Appendix.

Body Subnetwork. Table 2 compares the performance of the Body subnetwork across different methods on the 3DPW test set. We observe the competitiveness of RoboSMPLX in relation to other SMPL-based approaches. Besides, since the performance of various methods may significantly differ based on their backbone initialization, datasets and training strategies [38], we establish a baseline to evaluate the effectiveness of our added modules in Table 12 in Appendix. RoboSMPLX achieves a substantial improvement compared to the baseline.

Face Subnetwork. Table 3 compares the performance of the Face subnetwork for different methods on the Stirling3D test set. When training with the same dataset, RoboSMPLX outperforms ExPose. The performance of ExPose declines when training on multiple datasets, while RoboSMPLX can still keep low and consistent errors. Figure 8 in Appendix shows some qualitative results for the in-thewild scenarios, which demonstrates the high generalization of RoboSMPLX . Table 5 compares the robustness of ExPose and RoboSMPLX under different positional augmentations. We also observe that RoboSMPLX has lower errors with different translation and scaling operations. More visualizations are provided in Figure 21 in Appendix.

Whole-body Network. We further provide results of the whole-body network on two benchmarks: EHF val set and AGORA test set in Table 6. On EHF, RoboSMPLX outperforms other full-body approaches, particularly in hand and face performance evaluations, and under different positional augmentations (Table 7). It gives subpar performance on AGORA as the predominant source of

error is the misidentification of individuals under intense person-person occlusion. We give detailed investigation in Appendix D.

5.2 Ablation Studies

Contrastive loss . We validate prior contrastive SSL methods [63, 46, 5] are not particularly adept at learning useful embeddings for human pose and shape estimation. Figures 9 - 12 in Appendix visualize the retrieved images based on the top-5 embedding similarity. They show that without labels, the model primarily extracts features based on background information instead of pose information. Table 8 shows the estimation errors of top-1 retrieved pose (COCO-train) and query pose (COCOtest) with different methods and contrastive loss functions. We observe that SimCLR has higher mean errors than the supervised training method HMR. These results are aligned with [5] that the representations learned through SSL are not transferable for human pose and shape estimation tasks. RoboSMPLX incorporates contrastive loss and positive samples ("HMR + L con , +ve"), which can produce similar representations under varied augmentations, enhancing its robustness.

Table 9 shows the estimation errors when applying contrastive loss with different representations in Section 4.3: "pose" (a concatenated form of global orientation and rotational pose), "go+pose" (global orientation and rotational pose as separate entities), "keypoint" (3D joints regressed from the body model). We observe that regressed 3D joints are the most effective representation, as they encode both shape and pose information in a normalized space. In contrast, the representation of pose as relative rotation has a detrimental impact on the model performance. Incorporating positive samples ("pose, +ve" and "keypoint, +ve") bolsters contrastive learning, encouraging the model to generate similar representations under varied augmentations. Table 10 compares the model performance with different augmentations. Prior methods [63, 46] employed pose-variant augmentations (e.g., rotation and flipping), which can adversely affect the learning by altering the global orientation, and lead to increased errors ("pose") compared to "baseline". Conversely, color-variant, location-variant and their combination provide an improvement over the baseline, showing these augmentations are helpful.

Location features. Table 11 shows the ablation of different modules on the Hand subnetwork (ablation for the Body subnetwork is in Table 12 in Appendix). The baseline model is trained that randomly augments images with a scale factor of 0.2 and bounding box jitter of 0.2. We observe that training using strongaug with a larger scale and jitter factor harms the baseline performance. This is likely due to a domain shift. Hand4Whole [35] employs sampled features from positional pose-guided pooling (PPP) to predict

pose parameters while shape and camera parameters only utilize backbone features. Our method focuses on explicitly learning the location and part silhouettes, utilizing sparse and dense supervision methods. This proves advantageous as the location information ("LF") improve the performance of pose and shape estimations, with the reduced joint and vertex errors of the regressed mesh. Moreover,

we find that using location features "LF (all)" for predicting shape and camera parameters is also beneficial.

Pixel alignment. Tables 11 also shows that incorporating differential rendering and using projected segmentation loss ( L proj ) for the mesh in RoboSMPLX helps to achieve lower PVE and MPJPE errors. It facilitates the learning of more precise body model and camera parameters to improve the alignment between the rendered 3D model and 2D image. Notably, metrics such as PVE and MPJPE errors is calculated after root alignment and may not sufficiently reflect the quality of mesh projection onto the image space. To offer a more precise analysis, we evaluate the discrepancies between the projected 2D vertices of the ground-truth and projected meshes. More quantitative and qualitative comparisons can be found in Appendix F.

6 Conclusion

In this paper, we introduce a new framework RoboSMPLX to advance the field of whole-body pose and shape estimation. It enhances the whole-body pipeline by learning more precise localization for part crops while ensuring that part subnetworks are robust enough to handle suboptimal part crops and produce reliable outputs. It achieves this goal with three innovations: accurate subject localization by explicitly learning both sparse and dense predictions of the subject, robust feature extraction with supervised contrastive learning, and accurate pixel alignment of outputs with differentiable rendering. Nevertheless, it is important to acknowledge that there are instances in which our framework exhibits limitations, such as (1) inaccurate beta estimation due to out-of-distribution data (children), (2) challenges posed by severe object-occlusion, (3) difficulties arising from person-person occlusion, and (4) the potential for prediction errors in multi-person scenarios, as exemplified by the cases detailed in Appendix G. These challenges represent important avenues for future refinement of our approach.

There are several potential avenues for future research. First, the current approach does not deliberately select negative samples during training. Future work could explore if hard mining by intentionally selecting similar poses in a batch could enhance learning. Second, the careful selection of augmentations is essential. While augmentations that modify the global orientation, such as flipping and rotation, have proven detrimental and are not employed, the effects of individual augmentations and their combinations are not examined. Future research could explore the potential for automatically determining the optimal selection of augmentations to achieve improved performance. Additionally, simplifying the complex framework without sacrificing performance is a beneficial direction for future work. Lastly, considering that videos are a prevalent input format, the integration of video-based estimation can contribute to bolstering model robustness can enhance model robustness, alleviate depth ambiguity, and improve temporal consistency.

References

Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2D human pose estimation: New benchmark and state of the art analysis. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 3686-3693, 2014. ISSN 10636919. doi: 10.1109/CVPR.2014.471.

Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. CoKe: Contrastive Learning for Robust Keypoint Detection. Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023 , pp. 65-74, 2023. doi: 10.1109/WACV56688.2023.00015.

Adnane Boukhayma, Rodrigo De Bem, and Philip H.S. Torr. 3D hand shape and pose from images in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2019-June:10835-10844, 2019. ISSN 10636919. doi: 10.1109/CVPR. 2019.01110.

Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 12352 LNCS:769-787, 2020. ISSN 16113349. doi: 10.1007/ 978-3-030-58571-6_45.

Hongsuk Choi, Hyeongjin Nam, Taeryung Lee, Gyeongsik Moon, and Kyoung Mu Lee. Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimation. International Conference on Learning Representations (ICLR) , pp. 1-18, 2023. URL http://arxiv.org/abs/2303.05370 .

Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body-driven attention. In European Conference on Computer Vision (ECCV) , 2020. URL https://expose.is.tue.mpg.de .

MMHuman3D Contributors. Openmmlab 3d human parametric model toolbox and benchmark. https://github.com/open-mmlab/mmhuman3d , 2021.

Radek Danecek, Michael Black, and Timo Bolkart. EMOCA: Emotion Driven Monocular Face Capture and Animation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2022-June:20279-20290, 2022. ISSN 10636919. doi: 10.1109/CVPR52688.2022.01967.

Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops , 2019June:285-295, 2019. ISSN 21607516. doi: 10.1109/CVPRW.2019.00038.

Sai Kumar Dwivedi, Nikos Athanasiou, Muhammed Kocabas, and Michael J. Black. Learning to Regress Bodies from Images using Differentiable Semantic Rendering. In IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 11230-11239, 2021. ISBN 9781665428125. doi: 10.1109/iccv48922.2021.01106.

Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collaborative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV) , 2021.

Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from in-the-wild images. In ACMTransactions on Graphics, (Proc. SIGGRAPH) , volume 40, 2021. URL https://doi.org/10.1145/3450626.3459936 .

Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6M. Ieee Transactions on Pattern Analysis and Machine intelligence , pp. 1, 2014. ISSN 01628828. URL http://109.101.234.42/documente/publications/1-82.pdf .

Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-Body Human Pose Estimation in the Wild. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 12354 LNCS:196-214, 2020. ISSN 16113349. doi: 10.1007/978-3-030-58545-7_12.

Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 8320-8329, 2018. ISSN 10636919. doi: 10.1109/CVPR.2018.00868.

Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation. Proceedings - 2021 International Conference on 3D Vision, 3DV 2021 , pp. 42-52, 2021. doi: 10.1109/3DV53792.2021. 00015.

Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-End Recovery of Human Shape and Pose. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 7122-7131, 2018. ISBN 9781538664209. doi: 10.1109/CVPR.2018.00744.

Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43 (12):4217-4228, 2021. ISSN 19393539. doi: 10.1109/TPAMI.2020.2970919.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11362 , 2020.

Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In Proc. International Conference on Computer Vision (ICCV) , pp. 11127-11137, October 2021.

Muhammed Kocabas, Chun-Hao P. Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael J. Black. SPEC: Seeing people in the wild with an estimated camera. In Proc. International Conference on Computer Vision (ICCV) , pp. 11035-11045, October 2021.

Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Daniilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. Proceedings of the IEEE International Conference on Computer Vision , 2019-Octob:2252-2261, 2019. ISSN 15505499. doi: 10.1109/ICCV.2019.00234.

Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2019-June:4496-4505, 2019. ISSN 10636919. doi: 10.1109/CVPR.2019.00463.

Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic Modeling for Human Mesh Recovery. In International Conference on Computer Vision (ICCV) , pp. 11585-11594, 2021. ISBN 9781665428125. doi: 10.1109/iccv48922.2021.01140.

Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 19994-20006, 2023.

Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) , 36(6):194:1-194:17, 2017. URL https://doi.org/10.1145/3130800.3130813 .

Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. arXiv preprint arXiv:2303.16160 , 2023.

Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-End Human Pose and Mesh Reconstruction with Transformers. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 1954-1963, 2021. ISSN 10636919. doi: 10.1109/CVPR46437. 2021.00199.

Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 8693 LNCS(PART 5):740-755, 2014. ISSN 16113349. doi: 10.1007/978-3-319-10602-1_48.

Qihao Liu, Adam Kortylewski, and Alan L Yuille. Poseexaminer: Automated testing of out-ofdistribution robustness in human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 672-681, 2023.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) , 34(6):248:1-248:16, October 2015.

Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Transactions on Affective Computing , 10(1):18-31, 2019. ISSN 19493045. doi: 10.1109/TAFFC.2017.2740923.

Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 12352 LNCS:752-768, 2020. ISSN 16113349. doi: 10.1007/ 978-3-030-58571-6_44.

Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In European Conference on Computer Vision (ECCV) , 2020.

Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Accurate 3d hand pose estimation for whole-body 3d human mesh estimation. In Computer Vision and Pattern Recognition Workshop (CVPRW) , 2022.

Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops , 2022-June:2298-2306, 2022. ISSN 21607516. doi: 10.1109/CVPRW56347.2022.00256.

Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. Proceedings - 2018 International Conference on 3D Vision, 3DV 2018 , pp. 484-494, 2018. doi: 10.1109/3DV.2018.00062.

Hui En Pang, Zhongang Cai, Lei Yang, Tianwei Zhang, and Ziwei Liu. Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms. In NeurIPS , 2022.

Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael J. Black. AGORA: Avatars in geography optimized for regression analysis. In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2021.

Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. Learning to Estimate 3D Human Pose and Shape from a Single Color Image. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 459-468, 2018. ISSN 10636919. doi: 10.1109/CVPR.2018.00055.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Osman, DImitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , volume 2019-June, pp. 10967-10977, 2019. ISBN 9781728132938. doi: 10.1109/CVPR.2019.01123.

Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 11488-11499, 2021.

Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) , 36(6), November 2017.

Yu Rong, Takaaki Shiratori, and Hanbyul Joo. FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration. In Proceedings of the IEEE International Conference on Computer Vision , volume 2021-Octob, pp. 1749-1759, 2021. ISBN 9781665401913. doi: 10.1109/ICCVW54120.2021.00201.

Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning. Proceedings of the IEEE International Conference on Computer Vision , pp. 11210-11219, 2021. ISSN 15505499. doi: 10.1109/ICCV48922.2021.01104.

Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 11210 LNCS:536-553, 2018. ISSN 16113349. doi: 10.1007/978-3-030-01231-1_33.

Yating Tian, Hongwen Zhang, Yebin Liu, and Limin Wang. Recovering 3D Human Mesh from Monocular Images: A Survey. arXiv preprint arXiv:2203.01923 , pp. 1-20, 2022. URL http://arxiv.org/abs/2203.01923 .

Hsiao Yu Fish Tung, Hsiao Wei Tung, Ersin Yumer, and Katerina Fragkiadaki. Self-supervised learning of motion capture. Advances in Neural Information Processing Systems , 2017December(Nips):5237-5247, 2017. ISSN 10495258.

Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, and Gerard PonsMoll. Recovering accurate 3D human pose in the wild using IMUs and a moving camera. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 11214 LNCS:614-631, 2018. ISSN 16113349. doi: 10.1007/978-3-030-01249-6_37.

Jiahang Wang, Sheng Jin, Wentao Liu, Weizhong Liu, Chen Qian, and Ping Luo. When human pose estimation meets robustness: Adversarial algorithms and benchmarks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 11850-11859, 2021. ISSN 10636919. doi: 10.1109/CVPR46437.2021.01168.

Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness-aware reinforcement learning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 9319-9328, 2020. ISSN 10636919. doi: 10.1109/CVPR42600.2020.00934.

Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2019-June:10957-10966, 2019. ISSN 10636919. doi: 10.1109/ CVPR.2019.01122.

Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. GHUM GHUML: Generative 3D human shape and articulated pose models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 6183-6192, 2020. ISSN 10636919. doi: 10.1109/CVPR42600.2020.00622.

Yuanlu Xu, Song Chun Zhu, and Tony Tung. DenseRaC: Joint 3D pose and shape estimation by dense render-and-compare. Proceedings of the IEEE International Conference on Computer Vision , 2019-Octob:7759-7769, 2019. ISSN 15505499. doi: 10.1109/ICCV.2019.00785.

Lei Yang, Qingqiu Huang, Huaiyi Huang, Linning Xu, and Dahua Lin. Learn to propagate reliably on noisy affinity graphs. In European Conference on Computer Vision , pp. 447-464. Springer, 2020.

Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 12351 LNCS:465-481, 2020. ISSN 16113349. doi: 10.1007/978-3-030-58539-6_28.

Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Tianshu Zhang, Buzhen Huang, and Yangang Wang. Object-Occluded Human Shape and Pose Estimation from a Single Color Image. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 7374-7383, 2020. ISSN 10636919. doi: 10.1109/CVPR42600.2020.00740.

Yumeng Zhang, Li Chen, Yufeng Liu, Xiaoyan Guo, Wen Zheng, and Junhai Yong. Improving robustness for pose estimation via stable heatmap regression. Neurocomputing , 492:322-342, 2022. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2022.04.046. URL https: //www.sciencedirect.com/science/article/pii/S0925231222004131 .

Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu. Monocular Real-time Full Body Capture with Inter-part Correlations. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , number 61822111 in 61822111, pp. 4809-4820, 2021. ISBN 9781665445092. doi: 10.1109/CVPR46437.2021.00478.

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max J. Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single rgb images. Proceedings of the IEEE International Conference on Computer Vision , 2019-October: 813-822, 2019. ISSN 15505499. doi: 10.1109/ICCV.2019.00090.

Christian Zimmermann, Max Argus, and Thomas Brox. Contrastive Representation Learning for Hand Shape Estimation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 13024 LNCS:250-264, 2021. ISSN 16113349. doi: 10.1007/978-3-030-92659-5_16.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on improving whole-body pose and shape estimation from monocular images, a task that often struggles with complex, real-world scenarios. The authors argue that the performance of these models is significantly impacted by the quality of the predicted bounding box, such as the scale and alignment of body parts. The difference between ideal bounding box annotations and model detection results poses a substantial challenge to whole-body pose and shape estimation. To address this, the authors introduce RoboSMPLX, a framework to enhance the robustness of whole-body pose and shape estimation. RoboSMPLX integrates three new modules: a Localization Module to improve model awareness of the subject's location and semantics within the image space, a Contrastive Feature Extraction Module that uses contrastive loss with dedicated positive samples to help the model be invariant to robust augmentations, and a Pixel Alignment Module to ensure the reprojected mesh from predicted camera and body model parameters are accurate and pixel-aligned. The effectiveness of the method is showcased through comprehensive experiments on body, hands, face, and whole-body benchmarks.\n\nSTRENGTHS:\nThe paper is well-structured and reader-friendly.\n\nIt begins with a compelling motivation that efficiently highlights the issue of the current state-of-the-art methods' low robustness against varying crops and misalignments. The authors also convincingly justify potential solutions to these problems.\n\nThe proposed method by the authors is sound even if it lacks complete novelty.\n\nImpressively, the authors have performed extensive experiments and provided an in-depth ablation study, substantiating their proposed approach effectively.\n\nAlthough their method doesn't top all the benchmarks, its efficacy is undeniable, particularly in its demonstrated robustness to variations in crop size and alignment - a challenge that other methods fail to meet. The authors provide a thorough justification with robust qualitative and quantitative results.\n\nWEAKNESSES:\nThe claim made in lines 47-51 and 76-77, suggesting that contrastive learning is not used in the parametric estimation of human meshes, is inaccurate. Previous works, such as [1], have already incorporated a form of contrastive and triplet loss for 3D face shape and pose estimation from monocular face images. Thus, the application of contrastive loss for mesh recovery from monocular images isn't entirely novel. The authors should moderate their claims and include a thorough discussion concerning related work in this field.\n\nFurthermore, the pixel alignment loss achieved by projecting the mesh as a mask, a technique used by ICON [2] for improved SMPL pose parameter estimation, is not entirely new.\n\nThe authors should also include instances of the method's failure during testing within the main body of the paper.\n\n[1] Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision. CVPR 2019\n[2] ICON: Implicit Clothed humans Obtained from Normals. CVPR  2022\n\nQUESTIONS:\nThe paper seems sound to me and I do not have much questions.\n\nLIMITATIONS:\nAlthough some limitations have been discussed it might not be adequate. Please add some visual limitations of the model during the inference time.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of whole-body pose and shape estimation, including human mesh, hand gestures, and facial expressions, from monocular images. The author identifies the impact of predicted bounding box quality on the accuracy and reliability of existing methods. Based on this observation, a novel framework called RoboSMPLX is proposed to enhance robustness through three modules: a localization module, a contrastive feature extraction module, and a pixel alignment module. Comprehensive experiments are conducted to demonstrate the effectiveness of the framework.\n\nSTRENGTHS:\n1. Overall, the paper is of high quality, with clear motivation and well-organized sections. It is easy to follow and presents novel modules that address existing limitations. The paper includes comprehensive experiments and provides sufficient visualizations of the generated mesh. The author's contributions are highly appreciated.\n\n2. The empirical study on the impact of subject localization, feature extraction, and pixel alignment, as stated in lines 91-100, is crucial in breaking through barriers. The paper provides extensive qualitative and quantitative results, both in the main paper and supplementary material, making it solid and convincing.\n\n3. The three modules work together effectively, aligning intermediate representations such as pose, dense landmarks, segmentation, and silhouette masks. This enhances robustness through proper data augmentation techniques.\n\n4. The proposed framework consistently achieves impressive results for various tasks across multiple datasets, demonstrating superior performance in whole-body pose and shape estimation.\n\nWEAKNESSES:\n1. Since the author does not mention code release, it would be beneficial to release the code to contribute to the research community.\n\n2. The paper lacks a discussion on computational complexity and inference speed, which are important evaluation metrics, especially for practical deployment. I suggest reporting total parameters, FLOPs, and fps with a detailed discussion to strengthen the paper.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\n1.  Although the proposed framework achieves impressive performance, the overall framework and training strategy is somewhat complex. Simplifying the framework without sacrificing performance would be beneficial. \n\n2. Currently, the proposed framework only allows for whole-body estimation from a single image. Future work could focus on video-based estimation to further improve robustness, alleviate depth ambiguity, and enhance temporal smoothness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RoboSMPLX, a method for whole-body 3d human pose and shape estimation from a monocular image. Motivated by the poor robustness of existing methods, especially w.r.t. the quality of bounding boxes, three components are proposed: 1) a localization module, 2) a contrastive feature extraction module, and 3) a pixel alignment module. The robustness of existing methods is evaluated by applying different types of image, location and pose augmentations. By using these augmentations in a contrastive setting during training of RoboSMPLX, RoboSMPLX achieves higher robustness when evaluated on these augmentations. Additionally, the performance of Hand, Body, Face and Wholebody reconstructions are evaluated on typical benchmark datasets.\n\nSTRENGTHS:\nIncreasing the robustness of whole-body human pose and shape estimation methods is an important topic. The problem is well motivated, and the evaluation of the performance of existing methods under different augmentations is interesting.\n\nWEAKNESSES:\nI have the following main concerns about the paper:\n\n1) Evaluation seems inconsistent and incomplete.\nIt is unclear why some competitors are omitted in different experiments. For example, the performance of PyMAF[54] is reported in Table 1 and 2, but not in Table 3 and 6, even though [54] reports the relevant numbers. OSX[25] is also missing in Table 2. The competitors outperform RoboSMPLX in the experiments where they are omitted. \n\n2) Robustness of RoboSMPLX. \nThe robustness evaluation seems not realistic. It is not surprising that RoboSMPLX shows better performance under the different augmentations the method is trained on. However, most relevant is the performance on in-the-wild scenarios. This is not properly evaluated for the whole-body task. On the contrary, RoboSMPLX performs worse on AGORA than OSX. Although AGORA is a synthetic dataset, it contains realistic and diverse scenes with multiple persons and occlusions. On the other hand, EHF only consists of 100 images of a single subject recorded in a mocap studio. It is therefore questionable, if RoboSMPLX really succeeds in being more robust than its competitors. To better assess the in-the-wild performance, the methods could for example be evaluated on RICH (CVPR'22) (https://rich.is.tue.mpg.de/) or on BEDLAM (CVPR'23) (https://bedlam.is.tue.mpg.de/). Additionally, to show the effectiveness of the localization module, the accuracy of the predicted bounding boxes could be evaluated.\n\nFuther concerns:\n- The contrastive module is not well motivated. Why use the contrastive setting with all its overhead, instead of simply using the different transformations as data augmentation? Especially since adding positive samples only has small influence.\n\n- Why are the ablation studies in Table 11 not conducted with the whole body model? Mean+std of multiple runs should be reported since the differences are so small.\n\n- Why is the pixel alignment not evaluated quantitatively? This would be more meaningful instead of only showing some examples e.g. in Figure 13.\n\n- The experiments section should be revised. It is difficult to read due to the many experiments and referrals to the appendix. \n\n- The novelty is limited since the augmentations and the pixel alignment module already have been used extensively in the literature for estimating human pose and shape, e.g. [35] and [45].\n\n- IIn Table 10 7.18 PA-PVE should be in bold, and 14.38 MPJPE in Table 11.\n\nQUESTIONS:\nThe following requires clarification:\n- What is the estimation error reported in Table8?\n- Table 11: what does DR54 and KS stand for? What is the row named \"joints\"?\n- Regarding line 204: why should the model produce consistent representations for the same subject under different poses?\n- For how long is the model trained on what hardware? What's the inference time?\n- How are the joints for the body, hand and face determined? How does the 137 joints body skeleton look like?\n\nLIMITATIONS:\nMore failure cases could be shown in the paper. Also in comparisons to the competitors.\n\nThe potential societal impact is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a method to improve the robustness of whole-body pose and shape estimation, which mainly contains three components: 1) localization module to give the network awareness of location and semantic part; 2) contrastive feature extraction module to predict consistent representations under different augmentations; 3) pixel-alignment module to ensure alignment between projected mesh and 2d evidence.

STRENGTHS:
1) This paper is well-written and easy to understand

2) The topic of robustness of pose and shape estimation is meaningful and  the proposed method is effective against augmentations

3) The experiments are comprehensive and visualizations are very nice to help understand the method

WEAKNESSES:
1) Since the main topic is the robustness of whole body pose and shape estimation, the literature review of robustness in vision tasks, especially in pose estimation tasks should be included. For example, [1][2].

[1] Bai, Yutong, et al. "CoKe: Contrastive Learning for Robust Keypoint Detection."
[2] Zhang, Yumeng, et al. "Improving robustness for pose estimation via stable heatmap regression."


2) I wonder why the target task is whole-body pose and shape estimation rather than body/hand pose estimation?
3) Why does ‘baseline’ only appear in table2 but not in table1 and 3? Also, the definition of baseline is not clear. The definition of baseline is not clear either in table 9
4) Where is the result of robustness of body subnetwork again augmentations?
5) Where is the details of table 7?
6) Eq,2 typo

QUESTIONS:
see weaknesses

LIMITATIONS:
see weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes RoboSMPLX for whole-body pose and shape estimation. RoboSMPLX incorporates three modules, including a localization module, a contrastive feature extraction module, and a pixel alignment module. The localization module is aware of the location and semantics of body parts so that cropping could be more accurate. The contrastive feature extraction module incorporates a pose- and shape-aware contrastive loss, along with positive samples, for better feature extraction under robust augmentations. The pixel alignment  module applies differentiable rendering to inherence of the re-projection alignment of the mesh.\n\nSTRENGTHS:\nThe motivation of the proposed method is clear: using a localization module to improve the robustness of the cropping. Overall, the proposed method technically makes sense and can be easily reproduced. The experiments also show comparable or better performances with previous methods.\n\nWEAKNESSES:\nThe most severe weakness is the novelty of the proposed method and the lack of comparisons with recent state-of-the-art solutions.  \n\nTo be more specific, there are several major issues:\n\n- The proposed localization and pixel alignment modules have very limited contributions to the community, as these operations are commonly used in this field. For instance, similar localization strategies are used in [39,56], and differentiable modules are used in [i].\n[i] SK Dwivedi, N Athanasiou, M Kocabas, MJ Black, Learning to regress bodies from images using differentiable semantic rendering, ICCV 2021.\n\n- There is a lack of discussion and comparison with the recent state-of-the-art method PyMAF-X [54]. An in-depth discussion and comparison with [54] is necessary to support the claim of the proposed method. It is recommended to include [54] in the Related Work section, compare results with [54] in Tables 3,4,5, and Table 7, and show qualitative results of the proposed method and [54] for compressive comparisons.  \n\n- This paper claims robust performances of whole-body pose and shape estimation, but no video result is provided in the supplementary materials. To convincingly demonstrate the robustness of the proposed method, it is also recommended to include a side-by-side comparison video [54].\n\nQUESTIONS:\nHow about the run time of the proposed method?\n\nLIMITATIONS:\nThe limitations of this paper are mainly the novelty and the lack of comprehensive comparisons with recent state-of-the-art solutions. Given such clear defects in the experimental results, I rate this paper as the one below the acceptance bar.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}