SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hui En Pang $^{1}$ , Zhongang Cai $^{1,2}$ , Lei Yang $^{2}$ , Qingyi Tao $^{2}$ , Zhonghua Wu $^{2}$ , Tianwei Zhang $^{1\boxtimes}$ , Ziwei Liu $^{1}$
$^{1}$ S-Lab, Nanyang Technological University  $^{2}$ SenseTime Research {huien001, tianwei.zhang, ziwei.liu}@ntu.edu.sg {caizhongang, yanglei, taoqingyi, wuzhonghua}@sensetime.com
Whole-body pose and shape estimation aims to jointly predict different behaviors (e.g., pose, hand gesture, facial expression) of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We argue that the accuracy and reliability of these models are significantly affected by the quality of the predicted bounding box, e.g., the scale and alignment of body parts. The natural discrepancy between the ideal bounding box annotations and model detection results is particularly detrimental to the performance of whole-body pose and shape estimation. In this paper, we propose a novel framework RoboSMPLX to enhance the robustness of whole-body pose and shape estimation. RoboSMPLX incorporates three new modules to address the above challenges from three perspectives: 1) Localization Module enhances the model's awareness of the subject's location and semantics within the image space. 2) Contrastive Feature Extraction Module encourages the model to be invariant to robust augmentations by incorporating contrastive loss with dedicated positive samples. 3) Pixel Alignment Module ensures the reprojected mesh from the predicted camera and body model parameters are accurate and pixel-aligned. We perform comprehensive experiments to demonstrate the effectiveness of RoboSMPLX on body, hands, face and whole-body benchmarks. Codebase is available at https://github.com/robosmplx/robosmplx.
Human pose and shape estimation tries to build human body models from monocular RGB images or videos. It has gained widespread attention owing to its extensive applications in various fields, including robotics, computer graphics, and augmented/virtual reality. Early works use various statistical models (e.g., SMPL [31], MANO [43], FLAME [26]) to individually reconstruct different parts, including human body [17, 22, 4, 16, 24, 10, 21, 20], face [9, 8, 12], and hand [28, 3, 63]. Recently, there is a growing interest in whole-body estimation [11, 6, 61, 44, 58], which jointly estimates the pose, hand gestures and facial expressions of the entire human body from the input. Commonly these methods first employ separate sub-networks to extract the features of body, hands and face. These features are then used to predict whole-body 3D joint rotations and other parameters (e.g., body shape, facial expression), which are further combined to generate the whole-body 3D mesh. This is a crucial step towards modeling human behaviors in an efficient and practical manner.
However, achieving accurate and robust whole-body estimation is particularly challenging as it requires precise estimation of each body part and the correct connectivity between them. In particular, due to the smaller sizes of hand and face images, they are typically localized, cropped and resized to higher resolutions before being processed by the relevant sub-network. To tackle the absence
of ground-truth bounding boxes in the real-world scenarios, existing whole-body methods utilize various detection techniques to obtain the crops. The accuracy of the whole-body estimation is highly sensitive to the quality of input crops. Our experiment results in Section 3 show that even minor fluctuations in the scale and alignment of input crops can significantly affect the model performance, indicating a limited ability to localize and extract meaningful features about the subject in the image.
The lack of robustness in existing whole-body pose and shape estimation methods highlights three critical aspects that can be improved upon: 1) accurate localization of the subject and its parts, 2) accurate extraction of useful features, and 3) accurate pixel alignment of outputs. Inspired by these findings, we propose three novel modules, each specifically designed to address a particular goal:
- Localization Module. This module implements sparse and dense prediction branches to ensure the model is aware of the location and semantics of the subject's parts in the image. The learned location of the joint positions are helpful in recovering the relative rotations.
- Contrastive Feature Extraction Module. This module incorporates a pose- and shape-aware contrastive loss, along with positive samples, to promote better feature extraction under robust augmentations. By minimizing the contrastive loss, the model can produce consistent representations for the same subject, even when presented with different augmentations, making it robust to various transformations and capable of extracting meaningful invariant features.
- Pixel Alignment Module. This module applies differentiable rendering to ensure a more precise pixel alignment of the projected mesh, and learn more accurate pose, shape and camera parameters.
By integrating these three modules, we build a more robust and reliable whole-body pose and shape estimation framework, RoboSMPLX. Comprehensive evaluations demonstrate its effectiveness on body, face, hands and whole-body benchmarks.
Whole-body Mesh Recovery. Despite significant progress in 3D body-specific [23, 22, 4, 16, 24, 10, 21, 20], hand-specific [28, 3], and face-specific [9] mesh recovery methods, there have been limited attempts to simultaneously recover all those parts. Early studies on whole-body pose and shape estimation primarily fit a 3D human model to 2D or 3D evidence [15, 53, 41, 54], which can be slow and susceptible to noise. Recent studies utilized neural networks to regress the SMPL-X parameters for a whole-body 3D human mesh. The model is composed of separate sub-networks to process body, hand and face, respectively. One-stage methods, e.g., OS-X [27], have the benefit of reduced computational costs and improved communication within part modules for more natural mesh articulation. However, the omission of hand and face experts makes it difficult for the model to leverage the widely available part-specific datasets, thus decreasing the hand and face performance. Multi-stage methods, e.g., ExPose [6], FrankMocap [44], FIXIE [11] and Hand4Whole [35], use different techniques to localize part crops.
Expose [41] and PixIE [11] localize hand and part crops from the body mesh, making them dependent on the accuracy of body poses. Minor rotation errors accumulated along the kinematic chain may result in deviations in joint locations and thus inaccurate part crops. In contrast, Hand4Whole [35] predicts hand and face bounding boxes using a network leveraging image features and 3D joint heatmaps, but the resulting crops have low resolution. PyMAF-X [11] relies on an off-the-shelf whole-body pose estimation model to obtain crops, which, while more accurate, incurs extra computation. More detailed comparison with PyMAF-X are in Appendix C.
Robustness in vision tasks. Efforts to tackle robustness in vision tasks have utilized diverse strategies such as data augmentation, architectural innovations, and training methodologies [25, 56, 42, 30, 51, 60, 2]. AdvMix [51] employs adversarial augmentation and knowledge distillation, challenging models with corrupted images to foster learning from complex samples. Architectural modifications, such as novel heatmap regression [60], have been introduced to mitigate the impact of minor perturbations. HuMoR [42] utilizes a conditional variational autoencoder to capture the dynamics of human movement, thereby achieving generalization across diverse motions and body shapes. Additionally, PoseExaminer [30] employs a multi-agent reinforcement learning system to uncover failure modes inherent in human pose estimation models, highlighting model limitations in real-world scenarios. Complementing these efforts, Robo3D [25] provides a comprehensive benchmark for assessing the robustness of 3D detectors and segmentors in out-of-distribution scenarios.
Figure 1: Wholebody PA-PVE errors under different augmentations (sorted in descending order). The dashed line indicates baseline performance without augmentation.
Furthermore, [56] utilize a confidence-guided framework to improve the accuracies of propagated labels. Contrastive learning, as demonstrated by CoKe [2], has also been employed to enhance robustness in keypoint detection, especially in occlusion-prone scenarios.
Contrastive Learning. Recently contrastive learning has demonstrated state-of-the-art performance among self-supervised learning (SSL) approaches. This strategy has been applied to 3D hand pose and shape estimation [46, 63]. Sanyal et al. [45] incorporate a novel shape consistency loss for 3D face shape and pose estimation that encourages the face shape parameters to be similar when the identity is the same and different for different people. Choi et al. [5] were the first to apply contrastive learning for 3D human pose and shape estimation. They found that SSL is not useful for this task, as the learned representations could be challenging to embed with high-level human-related information. Khosla et al. [19] proposed supervised contrastive learning for image classification tasks, which incorporates label information during training. Currently there is not attempt to apply this strategy to human pose and shape estimation, where the definition of positive samples is unclear, and data lie in a continuous space. We are the first to overcome these challenges and integrate supervised contrastive learning with whole-body pose and shape estimation.
Pixel Alignment in Pose and Shape Estimation. Many studies have been done to learn the subject's location in an image. Some works implicitly supervise the location. They primarily utilize projected meshes by supervising 2D joints regressed from the mesh [17, 23, 22, 4, 16, 24, 10, 21, 20]. Further supervision, such as dense body landmarks, silhouettes, and body part segmentation, is also employed to better align the predictions with the image [55, 37, 40, 49, 59, 57, 10]. Some other works explicitly learn the subject's location. Moon et al. [35] explicitly predict the keypoint locations in the image. Semantic body part segmentation is used as an explicit intermediate representation [41, 37]. PARE [41] employs a renderer to project the ground-truth mesh to the image space, and supervise the predicted part silhouette mask. However, dense part segmentation and differentiable rendering have not been employed in whole-body pose and shape estimation, which will be achieved in our framework.
As discussed in Section 1, existing whole-body pose and shape estimation approaches suffer from the robustness issue, due to the models' sensitivity to the quality of input crops. To investigate the reasons and disclose the influence factors, we conduct a comprehensive evaluation of four state-of-the-art methods: ExPose [6], PixIE [11], Hand4Whole [35] and OS-X [27]. We opt for a set of ten commonly encountered augmentations and vary their scales within a realistic range (see Appendix A for more details). The augmentations can be classed into three categories (1) image-variant augmentations: they affect the image without altering the objects' 3D poses or positions, such as color jittering; (2) location-variant augmentations: they modify the subject's location without changing its pose, involving operations like translation and scaling; (3) pose-variant augmentations: they simultaneously alter both the 3D pose and location, including rotation.
Impact of subject localization. We first reveal that existing models demonstrate high sensitivity to the subject's position, indicating potential difficulties in subject localization. Figure 1 reports the PA-PVE errors of the whole body under different augmentations. We observe that image-variant augmentations (contrast, sharpness, brightness, hue and grayscale) lead to an acceptable range of error rates (approximately in the 50s) and minimal fluctuation (around  $\pm 2$ ). In contrast, location-
Figure 3: Sensitivity of existing body and hand models to different alignments (left) and scales (right).
variant augmentations altering the subject's position within the frame, such as rotation, scaling, and horizontal or vertical translation, result in substantially higher error magnitudes. This demonstrates the heightened sensitivity of existing models to changes in the subject's position. In Appendix, we provide the results of other metrics and benchmarks in Figures 22 - 23, and visualizations of whole-body estimation under different settings in Figures 25 - 26.
Such position-altering augmentations are common in real-world scenarios, where the subject in the image is often localized using external detection models and control over the quality of crops is less feasible. In practice, to guarantee the visibility of the subject, crops are often made broader. This can lead to significant performance degradation, as errors increase with smaller augmentation scale factors (<1.0) (Figure 1). Besides, horizontal and vertical translations, which correspond to scenarios where the subject is not perfectly centralized or entirely visible within the frame, can further decrease
Figure 2: Crops from (a) ExPose [6] (b)_PIXE [11], (c) Hand4Whole [35] (d) RoboSMPLX.
the performance. Similarly, the alignment and scale of these crops also influence the pose and shape estimation systems targeting body, face and hands (Figure 3, more quantitative and qualitative evidence in Appendix M). Whole-body methods bear the additional responsibility of accurately localizing body parts such as hands and face. Inaccurate part crops (Figure 2) can adversely affect the performance of part subnetworks, and further the whole-body estimation.
Impact of feature extraction. The deterioration of performance in the face of such variations suggests that the model struggles to extract meaningful features. Under alterations in translation or scale, the subject remains within the image frame, though the proportion of background content may vary. It is difficult for existing methods to effectively disregard irrelevant background elements and extract relevant features related to the subject of interest. To enhance the model's robustness, it is critical to produce consistent features irrespective of various augmentations applied to the image.
Impact of output pixel alignment. Pixel alignment is a critical aspect of high model performance. In certain instances, despite having precise subject localization, the model fails to produce properly aligned results (Figure 25 in Appendix). This is often caused by the suboptimal camera parameter estimation. To address this issue, we need to accurately estimate the camera parameters, ensuring the projected mesh is precisely aligned with the ground-truth at the pixel level. Such precision would enhance the effectiveness of the model in producing accurate pose, shape and camera parameter predictions, improving the overall accuracy and reliability of the estimation process.
4 RoboSMPLX Framework
We design RoboSMPLX to enhance the robustness of whole-body pose and shape estimation. It provides three specialized modules to address each challenge in Section 3: 1) Localization Module (Section 4.2): explicitly learning the location information of the subject and incorporating it into model estimations for pose, shape and camera; 2) Contrastive Feature Extraction Module (Section 4.3): reliably extracting pertinent features under various augmentations, thereby improving the model's generalization ability and robustness to a broader range of real-world scenarios; 3) Pixel Alignment Module (Section 4.4): ensuring that the outputs are pixel aligned.
We start with the description of RoboSMPLX architecture with Body, Hand and Face subnetworks (Section 4.1). Each subnetwork is integrated with the Localization Module and Pixel Alignment
Figure 4: Pipeline of our RoboSMPLX framework consisting of Body, Hand and Face subnetworks.
Figure 5: Examples of keypoint and part segmentation supervision for Body, Hand and Face subnetworks.
Module, and applies the Contrastive Feature Extraction Module for learning more robust features. Figure 6 shows the Hand subnetwork architecture. The other two subnetworks have the same designs.
4.1 Architecture and Training Details
Figure 4 shows the overall pipeline of RoboSMPLX for whole-body 3D human pose and mesh estimation. The Body subnetwork outputs 3D body joint rotations  $\theta_{b} \in \mathbb{R}^{21 \times 3}$ , global orientation  $\theta_{bg} \in \mathbb{R}^{3}$ , shape parameters  $\beta_{b} \in \mathbb{R}^{10}$ , camera parameters  $\pi_{b} \in \mathbb{R}^{3}$ , and whole-body joints  $K \in \mathbb{R}^{137 \times 3}$ . Joints corresponding to the hand and face are used to derive bounding boxes. Subsequently, hand and face images are cropped from a high-resolution image to preserve details. The Hand subnetwork predicts left and right hand 3D finger rotations  $\theta_{h} \in \mathbb{R}^{15 \times 3}$ . Simultaneously, the Face subnetwork generates 3D jaw rotation  $\theta_{f} \in \mathbb{R}^{3}$  and expression  $\psi_{f} \in \mathbb{R}^{10}$ . When training Hand and Face subnetworks with part-specific datasets, additional parameters such as global orientation  $\theta_{fg} \in \mathbb{R}^{3}$ , shape  $\beta_{f} \in \mathbb{R}^{50}$ , and camera  $\pi_{f} \in \mathbb{R}^{3}$  are estimated. These branches are discarded during whole-body estimation and training. Additional information concerning each subnetwork can be found in Appendix B. Further details regarding the training and inference durations are elaborated upon in Appendix K.
Subnetworks are trained separately, then integrated in a multi-stage manner. Initial whole-body training runs for 20 epochs. The hand and face modules are substituted with the trained Hand and Face subnetworks, followed by 20 epochs of fine-tuning to better unify the knowledge from the Hand and Face subnetworks into the whole-body understanding. Each subnetwork is trained by minimizing the following loss function  $L$ :
$$
L = \lambda_ {3 D} L _ {3 D} + \lambda_ {2 D} L _ {2 D} + \lambda_ {B M} L _ {B M} + \lambda_ {p r o j} L _ {p r o j} + \lambda_ {s e g m} L _ {s e g m} + \lambda_ {c o n} L _ {c o n} \tag {1}
$$
Here  $L_{BM}$  is the L1 distance between the predicted and ground-truth body model parameters.  $L_{3D}$  denotes the L1 distance between 3D keypoints and joints regressed from the body model.  $L_{2D}$  signifies the L1 distance of the ground-truth 2D keypoints to predicted and projected 2D joints. The latter are obtained by projecting the regressed 3D coordinates from the 3D mesh to the image space using the perspective projection [17]. The part segmentation loss  $L_{segm}$  is the cross-entropy loss between  $P_{h,w}$  after softmax and  $P_{h,w}$  averaged over H×W elements, following [20].  $L_{proj}$  refers to the projected segmentation loss, which is the sigmoid loss between the projected mesh and the ground-truth segmentation map.  $L_{con}$  is the contrastive loss described in Section 4.3. For wholebody training,  $L_{box}$  is added to measure the L1 distance between the predicted and actual center and scale of the hands' and face's boxes.
4.2 Localization Module
This module focuses on subject localization by explicitly learning both sparse and dense predictions of the subject within the image. Figure 5 shows an example of the supervision used for each subnetwork.
Figure 6: Subnetwork Architecture with three modules. We use the Hand subnetwork as an example.  $z$  represents normalized  $J_{3D}$  while  $p$  corresponds to the ground-truth of  $z$ . Green and red dashed lines refers to contrastive loss for positive and negative samples respectively.
Figure 7: Augmentations for the Body subnetwork. Black, blue and red labels represent image-variant, location-variant and pose-variant augmentations, respectively.
In contrast to prior methods that directly output pose rotations from backbone features, this module aims to make the model explicitly conscious of the subject's location and semantics while predicting pose, shape and camera parameters. It can reduce the model's sensitivity to the variations of the subject's position, caused by minor shifts in the scale and alignment of the bounding box.
As shown in Figure 6, given an image, a convolutional backbone is utilized to extract its feature map  $F \in \mathbb{R}^{512 \times 32 \times 32}$ . Following [35], a  $1 \times 1$  convolutional layer is then used to predict 3D feature maps  $LF \in \mathbb{R}^{32J \times 32 \times 32}$  from  $F$ , where  $J$  represents the number of predicted joints with a feature map depth of 32.  $LF$  contains valuable information about the mesh's position in the image and semantics of various parts. It is concatenated with the backbone feature map  $F$  to predict pose  $\theta \in \mathbb{R}^P$ , shape  $\beta \in \mathbb{R}^{10}$  and camera translation  $\pi \in \mathbb{R}^3$ , where  $P$  is the number of body parts. Meanwhile,  $LF$  is also used to obtain extra information with two branches: (1) 3D joint coordinates  $K \in \mathbb{R}^{J \times 3}$  are obtained from  $LF$  using the soft-argmax operation [47] in a differentiable manner. (2) 2D part segmentation maps  $S \in \mathbb{R}^{P + 1 \times 64 \times 64}$  are extracted from  $LF$  with several convolution layers, which model  $P$  part segmentation and 1 background mask. Here, 64 represents the height and width of the feature volume, and each pixel  $(h, w)$  stores the likelihood of belonging to a body part  $P$ .
Note that learning part segmentation maps and 3D joint coordinates is complementary, as 3D joint coordinates encode depth information that may inform part ordering in segmentation maps. Additionally, joints often reside at the boundaries of part segmentation maps, serving as separators for distinct parts. The Body subnetwork utilizes 24 parts  $P$  and 137 joints  $J$ , the Hand subnetwork employs 16 parts  $P$  and 21 joints  $J$ , while the Face subnetwork employs 15 parts  $P$  and 73 joints  $J$ .
4.3 Contrastive Feature Extraction Module
This module incorporates a pose- and shape-aware contrastive loss, along with positive samples. By minimizing this loss, the model can produce consistent representations for the same subject, even when presented with different augmentations, thus fostering the extraction of meaningful features.
Conventional contrastive learning methods based on SSL (e.g., SimCLR) face challenges in unifying similar pose embeddings and distancing dissimilar ones in human pose and shape estimation tasks. Without labels for guidance, images with similar poses could be misidentified as negative samples and contrasted away, complicating the self-organization of the embeddings in pose space. Figures 9 to 12 in Appendix show their ineffectiveness for the 3D human pose and shape task [5] by visualizing the retrieved samples from the embeddings. The supervised contrastive learning approach by Khosla
et al. [19], though effective for image classification, might not extend well to human pose and shape estimation, which is a high-dimensional regression problem and poses exist in a continuous space rather than well-defined classes.
Our module overcomes the aforementioned issues with two innovations. First, we experiment with three human pose representations  $z$  and the corresponding distance functions: (1) A concatenated form of the global orientation and rotational pose; (2) global orientation and rotational pose as separate entities (3) 3D root-aligned joints regressed from the body model, derived from pose and shape inputs. For (1) and (2), we explore relative rotations in two forms: 6D vector and rotation matrix representation. For (3), L1, Smooth L1, and Mean Squared Error (MSE) was used (Table 9).