HMR-Adapter: A Lightweight Adapter with Dual-Path Cross Augmentation for Expressive Human Mesh Recovery

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 05:21:13

Abstract: Expressive Human Mesh Recovery (HMR) involves reconstructing the 3D human body, including hands and face, from RGB images. It is difficult because humans are highly deformable, and hands are small and frequently occluded. Recent approaches have attempted to mitigate these issues using large datasets and models, but these solutions remain imperfect. Specifically, whole-body estimation models often inaccurately estimate hand poses, while hand expert models struggle with severe occlusions. To overcome these limitations, we introduce a dual-path cross augmentation framework with a novel adaptation approach called HMR-Adapter that enhances the decoding module of large HMR models. HMR-Adapter significantly improves expressive HMR performance by injecting additional guidance from other body parts. This approach refines hand pose predictions by incorporating body pose information and uses additional hand features to enhance body pose estimation in whole-body models. Remarkably, a HMR-Adapter with only about 27M parameters achieves better performance in fine-tuning the large model on a target dataset. Furthermore, HMR-Adapter significantly improves expressive HMR results by combining the adapted large whole-body and hand expert models. We show extensive experiments and analysis to demonstrate the efficacy of our method.

Main paper: 
 introduction: 
 Expressive Human Mesh Recovery (HMR) is a long-standing problem in computer vision, aiming to estimate the whole-body 3D human shape and pose, facial expressions, and hand gestures from an RGB image. This is a crucial task as it underpins various downstream applications, e.g., clothed human reconstruction, virtual try-on, and AR / VR content generation. However, this task is quite challenging because humans are highly deformable, and there is usually blur or occlusion with their movement. Moreover, capturing the motion of hands is extremely hard since the image sizes of hands and faces are relatively small while exhibiting complex articulations.To address the aforementioned problems, existing research on expressive HMR generally falls into two categories: multiple-stage methods [4, 20, 39] and single-stage methods [3, 22, 24] . Multiplestage methods reconstruct the body [11, 17, 21, 43] , hands [2, 13, 14, 29] , and faces [5, 34, 35, 37, 46] with corresponding expert models individually, then different parts can be directly combined to achieve whole-body reconstruction due to the consistency of the whole-body parametric model, such as SMPL-X [28] . However, such a straightforward strategy may lead to unnatural integration due to the lack of holistic modeling. On the contrary, one-stage methods reconstruct the whole-body posture, face, and hand pose in a unified manner. However, minor rotation errors are accumulated along the kinematic chain, leading to noticeable drifts in distal joint positions, e.g., hand joints. Moreover, the hand joints suffer from small input resolutions, making it challenging to capture the variations of complex hand gestures. Both paradigms have not effectively addressed the collective modeling of hands and other body parts. Instead, we opt for a new framework that leverages cross-body-part cues to improve both body and hand modeling.Recent works [3, 11, 29] scale up the model size and training data to fully unlock the potential of massive human motion datasets. These methods adopt large-scale image encoders as their backbone to extract visual cues in the input image, followed by several independent decoder modules used to predict the human model parameters. However, these large HMR models remain limited. Specifically, SMPLer-X [3] , a whole-body mesh recovery model, inaccurately estimates hand poses by cropping small hand feature maps. It also fails to account for interactions between body parts, resulting in noticeable errors in distal joint positions, e.g., hand joints, because of the accumulated drifts along the kinematic chain. HaMeR [29] is a large hand expert model to process hand images exclusively. However, human movements are holistic and interconnected, with hand actions being significantly influenced by the body. HaMeR's limitation lies in its focus on only the hand area, lacking global semantic context. This results in performance degradation in scenarios where hands are severely occluded or blurred.To address these problems above, we introduce a promising, lightweight framework, named HMR-Adapter, with dual-path cross augmentation. We adapt the existing large-scale HMR models and further boost their performance for expressive HMR.First, we use the dual-path cross augmentation paradigm to enhance both the body pose and hand pose estimation. We observe that different body parts are inherently interconnected: Body poses significantly affect the global orientation of the hands in various ways, while the positions of hand joints can indicate the potential range of arm poses. We first adapt a hand expert model to strengthen its robustness under occlusion and blur cases. Specifically, we introduce the body pose as additional guidance for more precise hand [29] ) produces inaccurate hand prediction when the hand is severely occluded. In contrast, we adapt the large hand expert model and inject additional body pose guidance to help the hand expert understand the overall context, thus giving reasonable predictions under occlusion.pose estimation. In the meantime, we adapt a whole-body HMR model to refine the upper body pose estimation and encourage natural integration between the hand and body. Specifically, we use hand information to further refine the initial predicted body pose from the whole-body model's body pose regressor. To this end, we sample image features from the finger's root joints (MCP joints) as additional guidance to predict the body pose offset. This will not only enhance the body pose but also smooth out the poses at intersecting joints between different body parts (e.g., wrists).Second, we propose HMR-Adapter to adapt the target model while maintaining the strong capability of pre-trained expert models. In contrast to previous methods [8, 24, 45] that combine body feature and hand feature by concatenating or weighted sum, HMR-Adapter does not harm the original model's structure and knowledge. The adapter is parallel with the original hand expert's decoder, combining hand features and body context without modifying the original decoder. While the previous method [24] claimed that body feature contains much unnecessary information which may harm the hand pose estimation, we find that with our HMR-Adapter, critical information such as the motion of the wrists and arms are getting much greater attention when predicting hand poses. We argue that the main reason why directly adding body information harms hand estimation lies in the naive incorporation of body information, which can potentially confuse the model by obscuring critical information. In our proposed HMR-Adapter, body information from different parts is adaptively selected to aid hand prediction. In this way, the adapted hand expert model is more robust and can leverage useful body information to perform better in occlusion and truncation situations.Another interesting property of our HMR-Adapter design is that it is lightweight and very efficient. When fine-tuning the large HMR models, the performance of a HMR-Adapter with around 30M parameters is comparable to or even better than a fully finetuned large model. With the proposed HMR-Adapter, the large HMR models can converge faster, even without additional guidance injection.Quantitative and qualitative experimental results show that our HMR-Adapter is efficient and generalizable, which can be applied to large HMR models with different architectures. A small HMR-Adapter with about 30M parameters is comparable to or even better than the fully fine-tuned models for HMR fine-tuning. Besides, HMR-Adapter significantly improves the large whole-body model in terms of hand estimation. Our proposed dual-path cross augmentation with HMR-Adapter for expressive HMR significantly outperforms previous approaches on the challenging ARCTIC dataset, where severe occlusions occur.Our contributions can be summarized as follows.• We present a promising lightweight but efficient adaptation method called HMR-Adapter that can be applied across largescale HMR models. A small HMR-Adapter with around 30M parameters is comparable to or even better than fine-tuning the entire large model. • We propose a dual-path cross augmentation paradigm to jointly optimize the body pose and hand pose estimation in expressive HMR. This contributes to more robust hand reconstruction and more accurate body pose prediction. We highlight this on ARCTIC, a hand-object interaction dataset with heavy occlusion. • We demonstrate that our framework for expressive HMR outperforms several state-of-the-art methods for whole-body mesh reconstruction. 
related work 2.1 expressive human mesh recovery: 
 Expressive human mesh recovery focuses on reconstructing human meshes, including body, hands, and face, from a single image. Early research employed predefined parametric models like SMPL-X [28] for prediction. Current approaches to expressive human mesh recovery can be broadly divided into two groups: multi-stage and singlestage methods. Multi-stage approaches break down the reconstruction problem into parts, addressing the estimation of the human body [11, 17, 19, 40] , hands [2, 13, 14, 29] , and faces [5, 34, 35, 37, 46] separately using part-specific models, before combining with a body template layer. This strategy may produce artifacts at intersecting joints.In contrast, one-stage paradigm proposes to reconstruct all body components as a whole. ExPose [4] predicts hands, face, and body SMPL-X parameters, using body estimation to refine face and hand regions from high-resolution inputs. PIXIE [8] calculates partspecific feature confidence and merges face-body and hand-body features for robust regression, while Hand4Whole [24] identifies joint-level features essential for body and hand joint rotations. PyMAF-X [39] and HybrIK-X [20] propose smooth integration methods between different body parts to integrate results from subnetworks. OSX [22] uses a transformer-based end-to-end method for body part connections. RoboSMPLX [27] improves body part localization and feature extraction for robust whole-body model recovery. SMPLer-X [3] scales up models and data for whole-body recovery. Despite recent advancements in one-stage methods, effectively recovering whole-body models with realistic hand gestures remains challenging, particularly in scenarios involving occlusions, interacting hands, and motion blur. To this end, We adapt existing large-scale HMR models, incorporating a dual-path augmentation strategy between body parts and hands to enhance their performance.Standard full fine-tuning of large models is computationally intensive and can degrade generalization. A prevalent solution is to use adapters to keep the pre-trained backbone intact, incorporating only a minimal number of strategically positioned trainable parameters within the architecture. Adapters have been extensively utilized in the field of large language models [10, 16, 26, 33, 41, 42] and are increasingly applied in visual-language alignment models. CLIP-Adapter [9] integrates residual-style adapters after text and visual encoders of CLIP [30] to avoid the gradients back-propagation through CLIP's encoders. IP-Adapter [38] introduces a novel decoupled cross-attention mechanism that differentiates between text and image features, adding a trainable cross-attention layer for image features within each existing layer.Despite the advancement in large model adaptation, the area of adapting large human mesh recovery models remains rarely explored. We propose our light-weight HMR-Adapter to incorporate additional guidance information while retaining the extensive knowledge in the pre-trained HMR models. 
method 3.1 overview: 
 The pipeline of using our proposed HMR-Adapter for expressive human motion capture is presented in Fig. 2 . Given a human image, our task is to predict the pose and shape parameters of the SMPL-X [28] model. We adopt a dual-path pipeline to approach this: In one path, a pre-trained whole-body model predicts the initial body pose and the hand bounding boxes. These bounding boxes are then used to crop out the hand area from the input image. With the sampled hand image features, HMR-Adapter refines the initial body pose. In the other path, HMR-Adapter augments hand pose prediction from a pre-trained hand expert model by injecting body pose information. Finally, the refined body pose and hand pose are forwarded to the SMPL-X layer to get the recovered human mesh.3.2.1 Parametric Human Models. Our task is to recover the wholebody human mesh in a parameterized way, which is to predict the pose and shape parameters of the SMPL-X human model [28] , a unified model that models the human body, hands, and face. SMPL-X takes as input the pose parameters 𝜃 ∈ R 55×3 that includes body, hands, and jaw poses; shape parameters 𝛽 ∈ R 10 , and expression parameters 𝜓 ∈ R 10 . SMPL-X defines a function 𝑀 (𝛽, 𝜃,𝜓 ) that returns the mesh of the human body 𝑉 ∈ R (𝑁 ×3) , with 𝑁 = 10475 vertices. SMPL-X additionally returns the 𝐽 = 53 major joints, which include 22 main body joints, 1 jaw joint, and 15 joints per hand.We use the MANO parametric hand model [32] for hand modeling. MANO shares the same pose space with SMPL-X's hand part. MANO also defines a function that takes hand pose parameters 𝜃 ℎ ∈ R 16×3 that includes 3D hand global rotation and 3D finger rotations, and hand shape parameters 𝛽 ℎ ∈ R 10 . MANO returns the mesh of the hand 𝑉 ℎ ∈ R (𝑁 ′ ×3) , with 𝑁 ′ = 778 vertices.To better understand the use of HMR-Adapter, we first briefly introduce the original hand expert model. The hand model adopts a fully transformer-based design. Before entering the backbone, the hand image is resized to 𝐼 ℎ with a shape of 256 × 256. The image is then split into fixed-size patches with patch embedding to obtain the hand image tokens. The tokens are further processed with a ViT [6] backbone that follows the "huge" design (denoted as ViT-Huge). The backbone further processes the image tokens to get a series of output tokens F 𝐻 , followed by a transformer head that regresses the hand parameters. The transformer head has multiple transformer blocks. Each block consists of a multi-head self-attention, a cross-attention, and a feed-forward network (FFN). The transformer head receives a single query token while cross-attending to the hand image tokens F 𝐻 . Specifically, the single query token 𝑧 is first mapped to a higher dimension using a FFN, then encoded by a self-attention module to form the crossattention query features Z. Formally, the decoder incorporates the query features Z and the input image feature F 𝐻 in a cross-attention manner:EQUATIONwhere Q = SA(Z)W 𝑞 , K = F 𝐻 W 𝑘 , V = F 𝐻 W 𝑣 are the query, key, and values matrices of the cross-attention layer respectively, and W 𝑞 , W 𝑘 , W 𝑣 are the weight of the trainable linear projection layers. The output of the head Z new is passed to three independent linear layers to predict the hand pose parameters, camera parameters, and hand shape parameters, respectively.Similarly, the whole-body model preprocesses the image features with a ViT, which additionally learns a series of task tokens concatenated with the image tokens. Hand and face bounding boxes are predicted from the image tokens. The hand and body heads that predict body pose consist of a positional module to predict 3D keypoints and a regressor module to predict pose parameters. And we concentrate on adapting the body regressor module. The original body regressor predicts 3D body joint rotations 𝜃 0 (including the root joint rotation) by a linear layer, using a concatenation of 3D body joint coordinates and task tokens as input.Since we are putting our hand expert model in a whole-body HMR setting, it is natural to introduce information from the human body into hand pose prediction. Such body guidance information enables the hand model to predict reasonable hand pose even under severe occlusion.We use body pose as the guidance instead of the commonly used body image feature since we do not have access to the whole-body image when training HMR-Adapter on the hand expert. To prevent unrelated information (such as the motion of lower limbs) from corrupting hand pose estimation, we choose joints from the root joint to the Left / Right Wrist along the kinematic chain, named the Left / Right Arm joint group. Each joint's rotation is represented in the 6D rotational representation of [44] . The Arm joint group for each hand is denoted as 𝜃 𝑏 ∈ R 𝐽 ×6 , where 𝐽 is the number of arm joints. Then, we apply a linear layer to map 𝜃 𝑏 to produce the body joint conditioning features F 𝐵 ∈ R 𝐽 ×1280 aligned with the hand image feature. We set the additional body pose to zero when training on hand-only datasets that do not contain body pose labels.A straightforward method for incorporating the body pose feature involves concatenating it with the original hand image feature F 𝐻 and subsequently inputting them into the decoder head. Alternatively, following the method described in [8] , a moderator could be used to merge the body pose and image feature through a weighted sum. However, these simple techniques of concatenation or summation could improperly couple these features, failing to fully leverage the injected body pose feature and potentially causing the "catastrophic forgetting" [31] of the pre-trained hand expert model. In practice, we have observed that such naive approaches can deteriorate the model's performance.To this end, we propose to use a decoupled transformer module to incorporate the body pose guidance, with separate transformer blocks for the body pose feature and hand image feature. Specifically, we add a new self-attention and cross-attention layer for each transformer decoder block in the original hand decoder head to insert the body pose feature. Given the body pose feature F 𝐵 , the output of new cross-attention Z ′′ is computed as follows:EQUATIONwhereQ ′ = SA ′ (Z)W ′ 𝑞 , K ′ = F 𝐵 W ′ 𝑘 and V ′ = F 𝐵 W ′𝑣 are the query, key, and value matrices from the body pose feature. W ′ 𝑘 and W ′ 𝑣 are the corresponding weight matrices. SA ′ is the new self-attention layer. In order to speed up the convergence, SA ′ , W ′ 𝑞 , W ′ 𝑘 and W ′ 𝑣 are initialized from SA, W 𝑞 , W 𝑘 and W 𝑣 . Then, we add the output of body pose cross-attention to the output of hand image crossattention. Hence, the final formulation of the decoupled transformer layer is defined as follows:EQUATIONSince we freeze the original hand decoder, only the W ′ 𝑞 , W ′ 𝑘 , W ′ 𝑣 and SA ′ are trainable in the above motion-adapter. Finally, the output token Z new is passed to the original linear layer for predicting hand poses.We also augment body pose prediction using information from hands. Inspired by [24] , we use the hand image feature sampled by predicted hand joints to benefit body pose prediction. We extract hand joint features from the 2D hand feature map. Specifically, we use the fingers' root joints (i.e., hand MCP joints) from both hands, and the 2D hand feature map is obtained by using the ROI (regions of interest) module to crop features from the whole-body image's feature map with the predicted hand bounding boxes. A 1-by-1 convolution reduces the 2D map's channel dimension from 1280 (ViT-Huge feature dimension) to 512, followed by bilinear interpolation at the MCP joint coordinates, yielding joint features F 𝐽 ∈ R 8×512 for the four MCP joints per hand.Similar to adapting the hand expert, we use parallel transformer blocks to incorporate the additional hand joint feature, with the original body regressor fixed. The new transformer layer receives a single query token as an input token and predicts the body pose offset Δ𝜃 to the initial body pose prediction 𝜃 0 . Formally, the refined body pose Δ𝜃 can be defined as:EQUATION)where FFN is the body pose offset regressor,Q ′′ = SA ′′ (Z ′′ )W ′′ 𝑞 , K ′′ = F 𝐽 W ′′ 𝑘 , V ′′ = F 𝐽 W ′′𝑣 are the query, key, and value matrices of the attention operation respectively, and W ′′ 𝑞 , W ′′ 𝑘 , W ′′ 𝑣 are the weights of the trainable linear projection layers. SA ′′ is the trainable selfattention layer. The query token Z ′′ is an empty beginning token.We adopt a two-stage training style to train the two HMR-Adapters separately. During training, we freeze all the original modules in the hand expert model and the whole-body model and only train the proposed two HMR-Adapters.We use the ground truth body pose as the conditioning body pose input when training HMR-Adapter for the hand expert model. We aim to optimize the weights of HMR-Adapter to introduce body pose guidance properly. The objective function we aim to minimize is defined as:EQUATIONwhere 𝜆 2𝐷/3𝐷 stands for the loss weights for the 2D/3D joints error, 𝜆 𝑣 denotes the loss weight of vertex error, and 𝜆 𝜃 /𝛽 is for the MANO hand pose and shape parameters 𝜃 /𝛽. The hand pose parameter loss L 𝜃 is a combination of L1 and cosine distance between the predicted hand pose 𝜃 and the target pose 𝜃 * :EQUATIONwhere cos indicates the cosine similarity between two items. Note that we do not use hand pose adversarial loss, which reduces the training cost.When training HMR-Adapter for the wholebody large model, we use the ground truth 2D hand keypoints to sample from the hand image feature map and use the sampled joint features as the body HMR-Adapter's additional guidance. We optimize the weights of HMR-Adapter to encourage integrating crucial hand information into the whole-body model's body prediction head. We use the following loss function when adapting the whole-body model:EQUATION)where 𝜆 2𝐷/3𝐷 stands for the loss weights for the 2D/3D joints error, and 𝜆 𝜃 /𝛽 is for the SMPL-X pose and shape parameters 𝜃 /𝛽. The 2D keypoint loss L 2𝐷 consists of the loss of both predicted 2D keypoints and the projected 3D keypoints:EQUATIONwhere L 2𝐷 denotes the 2D keypoint prediction error and L 𝑝𝑟𝑜 𝑗 denotes the projected 3D keypoint prediction error. 
experiments 4.1 setup: 
 Datasets. We train HMR-Adapter for the whole-body model [3] and the hand expert [29] on datasets that provide accurate 3D hand annotations. Specifically, we use Human3.6M [18] , RICH [15] , and BEDLAM [1] . For training HMR-Adapter for the hand expert, we additionally use FreiHAND [47] and InterHand2.6M [25] . We crop out hand images from whole-body images using the ground-truth hand bounding boxes. For 3D whole-body reconstruction evaluation, we use ARCTIC [7] and EHF [28] . We additionally use 3DPW [36] to evaluate the body-only reconstruction quality. ARCTIC is a handobject interaction dataset fitted with accurate SMPL-X annotations, containing 10 subjects manipulating 11 objects. Notably, we follow [3] to exclude the egocentric frames in our testing as they only capture hands. The EHF dataset is a widely used dataset to test the performance of expressive HMR, providing SMPL-X aligned 3D mesh that accurately reflects the subject's diverse body, hand, and face articulations. 3DPW is one of the most famous outdoor datasets with 3D SMPL [23] annotations without hand or face annotations. Evaluation Metrics. For whole-body and hand mesh reconstruction, we use Mean Per Vertex Position Error (PVE) and Mean Per Joint Position Error (MPJPE) to evaluate the positions of 3D vertices and 3D joints, respectively. The average 3D joint distance (in 𝑚𝑚) and mesh vertex distance (in 𝑚𝑚) are computed between the predicted and ground-truth values, with an initial alignment of the pelvis joint's translation. The pelvis serves as the root joint for the whole body's 3D error computations. For hands, the root joints are the wrists. The Procrustes Aligned [12] Mean Per Joint Position Error (PA-MPJPE) and Procrustes Aligned Mean Per Vertex Position Error (PA-PVE) are calculated by additionally aligning the rotation and scale. The 3D joint coordinates required for the MPJPE and PA-MPJPE calculations are derived by applying a joint regression matrix from the SMPL-X model to the mesh, consistent with prior studies [3, 22, 24] . Average 3D errors for the left and right hands are reported as the 3D hands' error measurement. Implementation Details. In our experiments, we choose SMPLer-X [3] and HaMeR [29] as our baseline whole-body model and hand expert model, respectively. There are 6 attention layers in the HMR-Adapter. The total trainable parameters of our HMR-Adapter include a body pose feature projection network and 6 attention modules, amounting to about 27M in HaMeR and 37M in SMPLer-X. HMR-Adapter is trained on a single machine with 4 V100 GPUs for 10K steps with a batch size of 32 per GPU, which roughly takes 6 hours.The detailed architecture of HMR-Adapter and more training details can be found in the supplementary materials.Quantitative Comparison. Table 1 shows the quantitative results for our proposed method and previous state-of-the-art methods, and we evaluate their performance on whole-body reconstruction and hand reconstruction by PA-PVE, PVE, and PA-MPJPE. We follow [39] to construct a baseline model that combines the pre-trained SMPLer-X-H32 (SMPLer-X [3] with ViT-Huge backbone and trained on 32 different datasets) and HaMeR using Inverse Kinematics [39] .From table 1 : Quantitative comparisons against state-of-the-art methods on the EHF [28] , ARCTIC [7] dataset. The best performance is in boldface. We use PA-MPVPE (𝑚𝑚), MPVPE (𝑚𝑚), and MPVPE (𝑚𝑚) as evaluation metrics.13.4% and 39.5% on whole-body and hand PVE on ARCTIC, respectively. Our method also surpasses other competitors in terms of quality for both whole-body and hand reconstruction. On the ARCTIC dataset, our method shows notable improvements in hand reconstruction, showcasing the robustness of our proposed HMR-Adapter in occlusion scenarios. The results underscore the effectiveness of the additional guidance brought by the HMR-Adapter for both the large whole-body model and the hand expert model.Although our primary focus is on Expressive HMR, we additionally show the quantitative results on 3DPW in Table 2 to demonstrate the improvement of body reconstruction quality and the generalizability of our method. We evaluate the body-only reconstruction performance by PVE, PA-MPJPE, and MPJPE. We can still boost the performance of the baseline model SMPLer-X significantly in terms of body prediction accuracy, affirming that HMR-Adapter enhances body pose estimation in the large whole-body model. Qualitative Comparison. We also present qualitative results in Fig. 3 . We can see that large models with our HMR-Adapter can produce more reasonable results for body and hand motion under challenging cases, such as dense hand-object interactions or motion blur. For example, in the (a) instance, our predictions come with more reasonable hand poses for the occluded right hand. In the (c) instance, SMPLer-X gives the erroneous prediction for the person's left elbow, while ours gives a more accurate body pose prediction, proving the efficacy of the proposed dual-path cross augmentation pipeline. In the (f) instance, SMPLer-X produces inaccurate hand poses and upper body poses, while our prediction with refined body pose and hand pose fits the input image better.To validate the efficacy of our proposed HMR-Adapter on the hand expert model, we further show qualitative results of hand pose predictions in Fig. 4 , where we show the input hand images, predicted results by HaMeR with HMR-Adapter, and the original HaMeR's prediction. Compared to the baseline method, our model can give a more reasonable prediction for occluded hands. For example, in the (e) instance, our method gives a reasonable hand global orientation prediction due to the body pose guidance from the arm poses, while the hand-only baseline fails to achieve that. In the (f) instance, our method gives a reasonable finger pose prediction, while the hand-only baseline could only infer from the limited hand image cues, leading to erroneous finger predictions. [36] . We compare our HMR-Adapter with previous methods.In this section, we first validate our design's ability to fine-tune a target dataset in Table 3 and Table 4 on both hand model and hand whole-body models. Then, we show that the proposed HMR-Adapter is a better way to inject additional body pose guidance into the hand expert model to improve hand prediction by comparing different injection designs. Finally, we validate the design choices in HMR-Adapter for the large whole-body model. HMR-Adapter for Fine-tuning Table 3 demonstrates that using HMR-Adapter to fine-tune the existing large motion models is more efficient and can achieve competitive or superior results than the fully fine-tuned model with much less trainable parameters. First, we show quantitative results of fine-tuning HaMeR on ARCTIC. We crop out the hand images using the ground truth hand bounding boxes as training and testing data. In Table 3 uses HMR-Adapter, with the hand image feature as HMR-Adapter's input (the original input of HaMeR's decoder head). No.6 is ours, which uses HMR-Adapter to introduce the body pose feature. We can observe that: [29] .In Table 4 , we also conduct similar experiments for different fine-tuning methods on SMPLer-X. The experiment settings are identical for No.1-No.6. We use hand MCP joints' image features as additional input for No.4 and No.6, while No.5 receives the original body pose task tokens as HMR-Adapter's input. Body pose guidance for the Hand Expert. To verify the advantages of using HMR-Adapter to incorporate body features for predicting 3D hand poses, we further conduct comparison between different injection methods. Table 5 shows the quantitative results. To construct the baselines, we first follow [24] to concatenate hand image features and body pose features, then feed it to the original decoder. We also follow [8] to use a multi-layer perception as a moderator to calculate the confidence for different part features before adding them together. We can see from Table . 5 that HMR-Adapter achieves more accurate 3D hand poses by leveraging the extensive hand estimation training from the pre-trained hand expert and incorporating new body pose guidance. One reasonable explanation is that concatenation harms the model's ability to choose useful information from inputs. Although the weighted summation could avoid some unrelated information using the moderator, it still cannot fully leverage the body conditions. HMR-Adapter disentangles the body pose information and uses the attention mechanism to choose the information that is most related to hand motion while maintaining the original model's strong ability. Besides, as can be seen from the table, it is necessary to discard the lower body poses for body pose guidance. We believe that discarding the lower body poses helps the model to concentrate on the most related information in the upper body.Hand guidance for the Whole-body model. Similarly, we verify the efficacy of using HMR-Adapter on the whole-body model to incorporate hand features for predicting 3D body poses. The quantitative results are presented in Table 6 . 
conclusion: 
 We have proposed HMR-Adapter to boost the performance of existing human mesh recovery models. HMR-Adapter features a dualpath cross augmentation architecture that introduces additional guidance information for body and hand prediction refinement. We seamlessly combine whole-body and hand expert models to validate our approach in expressive human mesh recovery. Extensive experiments prove that HMR-Adapter significantly enhances the robustness and efficiency without requiring extensive fine-tuning, particularly in hand reconstruction quality.  
 
 Appendix:  


Decision: All reviewers have agreed to accept this paper. The authors need to address all the reviewers' comments in the revised manuscript.

Reviews: review: paper_summary: paper_summary: This paper introduces a method to estimate the 3D pose and shape of 3D human body and hand from a single image. The idea is to develop a module that integrates SOTA body and hand estimation methods at the feature level to obtain more stable results in occlusion situations. In this way, the hand poses could be inferred based on image feature and upper body poses. The proposed method has been evaluated on ARCTIC and EHF datasets to show the effectiveness. The ablation study shows that, compared with directly integrating the results, such feature-fusion design achieve better peformance on the two benchmarks.
strength_weakness: strength_weakness: 1. Promising results. The qualitative results under occlusion cases look really promising, which shows the suprior of the proposed feature-level integration module. 
2. The quantitative experiments, especially the ablation study compared with SMPLer-X + HaMeR, validate the effectiveness of the proposed method. 
3. The paper is easy to understand.
limitations: limitations: 1. Novelty. The proposed method can be a minimal implementation to verify the importance of overall body pose information for more stable hand pose estimation from a whole-body image. However, such idea has been well proved by many previous methods, such as Pymaf-X, Hybrik-X, and SMPLer-X. From this point of view, the technical novelty of the proposed method is quite limited. 
2. Overclaim. The proposed method is claimed to be effective due to the small size of the feature fusion module.  A more reasonable way to compare efficiency is to compare the runtime of the entire model.  It is more fair to compare the efficiency of the entire solution rather than just the integrated modules.  Please consider properly accounting for contributions.
paper_summary: paper_summary: This paper aims to improve the performance of large-scale Human Mesh Recovery (HMR) models. To this end, the authors propose a dual-path cross augmentation framework, dubbed HMR-Adapter. Specifically, hand features are used to improve body pose estimation while body pose features are adopted to refine hand pose predictions. Experimental results show that the proposed approach considerably improves the performance of baselines.
strength_weakness: strength_weakness: 1) This paper presents an efficient approach for finetuning large-scale HMR models.
2) The proposed approach achieves notable improvements over baselines, particularly in hand mesh recovery.
limitations: limitations: 1) Insufficient experiments. As this paper aims to develop an efficient adapter for large-scale HMR models, it is necessary to see how the proposed approach performs on foundation models with different capacities. However, the authors only apply their approach to SMPLer-X-H32.
2) The effectiveness of the whole-body adapter seems doubtful. As shown in Table 4, finetuning the decoder head already achieves good performance, where the trainable parameters are less than the proposed adapter.
3) Missing analysis on the effectiveness of the proposed approach. For example, in what scenarios can the proposed approach improve the performance of baseline? 
4) The performance gain seems to primarily come from the hand part. Perhaps adding the body pose feature is sufficient to achieve good performance. More justifications are required to validate the effectiveness of the hand adapter.
paper_summary: paper_summary: This paper presents a dual-path cross-augmentation framework featuring a novel adaptation technique called HMR-Adapter, which enhances the decoding module of large HMR models. The HMR-Adapter significantly boosts the performance of expressive HMR by incorporating guidance from other body parts. This method refines hand pose predictions by integrating body pose information and utilizes additional hand features to improve body pose estimation in whole-body models.
strength_weakness: strength_weakness: 1)This paper is easy to understand.

2)The idea of associating and integrating of body pose information  and hand pose information to enhance human mesh recovery is interesting.

3)The proposed method achieves state-of-the-art performance on the ARCTIC and 3DPW datasets.
limitations: limitations: 1)The novelty of the approach appears limited. The decoupled transformer module resembles the decoder of the Hand Expert Model. Additionally, the paper employs a two-branch structure and a large pretraining model, but their usage does not seem to differ significantly from other existing methods.To further delve into the idea of associating and integrating body and hand information, the article needs to propose innovative methods.

2)The paper utilizes three models: the Hand Expert Model, the Whole-Body Model, and the proposed HMR-Adapter. Although the HMR-Adapter is parameter-efficient, using three models instead of a single model increases the inference speed and computational resources required during the inference phase.

3)The network aims to improve hand pose estimation, but its performance on the EHF dataset is subpar. The previous method, PyMAF-X (HR48), shows comparable (PVE: 10.2 vs. 10.0, 10.3 vs. 9.9) or even superior (PVE: 29.7 vs. 32.4) results for hand pose estimation.
paper_summary: paper_summary: This paper proposes a dual-path cross augmentation framework with an HMR-Adapter approach to address the challenges in expressive human mesh recovery from RGB images. The HMR-Adapter enhances the decoding module of large HMR models by injecting additional guidance from other body parts, improving both hand pose and body pose estimations. By combining the adapted large whole-body and hand expert models, HMR-Adapter achieves significant improvements in expressive HMR performance, demonstrating its effectiveness through extensive experiments and analysis.
strength_weakness: strength_weakness: [Novelty] The proposed approach exhibits novelty by leveraging the adapter concept, which effectively builds upon the existing literature on human body and hand pose estimation for expressive human mesh recovery.

[Clarity] This paper is well organized, and descriptions are clear in general.
limitations: limitations: 1. It would be beneficial to discuss or clarify certain details of the methods to enhance the strength of the paper's contributions:

1.1 Why is the hand pose derived from the regression of the original decoder, whereas the body pose is predicted through an offset mechanism?

1.2 Why is it that body pose information is utilized to guide the learning of hand pose, whereas it is the hand features, rather than hand pose information, that are employed to direct the learning of body pose?

1.3 Why did the author not consider utilizing face pose information? Given that body pose and hand pose can enhance each other's estimation, how might the inclusion of face pose information contribute?

2. Some points remain unclear:

2.1 What does † refer to in Table 1?

2.2 How is the inference process executed? Are the experimental results for HMR-Adapter in Table 1 achieved by utilizing ground truth (GT) from different branches or solely by directly adopting the baseline model's output? Specifically, are the test results for hands based on reading the GT of body pose or on the prediction results of SMPLer-X-H32? Conducting ablation experiments with these two distinct settings would provide valuable insights.

2.3 “critical information such as the motion of the wrists and arms are getting much greater attention when predicting hand poses.” (lin 152) The paper lacks a corresponding theory or experiment to substantiate this conclusion. If the stated conclusion holds true, the body area influenced by the hand should primarily manifest in the pose expression of the wrist and arm. However, how do we interpret the effect of the hand pose on the overall upper body pose as depicted in Figure 3 (f)?


3. Minor points:

3.1 The representation in Figure 2 can be misleading, as the input for hand pose decoding comprises not only the body pose but also the image features of the hand, leading to a similar issue in the other branch.

3.2 Reference citation format is not uniform.

3.3 There is no discussion on the failure cases or limitations of the introduced method.