ARTS: Semi-Analytical Regressor using Disentangled Skeletal Representations for Human Mesh Recovery from Videos

Authors: tao  tang
hong  liu
yingxuan  you
ti  wang
wenhao  li
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 03:59:45

Abstract: Although existing video-based 3D human mesh recovery methods have made significant progress, simultaneously estimating human pose and shape from low-resolution image features limits their performance. These image features lack sufficient spatial information about the human body and contain various noises (e.g., background, lighting, and clothing), which often results in inaccurate pose and inconsistent motion. Inspired by the rapid advance in human pose estimation, we discover that compared to image features, skeletons inherently contain accurate human pose and motion. Therefore, we propose a novel semi-Analytical Regressor using disenTangled Skeletal representations for human mesh recovery from videos, called ARTS, which effectively leverages disentangled information in skeletons. Specifically, a skeleton estimation and disentanglement module is proposed to estimate the 3D skeletons from a video and decouple them into disentangled skeletal representations (i.e., joint position, bone length, and human motion). Then, to fully utilize these representations, we introduce a semi-analytical regressor to estimate the parameters of the human mesh model. The regressor consists of three modules: Temporal Inverse Kinematics (TIK), Bone-guided Shape Fitting (BSF), and Motion-Centric Refinement (MCR). TIK utilizes joint position to estimate initial pose parameters and BSF leverages bone length to regress bone-aligned shape parameters. Finally, MCR combines human motion representation with image features to refine the initial parameters of the human model and enhance temporal consistency. Extensive experiments demonstrate that our ARTS surpasses existing state-of-the-art video-based methods in both per-frame accuracy and temporal consistency on popular benchmarks: 3DPW, MPI-INF-3DHP, and Human3.6M. Code is available at https://github.com/TangTao-PKU/ARTS.

Main paper: 
 introduction: 
 Recovering human meshes from monocular images is a crucial yet challenging task in computer vision, with extensive applications in virtual reality, animation, gaming, and robotics. Different from 3D Human Pose Estimation (HPE) [1] that predicts the location of several skeleton joints, 3D Human Mesh Recovery (HMR) [2] is a more complex task, which aims to estimate the detailed 3D human mesh coordinates.Although there has been some progress in 3D HMR from a single image [3] [4] [5] [6] [7] , accurate and temporally consistent recovery of the human mesh from monocular videos remains challenging. With the success of the SMPL [8] , previous video-based HMR methods directly predict SMPL parameters from image features. The SMPL model is a parametric human model consisting of 72 pose parameters and 10 shape parameters, which control 6890 vertices to form a human mesh. As shown in Figure 1 (a), previous video-based HMR methods first use the pre-trained Convolutional Neural Networks (CNNs) backbone [9] to extract image features from video frames, then design temporal networks based on Recurrent Neural Networks (RNNs) [10] [11] [12] or Transformers [13] [14] [15] [16] [17] to extract the spatial-temporal coupled features. Finally, SMPL pose and shape parameters are obtained from the same low-resolution and coupled features using the fully connected layers. However, limited by the insufficient spatial information and noises in the image features, these methods tend to suffer from the following three problems: (i) Inaccurate pose estimation. The image features extracted by ResNet [9] after global pooling are low-resolution with significant loss of spatial information, making it difficult for the following network to learn the highly non-linear mapping from image features to SMPL pose parameters [18] [19] [20] .(ii) Ineffective shape fitting. Many datasets [21] [22] [23] only contain a small number of subjects, resulting in the scarcity of body shapes. Consequently, employing neural networks to directly regress SMPL shape parameters often leads to overfitting [24] , which often regresses average human shape during inference.(iii) Inconsistent human motion. The image features contain various noises (e.g., background, lighting, and clothing) that affect the human motion capturing. Meanwhile, the changes in image features cannot directly reflect human movements, leading to undesirable motion jitters.Inspired by the rapid advance in video-based HPE [25] [26] [27] [28] [29] , we discover that the skeletons estimated by HPE algorithms can be utilized to alleviate the problems above. Compared to low-resolution image features, the skeletons contain more accurate information about the human pose, human motion, and basic human shape (e.g., body height) [24] . However, HPE utilizes joint coordinates to represent human pose, while HMR estimates joint rotations as SMPL pose parameters. Due to the difference in pose representation between these two tasks, previous video-based HMR methods do not effectively utilize the skeletons. PMCE [30] first introduces HPE into video-based HMR by integrating the skeletons and image features. However, it treats the skeleton as a supplementary feature, ignoring the advantages of different structural information of the skeleton in pose, shape, and motion estimation. Moreover, PMCE directly predicts mesh coordinates without using the human prior, often leading to self-interactions, unreasonable poses and shapes.Based on the above observations, we propose a novel semi-Analytical Regressor using disenTangled Skeletal Representations (ARTS) as shown in Figure 1 (b), which incorporates skeletons to alleviate the problems above through analytics and learning methods. We divide the human mesh recovery task into two parts: 1) 3D skeleton estimation and disentanglement and 2) regressing SMPL parameters from the disentangled skeletal representations through a semi-analytical regressor. Firstly, we propose a skeleton estimation and disentanglement module to estimate 3D skeletons from a video and decouple them into disentangled skeletal representations (i.e., joint position, bone length, and human motion). This disentanglement allows the following modules to focus on different information about human pose, shape, and motion. Secondly, to fully utilize the disentangled skeletal representations, we introduce a semi-analytical SMPL regressor consisting of three modules, which combines analytics and learning methods to estimate SMPL parameters. Specifically, a Temporal Inverse Kinematics (TIK) module is proposed to derive initial SMPL pose parameters from the joint position. The inverse kinematics improves the accuracy and robustness of SMPL pose estimation. To alleviate the challenge of ineffective shape fitting, we design a Bone-guided Shape Fitting (BSF) module, which combines analytics with MLP to regress the initial SMPL shape parameters from the bone length. Besides, we propose a Motion-Centric Refinement (MCR) module that employs crossattention to obtain motion-centric features from image features and human motion representation. Then, the motion-centric features are utilized to refine the initial SMPL parameters and enhance the temporal consistency of human mesh. Our model is evaluated on popular 3D human mesh recovery benchmarks, outperforming previous state-of-the-art video-based HMR methods.Our main contributions are summarized as follows:• We propose a semi-Analytical Regressor using disenTangled Skeletal representations (ARTS) that combines analytics with learning methods, which effectively leverages structural information of skeletons to improve both per-frame accuracy and temporal consistency. • In semi-analytical regressor, we carefully design three components, i.e., Temporal Inverse Kinematics (TIK), Bone-guided Shape Fitting (BSF), and Motion-Centric Refinement (MCR), for learning accurate and temporally consistent human pose, shape, and motion, respectively. • Our method achieves state-of-the-art performance on multiple 3D human mesh recovery benchmarks. Particularly, in cross-dataset evaluation, ARTS reduces MPJPE and MPVPE by 10.7% and 10.5% on the 3DPW [21] dataset. 
related work 2.1 3d human pose estimation: 
 Benefiting from the advance of 2D pose detections [31, 32] , many recent works for 3D HPE are based on the 2D-to-3D lifting pipeline [1] . Some 3D pose estimation methods [33] [34] [35] [36] employ Graph Convolutional Networks (GCNs) [37] to extract spatial-temporal information from the skeletons by treating the human skeleton as a graph. In recent years, Transformer-based methods [25] [26] [27] 29] have become popular, which carefully design serial or parallel, global or local Transformer [38] networks to explore the spatial and temporal dependencies of 2D skeletons. The achievements in 3D HPE demonstrate that the skeletons contain sufficient spatial and temporal information about human body, which can be leveraged to produce accurate and temporally consistent human mesh.Due to the easy access to images, many human mesh recovery methods take a single image as input. The image-based methods can be divided into two paradigms. The first is parametric methods based on the human model (e.g., SMPL [8] ). Due to the complexity of the SMPL estimation, some methods incorporate prior knowledge to assist in estimating SMPL parameters, such as body silhouette [39] , semantic body part segmentation [40, 41] , bounding box [4] , and kinematic prior [19, 42, 43] . Moreover, the skeletons from 3D HPE can assist the human mesh regression. For example, Nie et al. [44] aim to learn a good pose representation that disentangles pose-dependent and view-dependent features from human skeleton data. However, the pose-dependent features capture only the overall pose and lack disentangled semantic details. PC-HMR [45] uses the estimated human pose to calibrate the human mesh estimated by the off-the-shelf HMR methods, serving as a time-consuming post-processing. The other paradigm is non-parametric methods, which directly estimate the coordinates of each mesh from an image. For instance, Pose2Mesh [3] designs a multi-stage MeshNet to upsample sparse 3D skeletons to human mesh. GTRS [5] introduces a graph Transformer network to reconstruct human mesh from a 2D human pose. Although these image-based methods have achieved remarkable performance in accuracy, they tend to produce unsmooth human motion when applied to videos.Compared to image-based HMR methods, video-based HMR methods simultaneously recover accurate and temporally consistent human mesh. The previous video-based methods mainly focus on designing temporal extraction and fusion networks to enhance temporal consistency. For instance, VIBE [10] , MEVA [11] , and TCMR [12] carefully design Gated Recurrent Units (GRUs) based temporal extraction networks, which are utilized to smooth human motion but lack sufficient ability to model long-term dependencies. Consequently, most methods utilize Transformer-based temporal networks. MEAD [46] proposes a spatial and temporal Transformer to parallel model these two dependencies. GLoT [14] proposes a global and local Transformer to decompose the modeling of longterm and short-term temporal correlations. Bi-CF [15] introduces a bi-level Transformer to model temporal dependencies in a video clip and among different clips. UNSPAT [17] proposes a spatiotemporal Transformer to incorporate both spatial and temporal information without compromising spatial information. Despite the complicated design of these temporal networks, the insufficient spatial information and noises of image features unavoidably lead to limited performance. Although skeleton prior is commonly used in imagebased methods, it is often ignored by video-based methods. Sun et al. [47] also uses skeleton-disentangled representation, but its disentanglement refers to only extracting the skeletons from image features without HPE methods. Besides, it does not decompose different information within skeletons so it may not fully utilize the skeleton data. PMCE [30] is the first to incorporate 3D HPE methods in video-based HMR. However, it directly utilizes crossattention to fuse image features and the skeletons, which ignores the structural information of the skeletons. Additionally, PMCE directly regresses mesh coordinates without incorporating sufficient human prior (e.g., SMPL), which often results in self-interactions, unreasonable body poses and shapes. In contrast, our ARTS decouples the skeletons into joint position, human motion, and bone length and exploits the advantages of the disentangled skeletal representations in regressing different parameters of the SMPL model. This makes our method achieve better per-frame accuracy and temporal consistency. 
methodology 3.1 overall framework: 
 The overall framework of our semi-Analytical Regressor using dis-enTangled Skeletal representations for HMR (ARTS) is illustrated in Figure 2 , which primarily consists of two parts: 1) 3D skeleton estimation and disentanglement and 2) regressing human mesh from the disentangled skeletal representations and image features through a semi-analytical SMPL regressor. In detail, given a video sequence 𝑉 = {𝐼 𝑡 } 𝑇 𝑡 =1 with 𝑇 frames. A pre-trained ResNet50 [9] from SPIN [48] is utilized to extract image features 𝐹 ∈ R 𝑇 ×2048 for each frame. For 3D skeleton estimation and disentanglement, we first employ the off-the-shelf 2D pose detector [31, 32] to estimate 2D skeletons 𝑆 2𝐷 ∈ R 𝑇 ×𝐾 ×2 and then design a dual-stream Transformer network to lift 2D skeletons to 3D skeletons 𝑆 3𝐷 ∈ R 𝑇 ×𝐾 ×3 , where 𝐾 denotes the number of human keypoints. After that, we further decouple the 3D skeletons into disentangled skeletal representations (i.e., joint position 𝐽 , human motion 𝑀, and bone length 𝐵). In the second part, the semi-analytical SMPL regressor regresses SMPL parameters from disentangled skeletal representations and image features. In regressor, we propose Temporal Inverse Kinematics (TIK) to regress initial SMPL pose parameters 𝜃 𝑖𝑛𝑖𝑡 ∈ R 72 of mid-frame from joint position and image features. The Bone-guided Shape Fitting (BSF) module is introduced to predict bone-aligned SMPL shape parameters 𝛽 𝑖𝑛𝑖𝑡 ∈ R 10 of mid-frame from bone length. Finally, we design a Motion-Centric Refinement (MCR) to fuse image features with human motion guidance and employ motioncentric features 𝐹 ′ ∈ R 2048 to align SMPL parameters with the image cues. We elaborate on each part in the following sections.3.2.1 3D Human Pose Estimation. We first utilize the off-theshelf 2D pose detectors [31, 32] to obtain the 2D skeletons from the image sequence. Subsequently, we design a simple yet effective 𝐿 1 blocks dual-stream Transformer network to lift the 2D skeletons to 3D skeletons. Specifically, we first project the 2D skeletons feed 𝑋 to the dual-stream Transformer and the output features 𝑋 ′ ∈ R 𝑇 ×𝐾 ×𝐶 1 are fused as the subsequent block's input, which can be expressed as:𝑆 2𝐷 ∈ R 𝑇 ×𝐾 ×2 into high-dimensional features 𝑋 ∈ R 𝑇 ×𝐾 ×𝐶EQUATIONwhere MSA ST denotes spatial-temporal Transformer, MSA TS denotes temporal-spatial Transformer, 𝑃𝐸 𝑆 and 𝑃𝐸 𝑇 represents spatial and temporal embedding, respectively. Finally, the high-dimensional features 𝑋 ′ are mapped into the 3D skeletons 𝑆 3𝐷 ∈ R 𝑇 ×𝐾 ×3 .3.2.2 Skeleton Disentanglement. Different from the previous video-based method [30] that treats the skeletons as supplementary features, we decouple the 3D skeletons 𝑆 3𝐷 into the disentangled skeletal representations (i.e., joint position 𝐽 , bone length 𝐵, and human motion 𝑀). Specifically, joint position 𝐽 ∈ R 𝑇 ×𝐾 ×3 represents the positions of all joints, which contains accurate information about human pose. The bone length 𝐵∈ R 𝑁 represents the temporally average length of each bone within the video sequence, where 𝑁 denotes the number of bones. This average bone length contains temporally consistent basic body shape information. For human motion, We calculate the detailed motion of each frame 𝑀 𝑡 ∈ R (𝑇 -1) ×𝐾 ×3 and then concat temporally average human motion 𝑀 0 ∈ R 1×𝐾 ×3 to get the final human motion 𝑀∈ R 𝑇 ×𝐾 ×3 , which represents the overall and detailed human movements. The equations of human motion and bone length are as follows:EQUATIONEQUATIONwhere 𝑡 represents 𝑡 𝑡ℎ frame, 𝑎𝑣𝑔(•) represents averaging over time, 𝑛 represents 𝑛 𝑡ℎ bone, 𝑖 and 𝑗 denote two joints of the 𝑛 𝑡ℎ bone.As shown in Figure 2 , we propose a semi-analytical SMPL regressor to predict the SMPL parameters from the disentangled skeletal representations and image features. Semi-analytical regressor combines accurate analytical methods with flexible learning methods, which can generate accurate human mesh from skeletons and maintain robustness to the errors of skeleton estimation. Specifically, the joint position can provide a more accurate human pose. The bone length contains consistent basic body shape and human motion can provide accurate and smooth human movement. Based on these observations, we design three modules (i.e., TIK, BSF, and MCR) to exploit the advantages of the disentangled skeletal representations in regressing different parameters of the SMPL model.Temporal Inverse Kinematics. We regress accurate SMPL pose parameters from the joint position and image features. Following HybrIK [19] , we first decompose the SMPL pose into swing among neighboring joints and twist that represents the angle along the bone direction. Swing can be accurately derived from joint position, while twist cannot be directly obtained from the coordinates of the joints, so we regress the twist of each joint from image features. This decomposition allows us to tackle the SMPL pose estimation by separately handling the joint-based swing calculation and imagebased twist estimation, significantly reducing the complexity of the SMPL pose estimation. Different from HybrIK, which solely relies on analytical methods and ignores temporal cues, we utilize MLP and self-attention to obtain temporally consistent SMPL joint rotation from swing and twist. This process is expressed as follows: where 𝑠𝑤𝑖𝑛𝑔∈ R 𝑇 ×𝐾 ′ ×6 denotes 6D rotations [49] among joints, 𝑡𝑤𝑖𝑠𝑡 ∈ R 𝑇 × (𝐾 ′ -1) ×2 denotes twist angles of each joints (except root joint), 𝐾 ′ denotes the number of SMPL keypoints, 𝜃 𝑖𝑛𝑖𝑡 ∈ R 72 denotes temporally consistent SMPL pose parameters of mid-frame, MSA(•) represents multi-head self-attention.EQUATIONEQUATIONBone-Guided Shape Fitting. As shown in Figure 3 , we regress bone-aligned SMPL shape parameters from the bone length.According to the SMPL [8] , the joint position in the rest pose is a function of shape parameters. Given the mean shape parameters 𝛽 𝑚𝑒𝑎𝑛 ∈ R 10 of SMPL template, the joint position is calculated as:EQUATIONwhere T ∈ R 6890×3 is the SMPL mesh template, 𝑆 ∈ R 6890×3×10 is the shape blend matrix that maps shape parameters to the offsets of template, 𝑊 ∈ R 𝐾 ×6890 is the joint regression matrix that obtains joints from human mesh. We can obtain rough shape parameters from the bone length since the bone length 𝐵 contains the body height and reflects body weight [3] . We first align the rest pose 𝐽 𝑚𝑒𝑎𝑛 with bone length to get the bone-aligned pose 𝐽 𝑎𝑙𝑖𝑔𝑛𝑒𝑑 ∈ R 𝐾 ×3 through an analytical transform 𝑓 (𝐵, 𝐽 𝑚𝑒𝑎𝑛 ) along the kinematic tree:EQUATIONwhere 𝐵 𝑖,𝑗 denotes the bone length between parent joint 𝑖 and child joint 𝑗. Then, we can derive the bone-aligned SMPL shape parameters 𝛽 ′ 𝑖𝑛𝑖𝑡 ∈ R 10 from the bone-aligned pose:EQUATIONHowever, relying solely on the analytical method makes the shape estimation highly sensitive to errors and noises in bone length. Therefore, instead of directly using the analytical transform, we initialize the MLP with the pseudo-inverse matrix 𝑆 -1 and 𝑊 -1 . Then, we train the analytical-MLP transform 𝑔(𝐽 𝑎𝑙𝑖𝑔𝑛𝑒𝑑 , T ) to finetune the mapping from bone-aligned pose to shape parameters. This process can be expressed as follows:EQUATIONwhere FC S -1 (•) and FC W -1 (•) are the full-connected layer initialized by pseudo-inverse shape blend matrix and joint regression matrix.Refinement. Due to the sparsity of skeleton information, the SMPL parameters regressed from joint position and bone length lack human details (e.g., accurate shapes), which still need to be refined from images. Therefore, we utilize human motion to guide the fusion of image features and then refine the initial SMPL parameters. In detail, we first project human motion 𝑀 to high-dimension features as the queries 𝑄 𝑚 ∈ R 𝑇 ×𝐶 2 and projects image features 𝐹 to the keys 𝐾 𝑖 ∈ R 𝑇 ×𝐶 2 and values 𝑉 𝑖 ∈ R 𝑇 ×𝐶 2 . Then, we employ 𝐿 2 layers cross-attention MCA(•) and feedforward networks FFN to obtain the motion-centric features 𝐹 ′ ∈ R 2048 , which can be expressed as follows:EQUATION)This fusion manner makes the network focused on human movement. Moreover, we utilize MLP heads to refine the initial SMPL parameters from motion-centric features. Finally, we utilize the refined SMPL pose and shape parameters 𝜃 𝑟𝑒 𝑓 𝑖𝑛𝑒𝑑 , 𝛽 𝑟𝑒 𝑓 𝑖𝑛𝑒𝑑 to generate the final human mesh.The skeleton estimation and disentanglement module is trained with the 3D joint loss L 𝑗𝑜𝑖𝑛𝑡 to supervise the 3D skeletons of all frames. Then, the whole network is supervised by four losses: mesh vertex loss L 𝑚𝑒𝑠ℎ , 3D joint loss L 𝑗𝑜𝑖𝑛𝑡 , SMPL pose loss L 𝑝𝑜𝑠𝑒 , SMPL shape loss L 𝑠ℎ𝑎𝑝𝑒 , The final loss is calculated as:EQUATION)where 𝜆 𝑚 = 1, 𝜆 𝑗 = 1, 𝜆 𝑝 = 0.06, and 𝜆 𝑠 = 0.06 in ARTS. The 𝜆 parameters ensure that the values of various loss functions are maintained at the same range. 
expriments 4.1 datasets and evaluation metrics: 
 4.1.1 Datasets. Following previous methods [14, 15, 30] , we train our model on the mixed 2D and 3D datasets. For 3D datasets, 3DPW [21] , MPI-INF-3DHP [23] , and Human3.6M [22] contain the annotations of 3D joints and SMPL parameters. For 2D datasets, COCO [50] and MPII [51] contain 2D joint annotation with pseudo-SMPL parameters from NeuralAnnot [52] . To compare with previous methods, we evaluate the performance of our model on the 3DPW, MPI-INF-3DHP, and Human3.6M datasets.To evaluate per-frame accuracy, we employ the mean per joint position error (MPJPE), Procrustesaligned MPJPE (PA-MPJPE), and mean per vertex position error (MPVPE). These metrics measure the difference between the predicted mesh position and ground truth in millimeters (𝑚𝑚). To evaluate temporal consistency, we utilize the acceleration error (Accel) proposed in HMMR [53] . This metric calculates the average difference in acceleration of joints, which is measured in 𝑚𝑚/𝑠 2 .Consistent with the previous methods [14, 15, 30] , we set the input sequence length 𝑇 to 16 and utilize the pre-trained ResNet50 from SPIN [48] to extract image features of each frame. We train the network in two stages. Firstly, we train the 3D skeleton estimation network with all of the 3D and 2D datasets with a batch size of 64 and a learning rate of 1 × 10 -5 for 60 epochs. For 2D pose detectors in the skeleton estimation network, we adopt CPN [32] for Human3.6M and ViTPose [31] for 3DPW and MPI-INF-3DHP. In the second stage, we load the weights of the 3D skeleton estimation network and train the whole model with only 3D datasets. We train the whole network for 30 epochs with a batch size of 32 and a 1 , we report the results of our model on popular HMR benchmarks: 3DPW, MPI-INF-3DHP, and Human3.6M. Our ARTS surpasses existing state-of-the-art video-based methods in both perframe accuracy and temporal consistency metrics. Specifically, compared to the previous state-of-the-art method PMCE [30] , our model achieves a reduction of 2.6% (from 69.5𝑚𝑚 to 67.7𝑚𝑚), 9.9% (from 79.7𝑚𝑚 to 71.8𝑚𝑚), and 3.6% (from 53.5𝑚𝑚 to 51.6𝑚𝑚) in MPJPE metric on 3DPW, MPI-INF-3DHP, and Human3.6M datasets, respectively. Next, we analyze the limitations of previous video-based methods. GLoT [14] , Bi-CF [15] , and UNSPAT [17] regress SMPL parameters based on the low-resolution image features. Despite the design of complex networks to extract spatial-temporal features, the image features extracted by the backbone lack sufficient spatial information about the human body and contain various noises, resulting in inaccurate and unsmooth human mesh. Although UN-SPAT [17] and MEAD [46] have slightly lower PA-MPJPE than ours on the 3DPW dataset, MEAD utilizes ViT [55] as the backbone and their performance on other error metrics and datasets is much higher. PMCE [30] also incorporates 3D skeletons, but it integrates skeletons as a low-dimension feature. This limitation results in lower reconstruction accuracy of PMCE. Although PMCE achieves a marginally better Accel than ours by 0.3𝑚𝑚/𝑠 2 on the MPI-INF-3DHP dataset, it utilizes more 2D datasets with diverse scenes for the training of the SMPL regression network and its estimation error MPJPE is significantly higher than ours by 7.9𝑚𝑚. Different from previous methods, our ARTS effectively utilizes the disentangled skeletal representations for different SMPL parameters regression, including accurate joint position (lower MPJPE and PA-MPJPE), temporally consistent bone length (lower MPVPE), and precise human motion (lower Accel). Therefore, we achieve more accurate and temporally consistent human mesh recovery.To evaluate the generalization ability of our model, we conduct cross-dataset evaluation. Following previous methods [14, 15, 30] , we train our model on the MPI-INF-3DHP and Human3.6M datasets and evaluate it on 3DPW dataset. As illustrated in Table 2 , compared with image-based and video-based methods that report cross-dataset evaluation results, our ARTS shows a significant improvement in both per-frame accuracy and temporal consistency. Specifically, compared to Bi-CF [15] , These results demonstrate the robust cross-domain generalization ability of our model, which is primarily attributed to our effective utilization of disentangled skeletal representations. The image features extracted by the CNN backbone [9] vary significantly across different datasets, while the representation of the skeleton is quite similar across datasets. Therefore, the full utilization of the skeleton enhances the overall robustness of our model. As illustrated in Table 4 (Row 10-12), utilizing GRU networks similar to VIBE [10] and TCMR [12] leads to a reduction in Accel by 1.1𝑚𝑚/𝑠 2 , whereas employing Self-Attention methods (SA) similar to GLoT [14] and Bi-CF [15] results in a reduction in Accel by 1.2𝑚𝑚/𝑠 2 . Furthermore, both approaches exhibit a decrease in accuracy. ARTS leverages Motion-centric Cross-Attention (Motion-CA) to guide the fusion of image features, which enables the network to estimate temporally consistent human movements. To investigate the impact of the 2D pose detectors used in the skeleton estimation and disentanglement, we conduct experiments using different 2D poses (e.g., SH [56] , Detectron [57] , CPN [32] ) and Ground Truth (GT) pose as inputs. Table 5 compares PMCE [30] with ours on the Human3.6M dataset. The results indicate that ARTS outperforms PMCE across different 2D pose inputs. Moreover, our model has the potential to benefit from the development of 2D pose detectors and the experiment with GT inputs shows the lower bound of our method.Figure 5 shows the qualitative comparison among the previous state-of-the-art methods GLoT [14] , PMCE [30] , and our ARTS on the in-the-wild 3DPW dataset. Compared to GLoT that solely relies on image features, and PMCE that does not utilize the SMPL model as prior, our ARTS achieves more accurate and reasonable human meshes by using robust skeletons in various challenging scenarios, including self-interactions (Row 1), self-occlusion (Row 2), occlusion by others (Row 3), severe object occlusion (Row 4), low-light scene (Row 5), and image truncation (Row 6).Figure 5 : Qualitative comparison among GLoT (green mesh), PMCE (pink mesh) and our ARTS (blue mesh) on the challenging 3DPW dataset. 
conclusion: 
 In this paper, we present a novel semi-Analytical Regressor using disenTangled Skeletal representations for human mesh recovery from videos (ARTS), which mainly consists of two parts: 1) 3D skeleton estimation and disentanglement and 2) regressing SMPL parameters based on a semi-analytical SMPL regressor using the disentangled skeletal representations and image features. ARTS first estimates the 3D human skeletons from images and decouples them into joint position, human motion, and bone length. To estimate SMPL parameters from disentangled skeletal representations, we propose a semi-analytical SMPL regressor, which contains temporal inverse kinematics, bone-guided shape fitting, and motion-centric refinement modules. ARTS achieves state-of-the-art performance on multiple 3D human pose and shape estimation benchmarks. The cross-dataset evaluation and qualitative evaluation demonstrate the generalization ability and robustness of our ARTS. We hope our ARTS can further bridge the gap between human pose estimation and video-based human mesh recovery. 
 
 Appendix:  


Decision: This paper presents ARTS, a semi-analytical Regressor using disentangled Skeletal representations for human mesh recovery from videos. Two reviewers agree to accept the paper with Weak Accept and Borderline. One reviewer changed the Borderline Accept to Borderline Reject because the technical novelty is incremental and the suitability of multimedia, and one reviewer gave a weak reject but did not read the author's rebuttal and give the final rating. All the reviewers agree that the proposed method is effective. It also offers a novel approach for effectively integrating multimodal data (i.e., skeletons and images) to improve the HMR. Therefore, the paper can be accepted.

Reviews: review: paper_summary: paper_summary: The paper presents ARTS, a semi-Analytical Regressor using disentangled Skeletal representations for human mesh recovery from videos. The key innovation of ARTS lies in its approach to estimating 3D human meshes by leveraging disentangled information from skeletal data, which is more accurate and robust compared to traditional methods that rely on low-resolution image features. The method is composed of two main components: a skeleton estimation and disentanglement module, and a semi-analytical regressor that includes Temporal Inverse Kinematics (TIK), Bone-guided Shape Fitting (BSF), and Motion-Centric Refinement (MCR). The authors claim good performance on benchmarks such as 3DPW, MPI-INF-3DHP, and Human3.6M.
strength_weakness: strength_weakness: 1) The semi-analytical regressor, which combines elements of traditional analytical methods with machine learning techniques, is an interesting hybrid approach that could potentially offer the benefits of other methodologies.
2) The paper provides a clear explanation of the limitations of current video-based human mesh recovery methods and presents a logical argument for why their approach addresses these issues.
3) The extensive experiments and comparisons to state-of-the-art methods demonstrate the effectiveness of ARTS.
limitations: limitations: 1) While the paper claims state-of-the-art results, it is not clear how the performance of ARTS compares to the latest methods in terms of computational efficiency and scalability, which are critical for practical applications.
2) The paper could benefit from a more detailed discussion on the choice of network architecture and hyperparameters for the various modules, as these decisions can significantly impact the results.
3) The paper does not discuss potential failure cases or scenarios where ARTS might not perform as expected, which would be valuable for understanding the limitations and robustness of the approach.
paper_summary: paper_summary: The paper introduces ARTS, a new method for 3D human mesh recovery from videos that uses disentangled skeletal representations. It includes a novel semi-analytical regressor with three modules: Temporal Inverse Kinematics, Bone-guided Shape Fitting, and Motion-Centric Refinement. The method demonstrates improved accuracy and temporal consistency over existing methods on benchmarks such as 3DPW, MPI-INF-3DHP, and Human3.6M.
strength_weakness: strength_weakness: 1. The paper introduces a semi-analytical Regressor using disentangled Skeletal representations (ARTS) that combines analytical and learning methods to enhance 3D human mesh recovery accuracy and consistency.

2. The method design is generally technically sound to me.

3. The author provides extensive experimental results.

4. The result of the proposed method exceeds SOTA.
limitations: limitations: I have only one concern about the design of the method. It seems that previous works have similar disentangled designs. Could you explain some non-trivial differences between the proposed methods with the previous methods, such as [1,2]?

[1] Nie, Qiang, Ziwei Liu, and Yunhui Liu. "Unsupervised 3d human pose representation with viewpoint and pose disentanglement." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16. Springer International Publishing, 2020.

[2] Luan, Tianyu, et al. "Pc-hmr: Pose calibration for 3d human mesh recovery from 2d images/videos." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021.
paper_summary: paper_summary: The paper is focused on the task of 3D human mesh recovery from videos. Specifically, the main idea is to estimate the 3D human mesh from videos with the help of the estimated 3D skeletons. Therefore, the proposed method leverages disentangled skeletal representations to effectively utilize the information contained in the skeletons, which are more accurate and robust than image features in representing human pose and motion. The method consists of two main parts: (1) the 3D skeleton estimation and disentanglement module, and (2) the semi-analytical SMPL regressor from the disentangled skeletal representations. Evaluations on three widely-used datasets show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Novelty: The idea of the semi-analytical regressor using disentangled skeletal representations to estimate SMPL parameters is novel compared to existing methods that directly predict SMPL parameters from image features.
2. Technical correctness: The proposed method is technically sound and outperforms existing state-of-the-art methods, demonstrating its effectiveness.
3. Adequate evaluation: The paper provides a comprehensive evaluation of the proposed method using both per-frame accuracy and temporal consistency.
4. Clarity: The paper is well-written and easy to understand. The figures and video demos are clearly presented.
5. Applications: The proposed method has potential applications in various fields such as virtual reality, animation, gaming, and robotics.
limitations: limitations: 1. Compared with the existing end-to-end pipeline, the proposed framework is complex with two CNN-based backbone networks and a regressor head. I suggest the authors compare the number of parameters among the proposed method and existing ones.
2. The performance of the proposed method relies on the performance of 2D pose estimation and 3D lifting. These two techniques may limit the applications of the proposed method.
3. The experiments are insufficient for the following aspects: 1) only ResNet-50 is used as the backbone, so it is better to use other backbones like the ViT-based ones to further improve the performance, 2) from Table 1 and Table 2, it seems that the temporal consistency of the ARTS is just competitive to the SOTA methods like GLoT and PMCE.
4. Basically, this paper considers a computer vision task using unimodal data, i.e., video, so it is moderately suitable for the MM conference.
paper_summary: paper_summary: This paper describes a method for 3D human pose and shape (HPS) estimation from a monocular video input. The key idea is to disentangle the human pose and shape information embedded in 2D and 3D human skeleton to guide the HPS estimation process. The proposed method is evaluated on common 3D human pose benchmarks, 3DPW, MPI-INF-3DHP, and Human3.6M. The results seem promising.
strength_weakness: strength_weakness: 1. Promising results. The quantitative results on common benchmarks are quite promising. The qualitative video results look good. 
2. The paper is easy to understand. 
3. The idea of integrating skeleton sequence to refine temporal feature seems reasonable.
limitations: limitations: 1. Novelty & Missing important reference. The key idea of using disentangled information of skeleton to guide the HPS estimation is quite similar to [1]. But the paper didn't mention this. Please cite and clearly tell the difference, most importantly the technical novelty. Except for the different architecture, what is the key innovation from a research view?
[1] Human Mesh Recovery From Monocular Images via a Skeleton-Disentangled Representation. ICCV, 2019. 

2.Unfair comparisons. During evaluation, the image feature extraction part and 3D pose estimation part are only trained with similar 3D pose dataset, which seems to be fair. But the 2D pose estimation part is trained using an additional dataset, which makes the comparison less fair.