Toward Approaches to Scalability in 3D Human Pose Estimation

Authors: jun-hee  kim
seong-whan  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:08:44

Abstract: In the field of 3D Human Pose Estimation (HPE), scalability and generalization across diverse real-world scenarios remain significant challenges. This paper addresses two key bottlenecks to scalability: limited data diversity caused by 'popularity bias' and increased 'one-to-many' depth ambiguity arising from greater pose diversity. We introduce the Biomechanical Pose Generator (BPG), which leverages biomechanical principles, specifically the normal range of motion, to autonomously generate a wide array of plausible 3D poses without relying on a source dataset, thus overcoming the restrictions of popularity bias. To address depth ambiguity, we propose the Binary Depth Coordinates (BDC), which simplifies depth estimation into a binary classification of joint positions (front or back). This method decomposes a 3D pose into three core elements—2D pose, bone length, and binary depth decision—substantially reducing depth ambiguity and enhancing model robustness and accuracy, particularly in complex poses. Our results demonstrate that these approaches increase the diversity and volume of pose data while consistently achieving performance gains, even amid the complexities introduced by increased pose diversity.

Main paper: 
 introduction: 
 3D Human Pose Estimation (HPE) is essential for determining the 3D locations of human joints in images or videos. It plays a pivotal role in applications such as person re-identification [1] , action recognition [2, 3, 4] , human mesh recovery [5, 6] , and virtual reality [7, 8] . As the importance of this technology grows, there is an increasing demand for more accurate and universally applicable models. Addressing the key challenges that limit the scalability of this field is therefore crucial.In this paper, we identify two primary bottlenecks impeding the scalability of 3D HPE:1. Difficulty in Collecting Diverse Datasets: Annotated 3D datasets for 3D HPE are typically collected in controlled indoor environments with limited motions performed by a few individuals. This results in a 'popularity bias' [9] , where the data primarily reflects a small subset of human activities and demographics, thus restricting the diversity needed for effective model performance in various real-world scenarios. This lack of diversity means that models trained on such data may fail to generalize to unseen poses and environments. 2. Exacerbation of the 'One-to-Many' Depth Ambiguity Problem: Depth ambiguity [10, 11, 12] arises when a single 2D pose can correspond to multiple 3D interpretations. Increased pose diversity in 3D HPE intensifies this issue, complicating the depth estimation process. This ambiguity significantly impacts the accuracy of 3D pose estimation, as the model must choose from several plausible 3D configurations for a given 2D input, often leading to incorrect depth predictions.Figure 1 : Spherical coordinate distribution for the vector connecting the right shoulder to the right elbow across datasets: H36M [20] , 3DHP [21] , 3DPW [22] , and BPG, with valid regions highlighted in grey (taken from [23] ).To address the first bottleneck, pose augmentation methods for 3D HPE [13, 14, 15, 16] have been proposed. These methods leverage knowledge of labeled (source) data to enhance the generalization capability of pose estimators on unseen data. However, relying solely on source data does not fully overcome the inherent limitations, as the derived knowledge also carries the same 'popularity bias,' limiting its utility outside laboratory environments. Therefore, previous methods cannot guarantee the generation of out-of-source poses that are crucial for effective real-world applications.To overcome these limitations, we propose a Biomechanical Pose Generator (BPG) that does not rely on source data but utilizes biomechanical knowledge. The concept of 'Normal Range of Motion (NROM)' [17] is commonly used in the medical field to describe the standard limits within which a joint can move without discomfort or injury. Our method applies the NROM constraint to the forward kinematic function [18] , enabling the autonomous generation of a wide range of plausible 3D poses. This approach enhances the diversity and reliability of pose data.Furthermore, recognizing that adult human proportions are generally consistent [19] , we allow slight variations in proportions to accommodate a broader range of body types. Consequently, BPG leverages biomechanical knowledge to generate a broader spectrum of feasible 3D poses, thereby avoiding the popularity bias inherent in existing datasets, as demonstrated in Figure 1 . This methodology provides the essential flexibility and adaptability required for accurately modeling and predicting the diverse range of human poses and activities in real-world environments.Figure 2 : The MPJPE (in mm) across 3D HPE models against the depth ambiguity ratio is detailed in Sec.3.2. The baseline* employs BDC on the baseline model [24] , and H36M* combines source data with additional data synthesized by BPG, expanding the total dataset to ten times the H36M training size.To mitigate the one-to-many problem in 3D HPE, we developed the Binary Depth Coordinates (BDC) that reconceptualizes depth estimation as a binary classification task. Instead of using traditional continuous depth coordinates, we limit each joint's depth to two options: front or back. This simplification is based on geometric principles; once the length of connected bones is determined, two possible depth positions (front or back) can be calculated based on the joint positions. This approach allows us to decompose each 3D pose into three main components: 2D pose, bone length, and binary depth decision. This decomposition effectively reduces depth ambiguity and significantly enhances the model's accuracy and robustness for complex poses.As demonstrated in Figure 2 , poses with significant depth ambiguity have shown poorer performance in the baseline model, with increased data volumes exacerbating this issue. However, when employing the BDC, performance remains consistent regardless of ambiguity, and notably, poses with greater depth ambiguity exhibit improved performance as the amount of data increases.This paper aims to advance the development of scalable solutions in 3D HPE by addressing these issues, ultimately contributing to the creation of more accurate and adaptable technologies. By combining our data generation method with a new approach to resolving depth ambiguity, we aim to establish a foundation for future research and applications in this dynamic field. 
biomechanical pose generator: 
 To address the limitations of current data collection methods, we propose the Biomechanical Pose Generator (BPG) that does not rely on source data but utilizes biomechanical knowledge. The concept of 'Normal Range of Motion (NROM)' is commonly used in the medical field to describe the standard limits within which a joint can move without discomfort or injury. Our method applies the NROM constraint to the forward kinematic function, enabling the autonomous generation of a wide range of plausible 3D poses. This approach enhances the diversity and reliability of pose data. Recognizing that adult human proportions are generally consistent, we allow slight variations in proportions to accommodate a broader range of body types. Consequently, BPG leverages biomechanical knowledge to generate a broader spectrum of feasible 3D poses, thereby avoiding the popularity bias inherent in existing datasets. This methodology provides the essential flexibility and adaptability required for accurately modeling and predicting the diverse range of human poses and activities in real-world environments.Figure 3 : Osteo-kinematic Model with Joints and Degrees of Freedom.Osteo-kinematic Model. The human body performs complex motions constrained by biomechanical principles. To capture these constraints, we utilize an osteo-kinematic model similar to [18] for generating realistic 3D human poses. This model is based on the biomechanical structure of the human body, including bones, joints, and their respective degrees of freedom (DOF). In the osteo-kinematic model, bones can rotate at connected joints, and each joint is assigned a specific number of DOF to represent natural human-like movements. For instance, elbows and knees each have one degree of rotational freedom (1-DOF), while shoulders, hips, neck, and pelvis each have three degrees of rotational freedom (3-DOF). Unlike typical osteo-kinematic models that use fixed bone lengths, our model allows for adjustments in human body proportions to generate poses for a diverse range of subjects. The model includes 17 joints, 16 bone lengths, and 25 angle parameters. This detailed configuration ensures that the generated poses adhere to the natural movement constraints of the human body while allowing for variability in body types. Figure 3 illustrates the joints and degrees of freedom in the osteo-kinematic model.Postural Noise. The osteo-kinematic model uses 41 parameters (lengths and angles) to create a pose. Unconstrained random sampling of parameters may generate unnatural 3D poses, such as a backward-bent arm or leg, which do not accurately represent natural human postures. To address this, we set boundaries using two predetermined human features: a NROM [17] and human limb bone length ratios [19] . The range of motion (ROM) refers to the extent to which bones can move through a complete spectrum of movements. Although each individual's ROM varies, each joint possesses a NROM representing healthy human movement. NROM encompasses three different axes of rotation, considering the movement direction of the joint. By aligning these axes with the DOF of the osteo-kinematic model, we define an interval [θ lower i , θ upper i ] for each i th angle parameter. Hence, these intervals allow a self-created pose to enable natural human motion. Different people exhibit distinct bone lengths. However, the proportions of the human body remain relatively consistency for most individuals. This constancy permits us to set a mean value for human limb length ratios lj for the j th bone. To obtain 3D human data with various ratios, we establish a ratio range [(1-α) lj , (1+α) lj ] for each j th bone parameter, where α ∈ (0, 1) is a predetermined parameter. Using these intervals, we randomly obtain the bone parameter set l = {l b } B j=1 and the angle parameter set θ = {θ i } P i=1 , where B is the number of bones, P is the number of angle parameters, l j ∼ U [(1-α) * lj ,(1+α) * lj ] represents the bone parameter of the j th bone, and θ i ∼ U [θ lower i ,θ upper i ] represents the i th angle parameter. Finally, we define these two sets as the postural noise N pose = {l, θ}, which can be acquired randomly within boundaries representing the characteristics of ordinary individuals. Forward Kinematic Function. Following [25] , we use the forward kinematic function T to map postural noise to joint positions:EQUATIONwhere P 3D ∈ R J×3 represents the synthesized 3D pose (see Appendix C.1). Each joint undergoes a local coordinate transformation based on its parameters. The global coordinates of each joint are calculated by multiplying a series of rigid transformation matrices obtained via the parameters from the root point to itself.Pose Confidence. Although our method utilizes the NROM of individual joints to create realistic human poses, the relative positions of all bones must be considered to ensure the plausibility of the pose. For instance, disregarding the bone position can result in a physical intersection, such as legs crossing or hands intersecting with the torso. For this reason, we calculate the bone distance d n,m between the n th and m th bones. We use the indicator function of the interval [ dn,m , ∞), where dn,m is the pre-calculated relative bone radius [19] . The pose confidence C for P 3D is given byEQUATIONIf any bones get abnormally close, the confidence value becomes 0; otherwise, it remains 1. This measure allows us to avoid poses that are anatomically possible but physically impossible. We eliminate poses with a value of 0 using the pose confidence C and only use poses with a value of 1.Projection. We project the generated 3D poses into 2D poses using camera parameters, including a rotation matrix, translation vector, focal length, and principal point. Without predetermined camera settings, we generate these parameters within our framework. We sample a rotation matrix R ∈ R 3×3 from a uniform distribution on all 3D rotation matrices and the translation vector T ∈ R 3 from a uniform distribution on the interval [T lower , T upper ] that sets a camera location to avoid abnormally close or far positions. Although other methods use a limited rotation distribution, we use a comprehensive distribution to fairly represent all potential camera settings. For simplicity, we decide the focal length to be one. The principal point is fixed at zero, as the translation vector and the principal point serve similar functions. Using the camera parameters c = R, T , we define the projection function P that projects the self-created pose P 3D to a 2D pose P 2D ∈ R J×2 following Drover et al. [26] as:EQUATIONThe projected 2D pose, along with the original 3D pose, constitutes the final output of our generator. Since this pose is generated solely using biomechanical constraints, this approach ensures the reliability and diversity of the pose and produces data free from popularity bias. 
binary depth coordinates: 
 Our goal is to mitigate the depth ambiguity that arises when lifting a 2D pose to 3D. Typical human pose estimators predict poses in the Camera Space Coordinates (CSC) system, which neglects perspective, often resulting in substantial projection errors. An alternative, the Normalized Device Coordinates (NDC), remaps the viewing frustum to a cube but still struggles with the inherent depth ambiguity in continuous space. The distinction between potential poses in continuous space is not merely a matter of measurement but one of understanding the multidimensional nature of human motion, where multiple poses can be correct in different contexts. For instance, a hand extended forwards towards the camera or backwards away from it could represent equally plausible positions in a given pose. However, the variance between such positions in continuous space is significant, introducing a level of complexity that can obfuscate the learning process, as the model must recognize and account for fundamentally different yet anatomically feasible poses that share similar 2D projections.To address this, we introduce the Binary Depth Coordinates (BDC), a novel framework that accommodates the binary nature of depth in images and the continuity of real-world space. The BDC encodes a joint's position using a tuple (x 2D i , y 2D i , l i , s i , P i ), where x 2D i and x 2D i specify the i th joint's location in the image plane. The additional elements l i , representing the bone length, and s i , a binary depth parameter indicating depth relative to the plane of the image, provide the necessary context for a mapping function to estimate the corresponding 3D coordinates (x 3D i , y 3D i , z 3D i ) within the CSC. The parameter P i = [P x i , P y i , P z i ] stores the 3D coordinates of the preceding joint in the skeletal tree, helping to maintain the anatomical structure during transformations. In the BDC, the 3D depth z 3D is computed by solving a quadratic equation derived from the geometric constraints of Figure 4 : Our estimation process using BDC begins by using a fully connected layer to generate the bone lengths l, refined 2D joint, trajectoriy z 3D 0 , and the binary depth parameters s from the input 2D joint positions. These bone lengths and 2D trajectories are then processed through the quadratic formula to determine possible depth values (z 3D+ ,z 3D-). The final 3D pose in the camera coordinate system is obtained by combining the binary depth parameter with the possible depth values, resulting in the accurate reconstruction of the 3D skeleton. the joint's location in space. The equation is formulated as follows:EQUATIONwhere x 3D i and y 3D i are the coordinates in the CSC calculated by back-projecting a ray from the image plane using the camera parameters:EQUATIONHere, c x and c y denote the camera principal points, while f x and f y represent the focal lengths along the x and y axes, respectively. The solution to the quadratic equation provides two potential values for z 3D i as z 3D+ i and z 3D- i . These values reflect the dual nature of depth perception in perspective projection. To resolve the ambiguity between these two depths, the binary depth parameter s i is utilized. This parameter, derived from the BDC, indicates whether the joint is closer to or further from the camera relative to its connected joint. Specifically: When s i = 1, it is interpreted that the joint is closer to the camera, thus selecting z 3D+ i as the depth value. Conversely, when s i = -1, the system chooses z 3D- i , indicating that the joint is positioned further away from the camera. Our method precisely maps 2D poses to their accurate 3D configurations, leveraging observed and computed depth cues to resolve ambiguities and maintain anatomical consistency. The BDC integrates perspective projection through binary parameters and detailed bone length measurements, facilitating the accurate reconstruction of skeletal structure and overcoming depth ambiguity. By accurately calculating skeletal bone lengths and relationships, BDC enables the precise estimation of 3D poses, demonstrating a significant advancement over traditional coordinate systems that fall short in such complex scenarios.The BDC is designed to integrate seamlessly into existing 3D human pose estimation frameworks with minimal modifications. This compatibility is achieved by adapting BDC to the final layers of commonly used neural network architectures. By adjusting the dimensionality of the output layer to accommodate our parameters, BDC enables the prediction of 2D poses, bone lengths, and the binary depth parameter s i , thereby facilitating its integration into existing models. Additionally, predicting s i independently for each joint may fail to capture the complex interdependencies between different body parts adequately. To address this, we predict s i collectively for anatomically grouped segments-specifically, the left arm, right arm, right leg, left leg, and torso. For example, when predicting s i for the left arm, which typically involves three joints, there are 2 3 = 8 possible combinations of depth configurations. Thus, the model's task is to select one of these eight possible states. To achieve this, we utilize the Gumbel-Softmax, which allows differentiable sampling from discrete probability distributions. This approach enables the model to efficiently manage the prediction complexity, enhancing both the accuracy and anatomical consistency of the 3D pose reconstructions. Figure 4 illustrates the integration of the BDC into the existing lifting model.This section introduces a Depth Ambiguity Ratio (DAR) to quantitatively express the uncertainty in depth for each joint in a 3D pose. The DAR quantifies the uncertainty by normalizing the sum of the potential depth values, z 3D+ i and z 3D- i , obtained from the quadratic equation solving for depth. To ensure the ratio falls between 0 and 1, we normalize this sum by dividing it by twice the bone length, denoted as 2l i :EQUATIONThis normalized metric serves as a universal index to measure depth uncertainty, enhancing the analysis of 3D modeling accuracy and consistency across different implementations. 
related work: 
 3D human pose estimation. Initial approaches [27, 28, 29, 30, 31, 32, 33, 34, 35, 36] in 3D human pose estimation attempted to extract 3D poses directly from monocular images without leveraging 2D pose estimations. However, with significant progress in 2D pose estimation techniques [37, 38, 39] , the method of lifting 2D poses to 3D has emerged as a predominant strategy due to its effectiveness and scalability. Our work focuses on advancing these lifting-based frameworks [40, 24, 41, 42] , particularly in terms of scalability and generalization to diverse real-world scenarios.To enhance the lifting-based framework, several complementary approaches have been explored. Leveraging temporal information apporches [42, 24, 43] across multiple frames is a critical area in 3D HPE, as it can improve the accuracy and smoothness of 3D pose predictions. [24] utilized temporal convolutions over sequences of 2D keypoints to model temporal dependencies and reduce ambiguities inherent in single-frame predictions. Another effective strategy involves using rich image features. Approaches such as [41] incorporate spatial relationships between keypoints directly from image features, enhancing the pose estimation process. Addressing the inherent depth ambiguity in 3D pose estimation, multi-hypothesis approaches [12, 44, 45] generate multiple plausible 3D pose predictions and select the best fit. These methods use an evolutionary training data approach to generate multiple hypotheses for 3D poses from monocular images, thereby handling ambiguity and improving robustness.Cross-domain learning for 3D HPE. Generalizing to rare or unseen poses presents a significant challenge. Various strategies have been developed to address this by adapting training methods. For example, [28] introduced a method for optimal network architecture selection for different body parts, while [46] enhanced robustness by processing local regions of the 2D pose independently. [47] proposed inference stage optimization to extract distributional knowledge specific to target scenarios. Despite these advancements, they often struggle with the inherent limitations of training datasets, and self-supervised training frequently falls short in performance. To further enhance the generalization ability of 3D HPE, extensive research has focused on pose augmentation methods. Given the limitations of existing pose datasets and the challenges in collecting 3D pose data, generating additional 2D-3D pose pairs has proven beneficial. [28] suggested using body-part crossover and mutation operations to create new 3D poses, though constrained by dependence on the source dataset and fixed mutation ranges. [48] employed reinforcement learning for joint-angle rotation control without original datasets, but this self-supervised approach's performance is limited. [15] and [13] used GANs for pose augmentation, showing improved cross-dataset performance, primarily due to variations in camera viewpoints and human positions rather than increased pose diversity.[49] generated 2D-3D pose pairs from 2D pose labels of the target domain, closely matching its distribution. [16] introduced network designs like interpolation sub-net and body-part grouping net. Despite these advances, generated datasets often lack diversity, remain heavily dependent on the source dataset, and do not adequately consider the pose estimator's adaptability to the generated data, leading to potential training instability with complex poses.Our work focuses on pose augmentation for 3D HPE. By synthesizing novel 3D poses and integrating them with source poses, it increases pose diversity and mitigates depth ambiguity issues. 
experiments: 
 Our experiments were conducted on the Human3.6M (H36M) [20] , MPI-INF-3DHP (3DHP) [21] , and 3DPW [22] benchmarks. H36M [20] , a widely used benchmark for 3D human pose estimation, includes over 3.6 million indoor video frames captured from four angles with 7 subjects denoted as S1, S5, S6, S7, S8, S9, and S11. 3DHP [21] consists of 1.3 million images and human pose data captured from 8 subjects using a markerless motion capture system in a green screen studio with 14 cameras, with six indoor and outdoor sequences evaluated following standard protocols. 3DPW [22] dataset is a challenging collection of in-the-wild 2D and 3D data, uniquely captured using a moving camera, unlike other datasets. We evaluate pose accuracy on H36M and 3DPW using mean per joint position error (MPJPE) and procrustes aligned mean per joint position error (PMPJPE), and on 3DHP with MPJPE, percentage of correct keypoints (PCK) within 150mm, and area under the curve (AUC), following established protocols.In this section, we assess the generalizability of the data generated by our Biomechanical Pose Generator (BPG) in comparison to previous Pose Augmentation (PA) methods. Consistent with prior research and previous works [13, 16, 15] , we utilize the single-frame VPose [24] network to lift 2D poses to 3D. Our training sources include subjects S1, S5, S6, S7, and S8 from the H36M dataset, and we employ a 16-keypoint human model for evaluation. The volume of synthesized poses matches that of the source data. Source-dataset evaluation results. We conducted a series of experiments to assess the impact of integrating poses synthesized by the BPG with source data on the performance within the source dataset. For the evaluation, we utilized all available training data from the source dataset and employed HRnet [37] to generate predicted 2D keypoints as input, ensuring consistency with prior approaches. As depicted in Table 1 , our method achieves improvements in MPJPE by 3.3mm and in P-MPJPE by 2.5mm. Furthermore, it is evident that using only the BPG still results in performance improvements over traditional methods. These results suggest that our synthesized data can produce a more diverse set of poses compared to traditional pose augmentation techniques. Furthermore, we examined the feasibility of achieving high accuracy with a reduced amount of training data.For this evaluation, we utilized subsets of the source dataset, specifically S1 or S1 + S5, and used ground truth 2D keypoints as input, maintaining consistency with previous methods. As illustrated in Table 2 , our experiments demonstrate that leveraging medical knowledge to generate poses allows our method to significantly enhance performance over existing methods, even with less training data. Notably, the relative performance gains observed with reduced training data were substantially greater than those achieved with the full training dataset. Furthermore, it is evident that using only the BPG still results in performance improvements over traditional methods. This finding underscores the robustness of our approach, which maintains high accuracy and significantly outperforms conventional methods even when the training data is limited. Cross-dataset evaluation results. To evaluate whether the poses synthesized by the BPG contribute to improved generalization performance, we conducted experiments in a cross-dataset environment using the 3DPW and MPI-INF-3DHP datasets. For the evaluation, we utilized all available training data from the source dataset and employed ground truth 2D keypoints as input, consistent with previous methods. Our approach demonstrated performance improvements across both datasets. As shown in Table 3 , our method improved PCK by 2%, AUC by 7%, and MPJPE by 7.9mm. Notably, significant performance gains were observed in the 3DPW dataset, which features outdoor scenes with moving cameras. As indicated in Table 4 , our approach achieved improvements of 15.7mm in P-MPJPE. Furthermore, it is evident that using only the BPG still results in performance improvements over traditional methods. These results underscore the effectiveness of our synthesized data in enhancing generalization performance, particularly in challenging real-world environments.We evaluate the performance of our BDC by comparing poses across varying levels of depth ambiguity, using the proposed depth ambiguity ratio to quantify and analyze the results. In this experiment, we utilize multi-hypothesis model [45] for 3D HPE, which are designed to address depth ambiguity. As illustrated in Figure 2 , baseline model exhibit decreased performance on poses with high depth ambiguity. This trend is exacerbated when additional training data is introduced. However, when applying the BDC to existing models, performance remains consistent regardless of depth ambiguity. Furthermore, adding more data improves performance, even for poses with increased depth ambiguity. This observation highlights the robustness of the BDC in mitigating depth ambiguity problem and enhancing overall model accuracy. Scalability of integrating BDC and BPG. In this experiment, we analyze performance trends as the size of the training dataset increases using BPG. We utilize a multi-hypothesis model [45] for 3D HPE, designed to address depth ambiguity. By leveraging BPG, we expand the dataset size based on a fixed source dataset. As shown in Figure 5 , increasing the dataset size using BPG enhances cross-domain performance. However, for baseline models, performance on the source dataset declines as the dataset size increases, indicating potential overfitting and increased pose complexity. Conversely, integrating BDC with baseline models results in improved cross-domain performance and consistent performance gains on the source dataset as the dataset size increases. This highlights the effectiveness of BDC in enhancing model robustness and scalability, mitigating the challenges of depth ambiguity, and improving overall performance in both source and cross-domain evaluations. As shown in Table 6 , our approach consistently improved performance across all tasks in source dataset evaluations. The integration of BDC resulted in measurable improvements in key metrics, demonstrating the robustness and effectiveness of our method in enhancing model accuracy.Ablation study on BPG. In our ablation study, we evaluated various strategies within our framework, including normal range of motion (NROM), rotational restriction (RR), body ratio variation, and Pose Confidence (PC). Only data synthesized with the BPG was used as training data. The results presented in Table 5 highlight a consistent trend in performance enhancement as each strategy is successively incorporated. Specifically, NROM, as indicated by Variant A, has the most significant influence on performance. However, both Variants A and C indicate that introducing RR can somewhat diminish performance. The findings from Variants C and D emphasize the benefits of incorporating diverse pose proportions for better generalization. Finally, the outcome of Variant D and BPG warn of the pitfalls of including physically improbable postures without PC, suggesting that such postures can hinder effective learning. From these observations, it is clear that the confluence of all strategies is essential for achieving peak performance. Ablation study on BDC. In this ablation study, we compare the performance of various coordinate systems on a multi-hypothesis model [45] for 3D HPE, designed to address depth ambiguity. Specifically, we evaluate the Camera Space Coordinate (CSC), Normalized Device Coordinate (NDC), and BDC systems. The results, presented in Table 7 , highlight the effectiveness of the BDC system in improving model performance. The CSC and NDC systems serve as baselines for comparison. Additionally, we examine the impact of applying BDC without and with grouped segments (GS). As indicated, the BDC system, particularly when integrated with GS, demonstrates superior performance in reducing both MPJPE and P-MPJPE. This underscores the potential of BDC with GS in addressing depth ambiguity and enhancing the overall accuracy of 3D HPE models.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code can be found in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this study, we introduced the BDC and the BPG to address key challenges in 3D HPE, particularly the issues of dataset diversity and depth ambiguity. Our approach leverages biomechanical principles and a binary classification framework to generate a wide array of plausible 3D poses and simplify depth estimation, respectively.We demonstrated that integrating BDC and BPG into state-of-the-art 3D HPE models significantly improves accuracy and generalization across various datasets and tasks. Experiments on the H36M, 3DHP, and 3DPW datasets showed consistent performance enhancements, indicating that our synthesized data offers substantial benefits over traditional pose augmentation techniques, leading to more reliable 3D pose estimations in diverse scenarios.Limitations. Despite the promising results, our approach has certain limitations. The BPG generates poses based on single-frame data only, missing temporal dynamics crucial for motion analysis.Extending BPG to handle multi-frame data could generate more realistic poses. Additionally, while integrating BDC and BP enhances model performance, generating additional data beyond a certain point does not lead to further improvements, suggesting a saturation point. Future work should optimize the amount and type of synthetic data to maximize performance without unnecessary computational overhead. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies training and testing details, including data splits and evaluation metrics across different datasets and tasks.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA]Justification: The paper includes tables and figures that report performance metrics with appropriate statistical measures, such as MPJPE and P-MPJPE, though specific error bars are not mentioned .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The computer resources we used are provided in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper mentions the broader impacts, focusing on the improvements in 3D pose estimation which can benefit various applications such as virtual reality and action recognition.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: C.1 Forward Kinematics Function.T(N pose = {l, θ}) takes a joint variable vector θ = [θ 1 , θ 2 , . . . , θ n ] and a link length vector l = [l 1 , l 2 , . . . , l n ] as input, and returns a transformation matrix representing the position and attitude of the end effector. The mathematical representation of this isT(θ, l) = R(θ, l) d(θ, l) 0 ⊤ 1where R(θ, l) is a 3 × 3 rotation matrix, and d(θ, l) is a 3 × 1 position vector. This matrix represents the position and attitude of the end effector in the robot's reference frame.The forward kinematics function is defined as followsT(θ, l) = A 1 (θ 1 , l 1 )A 2 (θ 2 , l 2 ) • • • A n (θ n , l n )where A i (θ i , l i ) is the individual transformation matrix for each joint i. This transformation matrix represents the rotation and displacement of each joint, and typically has the form where l i is the length of link i, and d i and α i are the Denavit-Hartenberg (DH) parameters of the robot.A i (θ i , l i ) =    cos(θ i ) -sin(θ i ) 0 l i sin(θ i ) cos(α i ) cos(θ i ) cos(α i ) -sin(α i ) -d i sin(α i ) sin(θ i ) sin(α i ) cos(θ i ) sin(α i ) cos(α i ) d i cos(α i ) 0 0 0 0 1   So, the full formula for the forward kinematics function isT(θ, l) = n i=1 A i (θ i , l i )This formula represents sequentially multiplying the individual transformation matrices for each joint to compute the overall transformation matrix. This allows us to get the position and pose of the end effector.The forward kinematics function T(θ, l) takes a joint variable vector θ = [θ 1 , θ 2 , . . . , θ n ] and a link length vector l = [l 1 , l 2 , . . . , l n ] as input, and returns a transformation matrix representing the position and attitude of the end effector. The mathematical representation of this isT(θ, l) = R(θ, l) d(θ, l) 0 ⊤ 1where R(θ, l) is a 3 × 3 rotation matrix, and d(θ, l) is a 3 × 1 position vector. This matrix represents the position and attitude of the end effector in the robot's reference frame.The forward kinematics function is defined as followsT(θ, l) = A 1 (θ 1 , l 1 )A 2 (θ 2 , l 2 ) • • • A n (θ n , l n )where A i (θ i , l i ) is the individual transformation matrix for each joint i. This transformation matrix represents the rotation and displacement of each joint, and typically has the formA i (θ i , l i ) =    cos(θ i ) -sin(θ i ) 0 l i sin(θ i ) cos(α i ) cos(θ i ) cos(α i ) -sin(α i ) -d i sin(α i ) sin(θ i ) sin(α i ) cos(θ i ) sin(α i ) cos(α i ) d i cos(α i ) 0 0 0 0 1   where l i is the length of link i, and d i and α i are the Denavit-Hartenberg (DH) parameters of the robot.So, the full formula for the forward kinematics function isT(θ, l) = n i=1 A i (θ i , l i ) = P 3DThis formula represents sequentially multiplying the individual transformation matrices for each joint to compute the overall transformation matrix. This allows us to get the position and pose of the end effector.In summary, the forward kinematics function T maps the initial joint positions and link lengths (which constitute the kinematic model) to the 3D position and orientation of the end effector:Let X be a set, and let A be a subset of X. The indicator function 1 A (x) of a subset A of a set X is defined as:1 A (x) = 1 if x ∈ A 0 if x / ∈ A C.3 Camera Projection Function.The camera projection function P maps a 3D point X = [X, Y, Z, 1] ⊤ in homogeneous coordinates to a 2D point x = [u, v, 1] ⊤ on the image plane. The mathematical representation of this function isx = PXwhere P is the camera projection matrix, defined asP = K[R|T ]Here, K is the camera intrinsic matrix, R is the rotation matrix representing the camera orientation, and T is the translation vector representing the camera position. The intrinsic matrix K is given byK = f x 0 c x 0 f y c y 0 0 1where f x and f y are the focal lengths in the x and y directions, respectively, and c x and c y are the coordinates of the principal point.The full camera projection function is thus represented asx = K[R|t]XThis formula maps a 3D point X in the world coordinate system to a 2D point x on the image plane, taking into account the camera's intrinsic parameters and its pose in the world.C.4 Solving the quadratic equation for z 3D i .Given the equation:(x 3D i -P x i ) 2 + (y 3D i -P y i ) 2 + (z 3D i -P z i ) 2 = l 2 iand the transformations:x 3D i = x 2D i -c x f x • z 3D i , y 3D i = y 2D i -c y f y • z 3D iwe substitute these into the original equation:x 2D i -c x f x • z 3D i -P x i 2 + y 2D i -c y f y • z 3D i -P y i 2 + (z 3D i -P z i ) 2 = l 2 iExpanding and rearranging terms, we get a quadratic equation in z 3D i :A(z 3D i ) 2 + Bz 3D i + C = 0where:A = (x 2D i -c x ) 2 f 2 x + (y 2D i -c y ) 2 f 2 y + 1 B = -2 (x 2D i -c x )P x i f x + (y 2D i -c y )P y i f y + P z i C = (P x i ) 2 + (P y i ) 2 + (P z i ) 2 -l 2 iThe solutions for z 3D i are given by the quadratic formula:EQUATIONTherefore, the two roots are:z 3D+ i = -B + √ B 2 -4AC 2A z 3D- i = -B - √ B 2 -4AC 2A D Pseudo-Code Algorithm 1 Biomechanical Pose Generator (BPG)Algorithm 2 Binary Depth Coordinates (BDC) 1: Input: 2D joint positions P 2D , bone lengths l, binary depth parameter s, camera parameters (c x , c y , f x , f y ), root depth z 3D 0 , Skeleton Tree T 2: Output: 3D joint positions P 3D 3: Initialize P 3D ← {} 4: Compute the 3D coordinates of the root joint:5: x 3D 0 ← x 2D 0 -cx fx • z 3D 0 6: y 3D 0 ← y 2D 0 -cy fy• z 3D 0 7: Append (x 3D 0 , y 3D 0 , z 3D 0 ) to P 3D 8: for each joint i in tree order T do 9:Compute the 3D coordinates x 3D i , y 3D i using:EQUATIONEQUATIONSolve the quadratic equation for z 3D i :(x 3D i -P x,parent(i) ) 2 + (y 3D i -P y,parent(i) ) 2 + (z 3D i -P z,parent(i) ) 2 = l 2 i (9)11:This yields two potential values: z 3D+ i and z 3D- Use the binary depth parameter s i to select the correct depth:EQUATIONCompute x 3D i , y 3D i with the selected z 3D i :EQUATION)EQUATIONAppend (x 3D i , y 3D i , z 3D i ) to P 3D 15: end for 16: return P 3D Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: There is no indication that the research violates the NeurIPS Code of Ethics.The methodologies used align with ethical guidelines for research.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: All reviewers agreed to accept the paper. The paper originally got BA, BR, BR, and WA, and, after the rebuttal period, two reviewers raised the ratings, leading to final ratings of BA, BA, BA, and WA. 

Reviewers recognized the major contribution of this paper in leveraging biomechanical prior for synthesizing physically plausible human samples, in the framework for  3D human pose estimation. While reviewers were concerned about the similarity of the proposed one with existing approaches, after the authors' rebuttal and discussion period, most reviewers’  concerns have been resolved, and all reviewers agreed that the paper's shape is above the bar for NeurIPS. 

The AC also supports accepting the paper and strongly suggests the authors to follow the reviewers’ feedback in the camera-ready version. In particular, the AC suggests the authors significantly improve the clarity and readability of the paper, trying to include better and higher quality figures to better support the proposed method.

Reviews: review: paper_summary: paper_summary: This paper address the task of 3D Human Pose Estimation from monocular RGB. The authors make two main contributions: The Biomechanical Pose Generator (BPG) and the Binary Depth Coordinates (BDC). BPG is a 3D human pose generator that leverages the "Normal Range of Motion" (NROM) that is used in the medical field to describe standard biomechanical limitations. With it, BPG is capable of generating biomechanically sound 3D human poses by randomly sampling joint angles and bones that lie within a certain ratio to each other.
BDC is a coordinate system that decompose a 3D pose into constituents. Specifically, it decomposes it into the 2D coordinate, bone length, a binary depth parameter indicating the closeness to the image plane as well as the 3D coordinates of the parent joint. This decomposition, so the authors claim, allows models to better deal with depth ambiguity. 
Experimental results demonstrate that the proposed approach achieves better performance over the compared related work on a variety of datasets (cf. Tbl 1-4). Ablative studies demonstrate that BDC helps keep performance steady even in the face of larger depth ambiguity (Tbl. 5) and that related work can benefit as well from switching to the proposed coordinates (Tbl 6.)
strength_weakness: strength_weakness: - My biggest concern about the paper is that BDC is very similar conceptually to "Hand Pose Estimation via Latent 2.5D Heatmap
Regression", Iqbal et al., ECCV'18. Yet there is no mention of the paper, let alone any comparisons. The mentioned paper also addresses with depth ambiguity by decomposing the 3D pose into 2D pose and a root-relative depth vector. Addressing the differences, performing comparisons with this approach would better contextualize as well as strengthen the contribution of the paper.
- BPG shows to improve performance by improving the 2D to 3D lifting component. Yet, it's contribution is rather sparse, as it essentially amount to performing forward kinematics on bounded joint angle and bone lengths. It does not take into consideration statistics on poses. Certain poses are more common, due to them corresponding to actual human movement patterns (such as walking) that are affected by gravity. Randomly sampling poses without taking such statistics into consideration may generate a range of synthetic poses that are unrealistic, leading to non-optimal improvements.
questions: questions: - How would BPG compare to randomly sampling SMPL poses?
limitations: limitations: The authors address limitations of their methods, such as not taking temporal dynamics into account.
paper_summary: paper_summary: The authors propose a 3D human pose estimation framework that incorporates data augmentation and depth ordering information. The main contributions are two-fold: First, the proposed Biomechanical Pose Generator (BPG) generates plausible body poses based on kinematic constraints, which is used for data augmentation. Second, the Binary Depth Coordinates (BDC) disambiguate the projective depth of each joint by classifying whether the joints are positioned towards or away from the camera. The proposed framework achieved state-of-the-art performance in single-frame 3D human pose estimation settings.
strength_weakness: strength_weakness: My major concern lies on the novelty of the contribution.

- There are numerous research papers that regularize 3D human pose based on kinematic constraints. The authors did not clarify the distinctiveness of BPG from these conventional works, except for stating that BPG achieved better performance. An analysis showing how the proposed BPG generates more plausible poses compared to previous augmentation methods is required, either by displaying the generated poses or by showing qualitative estimation results.
- The concept of BDC is similar to [1] which learns ordinal depth information. The authors should cite the paper and discuss the difference.

The paper also contains	ambiguously explained parts or lacks details about their methods. Please refer to Questions section.

[1] G. Pavlakos et al., "Ordinal depth supervision for 3d human pose estimation", CVPR 2018
questions: questions: Method
- In line 155, the focal length of the camera matrix is set to 1 for BPG, is it also the case for the datasets used? Or the camera matrix provided in the datasets are used?
- In line 179, what is the meaning of "depth relative to the plane of the image". I guess $s_i$ is the depth relative to the preceding joint not the image plane.

Experiments
- Why did the authors use different baseline architectures in Sec. 5.1 and 5.2?
- How much portion of augmented data from BPG used for experiments in Sec. 5.1?
- Given that using only BPG increases the error in Fig. 4 left, how could it be possible to achieve better performance in Table 1 and 2 when only BPG is used?
- Why didn't the authors use BPG in Table 6?
- What is the difference between Variant E and BPG in Table 8?
limitations: limitations: Suggestions
- In Fig. 4, it should be clearly stated what * means. It would be better to use GFpose+BDC instead of GFpose*.
- More detailed explanation about how $T$ in Eq (1) and ${d}_{m,n}$ in Eq (2) are formulated would clarify the methods.

Typos
- Line 185, by the projection from -> by back-projecting a ray from
- Line 212, duplicated sentences.
- Line 115, to peed -> of
paper_summary: paper_summary: This paper introduces two components aimed at addressing challenges in 3D human pose estimation, specifically in terms of scalability and generalization. The authors propose a Biomechanical Pose Generator (BPG), which incorporates biomechanical principles to generate plausible new 3D poses. They also introduce Binary Depth Coordinates (BDC), a component designed to mitigate the depth ambiguity encountered when lifting a 2D pose to 3D. The paper includes ablation studies to demonstrate the impact of each component, and compares these new approaches to existing pose augmentation methods.
strength_weakness: strength_weakness: 1- The paper is generally well-written, but some parts could be clearer. Including a figure to illustrate the entire system could significantly help reader comprehension. For example, a diagram showing the VPose (or any baseline) architecture and the integration of the BDC component might be more effective than a text-only description. Additionally, including some implementation details about the BDC component in the main paper could improve the flow of information.

2- There are some ambiguities in the experiment section that need clarification. When referring to the “source-dataset”, it would be helpful to specify whether this refers to the Human 3.6M dataset or the newly synthesized poses. Similarly, when discussing evaluations on 3DHP and 3DPW, it would be beneficial to mention the specific subset used, such as the test set.

3- There appears to be some confusion between Table 1 and the results in Figure 4 (left). While Table 1 shows improvements in the Human 3.6M results when adding new poses generated from BPG, Figure 4 (left) indicates that adding more data increases the MPJPE error (without integrating BDC). This seems contradictory and could benefit from further explanation.

4- Typos: There are a couple of typographical errors that need correction. On Line 177, (xi) is repeated twice instead of yi. On Line 287, BDC should be corrected to BPG.
questions: questions: As I mentioned in the weaknesses section, I would like to learn more about the effect of adding more synthesized poses to Human 3.6M and evaluating on the same source of data as currently the results in the Table 1 and Figure 4 are a bit confusing to me.
limitations: limitations: Yes
paper_summary: paper_summary: Existing data in 3D human pose estimation are typically collected indoors with human actors. To address this scalability issue, the authors propose to synthesize 3D human pose data via an Osteo-kinematic model and introduce biochemical constraints for better physical plausibility. Additionally, to deal with the inherent ambiguity in single-view depth estimation, the authors introduce Binary Depth Coordinates to explicitly model the relative spatial relation between adjacent joints. Extensive experiments verify the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. $\textbf{Repeated text}$: The first paragraph of Sec.2 appears to be a copy-paste from the abstract, which is highly discouraged.
2. $\textbf{Requirement of camera intrinsics}$: While BDC shows notable performance gains to baselines, solving depth requires camera intrinsics (principal point and focal length), typically not required by current 3D HPE methods. This requirement may introduce additional constraints for in-the-wild inference.
questions: questions: 1. In Fig. 2 and Fig.4, adding synthesized data consistently decreased performance for some baseline methods, e.g., GFpose. This seems counterintuitive to me. As the authors mentioned, overfitting might be a reason; do the authors have any other insights regarding this? Does this phenomenon indicate there is still a gap between the real data and synthesized data?
limitations: limitations: The authors have included the limitations in Sec.6. However, the requirement for camera intrinsics may also be considered a limitation and discussed.

Comments: comment: Dear Reviewer 1aXZ,

Thank you very much for your thorough review and valuable feedback on our submission. We have carefully considered your comments and have submitted a detailed response addressing the points you raised.

As the discussion period is nearing its end, we would greatly appreciate it if you could review our response soon and confirm whether it addresses your concerns. If you have any further questions or need additional clarification, please feel free to let us know.

Thank you again for your time and consideration.
I sincerely appreciate the authors' thorough response to my concerns and questions. I acknowledge the novelty of the method in generating samples from the distribution of possible 3D poses rather than from the dataset distribution. The response clarified the methods and experiments for me. Based on this, I am raising my score to borderline accept.
Thanks for the clarifications. I still believe that while the idea of the paper is novel in some aspects, the paper contains some ambiguities in several parts and it needs revision to be ready for a good publication. Given that and also according to other concerns raised by other reviewers, I'd rather keep my original rating.
I would like to thank the authors' efforts to prepare the response. Most of my concerns were addressed.