Title: Toward Approaches to Scalability in 3D Human Pose Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Human Pose Estimation

Data generation

Pose Decompression
Abstract: In the field of 3D Human Pose Estimation (HPE), scalability and generalization across diverse real-world scenarios remain significant challenges. This paper addresses two key bottlenecks to scalability: limited data diversity caused by 'popularity bias' and increased 'one-to-many' depth ambiguity arising from greater pose diversity. We introduce the Biomechanical Pose Generator (BPG), which leverages biomechanical principles, specifically the normal range of motion, to autonomously generate a wide array of plausible 3D poses without relying on a source dataset, thus overcoming the restrictions of popularity bias. To address depth ambiguity, we propose the Binary Depth Coordinates (BDC), which simplifies depth estimation into a binary classification of joint positions (front or back). This method decomposes a 3D pose into three core elements—2D pose, bone length, and binary depth decision—substantially reducing depth ambiguity and enhancing model robustness and accuracy, particularly in complex poses. Our results demonstrate that these approaches increase the diversity and volume of pose data while consistently achieving performance gains, even amid the complexities introduced by increased pose diversity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting data in 3D human pose estimation are typically collected indoors with human actors. To address this scalability issue, the authors propose to synthesize 3D human pose data via an Osteo-kinematic model and introduce biochemical constraints for better physical plausibility. Additionally, to deal with the inherent ambiguity in single-view depth estimation, the authors introduce Binary Depth Coordinates to explicitly model the relative spatial relation between adjacent joints. Extensive experiments verify the effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. Leveraging biomechanical prior knowledge to synthesize physically plausible human data is $\\textbf{intuitive}$ and $\\textbf{interesting}$.\n2. Comprehensive experiments verify the effectiveness of the proposed data augmentation approach (BPG) and Binary Depth Coordinates (BDC). Specifically, BDC can be applied to different methods, e.g., image-based and lifting-based, showing superior generalization ability.\n\nWEAKNESSES:\n1. $\\textbf{Repeated text}$: The first paragraph of Sec.2 appears to be a copy-paste from the abstract, which is highly discouraged.\n2. $\\textbf{Requirement of camera intrinsics}$: While BDC shows notable performance gains to baselines, solving depth requires camera intrinsics (principal point and focal length), typically not required by current 3D HPE methods. This requirement may introduce additional constraints for in-the-wild inference.\n\nQUESTIONS:\n1. In Fig. 2 and Fig.4, adding synthesized data consistently decreased performance for some baseline methods, e.g., GFpose. This seems counterintuitive to me. As the authors mentioned, overfitting might be a reason; do the authors have any other insights regarding this? Does this phenomenon indicate there is still a gap between the real data and synthesized data?\n\nLIMITATIONS:\nThe authors have included the limitations in Sec.6. However, the requirement for camera intrinsics may also be considered a limitation and discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces two components aimed at addressing challenges in 3D human pose estimation, specifically in terms of scalability and generalization. The authors propose a Biomechanical Pose Generator (BPG), which incorporates biomechanical principles to generate plausible new 3D poses. They also introduce Binary Depth Coordinates (BDC), a component designed to mitigate the depth ambiguity encountered when lifting a 2D pose to 3D. The paper includes ablation studies to demonstrate the impact of each component, and compares these new approaches to existing pose augmentation methods.\n\nSTRENGTHS:\nThe paper’s focus on addressing the challenge of limited datasets and enhancing the generalizability of the method is interesting and to the best of my knowledge the idea of biomechanical pose generator which does not rely on a source dataset is novel. Also, the authors’ attention to the depth ambiguity in 3D pose estimation from a single image adds a value to the field. The authors have conducted comprehensive experiments and ablation studies, which provide valuable insights into the effectiveness of the proposed components. The inclusion of cross-dataset evaluation is crucial, as it allows for a robust assessment of the Biomechanical Pose Generator (BPG) component’s effectiveness.\n\nWEAKNESSES:\n1- The paper is generally well-written, but some parts could be clearer. Including a figure to illustrate the entire system could significantly help reader comprehension. For example, a diagram showing the VPose (or any baseline) architecture and the integration of the BDC component might be more effective than a text-only description. Additionally, including some implementation details about the BDC component in the main paper could improve the flow of information.\n\n2- There are some ambiguities in the experiment section that need clarification. When referring to the “source-dataset”, it would be helpful to specify whether this refers to the Human 3.6M dataset or the newly synthesized poses. Similarly, when discussing evaluations on 3DHP and 3DPW, it would be beneficial to mention the specific subset used, such as the test set.\n\n3- There appears to be some confusion between Table 1 and the results in Figure 4 (left). While Table 1 shows improvements in the Human 3.6M results when adding new poses generated from BPG, Figure 4 (left) indicates that adding more data increases the MPJPE error (without integrating BDC). This seems contradictory and could benefit from further explanation.\n\n4- Typos: There are a couple of typographical errors that need correction. On Line 177, (xi) is repeated twice instead of yi. On Line 287, BDC should be corrected to BPG.\n\nQUESTIONS:\nAs I mentioned in the weaknesses section, I would like to learn more about the effect of adding more synthesized poses to Human 3.6M and evaluating on the same source of data as currently the results in the Table 1 and Figure 4 are a bit confusing to me.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a 3D human pose estimation framework that incorporates data augmentation and depth ordering information. The main contributions are two-fold: First, the proposed Biomechanical Pose Generator (BPG) generates plausible body poses based on kinematic constraints, which is used for data augmentation. Second, the Binary Depth Coordinates (BDC) disambiguate the projective depth of each joint by classifying whether the joints are positioned towards or away from the camera. The proposed framework achieved state-of-the-art performance in single-frame 3D human pose estimation settings.\n\nSTRENGTHS:\n- The proposed method achieves state-of-the-art results in various 3D HPE datasets.\n- The effect of data augmentation is validated in cross-domain learning settings.\n\nWEAKNESSES:\nMy major concern lies on the novelty of the contribution.\n\n- There are numerous research papers that regularize 3D human pose based on kinematic constraints. The authors did not clarify the distinctiveness of BPG from these conventional works, except for stating that BPG achieved better performance. An analysis showing how the proposed BPG generates more plausible poses compared to previous augmentation methods is required, either by displaying the generated poses or by showing qualitative estimation results.\n- The concept of BDC is similar to [1] which learns ordinal depth information. The authors should cite the paper and discuss the difference.\n\nThe paper also contains\tambiguously explained parts or lacks details about their methods. Please refer to Questions section.\n\n[1] G. Pavlakos et al., \"Ordinal depth supervision for 3d human pose estimation\", CVPR 2018\n\nQUESTIONS:\nMethod\n- In line 155, the focal length of the camera matrix is set to 1 for BPG, is it also the case for the datasets used? Or the camera matrix provided in the datasets are used?\n- In line 179, what is the meaning of \"depth relative to the plane of the image\". I guess $s_i$ is the depth relative to the preceding joint not the image plane.\n\nExperiments\n- Why did the authors use different baseline architectures in Sec. 5.1 and 5.2?\n- How much portion of augmented data from BPG used for experiments in Sec. 5.1?\n- Given that using only BPG increases the error in Fig. 4 left, how could it be possible to achieve better performance in Table 1 and 2 when only BPG is used?\n- Why didn't the authors use BPG in Table 6?\n- What is the difference between Variant E and BPG in Table 8?\n\nLIMITATIONS:\nSuggestions\n- In Fig. 4, it should be clearly stated what * means. It would be better to use GFpose+BDC instead of GFpose*.\n- More detailed explanation about how $T$ in Eq (1) and ${d}_{m,n}$ in Eq (2) are formulated would clarify the methods.\n\nTypos\n- Line 185, by the projection from -> by back-projecting a ray from\n- Line 212, duplicated sentences.\n- Line 115, to peed -> of", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper address the task of 3D Human Pose Estimation from monocular RGB. The authors make two main contributions: The Biomechanical Pose Generator (BPG) and the Binary Depth Coordinates (BDC). BPG is a 3D human pose generator that leverages the \"Normal Range of Motion\" (NROM) that is used in the medical field to describe standard biomechanical limitations. With it, BPG is capable of generating biomechanically sound 3D human poses by randomly sampling joint angles and bones that lie within a certain ratio to each other.\nBDC is a coordinate system that decompose a 3D pose into constituents. Specifically, it decomposes it into the 2D coordinate, bone length, a binary depth parameter indicating the closeness to the image plane as well as the 3D coordinates of the parent joint. This decomposition, so the authors claim, allows models to better deal with depth ambiguity. \nExperimental results demonstrate that the proposed approach achieves better performance over the compared related work on a variety of datasets (cf. Tbl 1-4). Ablative studies demonstrate that BDC helps keep performance steady even in the face of larger depth ambiguity (Tbl. 5) and that related work can benefit as well from switching to the proposed coordinates (Tbl 6.)\n\nSTRENGTHS:\n- The authors properly motivate and evaluate their approach. Depth ambiguity in monocular RGB is a challenging problem to address. I particularly liked Tbl. 5 that demonstrated that BDC is capable of handling even larger depth ambiguities.\n- The paper was easy to digest and understand.\n- One of the main strength of this paper is that BDC can be combined with other related work, yielding improvements (Tbl. 6)\n\nWEAKNESSES:\n- My biggest concern about the paper is that BDC is very similar conceptually to \"Hand Pose Estimation via Latent 2.5D Heatmap\nRegression\", Iqbal et al., ECCV'18. Yet there is no mention of the paper, let alone any comparisons. The mentioned paper also addresses with depth ambiguity by decomposing the 3D pose into 2D pose and a root-relative depth vector. Addressing the differences, performing comparisons with this approach would better contextualize as well as strengthen the contribution of the paper.\n- BPG shows to improve performance by improving the 2D to 3D lifting component. Yet, it's contribution is rather sparse, as it essentially amount to performing forward kinematics on bounded joint angle and bone lengths. It does not take into consideration statistics on poses. Certain poses are more common, due to them corresponding to actual human movement patterns (such as walking) that are affected by gravity. Randomly sampling poses without taking such statistics into consideration may generate a range of synthetic poses that are unrealistic, leading to non-optimal improvements.\n\nQUESTIONS:\n- How would BPG compare to randomly sampling SMPL poses?\n\nLIMITATIONS:\nThe authors address limitations of their methods, such as not taking temporal dynamics into account.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}