Title: Dual-Diffusion for Binocular 3D Human Pose Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Human Pose Estimation

Binocular Vision

Diffusion Model

Pose Priors
Abstract: Binocular 3D human pose estimation (HPE), reconstructing a 3D pose from 2D poses of two views, offers practical advantages by combining multiview geometry with the convenience of a monocular setup. However, compared to a multiview setup, the reduction in the number of cameras increases uncertainty in 3D reconstruction. To address this issue, we leverage the diffusion model, which has shown success in monocular 3D HPE by recovering 3D poses from noisy data with high uncertainty. Yet, the uncertainty distribution of initial 3D poses remains unknown. Considering that 3D errors stem from 2D errors within geometric constraints, we recognize that the uncertainties of 3D and 2D are integrated in a binocular configuration, with the initial 2D uncertainty being well-defined. Based on this insight, we propose Dual-Diffusion specifically for Binocular 3D HPE, simultaneously denoising the uncertainties in 2D and 3D, and recovering plausible and accurate results. Additionally, we introduce Z-embedding as an additional condition for denoising and implement baseline-width-related pose normalization to enhance the model flexibility for various baseline settings. This is crucial as 3D error influence factors encompass depth and baseline width. Extensive experiments validate the effectiveness of our Dual-Diffusion in 2D refinement and 3D estimation. The code and models are available at https://github.com/sherrywan/Dual-Diffusion.

Dual-Diffusion for Binocular 3D Human Pose Estimation

Abstract

Binocular 3D human pose estimation (HPE), reconstructing a 3D pose from 2D poses of two views, offers practical advantages by combining multiview geometry with the convenience of a monocular setup. However, compared to a multiview setup, the reduction in the number of cameras increases uncertainty in 3D reconstruction. To address this issue, we leverage the diffusion model, which has shown success in monocular 3D HPE by recovering 3D poses from noisy data with high uncertainty. Yet, the uncertainty distribution of initial 3D poses remains unknown. Considering that 3D errors stem from 2D errors within geometric constraints, we recognize that the uncertainties of 3D and 2D are integrated in a binocular configuration, with the initial 2D uncertainty being well-defined. Based on this insight, we propose Dual-Diffusion specifically for Binocular 3D HPE, simultaneously denoising the uncertainties in 2D and 3D, and recovering plausible and accurate results. Additionally, we introduce Z-embedding as an additional condition for denoising and implement baseline-width-related pose normalization to enhance the model flexibility for various baseline settings. This is crucial as 3D error influence factors encompass depth and baseline width. Extensive experiments validate the effectiveness of our Dual-Diffusion in 2D refinement and 3D estimation. The code and models are available at https://github.com/sherrywan/Dual-Diffusion.

1 Introduciton

3D human pose estimation (HPE) aims to localize the 3D position of human joints, which has a broad range of downstream applications [49]. To date, monocular 3D HPE [25, 30, 20, 60, 22, 35, 59, 58] has received a great deal of attention due to its convenient for practical applications, while multiview (more than two cameras) 3D HPE [31, 14, 56, 5, 46] has earned popularity due to its absolute localization under geometric constraints. However, the pros and cons of these two setups are 'conjugat' to each other, with monocular suffering from depth ambiguity, while multiview is hindered by strict scene constraints. Binocular setup [45] offers both advantages, yet has long been ignored by the community. This motivates us to focus on the Binocular 3D HPE, which lifts to a 3D pose from binocular 2D poses.

image plane

optical center

increases

！

(a) 4 views

(b) 2 views

Observing the uncertainty range of a 3D point reconstructed under different numbers of cameras, shown in Fig. 1, it is evident that although binocular setups significantly reduce 3D reconstruction uncertainty compared to monocular configurations due to geometric constraints, they still have higher ambiguity compared to multiview setups. In previous works on monocular 3D HPE, to alleviate ambiguity, human pose priors such as joint angle limits [1] and physical plausibility [54] are commonly modeled. Nowadays, with advancements in probabilistic methods in machine learning, many works aim to leverage data to model the distribution of real human poses as a representation of pose prior, including V AE [23, 29, 48], GAN [47, 7], normalizing flow [19, 50], and diffusion models [8, 36, 16]. Among these, diffusion models stand out due to their advantages like indirect likelihood correlation, simple training, and network flexibility [12, 38]. Given the success of diffusion models in modeling pose priors, can they be cleverly leveraged to reduce uncertainty in binocular 3D HPE?

The diffusion model [11] comprises two processes: the forward diffusion process, which perturbs the real data to a diffused distribution; and the reverse denoising process, which denoises the noisy data to match the real distribution. In monocular 3D HPE, the diffusion model is employed to recover 3D poses from noisy data sampled from random noise [34, 12] or from an initial 3D pose distribution with high uncertainty [8]. The first category is excluded due to high time consumption, as it fails to leverage geometric constraints [9] to narrow down the search space. The second category starts from the initial 3D poses with higher uncertainty (estimated by off-the-shelf 3D HPE methods) and converges to the lower uncertainty distribution representative of more plausible and accurate 3D poses, which is more efficient. However, the main problem lies in the unknown distribution of the initial 3D uncertainty. The statistical method is used in [8] to solve it which is unrealistic in practice. Unlike monocular setups, 3D errors stem from errors in 2D estimation in binocular within a geometric framework. In other words, 3D uncertainty can be reconstructed from 2D cause they are intrinsically linked by geometric constraints. Therefore, we propose a diffusion-based method specifically for binocular 3D HPE, named Dual-Diffusion, capable of simultaneously denoising initial 2D and 3D uncertainties.

From a diffusion perspective, the uncertainty distribution in initial 2D joints is well-defined, typically modeled as a Gaussian distribution centered at the ground truth with a specified standard deviation [26, 51, 40]. However, the uncertainty in initial 3D poses remains unknown. Nevertheless, from a denoising standpoint, it is preferable to recover poses in 3D space [6, 61] rather than in 2D, given that human pose priors are inherently 3D. To bridge the gap between diffusion and denoising across different domains, we leverage geometric projection techniques to couple the 2D plane and 3D space. The specific modeling of Dual-Diffusion is illustrated in Fig. 2(a). In the forward diffusion process, noisy 2D binocular poses are generated from ground truth 2D poses by adding noise stepby-step. Subsequently, Triangulation [9] is used to reconstruct noisy 3D poses, thereby defining the uncertainty distribution in 3D. During the reverse denoising process, we sample noisy 3D poses and remove the noise to recover plausible and accurate 3D poses. Reprojection is utilized to estimate the corresponding 2D poses back, enabling simultaneous denoising in both 2D and 3D spaces.

Reviewing our Dual-Diffusion model, the essential objective of the denoiser network is to remove the noise of noisy 3D poses under different perturbed distributions. The noise-perturbed level is determined by the noise addition step t in diffusion and is reflected in the denoiser via timestamp embedding. While t is directly associated with 2D noise, 3D noise depends not only on 2D factors but also on 3D depth and the baseline width of the binocular setup. To enable flexible denoising of 3D noisy data under the same t but with varying uncertainties, we introduce 3D depth as an additional condition, named Z-embedding. Besides, the baseline-width-related normalization (BaseL-norm) is applied to 3D poses, which allows the model to flexibly adapt to different baseline width settings.

To validate the efficacy of our Dual-Diffusion model in denoising both 2D and 3D poses, we conducted experiments on the binocular H36M [13] and MHAD [27] datasets, utilizing only 2-view camera pairs. Our model outperforms the baseline Triangulation and other state-of-the-art methods. Additionally, we establish the 'random-noise' and '2D-Diff' models for comparison, demonstrating the effectiveness of starting from initial pose distributions and leveraging 3D pose priors.

Our contributions can be summarized as follows:

Dual-Diffusion Framework . We propose a novel framework, Dual-Diffusion, specifically designed for binocular 3D HPE. This model simultaneously removes noise from both 2D and 3D poses by leveraging geometric mapping.

Uncertainty Analysis and Denoiser Enhancement . We analyze the relationship between 3D and 2D uncertainties and introduce Z-embedding and BaseL-norm to enhance the flexibility of the denoiser.

Benchmark Performance . Our method achieves superior performance on the evaluated benchmarks, demonstrating effectiveness.

2 Related Work

Binocular and Multiview 3D HPE. Binocular 3D HPE aims to estimate 3D poses from singleframe 2D poses captured from two perspectives. This task has been sparsely studied. RSB-Pose [45] utilizes stereo volume features to enhance binocular coherence in 2D poses and employs a spatial Transformer for 3D pose refinement. While the Transformer excels at establishing correlations between nodes, its ability to effectively identify and denoise incorrect joints is uncertain. Given the geometric constraints shared in multiview setups, we review multiview 3D HPE methods, typically involving two stages: 1) estimating 2D poses from images, and 2) lifting these to 3D poses using geometric constraints, with Triangulation [9] being the most common approach. Existing methods can be categorized into two stages of improvement. The first category [32, 10, 33, 46] enhances 2D poses using 3D-aware features from multiview fusion. The second category [3, 31, 14, 5, 15] focuses on lifting 2D poses to 3D. Explicit pose priors, such as bone length, are incorporated into the lifting process using Pictorial Structure Models [3, 31] or closed-form Structural Triangulation [5]. Some methods [14, 15] treat lifting as a 3D regression task using volume representations, implicitly considering pose coherence. However, these approaches either rely on limited explicit pose priors or involve computationally intensive 3D convolutions. Actually, due to geometric constraints, the 3D poses reconstructed from initial 2D poses in multiview are generally reliable, with limited studies focusing on 2D-3D lifting. Recognizing that the primary challenge in binocular setups is increased 3D uncertainty, this work specifically focuses on the 2D-3D lifting process. Hence, we review monocular 3D HPE methods for additional insights.

Monocular 3D HPE. Monocular 3D HPE methods can be categorized into one-stage and two-stage approaches. One-stage methods [30, 43, 62, 42] directly regress the 3D pose from the image, relying on extensive datasets and complex network architectures. Two-stage [25, 18, 4] first estimate the 2D pose using a 2D detector, then lift it to 3D through deep network mapping. Considering the joints are connected within a skeleton, Graph Convolutional Networks (GCNs) [57, 2, 60] and Transformer architectures [35, 55, 53, 41, 59] are introduced to establish correlations between joints during the lifting process. However, despite establishing joint relationships, monocular methods still suffer from inherent depth ambiguity, which utilizes temporal consistency or poses priors to overcome.

Pose Priors in 3D HPE. Since our task involves single-frame pose estimation, we primarily review works focusing on modeling pose priors in monocular 3D HPE. Previous approaches explore explicit pose priors, such as joint angle limits [1], physical plausibility [54], or defined pose models [21]. However, explicit priors, despite their interpretability, may not be comprehensive. With advancements in image generation, probabilistic methods capable of modeling data distributions gain attention. VAE [23, 29, 48] encodes poses into a latent space following a normal distribution, then decodes them back to the original pose. However, inference through recurrently optimizing latent parameters can be time-consuming. Normalizing flows [19, 50] use invertible transformations to map latent features to 3D poses, aiding inference, but with complex network architectures. Generative adversarial networks (GANs) [47, 7] learn pose distributions by distinguishing fake and real poses, but face challenges in training. The diffusion models [11, 38, 39] gain popularity due to its network flexibility and training simplicity. It learns data distribution by iteratively removing noise which is incrementally added to real data until a fully diffused noise distribution is achieved. Several methods [6, 12, 36, 34, 61] condition the 3D pose distribution on estimated 2D points. However, directly applying this framework to binocular 3D HPE disregards geometric constraints, acquiring more diffusion and denoising steps. Another approach [8] treats initial 3D poses as containing high uncertainty and accurate 3D poses as maintaining low uncertainty, and employs the diffusion model to denoise noisy 3d pose from an initial high-uncertainty distribution to a more accurate result under a certain distribution. However, the initial distribution of 3D poses is unknown which is estimated statistically in [8]. Considering geometric constraints in binocular configuration, the initial 3D pose distribution can be derived from

……

……

……

……

……

……

……

……

～

～

～

～

𝑢

!

𝑢

"

𝑢

#

𝑦

!

𝑦

"

𝑦

#

2D pose initial

distribution

𝑈

#

2D pose real

distribution

𝑈

!

3D pose initial

distribution

𝑌

#

3D pose real

distribution

𝑌

!

Denoiser

𝑢

"

𝑦

"

Z

𝑧

"

embedding

Timestamp

𝑡

embedding

DDIM

generate

𝑢

"

K times

Triangulation

Reprojection

real diffuse

real denoise

virtual diffuse

virtual denoise

(a) Modeling

(b) Inference

Forward Diffusion

Reverse Denoising

𝑢

#

' 𝑦

!

' 𝑢

!

𝑦

#

BaseL-norm

BaseL-denorm

the initial 2D distribution which is more straightforward. Therefore, we propose the Dual-Diffusion model to denoise both 2D and 3D poses, specifically for binocular 3D HPE.

3 Method

We propose Dual-Diffusion to simultaneously optimize 2D poses and 3D poses, addressing the high uncertainty issue in binocular 3D HPE. The modeling framework is illustrated in Fig. 2(a). Starting with the initial 2D poses estimated by the off-the-shelf 2D detector, which are considered as the diffused data, we apply a diffusion model to denoise it. However, the denoising is applied to the 3D pose rather than the 2D pose, given the inherently 3D nature of pose priors. The geometric projection relationship is leveraged to bridge the gap and enable dual-denoising of 2D and 3D space. In the subsequent sections, we first briefly introduce the diffusion models and then describe our Dual-Diffusion model in detail.

3.1 Revisiting Diffusion Models

Diffusion models [11, 38] are a type of probabilistic method that can recover data that satisfy the underlying distribution p data ( x ) from noisy data. It comprises two processes: the forward diffusion process and the reverse denoising process. During the forward process, the real data x 0 is diffused by a Gaussian noise step-by-step over T steps under a Markov chain. The formulation can be written as:

where the variance schedule β t determines the perturbed level of the noisy data x t . Using the notation α t := 1 -β t and ¯ α t := ∏ T t =1 α t , x t can be sampled from x 0 skipping timestamps 1 : t -1 :

When T is large enough, x T ∼ N (0 , I ) can be satisfied. Hence, the reverse process starts at samples from N (0 , I ) , and the purpose is to recover x 0 ∼ p data ( x ) . According to ELBO [17], the target max logp θ ( x 0 ) can be simplified to minimize the KL divergence between the reverse conditional distribution p θ ( x t -1 | x t ) and the posterior of the diffusion q ( x t -1 | x t , x 0 ) which is formulated as:

where ˜ µ t is a linear combination of x t and x 0 . The reverse conditional distribution can be ensured in Gaussian form p θ ( x t -1 | x t ) ∼ N ( x t -1 ; µ θ ( x t , t ) , Σ θ ( x t , t ) if β t are small. Thus, the key to KL divergence is the L 2 distance between µ θ and ˜ µ t , which drives the denoiser network to learn µ θ to predict ˜ µ t . According to Eq. 2, the loss of the denoiser training is finally simplified to:

The essential of the denoiser is to predict the noise ϵ t at any perturbed level t added to the real data, and then can recover the x 0 .

3.2 Dual Diffusion

Starting from N (0 , I ) is highly time-consuming due to the high diffusion of noise. [8] suggests that the 3D pose y T predicted by the off-the-shelf methods is a kind of noisy data under the distribution with high uncertainty Y T , and diffusion models can be used to reduce uncertainty to generate the accurate result y 0 . Starting from Y T with limited diffusion is more efficient. The major problem is that the 3D pose initial uncertainty is unknown, which is solved by the statistical method in [8]. However, if the uncertainty of the 3D pose follows a Gaussian distribution is still confusing. And the statistical results can be easily influenced by the models and training dataset, posing challenges in practical applications. In this work, considering the binocular geometry framework, we design an elegant method to build uncertainty distribution of initial 3D poses from 2D uncertainty, and conversely, to achieve 2D denoising by denoising 3D poses, which we name Dual-Diffusion.

Forward Dual Diffusion process gradually adds noise to ground truth binocular 2D poses u 0 = { u 0 ,v } ∈ R J × 4 in a Markov chain. Here, J is the number of joints and v ∈ 0 , 1 indicates the left and the right perspectives. The diffusion domain is localized in 2D based on two reasonable assumptions: 1) the uncertainty distribution of initial 2D poses is known, and 2) it is in Gaussian form. We predict the initial 2D poses using a 2D detector. The training objective of this detector is to predict heatmaps of 2D joints whose supervision is a Gaussian distribution centered at the ground truth with a fixed standard deviation σ T . The maturity of 2D detectors ensures the quality of the heatmap generation. Therefore, the initially predicted binocular 2D poses can be treated as the noisy data following the Gaussian distribution N (0 , σ T 2 I ) around the ground truth 2D poses u 0 . This indicates that the diffusion target at T step should satisfy the distribution U T ∼ N ( u 0 , σ T 2 I ) . The formulation to create the diffused 2D poses can be written as:

Here, x 0 and x t in Eq. 2 are replaced by u 0 -u 0 and u t -u 0 correspondingly to maintain the distribution consistency of ϵ t . When t = T , u T ∼ U T is satisfied. The diffusion is transmitted to 3D space by Triangulation [9], which is a geometric tool to reconstruct 3D poses y ∈ R J × 3 from binocular 2D poses using camera parameters. The diffused 3D poses can be generated by:

where Tri represents the closed-form solver using Linear Triangulation. Up to now, the uncertainty distribution of initial 2D poses is determined. Although the initial uncertainty distribution Y T in 3D has not yet been formulated, it can be reconstructed by sampling from noisy 2D poses.

Reverse Dual Denoising should recover accurate binocular 2D poses u 0 from the estimated 2D poses u T according to the definition in diffusion models. However, we argue that 3D poses inherently exhibit a more constrained distribution compared to 2D poses. 2D poses may be various under different camera parameters and perspectives. Hence, we modify the denoising network to recover the original 3D poses y 0 . As above, the denoising is transmitted to the 2D plane by Reprojection, given the camera extrinsic and intrinsic matrix. Then the noise ϵ t added to 2D poses in Eq. 5 can be predicted. However, taking into account the depth ambiguity problem in the correspondence between 3D and 2D noises, the denoiser objective in Eq. 4 is adjusted to predict the original data rather than noise, as stated:

where Rpj v is the Reprojection from v perspective, and f θ represents the denoiser network.

Z-embedding Condition. Even though the denoising formulation of diffusion modeling is to denoise the 2D noisy poses, the essential task of the denoiser focuses on removing noise from the 3D noisy poses under t perturbed level. We leverage the geometric and experiment analyses to explore the relationship between 3D and 2D uncertainty (details in Appendix A). Our findings reveal that the 3D uncertainty range is not only influenced by the 2D uncertainty range but also relative to the depth z of the 3D point and the baseline width of the binocular setting. However, the diffused data in the fixed t shares the same perturbed level, as defined in Eq. 2. To facilitate our denoiser to learn the noise within different uncertainty but under the same timestamp t , we introduce the z t , the absolute depth of root joint of 3D poses y t , as an additional condition. This is based on the assumption that the z t is close to the ground truth z because of the initial results with limited diffused. The denoiser is modified as:

Baseline-width-related Pose Normalization. The depth z affects the noise along the x-axis, y-axis, and z-axis, while the baseline width only affects the noise on the z-axis. Considering the baseline width of binocular cameras will be changed in practical application. We propose a simple method to normalize and denormalize the 3D poses under different baseline width settings:

where z θ shares the same definition with z t , B is a scalar representing the baseline width, and ¯ z is the normalized z . Through normalization and denormalization, the input to the denoiser is baseline width independent, but the 3D pose estimation results are not affected.

Inference. During the inference process, shown in Fig. 2(b), the initial binocular 2D poses u T estimated by a 2D detector are the input to our Dual-Diffusion model. Subsequently, the initial 3D pose y T is reconstructed and normalized, then fed into the denoiser to generate the plausible and accurate 3D pose ˜ y 0 , along with their corresponding binocular 2D poses ˜ u 0 . These poses ˜ u 0 are then diffused to u t using the DDIM strategy [37] for input to the next denoinsing. After iterative denoising for K times. The final 3D pose ˜ y 0 and 2D poses ˜ u 0 are estimated. It should be noted that the 3D pose is converted to a root-relative format before being processed by the denoiser and then converted back afterward. For the experiments described below, we set T = 25 and K = 1 according to the ablation study in Sec. 4.3 and Appendix D.3. The denoiser follows a GCN-Transformer structure. Detailed architecture and training information can be found in Appendix B.

4 Experiments

Dual-Diffusion aims to reconstruct 3D pose from binocular 2D poses estimated from one off-theshelf 2D detector. We refer to it as 'Dual-Duffison-2D POSE DETECTOR' in tables. The baseline method for comparison is the Triangulation [9] (Tri), referred to as 'Tri-2D POSE DETECTOR'. The experiments are conducted on two benchmarks: 1) the short-baseline binocular benchmark, MHAD Berkeley dataset [27], and 2) the wide-baseline benchmark, H36M dataset [13]. MHAD [27] is a multi-modal dataset that encompasses 11 actions performed by 12 subjects. We choose the binocular camera pairs ( 1 -3 , 2 -4 ) in the L1 quad camera, with approximate 200 mm baseline width. H36M [13] is one of the most popular datasets for 3D HPE. To simulate the binocular setup, the camera pairs ( 1 -3 , 2 -4 ) are selected, with about 3000 mm baseline width. Mean Per Joint Position Error (MPJPE) is used to assess the accuracy, while Bone Length error (BL) and Symmetry error (Sym) are employed to evaluate the plausibility of 3D poses. Joint Detection Rate (JDR) is applied to assess 2D poses. The details of the metric and implementation can be found in Appendix C.

4.1 Comparison on MHAD

There are few methods designed particularly for binocular 3D HPE, except RSB-Pose [45]. We reproduce the state-of-the-art multiview 3D HPE methods and fine-tune them in binocular datasets, including TPPT [24], Epipolar_Tri [10], Algebraic_Tri [14] and AdaFuse [56]. Among them, RSBPose [45] utilizes Triangulation to lift 3D pose and refine it with Pose Transformer, Algebraic_Tri [14] employs a weighted Triangulation to reconstruct 3D pose. Other methods focus on estimating the 2D poses from fused features in a geometric or an attention mechanism based on the backbone ResNet [51], but the 3D pose is lifted solely through Triangulation. We use 'METHOD NAME*' in tables to indicate the methods that primarily focus on improving multiview 2D pose estimation.

To evaluate the performance of our Dual-Diffusion in 2D-3D lifting, we employ state-of-the-art binocular or multiview models as 2D detectors. Results are shown in Table 1. Our method generates more accurate and plausible 3D poses compared to the baseline. For instance, based on the 2D poses generated by ResNet50, Dual-Diffusion reduces the MPJPE by 5 . 53 mm (a 9 . 2% error reduction), and the BL and Sym by 5 . 59 mm ( 23 . 6% ) and 12 . 01 mm ( 32 . 8% ) respectively. This improvement is consistently observed with the 2D poses generated by RSB-Pose50* as well. When the image resolution increases to 384 , the 2D poses estimated are more accurate. Our method still outperforms the baseline with 2 . 02 mm ( 6 . 8% ) in MPJPE, 2 . 28 mm ( 23 . 2% ) in BL and 1 . 78 mm ( 15 . 3% ) in Sym using the RSB-Pose152* 2D detector. Additionally, Dual-Diffusion generates more accurate 2D results, with increases of 2 . 32% and 1 . 80% compared to the initial results of 2D detectors RSB-Pose50* and RSB-Pose152*, respectively. The performance enhancement demonstrates our Dual-Diffusion can simultaneously denoise both 2D and 3D noisy poses and generate more accurate and plausible results. Regardless of the input resolution, our method consistently achieves the best results across all four metrics. This demonstrates the effectiveness of our approach in short-baseline binocular 3D HPE, making it promising for practical applications.

4.2 Comparison on H36M

In Table 2, the comparison is conducted on H36M [13]. Even though the uncertainty of initial 3D poses in the wide-baseline binocular is reduced compared to the short-baseline setup [45], our Dual-Diffusion still outperforms the baseline and achieves the best results across all metrics. Based on the RSB-Pose50* 2D detector, Dual-Diffusion

achieves an error reduction of 13 . 0% , 28 . 8% , 30 . 1% and 0 . 09% in MPJPE, BL, Sym of 3D poses and JDR of 2D poses, respectively. Even at 384 resolution, where other methods demonstrate effectiveness with about 30 mm MPJPE results, our model still outperforms them by at least 1 . 87 mm in MPJPE, achieving a result of 28 . 67 mm . It is demonstrated that Dual-Diffusion can enhance 3D poses under wide-baseline binocular configurations, showcasing its generalization capabilities.

Additionally, Dual-Diffusion requires only 0.74 million parameters, significantly fewer than other methods. This demonstrates that the effectiveness of pose refinement in our method does not rely on a large number of parameters, but rather on the dual diffusion modeling and training.

4.3 Ablation Study

The ablation experiments are conducted on MHAD, as the high uncertainty issue is more pronounced in short-baseline setups compared to wide-baseline ones [45].

Impact of Each Module. We first investigate the improvements provided by the pure DualDiffusion model described in Sec.3.2, denoted as 'Dual-Diff' in the tables. Then, two additional modules, Z-embedding and BaseL-norm, are assessed. As depicted in Table3, the pure Dual-Diff significantly enhances the accuracy of 3D poses, achieving a 5 . 3% reduction in MPJPE and a 10 . 5% reduction in BL, indicative of its capability to generate more precise and plausible 3D poses. With the incorporation of Z-embedding as an additional condition, there are further 10 . 9% and 9 . 9% relative improvements in BL and Sym. It is worth noting that subjects often perform actions while tilting towards the camera rather than facing it, leading to differences in the depth of joints on the left and right sides, resulting in variations in their 3D uncertainty regions. Z-embedding is specifically designed to enhance the adaptability of the denoiser to different perturbed noise levels at the same time t , as demonstrated by the significant enhancement in Sym. Finally, the addition of the BaseL-norm brings slight improvements in three metrics. Additionally, the computational cost is limited.

Zero-Shot 2D-3D Lifting. The purpose of BaseL-norm is to enhance the flexibility of the DualDiffusion to various baseline width settings. To evaluate this, we adapt two additional camera pairs ( 1 -2 , 1 -4 ) on MHAD, representing binocular baseline width of 100 mm and 300 mm , and conduct zero-shot 2D-3D lifting on them. Firstly, we employ ResNet50 to detect 2D poses. Then, the denoiser trained on 200 mm -baseline training dataset is directly utilized to generate 3D poses on 100 mm -baseline and 300 mm -baseline testing sets without fine-tuning. The results are presented in Table 4. Without BaseL-norm, the accuracy even worsens. But with BaseL-norm, there are 4 . 25 mm and 3 . 1 mm reductions in MPJPE for two settings, respectively. The improvement illustrates the zero-shot adaptability of the BaseL-norm module to various baseline widths, which is beneficial for further application in practice.

Efficiency of the Uncertainty Distribution Initialization. We use the initialized uncertainty distribution as the target distribution for diffusion instead of random noise. To evaluate its efficiency, we establish a baseline diffusion model that generates 3D poses by denoising random noise conditioned on 2D binocular poses, referred to as 'random-noise'. To mitigate the effect of camera projection matrices P v , we translate 2D keypoints to 3D-aware vectors by P -1 v . The comparison results are shown in Table 5. Under the same inference iteration K = 1 , across the diffusion steps from 25 to 125 , the random-noise consistently yields inferior results, while Dual-Diff achieves promising results even with T = 25 . A smaller T indicates a reduced time cost of training and inference.

Moreover, there remains some ambiguity regarding whether the initial uncertainty distribution of the 3D estimation is effectively modeled. To investigate this, we perform denoising on simulated noisy 3D poses. Specifically, we first calculate the error between the 3D estimation and the 3D GT along each axis in the MHAD training set, storing this as the noise set. Then, we add noise sampled randomly from this set to the 3D GT along each axis. Finally, we use the denoiser to refine both the 'GT + noise' and 'estimated' 3D poses, comparing the MPJPE results. The hypothesis is that if the 3D uncertainty is well-modeled, the performance in refining both estimated 3D poses and simulated 3D poses should be similar. As shown in Table 6, regardless of the 2d pose detector, the accuracy of denoised 'GT+noise' in both training and testing sets is all close to the 'estimated' in the training set, demonstrating that the 3D uncertainty distribution is well-modeled.

Uncertainty Reconstructing v.s. Uncertainty Statistics. We argue that reconstructing 3D pose uncertainty from 2D pose uncertainty is a more practical approach. As discussed in Appendix A, the depth uncertainty of a 3D point increases with greater depth. Consequently, the statistical approach in DiffPose [8] tends to constrain the model to a narrow depth range, while Dual-Diffusion leverages more reliable 2D results. To evaluate this, we divide the MHAD into two subsets: large-dataset and small-dataset, based on the average depth of 3D poses and compare Dual-Diffusion with DiffPose. The results are illustrated in Table 7. Compared to the Triangulation baseline, performance improves with Dual-Diffusion but decreases with DiffPose. DiffPose suffers from the change of 3D poses uncertainty distribution while our Dual-Diffusion remains stable. This stability is due to the fact that Dual-Diffusion models the diffusion target using 2D uncertainty, which is significantly more stable than 3D uncertainty. The comparison of the stability in uncertainty between 2D poses and 3D poses can be found in Appendix D.3.

3D Pose Priors v.s. 2D Pose Priors. We argue that 3D pose priors are more easily captured compared to 2D pose priors because 2D poses vary under different perspectives. To validate this, we establish a diffusion model directly for denoising 2D poses, named '2D-Diff', and then reconstruct 3D poses. As shown in Table 5, Dual-Diff consistently outperforms 2D-Diff in terms of 3D pose accuracy. Furthermore, we compare the plausibility of 3D poses. As illustrated in Fig. 3, Dual-Diff exceeds 2D-Diff in both BL and Sym. These results collectively demonstrate the necessity of denoising in the 3D domain.

4.4 Visulization

Dual-Diffusion Denoises the Binocular 2D Poses. To understand the denoising process of binocular 2D poses, we conduct a simulation experiment. First, we gradually add noise to the ground truth 2D poses over T = 25 steps and generate the final noisy 2D poses. Then, we set the reverse iteration K = 25 to recurrently denoise the noisy poses. Fig. 4 illustrates the absolute distance between noisy data and the ground truth at each step t in diffusion and each iteration k in denoising. The noise added to the joint during the diffusion process is incrementally removed during the denoising process.

(a) diffusion

(b) denoising

0.002

u_left

u_left

0. 002

v_left

v_left

0. 001

v_right

u_right

0. 001

u_right

v_right

0. 000

0. 000

-0. 001

-0. 001

-0. 002

-0. 002

-0. 003

-0. 003

25

15

10

(a) H36M

(b) MHAD

Triangulation

RSB-Pose

Ours

Triangulation

RSB-Pose

Ours

Dual-Diffusion Denoises the 3D Poses. We provide a qualitative comparison with Tri-RSB152 and RSB-Pose152 in Fig. 5. Our Dual-Diffusion achieves more accurate 3D poses, particularly in cases of self-occlusion. For instance, in the 3 rd row of the 1 st column, the right hip is occluded, resulting in an inaccurate 3D pose using baseline Triangulation. Dual-Diffusion effectively corrects the 3D pose. This highlights our method's ability to denoise noisy 3D poses. Additionally, when baseline results are poor, such as in the 1 st row of the 2 nd column, RSB-Pose can only partially correct some joints, whereas our method corrects the entire right skeleton. More visualization is in Appendix D.4.

5 Conclusion and Discussion

This work introduces a novel framework, Dual-Diffusion, to reconstruct 3D poses from 2D poses estimated by off-the-shelf 2D pose detectors in a binocular configuration. Dual-Diffusion simultaneously denoises initial 2D and 3D poses within a single diffusion model. The diffusion process operates on 2D poses, while the denoising process occurs in 3D space, utilizing geometric mapping to connect the 2D and 3D domains. Comparisons with state-of-the-art methods demonstrate that our approach effectively denoises both 2D and 3D poses, yielding superior results and making it particularly suitable for binocular 3D HPE, especially in short-baseline configurations.

Discussion. We further extend the applicability of Dual-Diffusion to multiview settings, as shown in the Appendix D.2. The observed performance improvements validate the scalability of our method. Additionally, we incorporate 3D supervision (see Appendix D.3), revealing further advantages that will be explored in future work. However, we also acknowledge two main limitations of our method. Firstly, while simulation experiments are conducted to validate our hypothesis regarding 3D uncertainty factors (see Appendix A.2 for details), computing the boundary points of the 3D uncertainty region to explore it may lack rigor. We plan to leverage an algebraic approach to derive the range of the 3D uncertainty caused by 2D errors in future work. Secondly, our denoising network takes the root-relative 3D pose as input, neglecting optimization of the root joint. Although the root joint is finally refined (see Appendix D.1), another module to optimize the root joint is preferred.

References

Ijaz Akhter and Michael J Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1446-1455, 2015.

Soubarna Banik, Alejandro Mendoza GarcÍa, and Alois Knoll. 3d human pose regression using graph convolutional network. In 2021 IEEE International Conference on Image Processing (ICIP) , pages 924-928. IEEE, 2021.

Magnus Burenius, Josephine Sullivan, and Stefan Carlsson. 3d pictorial structures for multiple view articulated pose estimation. In 2013 IEEE Conference on Computer Vision and Pattern Recognition , pages 3618-3625. ISSN: 1063-6919.

Xipeng Chen, Kwan-Yee Lin, Wentao Liu, Chen Qian, and Liang Lin. Weakly-supervised discovery of geometry-aware representation for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10895-10904, 2019.

Zhuo Chen, Xu Zhao, and Xiaoyue Wan. Structural triangulation: A closed-form solution to constrained 3d human pose estimation. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 , Lecture Notes in Computer Science, pages 695-711. Springer Nature Switzerland.

Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang. Gfpose: Learning 3d human pose prior with gradient fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4800-4810, 2023.

Mohsen Gholami, Bastian Wandt, Helge Rhodin, Rabab Ward, and Z Jane Wang. Adaptpose: Cross-dataset adaptation for 3d human pose estimation by learnable motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13075-13085, 2022.

Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. DiffPose: Toward more reliable 3d pose estimation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13041-13051. IEEE.

Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision . Cambridge University Press, 2 edition, 2004.

Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages 7779-7788, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1597715987, 2023.

Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence , 36(7):1325-1339, 2013.

Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. Learnable triangulation of human pose. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7718-7727, 2019.

Boyuan Jiang, Lei Hu, and Shihong Xia. Probabilistic triangulation for uncalibrated multi-view 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14850-14860, 2023.

Zhongyu Jiang, Zhuoran Zhou, Lei Li, Wenhao Chai, Cheng-Yen Yang, and Jenq-Neng Hwang. Back to optimization: Diffusion-based zero-shot 3d human pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6142-6152, 2024.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Muhammed Kocabas, Salih Karagoz, and Emre Akbas. Self-supervised learning of 3d human pose using multi-view geometry. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1077-1086, 2019.

Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling for human mesh recovery. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11605-11614, 2021.

Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. HybrIK: A hybrid analyticalneural inverse kinematics solution for 3d human pose and shape estimation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3382-3392. IEEE.

Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analyticalneural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3383-3393, 2021.

WLi, H Liu, H Tang, P Wang, and L MHFormer Van Gool. Multi-hypothesis transformer for 3d human pose estimation. arxiv 2021. arXiv preprint arXiv:2111.12707 .

Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. Deformable shape completion with graph convolutional autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1886-1895, 2018.

Haoyu Ma, Zhe Wang, Yifei Chen, Deying Kong, Liangjian Chen, Xingwei Liu, Xiangyi Yan, Hao Tang, and Xiaohui Xie. Ppt: token-pruned pose transformer for monocular and multi-view human pose estimation. In European Conference on Computer Vision , pages 424-442. Springer, 2022.

Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision , pages 2640-2649, 2017.

Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 483-499. Springer, 2016.

Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena Bajcsy. Berkeley mhad: A comprehensive multimodal human action database. In 2013 IEEE workshop on applications of computer vision (WACV) , pages 53-60. IEEE, 2013.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10975-10985, 2019.

Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniilidis. Coarse-to-fine volumetric prediction for single-image 3d human pose. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1263-1272. IEEE.

Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. Harvesting multiple views for marker-less 3d human pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6988-6997, 2017.

Haibo Qiu, Chunyu Wang, Jingdong Wang, Naiyan Wang, and Wenjun Zeng. Cross view fusion for 3d human pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4342-4351, 2019.

Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, and Robert Wang. Lightweight multi-view 3d pose estimation through camera-disentangled representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6040-6049, 2020.

Cédric Rommel, Eduardo Valle, Mickaël Chen, Souhaiel Khalfaoui, Renaud Marlet, Matthieu Cord, and Patrick Pérez. Diffhpe: Robust, coherent 3d human pose lifting with diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3220-3229, 2023.

Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. P-stmo: Pretrained spatial temporal many-to-one model for 3d human pose estimation. In European Conference on Computer Vision , pages 461-478. Springer, 2022.

Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang, Kai Han, Shanshe Wang, Siwei Ma, and Wen Gao. Diffusion-based 3d human pose estimation with multi-hypothesis aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14761-14771, 2023.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5693-5703, 2019.

Zhenhua Tang, Zhaofan Qiu, Yanbin Hao, Richang Hong, and Ting Yao. 3d human pose estimation with spatio-temporal criss-cross attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4790-4799, 2023.

Bugra Tekin, Isinsu Katircioglu, Mathieu Salzmann, Vincent Lepetit, and Pascal Fua. Structured prediction of 3d human pose with deep neural networks. arXiv preprint arXiv:1605.05180 , 2016.

Denis Tome, Chris Russell, and Lourdes Agapito. Lifting from the deep: Convolutional 3d pose estimation from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2500-2509, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Xiaoyue Wan, Zhuo Chen, Yiming Bao, and Xu Zhao. Rsb-pose: Robust short-baseline binocular 3d human pose estimation with occlusion handling. arXiv preprint arXiv:2311.14242 , 2023.

Xiaoyue Wan, Zhuo Chen, and Xu Zhao. View consistency aware holistic triangulation for 3d human pose estimation. Computer Vision and Image Understanding , 236:103830, 2023.

Bastian Wandt and Bodo Rosenhahn. Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7782-7791, 2019.

Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, and Christian Theobalt. Estimating egocentric 3d human pose in global space. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11500-11509, 2021.

Jinbao Wang, Shujie Tan, Xiantong Zhen, Shuo Xu, Feng Zheng, Zhenyu He, and Ling Shao. Deep 3d human pose estimation: A review. Computer Vision and Image Understanding , 210:103225, 2021.

Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d human pose estimation with normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11199-11208, 2021.

Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV) , pages 466-481, 2018.

Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems , 35:38571-38584, 2022.

Youze Xue, Jiansheng Chen, Xiangming Gu, Huimin Ma, and Hongbing Ma. Boosting monocular 3d human pose estimation with part aware attention. IEEE Transactions on Image Processing , 31:4278-4291, 2022.

Petrissa Zell, Bastian Wandt, and Bodo Rosenhahn. Joint 3d human motion capture and physical analysis from monocular videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 17-26, 2017.

Jinlu Zhang, Zhigang Tu, Jianyu Yang, Yujin Chen, and Junsong Yuan. Mixste: Seq2seq mixed spatiotemporal encoder for 3d human pose estimation in video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13232-13242, 2022.

Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, and Wenjun Zeng. Adafuse: Adaptive multiview fusion for accurate human pose estimation in the wild. International Journal of Computer Vision , 129:703718, 2021.

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Semantic graph convolutional networks for 3d human pose regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3425-3435, 2019.

Qitao Zhao, Ce Zheng, Mengyuan Liu, and Chen Chen. A single 2d pose with context is worth hundreds for 3d human pose estimation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen. Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8877-8886, 2023.

Weixi Zhao, Yunjie Tian, Qixiang Ye, Jianbin Jiao, and Weiqiang Wang. Graformer: Graph convolution transformer for 3d pose estimation. arXiv preprint arXiv:2109.08364 , 2021.

Jieming Zhou, Tong Zhang, Zeeshan Hayder, Lars Petersson, and Mehrtash Harandi. Diff3dhpe: A diffusion model for 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2092-2102, 2023.

Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards 3d human pose estimation in the wild: a weakly-supervised approach. In Proceedings of the IEEE international conference on computer vision , pages 398-407, 2017.

A Relationship between 3D and 2D Uncertainty

A.1 Theoretical Analysis

We simplify the relationship between 3D and 2D uncertainty to the relationship between 3D uncertainty range ∆x , ∆z , ∆z ′ with 2D uncertainty range ∆u . We analyze it in the x-o-z cross-section of 3D space, as shown in Fig. 6. O l and O r are the optical centers of binocular cameras which are rectified and their image plane is parallel to the x-axis. The red point H is the ground truth of a 3D point with depth z . The analysis is based on the hypothesis that the uncertainty range of the estimated 2D point is the same in 2 views with ∆u along the x-axis. The ground truth 2D point is the projection of H .

Point A represents the intersection of the left boundaries of the 2D uncertainty regions from both viewpoints and serves as the left boundary along the x-axis in the 3D uncertainty region. B denotes the intersection of the right boundaries from both viewpoints, forming the right boundary along the x-axis in the 3D space. C is the intersection between the right boundary of the 2D uncertainty region from the left view and the left boundary of the 2D uncertainty region from the right view, establishing the lower bound along the z-axis of 3D uncertainty. Conversely, point D marks the upper bound along the z-axis.

Uncertainty along the x-axis. To analysis the relationship between ∆x and ∆u , we first prove that line segment AB is parallel to the image plain. Based on the disparity formula z = fB/ ∆ d , A and H have the same depth z since they share the same disparity, and similarly for points B and H . Therefore, line segment AB passes through H and is parallel to the image plane. Then, the segment AH can be derived using similar triangles:

A

B

C

D

𝑂

!

∆𝑢

∆𝑥

∆𝑧

∆𝑧′

𝑓

𝐵

𝑧

𝑂

"

E

F

G

Z

O

X

H

Uncertainty along the z-axis. We first prove that the midpoint E of O l O r and H lie on the line DC . Through point C , draw a line parallel to the image plane, intersecting O l D and O r D at points F and G respectively. Line DC intersects AB and O r O l at points E ′ and H ′ . According to the properties of similar triangles, FC = GC . Furthermore, cause △ DFC ∼ △ DO l E ′ and △ DGC ∼ △ DO r E ′ , we can deduce that O l E ′ = O r E ′ . Similarly, it can be proven that AH ′ = BH ′ . Then, the segment ∆z can be derived using similar triangles:

Combining with Eq. 10, it can be written as:

The segment ∆z ′ can be derived in the similar way:

We omit the proof of ∆y as it follows the same reasoning as ∆x , given that the standard deviation along the u-axis and v-axis in a 2D image plane are equivalent. In summary, the factors contributing to the 3D uncertainty range include the 2D uncertainty range, the depth of the 3D point, and the baseline width. The 3D depth affects uncertainty along the x-axis, y-axis, and z-axis. The baseline width only affects the z-axis uncertainty.

The error distribution along X-axis of reconstructed 3d point

X_error

X_error

The error distribution along Y-axis of reconstructed 3d point

The error distribution along Z-axis of reconstructed 3d point

500

1000

1500

2000

2500

3000

-10

-5

0

5

10

500.0

1000.0

1500.0

2000.0

2500.0

3000.0

3dpoint_depth

500

1000

1500

2000

2500

3000

-10

-5

0

5

10

500.0

1000.0

1500.0

2000.0

2500.0

3000.0

3dpoint_depth

Y_error

500

1000

1500

2000

2500

3000

-100

-50

0

50

100

500.0

1000.0

1500.0

2000.0

2500.0

3000.0

3dpoint_depth

200

400

600

800

-10

-5

0

5

10

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

The error distribution along X-axis of reconstructed 3d point

baseline_width

200

400

600

800

-6

-4

-2

0

2

4

6

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

The error distribution along Y-axis of reconstructed 3d point

baseline_width

Y_error

200

400

600

800

-200

-100

0

100

200

300

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

900.0

The error distribution along Z-axis of reconstructed 3d point

baseline_width

(a) with depth changing

(b) with baseline width changing

z

y

x

z

y

x

A.2 Experiment Analysis

The analysis above solely relies on the boundary points to explore the relationship between 3D and 2D uncertainty, which is not rigorous. Hence, we conduct the experiments to validate the conclusion.

We change the point depth from 500 mm to 3000 mm with 500 mm step to discover the relationship between 3D uncertainty and depth. The binocular 2D points within a fixed uncertainty range are sampled randomly and then reconstructed as a 3D point using Triangulation. We sample for 100000 times at each depth and then analyze the 3D uncertainty range, visualized in Fig. 7(a). The 3D depth affects the uncertainty of a 3D point along the x-axis, y-axis, and z-axis but with different relationships. Hence, we add z-embedding as an additional condition to guide the denoiser. The experiment investigating the impact of baseline width is depicted in Fig.7(b). Baseline width solely affects the uncertainty distribution along the z-axis. As depicted in Eq.12 and Eq.13, aside from the constant term, we introduce a normalization factor B to standardize the

The error distribution along Z-axis of reconstructed 3d point with or without baseline norm

200

400

600

800

-300

-200

-100

0

100

200

300

T

F

baseline_width

error distribution across different baseline widths. The distribution with and without normalization is illustrated in Fig. 8.

B Denoiser Architecture and Training Details

Denoiser Architecture. The denoiser architecture, illustrated in Fig. 9, is similar to the one proposed by [8] and essentially follows a GCN-Transformer structure. The 3D pose, represented as J joint locations, is first input into a GCN layer. The graph for the GCN is defined by the skeleton connections, which encode the topological features of each joint. Next, five stacked GCN-Attention modules are used to enhance the global-local perception of the joint features. Each GCN-Attention module comprises one Attention layer and two GCN layers. Each Attention layer includes a 4-head self-attention mechanism. Timestamp embedding and Z-embedding are added to each GCN-Attention module. The embedding schedule is sinusoidal [44]. Finally, the 3D pose is output after passing through another GCN layer.

Training Details. The training process is depicted in Fig. 9. We first generate diffused data for training as follows: 1) Initialization: Start with the ground truth 2D poses u 0 . 2) Noise Addition:

Z

𝒛

𝒕

embedding

Timestamp

𝒕

embedding

𝑢

"

𝑢

#

𝑦

#

& 𝑦

"

& 𝑢

"

Training

Denoiser

𝑳

𝜽

Loss

GCN Layer

GCN Layer

Attention Layer

GCN Layer

GCN Layer

……

……

X 5

Triangulation

Projection

diffuse

denoise

BaseL-norm

BaseL-denorm

Randomly choose a timestamp t ∈ [1 , T ] , add noise to generate noisy 2D poses u t . 3) Triangulation: Use triangulation to reconstruct the noisy 3D poses y t . Then we denoise to recover the data through the denoiser: 1) Denoising 3D Poses: The denoiser takes the noisy 3D poses y t as input and denoises them to recover original 3D poses ˜ y 0 . BaseL-norm is applied before the denoiser and BaseL-denorm is applied after. 2) Reprojection: Reproject the denoised 3D poses back to 2D poses ˜ u 0 . The training objective follows the L θ in Eq. 7 to facilitate the denoiser recovering the original input.

The Dual-Diffusion is implemented by PyTorch [28] using Adam optimizer with learning rate 0 . 00002 and other parameters are all default. We train our Dual-Diffusion in MHAD [27] for 80 epochs, and in H36M [13] for 40 epochs, respectively. There is no other schedule for the learning rate employed. All experiments are conducted on GeForce RTX 2080 Ti.

C More Details of Experiment Implementation

Evaluation Metrics. To evaluate 3D pose accuracy, we utilize Mean Per Joint Position Error (MPJPE). Besides, we employ two metrics to assess the plausibility of the 3D pose: Bone Length error (BL), which measures the average distance between the predicted and ground truth bone lengths; and Symmetry (Sym) [34], which quantifies the average difference between the lengths of corresponding left and right bones. The Joint Detection Rate (JDR) is applied for 2D pose evaluation with the detection threshold set at 2 . 5% of the bounding box width.

Implementation Details. We choose ResNet [51] and RSB-Pose* [45] as our 2D pose detectors and divide them into two categories with different input image resolutions. To acquire the initial uncertainty distribution of estimated 2D poses, the methods are distinct for different 2D detector. For ResNet, the standard deviation are set to 2 and 3 separately for 256 and 384 image resolution, as the training objective formulating. For RSB-Pose*, we obtain the standard deviation by statistically analyzing the estimation results on the training set. Because its training objective is MPJPE but not heatmap. Then, the standard deviation is scaled by image resolution to acquire σ T . The diffusion step T is set to 25, and the reverse step K is set to 1 decided by the ablation study in Sec. 4.3 and Appendix D.3. To acquire the deterministic results, the reverse sampling is followed to the ODEs [39].

D More Experiments

D.1 Comparison on MHAD

We compare the per-joint error with the baseline using the same 2D pose detector. The results are shown in Table 8. Our Dual-Diffusion method surpasses the baseline in most joints, particularly

the root joints and those close to the root. Interestingly, even though the input to the denoiser is the root-relative 3D pose, the root joint itself is still optimized as highlighted in green. We hypothesize that this optimization occurs due to the correlation established with the connected joints, which demonstrates that the pose prior is effectively captured.

D.2 Applicability to Multiview Settings

To evaluate the applicability of Dual-Diffusion in multiview settings, we conduct experiments by denoising the initially estimated 3D poses using ResNet152 across 2-view, 3-view, and 4-view configurations of the H36M testing set. The results, as shown in Table 9, demonstrate that DualDiffusion improves performance compared to the Triangulation baseline. However, the degree of improvement decreases as the number of views increases. This aligns with our explanation that 3D uncertainty is more ambiguous in binocular settings than in 3-view or 4-view setups, where 3D uncertainty is reduced, making denoising less impactful.

D.3 Ablation Study

Uncertainty Stability Comparison. We conducted a comparison of the stability in uncertainty between 2D and 3D poses. First, we evaluated the 2D MPJPE and 3D MPJPE estimated by ResNet50 under binocular settings with varying baseline widths, as shown in Table 10. Additionally, we compared the 2D and 3D uncertainties obtained from ResNet152 across different depths, detailed in Table 11. The results clearly demonstrate that 2D pose estimation is more robust than 3D pose estimation.

Impact of the Inference Iteration Time. To explore the impact of the inference iteration time K , we denoise the 3D poses for K ∈ { 1 , 2 , 5 , 10 , 20 } iterations and compare the MPJPE results. When K = 1 , we achieve the best result, as shown in Table 12. Additionally, our Dual-Diff outperforms the 2D-Diff across all K , demonstrating the effectiveness of modeling pose priors in the 3D domain.

Impact of Supervision. We implement experiments by using 3D supervision alone and in combination with 2D supervision to retrain

Triangulation

RSB-Pose

Ours

Triangulation

RSB-Pose

Ours

Triangulation

RSB-Pose

Ours

Triangulation

RSB-Pose

Ours

这这

Dual-Diffusion, based on the RSB152 backbone. As shown in Table 13, adding 3D supervision additionally enhances 3D accuracy and plausibility. Considering that Dual-Diffusion still has room for improvement, future work will focus on addressing the limitations discussed in the paper and exploring additional enhancement methods.

D.4 Visualization

We provide a qualitative comparison with Triangulation and RSB-Pose on H36M and MHAD, as illustrated in Fig.10 and Fig.11 respectively. Our Dual-Diffusion method noticeably corrects selfoccluded joints in MHAD. In the wide-baseline H36M, although the joints occluded in both views are minimal, our method still achieves enhancements compared to Triangulation and RSB-Pose.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are clearly stated in the abstract and the introduction. The contributions are summarized in the last paragraph in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the limitations paragraph of Sec. 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumption about 2D uncertainty distribution of Dual-Diffusion is provided in Forward Dual Diffusion paragraph of Sec. 3.2. The proof of the relationship between 3D and 2D uncertainty is provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Dual-Diffusion model is clearly described in Sec. 3.2. The training details and the implementation details are provided in Appendix B and Appendix C respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and models are available at https://github.com/sherrywan/DualDiffusion.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details are provided in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The metrics are clearly defined in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The model is implemented by PyTorch and the computation cost is illustrated in Table 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our Dual-Diffusion is specifically designed for binocular 3D HPE which has positive societal impacts due to its advantages, which is discussed in the first paragraph of Sec. 1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original paper and datasets are all cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a dual-diffusion model for Binocular 3D Human Pose Estimation, which models the uncertainties integrated in the binocular configuration. Since constructing the depth distribution of 3D poses is difficult, the authors use 2D poses with triangulation to initialize the 3D poses during the diffusion process. Additionally, they adopt the depth of the 3D root as an additional condition and introduce pose normalization to improve generalization ability. The experiments show that the method achieves superior performance on the MHAD and H36M datasets.\n\nSTRENGTHS:\n- Using multi-view 2D poses with triangulation to construct the initial distribution for 3D poses during the diffusion process is interesting and superior to previous designs in diffusion-based 3D pose estimation.\n- The Z-embedding Condition and Baseline-width-related Pose Normalization are simple but reasonable approaches that consider the varying depth of humans and the baseline-width in the binocular setting.\n\nWEAKNESSES:\n- The comparison between triangulation and common 3D initial distributions like DiffPose [8] should be conducted, which is necessary for revealing the strength of the proposed dual diffusion structure.\n- The scalability of the proposed method to address more camera views should be clarified. It would be helpful if the authors could discuss how their approach handles an increased number of cameras and any potential impacts on performance.\n- 3D poses can be directly obtained by triangulating multi-view 2D poses. What is the accuracy of the 3D poses from direct triangulation? This comparison is important to assess the necessity of the proposed diffusion process.\n\nQUESTIONS:\n- How does the proposed method perform in multi-view scenarios with more than 2 views?\n- The proposed framework does not require 3D supervision. Could the performance be improved with additional 3D pose supervision?\n- What is the performance gap between the proposed triangulation and common 3D initial distributions like DiffPose [8] in the 3D pose diffusion process?\n- How could the proposed diffusion framework enhance the results of direct triangulation?\n\nLIMITATIONS:\nThe limitations and societal impact have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method for binocular human pose estimation. The key idea is to employ diffusion model into this problem.  The authors propose a novel dual-diffusion process: jointly diffuse 2D pose uncertainty and 3D pose uncertainty. Because 3D uncertainty relates to both 2D noise (which directly relates to diffusion step t) and depth, the authors further proposed z-embedding and BaseL-norm. Results on bioncular H36M and MHAD datasets demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The dual-denoising is interesting for human pose estimation. The novelty of the proposed method is clear. \n\n2. The quantitative results seem good in the Table.1 and Table.2. \n\n3. The paper is clearly written.\n\nWEAKNESSES:\nI have some questions about the evalution. \n\n1. The paper compares a lot with RSB-pose. However, in the orignial paper of RSB-pose, the absolute MPJPE is not reported. How do the authors compute the results of RSB-pose? Why not compare with RSB-pose with MPJPE_re (relative to pelvis) ? \n\n2. MIssing comparison with recent paper \"Triangulation Residual Loss for Data-efficient 3D Pose Estimation (NeurIPS 2023)\". It optimizes both multiview 2D and 3D with an extension of epipolar loss.  at least the authors should discuss it. \n\n3. Why not use transformer based backbones, which are the mainstreams of 2D pose estimation? Accuracy may improve with transformer backbones. The table.1 may indicate that larger backbone may mitigate the gap among different methods. The table.2 may indicate that larger scale may narrow the gap too. \n\n4. I want to know the parameter size of different methods (parameters other than the same backbone part) in table.1 and table.2. I want to make sure that the better accuracy does not simply come from large parameter capability but come from the network design.\n\nQUESTIONS:\nNo  detailed question.\n\nLIMITATIONS:\nThe authors addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors suggest a 3D human pose estimation (HPE) in binocular (two-view) settings. In order to reduce the uncertainty of triangulation method, the authors leverage the diffusion model. They propose a method called Dual-Diffusion method, which simultaneously denoises uncertainties in both 2D and 3D to produce accurate results. Additionally, they introduce Z-embedding and pose normalization related to baseline width. Experiments validate the effectiveness of the proposed method, showing performance improvements over the triangulation method.\n\nSTRENGTHS:\n- The proposed method achieves state-of-the-art results in binocular 3D HPE settings.\n\nWEAKNESSES:\n- The concept of applying diffusion models to 3D HPE is similar to DiffPose[8].\n- While the method can be applied to any multi-view setting, the authors limited their experiments to binocular settings, which restricts the applicability of the proposed method. Additionally, the performance improvements due to the dual diffusion over triangulation are marginal. Since accurate 2D pose detection reduces uncertainty in the triangulation method, it is challenging to assess the significance of the dual-diffusion models. It would be more convincing to show how the proposed method performs better with noisy 2D inputs.\n- The clarity and readability of the paper need to be enhanced.\n\nQUESTIONS:\nMethod\n- What is the meaning of the sentence in Line 176? \"However, if the uncertainty of the 3D pose follows a Gaussian distribution is still confusing.\"\n- According to Fig2 (b), the denoiser estimate $y_0$ from $y_t$, not $y_{t-1}$. Why does the denoising process needs to be repeated $K$ times? I also found that only $K=1$ is used for the experiments.\n\nExperiments\n- What is the performance of the proposed method in 3-View or 4-View settings?\n\nLIMITATIONS:\nLimitations\n- The method can be used only for binocular settings.\n\nSuggestions\n- Extending the framework to multi-view 3D HPE and validating the superiority over other multi-view 3D HPE methods would strengthen the paper.\n\nTypos\n- Figure 2. Initial estimated -> initially estimated\n- Line 149. Inherently -> inherent\n- Line 182. $0, 1$ -> ${0,1}$\n- Line 320, by $P_v$ -> by multiplying $P_v$", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for 3D human body keypoint estimation from binocular images. Different from traditional multi-view settings, such methods only take two views as input, which suffer from larger uncertainty at detph wise. To alleviate the depth ambiguity, the paper describes a diffusion-based framework. The forward diffusion process is used to simulate the noisy 2D poses from GT 2D poses. The reverse denoising process refines the 3D poses obtained by triangulation of noisy binocular 2D poses. The denoised 3D poses are then projected back to 2D poses to make the loop. The proposed method has been added to previous SOTA method, RSB-Pose, and improves the performance on MHAD and Human3.6M in binocular settings.\n\nSTRENGTHS:\n1. The idea of using diffusion process to simulate noisy 2D poses and to refine the 3D poses via denising is interesting. The insights behind these designs have been clearly explained and sound reasonable. \n2. Experiments of adding the proposed method to previous method in Tab. 1 and 2 verify the effectiveness of the proposed method. The detailed ablation study futher shows the rationality of the architecture design. \n3. The paper is easy to understand and clearly written.\n\nWEAKNESSES:\n1. Relatively limited experiments. The proposed method is mainly added to one previous method, RSB-Pose. More such experiments will make the paper be more convincing. \n2. Lacking discussion about whether the proposed method can effectively model the 3D keypoint uncertainty.\n\nQUESTIONS:\n1. Could you provide more evidence to further verify the generalization ablity of the proposed refinement method, like equipping more SOTA method?\n2. The in-depth discussion about how the proposed method models the 3D keypoint uncertainty. Is there any evidence to prove this?\n\nIn rebuttal, these questions have been well addressed by the feedbacks.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}