Dual-Diffusion for Binocular 3D Human Pose Estimation
Authors: Xiaoyue Wan
Zhuo Chen
Bingzhi Duan
Xu Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Human Pose Estimation
Binocular Vision
Diffusion Model
Pose Priors
Primary area: machine_vision
TL;DR: To address the increasing uncertainty of binocular 3D HPE due to the reduction of views compared to multiview setups, we propose a Dual-Diffusion method to simutaneously denoise the 3D and 2D poses.

Abstract: Binocular 3D human pose estimation (HPE), reconstructing a 3D pose from 2D poses of two views, offers practical advantages by combining multiview geometry with the convenience of a monocular setup. However, compared to a multiview setup, the reduction in the number of cameras increases uncertainty in 3D reconstruction. To address this issue, we leverage the diffusion model, which has shown success in monocular 3D HPE by recovering 3D poses from noisy data with high uncertainty. Yet, the uncertainty distribution of initial 3D poses remains unknown. Considering that 3D errors stem from 2D errors within geometric constraints, we recognize that the uncertainties of 3D and 2D are integrated in a binocular configuration, with the initial 2D uncertainty being well-defined. Based on this insight, we propose Dual-Diffusion specifically for Binocular 3D HPE, simultaneously denoising the uncertainties in 2D and 3D, and recovering plausible and accurate results. Additionally, we introduce Z-embedding as an additional condition for denoising and implement baseline-width-related pose normalization to enhance the model flexibility for various baseline settings. This is crucial as 3D error influence factors encompass depth and baseline width. Extensive experiments validate the effectiveness of our Dual-Diffusion in 2D refinement and 3D estimation. The code and models are available at https://github.com/sherrywan/Dual-Diffusion.

Dual-Diffusion for Binocular 3D Human Pose Estimation
Xiaoyue Wan
Zhuo Chen Department of Automation
Bingzhi Duan Xu Zhao
Shanghai Jiao Tong University
(sherrywaan, chzh9311, DuanBingzhi, zhaoxu]@sjtu edu. cn
Abstract
Binocular 3D human pose estimation (HPE), reconstructing 3D pose from 2D poses of two views, offers practical advantages by combining multiview geometry with the convenience of a monocular setup. However, compared to a multiview setup, the reduction in the number of cameras increases uncertainty in 3D recon- struction. To address this issue, we leverage the diffusion model, which has shown success in monocular 3D HPE by recovering 3D poses from noisy data with high uncertainty. Yet, the uncertainty distribution of initial 3D poses remains unknown. Considering that 3D errors stem from 2D errors within geometric constraints, recognize that the uncertainties of 3D and 2D are integrated in binocular configu- ration, with the initial 2D uncertainty being well-defined. Based this insight, we propose Dual-Diffusion specifically for Binocular 3D HPE, simultaneously denoising the uncertainties in 2D and 3D, and recovering plausible and accurate results. Additionally, we introduce Z-embedding as an additional condition for denoising and implement sosliinewwith--rlltte pose normalization enhance the model flexibility for various baseline settings. This is crucial as 3D error influence factors encompass depth and baseline width. Extensive experiments validate the effectiveness our Dual-Diffusion in 2D refinement and 3D estimation. The code and models are available at http:/gghhbbbooom/////////w////a -ifffssio
Introduciton
3D human pose estimation (HPE) aims to localize the
3D position of human joints, which has broad range of downstream applications
uncertalntyy Incresses]
[49] To date, monocular
3D HPE (25] 30 20
G0
22
35
59
58 has received
great deal of attention due to its convenient for practical applications, while multiview (more than cameras)
30 point
5 46] has earned popularity due its absolute localization under geometric constraints. However, the pros and cons of these two setups are "conjugat" each other, with monocular suffering from depth ambiguity, while multiview is hindered by strict scene constraints. Binocular setup [45] offers both ad- vantages, yet has long been ignored by the community. This motivates us focus on the Binocular 3D HPE, which lifts to 3D pose from binocular 2D poses.
3D HPE [31 14 56
Iroge plase
20uniollailly Tange
optica center
(a) 4 views
[b] 2 viows
Figure 1: Binocular reconstruction has higher 3D uncertainty compared to multi- view configurations.
Corresponding author: Xu Zhao.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Observing the uncertainty range of a 3D point reconstructed under different numbers of cameras, shown in Fig. ! it is evident that although binocular setups significantly reduce 3D reconstruction uncertainty compared to monocular configurations due to geometric constraints, they still have higher ambiguity compared to multiview setups. In previous works on monocular 3D HPE, to alleviate ambiguity, human pose priors such as joint angle limits [1] and physical plausibility [54] are commonly modeled. Nowadays, s, with advancements in probabilistic methods in machine learning, many works aim to leverage data to model the distribution of real human poses as representation of
pose prior, including VAE [23 29 48). GAN 7], normalizing flow 19 50], and diffusion models
36 [6]. Among these, diffusion models stand out due to their advantages like indirect likelihood
correlation, simple training, and network flexibility [12 38). Given the success of diffusion models in modeling pose priors, can they be cleverly leveraged to reduce uncertainty in binocular 3D HPE?
The diffusion model 11 comprises two processes: the forward diffusion process, which perturbs the real data to diffused distribution; and the reverse denoising process, which denoises the noisy data to match the real distribution. In monocular 3D HPE, the diffusion model is employed to recover 31) poses from noisy data sampled from random noise [34] [1] or from an initial 3D pose distribution with high uncertainty [8] The first category excluded due to high time consumption, as it fails to leverage geometric constraints [9] to narrow down the search space. The second category starts from the initial 3D poses with higher uncertainty (estimated by off-the-shell 3D HPE methods) and converges to the lower uncertainty distribution representative of more plausible and accurate 3D poses, which is more efficient. However, the main problem lies in the unknown distribution of the initial 3D uncertainty. The statistical method is used in [8] to solve it which is unrealistic in practice. Unlike monocular setups, 3D errors stem from errors in 2D estimation in binocular within a geometric framework. In other words, 3D uncertainty can be reconstructed from 2D cause they are intrinsically linked by geometric constraints. Therefore, we propose a diffusion- based method specifically for binocular 3D HPE, named Dual-Diffusion, capable of simultaneously denoising initial 2D and 3D uncertainties.
From diffusion perspective, the uncertainty distribution in initial 2D joints is well-defined, typically modeled as a Gaussian distribution centered at the ground truth with a specified standard deviation 26 51 40). However, the uncertainty in initial 3D poses remains unknown. Nevertheless, from denoising standpoint, it is preferable to recover poses in 3D space 61 rather than in 2D, given that human pose priors are inherently 3D. To bridge the gap between diffusion and denoising across different domains, we leverage geometric projection techniques to couple the 2D plane and 3D In the forward diffusion process, noisy 2D binocular poses are generated from ground truth 2D poses by adding noise step- by-step. Subsequently, Triangulation [ is used to reconstruct noisy 3D poses, thereby defining the uncertainty distribution in 3D. During the reverse denoising process, we sample noisy 3D poses and remove the noise to recover plausible and accurate 3D poses. Reprojection utilized to estimate the corresponding 2D poses back, enabling simultaneous denoising in both 2D and 3D spaces.
space. The specific modeling of Dual-Diffusion is illustrated in Fig. (a). Fig.
model, the essential objective of the denoiser network is to remove the noise of noisy 3D poses under different perturbed distributions. The noise-perturbed level is determined by the noise addition step in diffusion and is reflected in the denoiser via timestamp embedding. While t is directly associated with 2D noise, 3D noise depends not only on 2D factors but also on 3D depth and the baseline width of the binocular setup. To enable flexible denoising of 3D noisy data under the same / but with varying uncertainties, we introduce 3D depth as an additional condition, named Z-embedding. Besides, the bassliine-iidh-relaae applied to 3D poses, which allows the model to flexibly adapt to different baseline width settings.
Reviewing
normalization (BaseL-norm) is
To validate the efficacy of our Dual-Diffusion model in denoising both 2D and 3D poses, conducted experiments on the binocular H36M [13 and MHAD (27] datasets, utilizing only 2-view camera pairs. Our model outperforms the baseline Triangulation and other state-of-the-ar methods. Additionally, we establish the "random-noise" and "2D-Diff" models for comparison, demonstrating the effectiveness of starting from initial pose distributions and leveraging 3D pose priors.
Our contributions can be summarized as follows:
Dufl-Diffusion Framework. We propose a novel framework, Dual-Diffusion, specifically designed for binocular 3D HPE. This model simultaneously removes noise from both 2D and 3D poses by leveraging geometric mapping.

---

Uncertainty Analysis and Denoiser Enhancement. We analyze the relationship between 3D and 2D uncertainties and introduce Z-embedding and BaseL-norm to enhance the flexibility of the denoiser.
Benchmark Performance. Our method achieves superior performance on the evaluated benchmarks, demonstrating effectiveness.
2 Related Work
Binocular and Multiview 3D HPE. Binocular 3D HPE aims to estimate 3D poses from single- frame 2D poses captured from two perspectives. This task has been sparsely studied. RSB-Pose [45 utilizes stereo volume features to enhance binocular coherence in 2D poses and employs spatial Transformer for 3D pose refinement. While the Transformer excels at establishing correlations between nodes, its ability to effectively identify and denoise incorrect joints is uncertain. Given the geometric constraints shared multiview setups, we review multiview 3D HPE methods, typically involving two stages: 1) estimating 2D poses from images, and 2) lifting these to 3D poses using geometric constraints, with Triangulation [9] being the most common approach. Existing methods can be categorized into two stages of improvement. The first category [32 2D poses using 3D-aware features from multiview fusion. The second category 3
10
33
46 enhances
31
[4,
[5 [5]
focuses on lifting 2D poses to 3D. Explicit pose priors, such as bone length, incorporated into the lifting process using Pictorial Structure Models 31] or closed-form Structural Triangulation [5]. treat lifting as 3D regression task using volume representations, implicitly considering pose coherence. However, these approaches either rely on limited explicit pose priors or involve computationally intensive 3D convolutions, Actually, due to geometric constraints, the 3D poses reconstructed from initial 2D poses in multiview are generally reliable, with limited studies focusing on 2D-3D lifting. Recognizing that the primary challenge in binocular setups is increased 3D uncertainty, this work specifically focuses on the 2D-3D lifting process. Hence, we review monocular 3D HPE methods for additional insights.
Some methods [14 15
Monocular 3D HPE. Monocular 3D HPE methods can be categorized into one-stage and two-stage
approaches. One-stage methods 30 43 on extensive datasets and complex network architectures. Two-stage (55 18 4 first estimate the 21) pose using 2D detector, then lift it to 3D through deep network mapping. Considering the joints are
62 42
directly regress the 3D pose from the image, relying
connected within a skeleton, Graph Convolutional Networks (GCNs) (57 2 60 and Transformer architectures [35 55 53 59] are introduced to establish correlations between joints during the lifting process. However, despite establishing joint relationships, monocular methods still suffer from inherent depth ambiguity, which utilizes temporal consistency poses priors to overcome.
Pose Priors in 3D HPE. Since our task involves single-frame pose estimation, we primarily review works focusing on modeling pose priors in monocular 3D HPE. Previous approaches explore explicit pose priors, such as joint angle limits (.. physical plausibility [54], defined pose models (2I). However, explicit priors, despite their interpretability, may not be comprehensive. With advancements in image generation, probabilistic methods capable of modeling data distributions gain attention. VAE [23 29 488 encodes poses into latent space following a normal distribution, then decodes them back the original pose. However, inference through recurrently optimizing latent parameters can be time-consuming. Normalizing flows 19 50 use invertible transformations to map latent features to 3D poses, aiding inference, but with complex network architectures. Generative adversarial networks (GANs) (7) learn pose distributions by distinguishing fake and real poses, but face challenges in gain popularity due to its network flcxibility and training simplicity. It learns data distribution by iteratively removing noise which is incrementally added to real data until fully diffused noise distribution is achieved. Several methods 66 [2] 36 34 61
training. The diffusion models 38 39
condition the 3D pose distribution on estimated 2D points. However, directly applying this framework to binocular 3D HPE disregards geometric constraints, acquiring more diffusion and denoising steps. Another approach g treats initial 3D poses as containing high uncertainty and accurate 3D poses as maintaining low uncertainty, and employs the diffusion model to denoise noisy 3d pose from an initial high-uncertainty distribution to a more accurate result under certain distribution. However, the initial distribution of 3D poses is unknown which is estimated statistically g. Considering geometric constraints in binocular configuration, the initial 3D pose distribution can be derived from
3

---

(a) Modeling
(b) Inference
2D pase initall
Forward Diffusion
20 pose real
distribution Uf
distributiinn
Uo
K times
Z embodking
Yt
Me
Geometric
Mapping
Denoiser
DDIM generate
Timestamp emshedding
yr
Yu
3D pose initial distribution
pose real distribution
Reverse Denoising
Trinngaaation
Reperjaction
Bood--oon
Figure 2: Overview of Dual-Diffusion Method. (a) Modeling: In the forward diffusion process, noise is added to the ground truth binocular 2D poses ug for T steps, aligning with the distribution of initial estimated 2D poses. During the reverse denoising process, noisy 3D poses are progressively denoised plausible poses. Geometric mapping is employed connect 2D and 3D domains. (b) Inference: The initial 3D pose у/, reconstructed from binocular 2D poses ur, is denoised to jo- Then yo is reprojected to the denoised 2D poses ug. The entire denoising process iterates for times.
the initial 2D distribution which is more straightforward. Therefore, we propose the Dual-Diffusion model to denoise both 2D and 3D poses, specifically for binocular 3D HPE.
3 Method
We propose Dual-Diffusion to simultaneously optimize 2D poses and 3D poses, addressing the high uncertainty issue in binocular 3D HPE. The modeling framework illustrated in Fig. [[a). Starting with the initial 2D poses estimated by the off-the-shelf 2D detector, which are considered as the diffused data, we apply a diffusion model to denoise it. However, 3D pose rather than the 2D pose, given the inherently 3D nature of pose priors. The geometric projection relationship is leveraged to bridge the gap and enable dual-denoising of 2D and 3D space. In the subsequent sections, we first briefly introduce the diffusion models and then describe our Dual-Diffusion model in detail.
denoising applied to
3.1 Revisiting Diffusion Models
Diffusion models
38
are a type of probabilistic method that can recover data that satisfy the
underlying distribution Pdata (z) from noisy data. comprises two processes: the forward diffusion process and the reverse denoising process. During the forward process, the real data zo is diffused by Gaussian noise step-by-ste step over T steps under Markov chain. The formulation can be written as:
T
(( = II a( 1=1
g( = N(a,; / 1, BrI),
(I)
where the variance schedule 3, determines the perturbed level of the noisy data xg. Using the notation
Or : 1- and a. := 11 ar, It. can be sampled from zo skipping timestamps 1 1:
(0x))x0)= = Naa;; Vorzo, (1 o+) and Se = / Crta + v Cett, 6 ~ (O,
(2)
When T large enough, FT ~ (0, I) can satisfied. Hence, the reverse process starts at samples from .N I). and the purpose is to recover zo (x). According to ELBO [7]. the target max logpe (zo) can be simplified to minimize the KL divergence between the reverse conditional distribution pa (x+-11 (xr)) and the posterior of the diffusion g( [[2 [I,, xo) which is formulated
1 5
q( vo) - (xxx (T,, 0o). B.I). 3, = - α

---

where by is a linear combination of Ie and To. The reverse conditional distribution can be ensured in Gaussian form po(2)-1(zz) ~ (2 po(a,, t)), Eo(z,, t) if B, are small. Thus, the key to KL divergence is the T22 distance between [4g and ft, which drives the denoiser network to learn Po to predict ji.. According to Eq. [ the loss of the denoiser training finally simplified to:
Lg Ex, co llle: calvonzo + v1 Cr )-
The essential of the denoiser is to predict the noise e, at any perturbed level & added to the real data, and then can recover the T0-
3.2 Dual Diffusion
Starting from (0, I) is highly time-consuming due the high diffusion of noise. 8 suggests that the 3D pose yr predicted by the fff-the-shelf methods is a kind of noisy data under the distribution with high uncertainty Vr, and diffusion models can be used reduce uncertainty to generate the accurate result yo- Starting from Drr with limited diffusion is more efficient. The major problem that the 3D pose initial uncertainty is unknown, which is solved by the statistical method in (8]. However, if the uncertainty of the 3D pose follows a Gaussian distribution is still confusing. And the statistical results can be easily influenced by the models and training dataset, posing challenges in practical applications. In this work, considering the binocular geometry framework, we design an elegant method to build uncertainty distribution of initial 3D poses from 2D uncertainty, and conversely, to achieve 2D denoising by denoising 3D poses, which we name Dual-Diffusion.
Forward Dual Diffusion process gradually adds noise to ground truth binocular 2D poses Ulo (u E /2 in Markov chain. Here, J is the number of joints and v E 0, 1 indicates the left and the right perspectives. The diffusion domain localized in 2D based on two reasonable assumptions: 1) the uncertainty distribution of initial 2D poses is known, and 2) it is in Gaussian form. We predict the initial 2D poses using a 2D detector. The training objective of this detector is to predict heatmaps of 2D joints whose supervision is Gaussian distribution centered at the ground truth with fixed standard deviation dr. The maturity of 2D detectors ensures the quality of the heatmap generation. Therefore, the initially predicted binocular 2D poses can be treated as the noisy data following the Gaussian distribution (0, of "I) around the ground truth 2D poses uo. This indicates that the diffusion target at T step should satisfy the distribution UT N(un, 2I). The formulation to create the diffused 2D poses can be written as:
= uo + V(1 āg)e,, E ~ NN
(5)
Here, xp and 3e in Eq. 2 are replaced by uo up and u4 uo correspondingly maintain the distribution consistency of Er. When t = T. uT ~ Ur is satisfied. The diffusion is transmitted to 3D space by Triangulation [9]. which is geometric tool to reconstruct 3D poses y e R from binocular 2D poses using camera parameters. The diffused 3D poses can be generated by:
yt = Tri((u,,, 1e,1);
(6)
where Tri represents the closed-form solver using Linear Triangulation. Up to now, the uncertainty distribution of initial 2D poses is determined. Although the initial uncertainty distribution Dr in 31) has not yet been formulated, it can be reconstructed by sampling from noisy 2D poses.
Reverse Dual Denoising should recover accurate binocular 2D poses uo from the estimated 2D poses ur according to the definition in diffusion models. However, we argue that 3D poses inherently exhibit more constrained distribution compared 2D poses. 2D poses may be various under different camera parameters and perspectives. Hence, we modify the denoising network to recover the original 3D poses yo- As above, the denoising is transmitted to the 2D plane by Reprojection. given the camera extrinsic and intrinsic matrix. Then the noise e, added to 2D poses in Eq. [] can be predicted, However, taking into account the depth ambiguity problem the correspondence between 3D and 2D noises, the denoiser objective in Eq. 4 is adjusted to predict the original data rather than noise, as stated:
L.g E., "g..1 Ng.1
Rpio (10))1, yo = fo(ye: 1):
where Rpj. is the Reprojection from perspective, and represents the denoiser network.
5

---

Z-embedding Condition. Even though the denoising formulation of diffusion modeling is to denoise the 2D noisy poses, the essential task of the denoiser focuses on removing noise from the 3D noisy poses under 1 perturbed level. We leverage the geometric and experiment analyses to explore the relationship between 3D and 2D uncertainty (details Appendix (A]. Our findings reveal that the 3D uncertainty range is not only influenced by the 2D uncertainty range but also relative to the depth z of the 3D point and the baseline width of the binocular setting. However, the diffused data in the fixed shares the same perturbed level, as defined in Eq. 2 To facilitate our denoiser to learn the noise within different uncertainty but under the same timestamp 6, we introduce the Ze, the absolute depth of root joint of 3D poses yt, as additional condition. This is based on the assumption that the Z, is close to the ground truth z because of the initial results with limited diffused. The denoiser is modified as:
yo = [o 24, L)-
rettl Pose Normalization. The depth z affects the noise along the x-axis, y-axis, and z-axis, while the baseline width only affects the noise on the z-axis. Considering the baseline width of binocular cameras will be changed in practical application. We propose simple method to normalize and denormalize the 3D poses under different baseline width settings:
Zz Biz,
Zy / B,
where Z9 shares the same definition with Zr, B is scalar representing the baseline width, and z is the normalized z. Through normalization and denormalization, the input to the denoiser is baseline width independent, but the 3D pose estimation results are not affected.
Inference. During the inference process, shown in Fig. Zb.). the initial binocular 2D poses ur estimated by 2D detector are the input to our Dual-Diffusion model. Subsequently, the initial 3D) pose yr is reconstructed and normalized, then fed into the denoiser to generate the plausible and accurate 3D pose jo. along with their corresponding binocular 2D poses ügg These poses üg are then diffuued to un using the DDIM strategy (37 for input to the next denoinsing. After iterative denoising for K times. The final 3D pose do and 2D poses uo are estimated. It should be noted that the 3D pose is converted to a root-relative format before being processed by the denoiser and then converted back afterward. For the experiments described below, we set T 25 and K according to the ablation study in Sec. 4.3 and Appendix D.3 The denoiser follows a GCN-Transformee structure. Detailed architecture and training information can be found in Appendix B
4 Experiments
Dual-Diffusion aims to reconstruct 3D pose from binocular 2D poses estimated from one off-the- shelf 2D detector. We refer to it as method for comparison is the Triangulation
POSE DETECTOR" in tables. The baseline g (Tri), referred to as "Tri-2D POSE DETECTOR". The
experiments are conducted on two benchmarks: 1) the short-baseline binocular benchmark, MHAD Berkeley dataset [27], and 2) the wide-baseline benchmark, H36M dataset [33] MHAD (27) is a multi-modal dataset that encompasses 11 actions performed by 12 subjects. We choose the binocular camera pairs (1 - 3, 2 4) in the LI quad camera, with approximate 200mm baseline width. H36M 13 is one of the most popular datasets for 3D HPE. To simulate the binocular setup, the camera pairs (1 - 3, 2 - 4) are selected, with about 3000mm baseline width. Mean Per Joint Position Error (MPJPE) is used to assess the accuracy, while Bone Length error (BL) and Symmetry error (Sym) employed to evaluate the plausibility of 3D poses. Joint Detection Rate (JDR) is applied to assess 2D poses. The details of the metric and implementation can be found in Appendix C
4.1 Comparison on MHAD
There are few methods designed particularly for binocular 3D HPE, except RSB-Pose 45 We reproduce the state-of- oftte-of-the-ane the-art multiview 3D HPE methods and fine-tune them in binocular datasets, including TPPT (24), Epipolar Tri [10]. Algebraic Tri [14] and AdaFuse [56]. Among them, RSB- Pose [45 utilizes Triangulation to lift 3D pose and refine it with Pose Transformer, Algebraic_ Tri [4] employs a weighted Triangulation to reconstruct 3D pose. Other methods focus on estimating the 2D poses from fused features in geometric or an attention mechanism based on the backbone ResNet [5 but the 3D pose is lifted solely through Triangulation. We use "METHOD NAME in tables to indicate the methods that primarily focus improving multiview 2D pose estimation.

---

Table 1: Quantitative Comparison on MHAD. Scale is the resolution of image input to the 2D pose detector. The best results are highlighted in bold, and the second results are underlined. The results of the baseline comparison are in light blue, while the results of
are in dark blue.
Method
Venue
Pose Detector
Scale
MPIIE
BL
Sym
J
(mm)
(mm)
(mm)
(56
TPPT [24]
ECCV222
TPPT*
256
209.03
134.05
248.93
RSB-Pos:50
urXiv'23 23
RSB-Pose50
256
32.10
10.21
12.13
96.62
Hipipolar_ Tri 10
CVPR 20
Epipolarr_ Tri*
256
90.73
33.67
34,21
Dual- Dual-Diffusion-Fa
Epipolar Tri*
256
76.42 14a1)
27.02 6.5.
26.42 7.79L
Tri Vil
NeurlPS'22 22
ViTPese [52
256
70.84
42.55
48.43
95.83
Dual-Diffusion VIT
ViTPose 52
256
61.02 9.82,
37.90 4.65
30.09 18.34)
95.88 pa5t
Tri-ResNet50
ResNet50
256
60.04
23.68
36.65
95.95
ResNe150
256
54.51 1.5%
18.09
24.64 12:011
98.86
Tri-RB550
RSB-Pose50*
256
35.40
11.36
14.25
96.62
Dual Diffusion
RSB-Pose50
444
9.60 1.6]
11.53 2.71L
Algehraic-Tr 14
ICCV
ResNet152 [51]
384
51.69
27.11
45.69
95,95
RSB-Pose 152 45
urXiv'23
RSB-Pose152
384
29.33
8,70
2.94
97.40
AdaPuse [66
UCC"20
AdaFuse
30.08
83.46
Dual- -ifaifussion-Ada
AdaFuse
256
53.77 16.30)
24.59 11
23.19 689L
95,37 11:911
Tri- Tri-ResNet152
15s (51)
384
48.26
19.22
27.73
95.95
Dual- Saal--iiffusio-ReeNtttl
ResNet152 [5]
384
43.57 4,69,
16.20
14.91 12:21
97.26 13:1
Tri-RSB152 Duul-DDiflusion-RS8155
RSB RSB-Pose155 15s
384
29.78
9.84
11.61
97.40
RSB--Pose152
384
27.76 2/00
7.56 2.241
9.83 1.78]
99,20 1.00
To evaluate the performance of our Dual-Diffusion in 2D-3D lifting, we employ state-of-the-ar binocular or multiview models 2D detectors. Results are shown in Table Our method generates more accurate and plausible 3D poses compared to the baseline. For instance, based on the 2D poses generated by ResNet50, Dual-Diffusion reduces the MPJPE by 5.53mm 9.2% error reduction), and the BL and Sym by 5.59mm (23.6%) and 12.01mm (32.8%) respectively. This improvement consistently observed with the 2D poses generated by RSB-Pose50* well. When the image resolution increases to 384, the 2D poses estimated are more accurate. Our method still outperforms the baseline with 2.02mmm (6.8%) in MPJPE, 2.28mmm (23.2%) in BL and 1.78mmm (15.3%) in Sym using the RSB-Pose152 2D detector. Additionally, Dual-Diffusin generates more accurate 2D results, with increases of 2.32% and 1.80% compared to the initial results of 2D detectors RSB- SSB-Posc50* and Dual-Diffusion can simultancously denoise both 2D and 3D noisy poses and generate more accurate and plausible results. Regardless of the input resolution, results across all four metrics. This demonstrates the effectiveness our approach in short-baseling binocular 3D HPE, making it promising for practical applications.
152*, respectively. The performance enhancement demonstrates
method consistently achieves the best
4.2 Comparison on H36M
In Table 2 the com-
Table 2: Quantitative Comparison on H36M. Params is the number of model parameters excluding the backbone.
parison is conducted on H36M [13]. Even though the uncertainty of initial 3D poses in the baselie binocular reduced compared the short-baseline setup 45), our Dual-Diffusion still outperforms the baseline and achieves the best results across all metrics. Based on the RSB-Pose50* 2D detector, Dual-Diffusion achieves an eтror reduction of 13.0%. 28.8%, 30.1% and 0.09% in MPJPE, BL, Sym of 3D poses and JDR of 2D poses, respectively. Even at 384 resolution, where other methods demonstrate effectiveness with about 30mm MPJPE results, our model still outperforms them by at least 1.87mm in MPJPE, achieving result of 28. m.. It is demonstrated that Dual-Diffusion can enhance 3D poses under bidebbas binocular configurations, showcasing its generalization capabilities.
Method
Scale
Params
(M)
MPJPE
(mm)
(mm)
BL.
Sym (mm)
JDR 1
TPPT 24
256
9.70
40.72
22.49
25.44
RSB-PoseSS
45
256
9.25
35.01
14.16
13.54
94.82
Fpipolar Thi
10
256
0.08
41.22
20.39
20.18
Ranl-Diffusion-F Epi
256
0.74
37.03 4.99
16.90 5490
18.08 am
fi-Ve
256
41.99
18.09
20.75
93.33
Dual- mall--iffsion VIT
256
0.74
35.20 629,
16.02 2001
19.66 1800
95.772.41
RSB50
256
38.13
Dunl-Dift Diff RiSf-
256
0.74
33.17
12.29 4.9.
11.75 1
94,91 0.91
Algebraic Tn [14
384
10.88
31.24
13.52
13.59
95.81
RSB 15s 45
384
9.25
30,07
13.33
12.86
95.93
AdaFuse (56
384
1.02
30.27
15.23
14,36
94.25
Dual- Diffusion-Ada
384
0.74
29.17 1100
13.85
13.57 6.70)
96.06
In-RSB152
384
30.54
13.65
13.42
95
Dual-Diff-RSBel
384
0.74
28.67 1.87L
12.06 1.5%)
12.35
18FL
95.97
7

---

Table 3: Impact of Each Module. Experiments are conducted on MHAD with 2D poses estimated from RSB-Pose1523 The first row the result generated by Tri.
Params + (M)
MACS (G)
MPJPE
BL
Sym +
Dual-Diff
Z-embedding
BaseL-norm
(mm)
(mm)
(mm)
X
X
29.78
9.84
11.61
X
0.74
0.42
28.20 1.584
8.81 1.034
11.23 0.381
X
0.74
0.42
27.91 1.781
7.85 1.991
10.12
1.491
0.74
0.42
27.76 2.02.
7.56 2.28.
9.83 1.78)
Table 4: Impact of BaseL-norrm The results MPJPE of 3D poses generated with 2D poses estimated from ResNet50.
Table 5: Comparison of Diffusion Models in MPJPE. The 2D poses are estimated from RSB-
Pose 152*. T is the overall diffusion steps.
Baseline width
Dual-Dift
T(=1)
25
50
75
100
125
(mm)
BaseL--arm
BaseL-nnorm
mando
328.82
270.51
234.41
135.72
70.19
97.57
10387
2D-Diff
29.40
29.55
28.81
31.09
29.25
300
$4.36
62.13
51.26
Dual-Diff
28.20
28.31
28.17
28.19
28.25
Additionally, Dual-Diffusion requires only 0.74 million parameters, significantly fewer than other methods. This demonstrates that the effectiveness of pose refinement in our method does not rely on large number of parameters, but rather the dual diffusion modeling and training.
4.3 Ablation Study
The ablation experiments are conducted on MHAD, as the high uncertainty issue is more pronounced in short-baseline setups compared to wide-baseline ones 45
Impact of Each Module. We first investigate the improvements provided by the pure Dual- Diffusion model described in Sec 3.2 denoted as "Dual-Diff" in the tables. Then, two additional modules, Z-embedding and nore,-norm, are assessed. As depicted in Table3 the pure Dual-Diff significantly enhances the accuracy of 3D poses, achieving a 5.3% reduction in MPJPE and 10.5% reduction in BL, indicative of its capability generate more precise and plausible 3D poses. With the incorporation of Z-embedding as an additional condition, there are further 10.9% and 9.9% relative improvements in BL and Sym. It is worth noting that subjects often perform actions while tilting towards the camera rather than facing it, leading to differences in the depth of joints on the left and right sides, resulting variations their 3D uncertainty regions. Z-embedding specifically designed to enhance the adaptability of the denoiser to different perturbed noise levels at the same time as demonstrated by the significant enhancement in Sym. Finally, the addition of the nose brings slight improvements in three metrics. Additionally, the computational cost is limited.
Zero-Shot 2D-3D Lifting. The purpose of BaseL-norm is to enhance the flexibility of the Dual- Diffusion to various baseline width settings. To evaluate this, we adapt two additional camera pairs (1-2,1-4) 4-4) on MHAD, representing binocular baseline width of 100mm and 300mm, and conduct zero-shot 2D-3D lifting on them. Firstly. we employ ResNet50 to detect 2D poses. Then, the denoiser trained on 200mm-baseline training dataset is directly utilized to generate 3D poses on 100mm-baseline and 300mem-baseline testing sets without fine-tuning. The results are presented in Table Without BaseL-norm. the accuracy even worsens. But with BaseL-norm, there are 4.25mm and 3. Immm reductions in MPJPE for two settings, respectively. The improvement illustrates the zero-shot adaptability of the BaseL-norm module to various baseline widths, which is beneficial for further application practice.
Efficiency of the Uncertainty Distribution Initialization. We the initialized uncertainty distri- bution as the target distribution for diffusion instead of random noise. To evaluate its efficiency, we establish a baseline diffusion model that generates 3D poses by denoising random noise conditioned on 2D binocular poses, referred to as "random-noise" To mitigate the effect of camera projection matrices P., we translate 2D keypoints to 3D-aware vectors by P.- The comparison results are shown in Table 5 Under the same inference iteration K = 1, across the diffusion steps from 25 to 125, the random-noise consistently yields inferior results, while Dual-Diff achieves promising results even with T 25. A smaller T' indicates reduced time cost of training and inference.
8

---

Table 6: Validation of 3D Uncertainty Distribution Modeling, The results are MPJPE of 3D poses denoised by the denoiser trained with the MHAD training set.
2D Pose Detector
Dataset
3D Posc
MPJPE (mm)
2D Posc Detector
Dataset
3D Pose
MPJPE (mm)
estimated
13,93
estimated
10.96
ResNet152 [1
training
GT+noise
17.07
RSB-Pose1522
45
training
GT++noise
11.95
estimated
43.57
estimated
27.76
testing
testing
GT+noise
17.51
Go+so
12.46
Table 7: Comparison of Uncertainty Reconstructing and Uncertainty Statistics. The 2D poses are estimated form ResNet152.
Setting
Method
MPJРE (mm)
Sctting
Method
MPIPE (mm)
training in small-dataset
Tri
40.39
training in large-dataset
In Dual-Diff
38.17
and testing in large-dataset
Dual-Diff
39.11
and testing small-
35.23
DiffPose
8
54.12
DiftPPse
40.62
Moreover, there remains some ambiguity regarding whether the initial uncertainty distribution of the 3D estimation is effectively modeled. To investigate this, we perform denoising on simulated noisy 3D poses. Specifically, we first calculate the error between the 3D estimation and the 3D GT along each axis in the MHAD training set, storing this as the noise set. Then, we add noise sampled randomly from this set to the 3D GT along each axis. Finally, we use the denoiser to refine both the
"GT + noise" and "estimated" 3D poses, comparing the MPJPE results. The hypothesis is that if the 3D uncertainty is well-modeled, the performance in refining both estimated 3D poses and simulated 3D poses should be similar. As shown Table regardless of the 2d pose detector, the accuracy of denoised "GT+noise" in both training and testing sets is all close to the "estimated" in the training set, demonstrating that the 3D uncertainty distribution is well-modeled.
Uncertainty Reconstructing v.s. Uncertainty Statistics. We argue that reconstructing 3D pose uncertainty from 2D) pose uncertainty is more practical approach. As discussed in Appendix A the depth uncertainty of a 3D point increases with greater depth. Consequently, the statistical approach in DiffPose [8] tends to constrain the model to a narrow depth range, while Dual-Diffusion leverages more reliable 2D results. To evaluate this, we divide the MHAD into two subsets: large-dataset and small-dataset, based on the average depth of 3D poses and compare Dual-Diffusion with DiffPose. The results are illustrated in Table Compared to the Triangulation baseline, performance improves with Dual- Diffuusifn but decreases with DiffPose. DiftPose suffers from the change of 3D poses uncertainty distribution while our Dual-Diffusiin remains stable. This stability is due to the fact that Dual-Diffuse models the diffusion target using 2D uncertainty, which is significantly more stable than 3D uncertainty. The comparison of the stability in uncertainty between 21) poses and 3D) poses can be found in Appendix [D.3
3D Pose Priors v.s. 2D Pose Priors. We argue that 3D pose priors are more easily captured compared to 2D pose priors be- cause 2D poses vary under different perspectives. To validate this, we establish a diffusion model directly for denoising 2D poses, named "2D-Diff", and then reconstruct 3D poses. As shown in Table 5, Dual-Diff consistently outperforms 2D-Diff in terms of 3D pose accuracy. Furthermore, we compare the plausibility of 3D poses. As illustrated in Fig. Dual-Diff ex- ceeds DifD in both BI. and Sym. These results collectively demonstrate the necessity of denoising in the 3D domain.
200
Figure 3: Dual-Diff (red) v.s. 2D- Diff (blue) under various T and K=
4.4 Visulization
Dual-Diffusion Denoises the Binocular 2D Poses. To understand the denoising process of binocu- lar 2D poses, we conduct simulation experiment. First, we gradually add noise to the ground truth 2D poses over = 25 steps and generate the final noisy 2D poses. Then, we set the reverse iteration K = 25 to recurrently denoise the noisy poses. Fig. iillustrates the absolute distance between noisy data and the ground truth at each step / in diffusion and each iteration & in denoising. The noise added to the joint during the diffusion process is incrementally removed during the denoising process.

---

diffusion
denolsing
Figure 4: Step-wise errors of binoculrr 2D joints, (u, v)teft and (u, Dright, during the diffusion and denoising processes. The joint analyzed is the right knee.
Triangulation
-Ss-P
Triangulation
RSB-Poue ASB-Pose
Ours
H36M
(b) MHAD
Figure 5: Qualitative Comparison with Triangulation and RSB-Pose in 3D Pose Estimation. 2D poses are estimated by RSB-Pose152* The gray skeleton is the ground truth, while the black represents the estimates. Red and blue points correspond to joints on the right and left sides, respectively. Yellow arrows indicate parts of significant improvement achieved by our method.
Dual-Diffusion Denoises the 3D Poses. We provide qualitative comparison with Tri-RSB15 1522 and RSB-Posel52 in Fig. [ Our mull-Diffusion achieves more accurate 3D poses. particularly in cases of self-occlusion. For instance, in the 3rd row of the 1st column, the right hip is occluded, resulting in an inaccurate 3D pose using baseline Triangulation. Dual-Diffusion effectively corrects the 31) pose. This highlights our method's ability to denoise noisy 3D poses. Additionally, when baseline results are poor, such as in the 1a row of the 2nd column, RSB-Pose can only partially correct some joints, whereas method corrects the entire right skeleton. More visualization is in Appendix [D.4
5 Conclusion and Discussion
This work introduces a novel framework, Dual-Diffusion, to reconstruct 3D poses from 2D poses estimated by off-the-shelf 2D pose detectors binocular configuration. Dual-Diffusion simulta- neously denoises initial 2D and 3D poses within single diffusion model. The diffusion process operates on 2D poses, while the denoising process occurs in 3D space, utilizing geometric mapping to connect the 2D and 3D domains. Comparisons with state-of-the-a methods demonstrate that our approach effectively denoises both 2D and 3D poses, yielding superior results and making particularly suitable for binocular 3D HPE, especially in short-baseline configurations.
Discussion. We further extend the applicability of Dual-Diffusion to multiview settings, as shown in the Appendix [.2] The observed performance improvements validate the scalability of our method. Additionally, we incorporate 3D supervision (see Appendix D.3) revealing further advantages that will be explored in future work. However, we also acknowledge two main limitations of our method. Firstly, while simulation experiments are conducted to validate our hypothesis regarding 3D uncertainty factors (see Appendix A.. for details), computing the boundary points of the 3D uncertainty region to explore may lack rigor. We plan to leverage an algebraic approach to derive the range of the 3D uncertainty caused by 2D errors in future work. Secondly, our denoising network takes the root-relative 3D pose as input, neglecting optimization of the root joint. Although the root joint is finally refined (see Appendix D.I). another module to optimize the root joint is preferred.
10

---

Acknowledgement
This work has been funded in part by the NSFC grants 62176156 and the Medical Engineering Cross Research Fund of Shanghai Jiao Tong University (YG2023ZD12).
References
Ijaz Akhter and Michael J Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1446-1455 2015.
[2] Soubarna Banik, Alejandro Mendoza Garela, and Alois Knoll. 3d human pose regression using graph convolutional network. In 2021 IEEE International Conference on Image Processing (ICIP), pages 924 928. IEEE, 2021.
[3] Magnus Burenius, Josephine Sullivan, and Stefan Carlsson. 3d pictorial structures for multiple view articulated pose estimation. 2013 IEFE Conference on Computer Vision and Pattern Recognition, pages 3618-3624 ISSN: 1063-6919.
[4] Xipeng Chen, Kwan-Yee Lin, Wentao Liu, Chen Qian, and Liang Lin, Weakly-superrised discovery of geometry-aware representation for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10895-10904, 2019.
[5] Zhuo Chen, Xu Zhao, and Xiaoyue Wan. Structural triangulation: A closed-form solution to constrained 3d human pose estimation. In Shai Avidan, Gabriel Brostow. Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, Lecture Notes in Computer Science, pages 695-711. Springer Nature Switzerland.
[6] Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang. Gfpose: Learning 3d human pose prior with gradient fields. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4800 4810, 2023.
[7] Mohsen Gholami. Bastian Wandt, Helge Rhodin, Rabab Ward, and Z Jane Wang. Adaptpose: Cross-dataset adaptation for 3d human pose estimation by learnable motion generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13075-13088 13085, 2022.
[8] Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. DiftPose: Toward more reliahle 3d pose estimation. In 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13041-13051. IEEE.
Richard Hartley and Andrew Zisserman, Multiple View Geometry in Computer Vision Cambridge University Press, 2 edition, 2004.
[10] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. Proceedings of the ieee/cyf conference on computer vision and pattern recognition, pages 7779-7788, 2020.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6853 2020.
[12] Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 15597 1,587, 2023.
[13] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133 1339- 2013.
[14] Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkow. Learnalee triangulation of human pose. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1718-7722 2019.
[15] Boyuan Jiang. Lei Hu, and Shihong Xia. Probabilistic triangulation for uncalibrated multi-view 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14850- 14850-14860, 2023.
[16] Zhongyu Jiang. Zhuoran Zhou, Lei Li, Wenhao Chai, Cheng-Yer Yen Yang. and Jenq-Neng Hwang. Back to optimization: Difffusion based zero-shot 3d human pose estimation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 6142 2152,112,,10 2024.
11

---

[17] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[18] Muhammed Kocabas. Salih Karagoz, and Emre Akbas. Self-supervised learning of 3d human pose using multi-view geometry. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 1077-1086, 2019.
[19] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Danillidis. Probabilistic modeling for human mesh recovery. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11605-116/1 2021.
[20] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. HybrIK: A hybrid analytical- neural inverse kinematies solution for 3d human pose and shape estimation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3382-3392. IEEE.
[21] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: hybrid analytical- neural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3383-3393, 2021.
[22] W Li, Liu, H Tang, Wang, and MHFormer Van Gool. Multi-hypothesis transformer for 3d human pose estimation. arxiv 2021. arXiv preprint 1rXiv:2111.12700
[23] Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. Deformable shape completion with graph convolutional autoencoders. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1886-1895, 2018.
[24] Haoyu Ma, Zhe Wang. Yifei Chen, Deying Kong, Liangjian Chen, Xingwei Liu, Xiangyi Yan, Hao Tang. and Xiaohui Xie, Ppt: token-pruned pose transformer for monocular and multi-view human pose estimation. In European Conference on Computer Vision, pages 424 4424. Springer, 2022.
[25] Julieta Martinez, Rayat Hossain, Javier Romero, and James Little. simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision, pages 2640-2644 2017.
[26] Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In Computer Vision-ECCO 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part VIII 14, pages 483-499. Springer, 2016.
[27] Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena Bajesy. Berkeley mhad: A comprehensive multimodal human action database. In 2013 IEEE workshop applications of computer vision (WACV), pages 53-60. IEEE, 20133
[28] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[29] Georgios Pavlakos, Vasileios Choutas, Nima Ghorhani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10975-10985, 2019.
[30] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniliids. Coarse-to fine volumetric prediction for single-image 3d human pose. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1263-1272. IEFF.
[31] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Danillidis. Harvesting multiple views for marker-less human pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6988-6997, 2017.
[32] Haibo Qiu, Chunyu Wang, Jingdong Wang. Naiyan Wang, and Wenjun Zeng. Cross view fusion for 3d human pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4342 0422-331, 2019.
[33] Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, and Robert Wang. Lightweight multi-view 3d pose estimation through camera-disentangle representation. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pages 6040-6049, 2020.
[34] Cédric Rommel, Eduardo Valle, Mickaël Chen, Souhaiel Khalfaoui, Renaud Marlet, Matthieu Cord, and Patrick Pérez. Diftbpe: Robust, coherent 3d human pose lifting with diffusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1220-3229 2023.
12

---

[35] Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. P-stmo: Pre- trained spatial temporal many-to-one model for 3d human pose estimation. In European Conference on Computer Vision, pages 461-478, Springer, 2022.
[36] Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang. Kai Han, Shanshe Wang, Siwei Ma, and Wen Gao. Diffusion-based 3d human pose estimation with multi-hypothesis aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14761 117711, 2023.
[37] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2XXi::0010002002. 2020.
[38] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[39] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma. Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
[40] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high- righurtcolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (693357033 2019.
[41] Zhenhua Tang, Zhaofan Qiu, Yanbin Hao, Richang Hong. and Ting Yao. 3d human pose estimation with spatio-temporal criss-cross attention. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4790-4799, 2023.
[42] Bugra Tekin, Isinsu Katircioglu, Mathieu Salzmann, Vincent Lepetit, and Pascal Fua. Structured prediction of human pose with deep neural networks. arXiv preprint 160Xiiv:60000555 10800 2016.
[43] Denis Tome, Chris Russell, and Lourdes Agapito. Lifting from the deep: Convolutional pose estimation from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1550025099 2017.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[45] Xiaoyue Wan, Zhuo Chen, Yiming Bao, and Xu Zhao. Rsb-pose: Robust short-baseline binocular 3d human pose estimation with occlusion handling. arXiv preprint arXiv:2311. 14222,, 2023.
[46] Xiaoyue Wan, Zhuo Chen, and Xu Zhao. View consistency aware holistic triangulation for 3d human pose estimation. Computer Vision and Image Understanding, 236:103830, 2023.
[47] Bastian Wandt and Bodo Rosenhahn. Repnet: Weakly supervised training of adversarial reprojection network for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7782-7791, 2019.
[48] Jian Wang, Lingjee Liu, Weipeng Xu, Kripasindhu Sarkar, and Christian Theobalt. Estimating egocentric 3d human pose in global space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15500 1 11509, 2021.
[49] Jinbao Wang, Shujie Tan, Xiantong Zhen, Shuo Xu, Feng Zheng, Zhenyu He, and Ling Shao. Deep 3d human pose estimation: review. Computer Vision and Image Understanding, 210: 210:103225, 2021.
[50] Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular human pose estimation with normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11 1199-11200, 2021.
[51] Bin Xiao, Haiping Wa, and Yichen Wei. Simple haselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 466-481, 2018.
[52] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571-333 -38584,, 2022.
[53] Youze Xue, Jiansheng Chen, Xiangming Gu, Huimin Ma, and Hongbing Ma. Boosting monocular 3d human pose estimation with part aware attention. IEEE Transactions on Image Processing, 114278--29) 2022.
13

---

[54] Petrissa Zell, Bastian Wandt, and Bodo Rosenhahn. Joint 3d human motion capture and physical anal- ysis from monocular videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pages 17-26, 2017.
[55] Jinlu Zhang, Zhigang Tu, Jianyu Yang. Yujin Chen, and Junsong Yuan. Mixste: Seq2seq mixed spatio- temporal encoder 3d human pose estimation video. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 33232-13242 2022.
[56] Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, and Wenjun Zeng. Adafuse: Adaptive multiview fusion for accurate human pose estimation in the wild. International Journal of Computer Vision, 129:703- 718, 2021.
[57] Long Zhao, Xi Peng. Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Semantic graph convolutional networks for 3d human pose regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3425 3435, 2019.
[58] Qitao Zhao, Ce Zheng, Mengyuan Liu, and Chen Chen. A single 2d pose with context is worth hundreds for 3d human pose estimation. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[59] Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen, Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8877-8886, 2023.
[60] Weixi Zhao, Yunjie Tian, Qixiang Ye, Jianbin Jiao, and Weiqiang Wang. Graformer: Graph convolution transformer for 3d pose estimation. arXiv preprint Xrx:2 109, 08364, 2021.
[61] Jieming Zhou, Tong Zhang, Zeeshan Hayder, Lars Petersson, Mehrtash Harandi. Diff3dhpe: diffusion model 3d human pose estimation. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2092-2102, 2023.
[62] Xingyi Zhou, Qixing Huang. Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards 3d human pose estimation the wild: a seaell-- approach. In Proceedings of the IEEE international conference computer vision, pages 398 2017.
14

---

A Relationship between 3D and 2D Uncertainty
A.1 Theoretical Analysis
We simplify the relationship between 3D and 2D uncertainty to the relationship between 3D uncer- tainty range Ax, Az, Az' with 2D uncertainty range Au. We analyze it in the X-0-Z cross- sectsonection of 3D space, as shown in Fig. 6 O and O, are the optical centers of binocular cameras which are rectified and their image plane parallel to the x-axis. The red point H is the ground truth of 3D point with depth z. The analysis is based on the hypothesis that the uncertainty range the estimated 2D point the same in 2 views with Au along the x-axis. The ground truth 2D point is the projection of H.
Point A represents the intersection of the left boundaries of the 2D uncertainty regions from both viewpoints and serves as the left boundary along the x-axis in the 3D uncertainty region. B denotes the intersection of the right boundaries from both viewpoints, forming the right boundary along the x-axis in the 3D space. C' is the intersection between the right boundary of the 2D uncertainty region from the left view and the left boundary of the 2D uncertainty region from the right view, establishing the lower bound along the z-axis of 3D uncertainty. Conversely, point D marks the upper bound along the z-axis.
A2
Uncertainty along the x-axis. To analysis the relation- ship between Ax and Au, we first prove that line segment AB is parallel to the image plain. Based on the disparity formula
f /B/ Ad. A and H have the same depth z since they share the same disparity. and similarly for points B and H. Therefore, line segment AB passes through H and is parallel to the image plane. Then, the segment AH can be derived using similar triangles:
Figure 6: 3D reconstructing uncertainty range of binocular configuration.
Ax Au
(10)
f
Uncertainty along the z-axis. We first prove that the midpoint E of O,O, and lie on the line DC. Through point C, draw line parallel to the image plane, intersecting O,D and O, at points F and G respectively. Line DC intersects AB and O,Or at points E' and H'. According to the properties of similar triangles, FC GC. Furthermore, cause ADFC DOOE and A DGC ~ AD.,E' we can deduce that O,E" = O,E'. Similarly, it can be proven that AH' BH'. Then, the segment Az can be derived using similar triangles:
Az Ax
(11)
Az +
B/2
Combining with Eq. 10 it can be written as:
1
Bf 1
(12)
Az
2z
Au
Z
The segment Az' can be derived in the similar way:
1
Bf 1
1 z
(13)
Az'
22
Au
We omit the proof of Ay as it follows the same reasoning as Ax, given that the standard deviation along the u-axis and v-axis in a 2D image plane are equivalent. In summary, the factors contributing to the 3D uncertainty range include the 2D uncertainty range, the depth of the 3D point, and the baseline width. The 3D depth affects uncertainty along the x-axis, y-axis, and z-axis. The baseline width only affects the z-axis uncertainty.
15

---

Stand (4) with depth changing
with baseline width changing
Figure 7: Uncertainty distribution of 3D point as (a) depth or (b) baseline width changes.
A.2 Experiment Analysis
The analysis above solely relies on the boundary points to explore the relationship between 3D and 2D uncertainty, which We change the point depth from 500mm to 3000mm with 500mm step to discover the relationship between 3D uncertainty and depth. The binocular 2D points within a fixed uncertainty range are sampled randomly and then reconstructed as a 3D point using Triangula- tion. We sample for 100000 times at each depth and then analyze the 3D uncertainty range, visualized in Fig. a[). The 3D depth affects the uncertainty of 3D point along the x-axis, y-axis, and z-axis but with different relationships. Hence, we add z-embedding as additional condition to guide the denoiser. The experiment investigating the impact of baseline width is depicted in Fig bb). Baseline width solely affects the uncertainty distribution along the z-axis. As depicted in Eq 12 and [4[13] aside from the constant term, introduce normalization factor B to standardize the error distribution across different baseline widths. The distribution with and without normalization is illustrated Fig. g
rigorous. Hence, we conduct the experiments to validate the conclusion.
Figure 8: The difference in z-axis un- certainty distribution between with and without normalization as baseline width changes.
B Denoiser Architecture and Training Details
Denoiser Architecture. The denoiser architecture, illustrated in Fig. 9 is similar to the one proposed by (8) and essentially follows GCN-Transformer structure. The 3D pose, represented as joint locations, is first input into GCN layer. The graph for the GCN is defined by the skeleton connections, which encode the topological features of each joint. Next, five stacked GCN-Attention modules are used to enhance the global-local perception of the joint features. Fach GCN-Attentin module comprises Attention layer and two GCN layers. Each Attention layer includes 4-head self-attention mechanism. Timestamp embedding and Z-embedding are added to each GCN-Attention module. The embedding schedule sinusoidal (44) Finally, the 3D pose is output after passing through another GCN layer.
Training Details. The training process is depicted Fig. We first generate diffused data for training as follows: 1) Initialization: Start with the ground truth 2D poses ug. 2) Noise Addition:
16

---

Training
Lg Loss
diflase
Z 2, embedding
Timestamp embedding
%
9
denoise
Denoiser
Tranguuation
X5
Projectios
Attention Layer
GCN Layer
GCN Layer
CCN Layer
Layer
BaseL norm
BaseL-desom
Figure An overall training process and denoiser architecture for Dual-Diffusion.
Randomly choose a timestamp t E [1, T). add noise to generate noisy 2D poses u4. 3) Triangulation: Use triangulation reconstruct the noisy 3D poses yr. Then we denoise recover the data through the denoiser: 1) Denoising 3D Poses: The denoiser takes the noisy 3D poses y/r as input and denoises them recover original 3D poses 5o- BaseL-norm is applied before the denoiser and BaseL-denorm is applied after. 2) Reprojection: Reproject the denoised 3D poses back to 2D poses Bg. The training objective follows the Lg Eq. 7 facilitate the denoiser recovering the original input.
The Qual-Diffussinn is implemented by Py Torch (88] using Adam optimizer with learning rate 0.00002 and other parameters are all default. We train Dual-Diffusion in MHAD 27 for 80 epochs, and in H36M 13 for 40 epochs, respectively. There is no other schedule for the learning rate employed. All experiments conducted on GeForce RTX 2080 Ti.
C More Details of Experiment Implementation
Evaluation Metrics. To evaluate 3D pose accuracy, we utilize Mean Per Joint Position Error (MPJPE). Besides, we employ metries assess the plausibility of the 3D pose: Bone Length error (BL), which measures the average distance between the predicted and ground truth bone lengths; and Symmetry (Sym) 34I, which quantifies the average difference between the lengths of corresponding left and right bones. The Joint Detection Rate (JDR) is applied for 2D pose evaluation with the detection threshold set 2.5% the bounding box width.
Implementation Details. We choose ResNet [5]] and RSB-Pose 45 as our 2D pose detectors and divide them into two categories with different input image resolutions. To acquire the initial uncertainty distribution of estimated 2D poses, the methods are distinct for different 2D detector. For ResNet, the standard deviation are set and 3 separately for 256 and 384 image resolution, as the training objective formulating, For RSB-Pose", we obtain the standard deviation by statistically analyzing the estimation results on the training set. Because its training objective is MPJPE but not heatmap. Then, the standard deviation is scaled by image resolution to acquire ør. The diffusion step T is set 25, and the reverse step K is set to decided by the ablation study in Sec. 4.3 and Appendix D.3 To acquire the deterministic results, the reverse sampling is followed to the ODEs
D
More Experiments
D.1
Comparison MHAD
compare the per-joint error with the baseline using the same 2D pose detector. The results are shown in Table 8 Our Dual-Diffusion method surpasses the baseline in most joints, particularly
17

---

Table 8: Quantitative Comparison of Per-Joint Error on MHAD. Scale is the resolution of image
input to the 2D pose detector. The column in green is the root joint.
MPJP
Scale
Shiddr
Elhow
Wrist
Hip
Knee
Ankle
Pelvi
Belly
Nock
Nosc
Head
Avg.
(mm)
TH-RRSB0
256
29.52
41.76
49.20
25.12
22.56
24.95
103.20
25.19
29.20
27.98
30.68
35.40
Dual--Difusioo-R8550
256
26.18
24.21
21.54
30.54
40.33
23.12
25.11
26.40
30.96
Tri-RSB152
384
29.06
37.81
48.56
25.02
17.85
23.46
41.54
24.30
25.66
23.35
28.83
29.78
Dual-DDifusiooRSBB155
384
26.26
35,95
23.72
17.09
31.22
23.95
22.79
28.09
27.76
Table 9: Applicability to Multiview Settings. The results are MPJPE of 3D poses denoised from the
initial estimated 3D poses using ResNet152 2-view, 3-view, and 4-view H36M testing sets.
View
SIPIPE
Nym
J0K
View
MPPP
BL
sym
10K
View
XPIPE
BL
N5m
(mm)
(um
P9)
tnmm
(mu
(mm))
(em
Table 10: Impact of Baseline Width to Table 11: Impact of Depth to Uncertainty. The 2D poses are estimated by ResNet152. STD is the standard deviation.
Uncertainty.
Baseline width
2D MPJPE
3D MPJPE
2D MPIPE
3D MPIPE
3D MPIPE
(mm)
(pixel)
(mm)
Dateset
(pixel)
(mm)
STD (mm)
100
3.084
120.37
large-datasct
5.26
40.39
1025.88
200
3.079
63.18
300
3.079
54.69
small-dataset
5.33
38.17
161.88
the root joints and those close to the root. Interestingly, even though the input to the denoiser is the root-relative 3D pose, the root joint itself is still optimized as highlighted in green. We hypothesize that this optimization occurs due to the correlation established with the connected joints, which demonstrates that the pose prior effectively captured.
D.2 Applicability Multiview Settings
To evaluate the applicability of Dual-Diffusion in multiview settings, we conduct experiments by denoising the initially estimated 3D poses using ResNet152 across 2-view, 3-view, and 4-view configurations of the H36M testing set. The results, as shown in Table | demonstrate that Dual- Diffusion improves performance compared the Triangulation baseline. However, the degree of improvement decreases as the number of views increases. This aligns with our explanation that 3D uncertainty more ambiguous binocular settings than in 3-view or 4-view setups, where uncertainty is reduced, making denoising less impactful.
D.3 Ablation Study
Uncertainty Stability Comparison. We conducted comparison of the stability in uncertainty between 2D and 3D poses. First, we evaluated the 2D MPJPE and 3D MPJPE estimated by ResNet50 under binocular settings with varying baseline widths. as shown in Table [0] Additionally, we compared the 2D and 3D uncertainties obtained from ResNet152 across different depths, detailed in Table IT The results clearly demonstrate that 2D pose estimation more robust than 3D pose estimation.
Impact the Inference Iteration Time. To explore the impact of the inference iteration time K, we denoise the 3D poses for E (1, 5, 10, 20} iterations and compare the MPJPE results. When K 1, we achieve the best result, as shown in |2 Additionally, our Dual-Diff outperforms the 2D-D across all K, demonstrating the effectiveness of mod- eling pose priors in the 3D domain.
Table 12: Impact of Inference Iteration Times K.
The results are based on RSB-Pose 152*
K (T=25)
2
10
20
Dual Diff
28.20
28.36
28.80
28.97
29.08
2D-Diff
29.40
33.75
35.75
39.15
40.30
Table 13: Impact of Supervision. The experi-
ments are conducted in MHAD, based on RSB-
Pose152*
Loss
MPJPE (mm)
BL (mm)
Sym (mm)
2D
27.76
7.56
9.83
Impact of Supervision. We implement ex- periments by using 3D supervision alone and in combination with 2D supervision to retrain
3D
29.38
8.62
10.57
2D+3D
27.73
7.43
9.43
18

---

Triangulation
Pose
Triangulation
Figure 10: Qualitative Comparison on H36M. 2D poses estimated by RSB-Pose152* The gray skeleton is the ground truth, while the black represents the estimates. Red and blue points correspond to joints on the right and left sides, respectively, Yellow arrows indicate parts of significant improvement achieved by our method.
Triangulation
RSB-Pose
Ours
Triangulation
RSB-Pose
Ours
Figure 11: Qualitative Comparison on MHAD. 2D poses are estimated by RSB-Pose 152#, The gray skeleton is the ground truth, while the black represents the estimates. Red and blue points correspond joints on the right and left sides, respectively. Yellow arrows indicate parts of significant improvement achieved by our method.
Dual-Diffusion, based on the RSB152 backbone. As shown in Table [3] adding 3D supervision additionally enhances 3D accuracy and plausibility. Considering that Dual-Diffusion still has room for improvement, future work will focus on addressing the limitations discussed in the paper and exploring additional enhancement methods.
D.4 Visualization
We provide a qualitative comparison with Triangulation and RSB-Pose on H36M and MHAD, as illustrated in Fig 10 and Fig IT respectively. Our Dual-Diffusion method noticeably corrects self- occluded joints MHAD. In the wide-baseline H36M, although the joints occluded in both views are minimal, our method still achieves enhancements compared to Triangulation and RSB-Pose
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are clearly stated in the abstract and the introduction. The contribu- tions are summarized the last paragraph the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the limitations paragraph of Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
The assumption about 2D uncertainty distribution of Dual- Diffuusiooo is pro- vided in Forward Dual Diffusion paragraph of Sec. 3.2 The proof of the relationship between 3D and 2D uncertainty is provided in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Dual-Diffusion model is clearly described in Sec. 3.2 The training details and the implementation details are provided in Appendix and Appendix C respectivvey.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and models are available at https//ggthab.com//sierryww/auuu
Diffusion
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
ooiicyllyyyccc for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle wwwppppgpuuiiieeeee /otpc//uuuiee//oeeuubiiiiii SooodeSubiissiee ssioopolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The implementation details are provided in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The metrics are clearly defined in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The model is implemented by PyTorch and the computation cost is illustrated in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neur//: wttpuuuis..cpppullii.eee httppplcc//thcssuuuiuulpe wwwhhhhhhhiieeeeeeeeee
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our Dual-Diffusion is specifically designed for binocular 3D HPE which has positive societal impacts due its advantages, which is discussed in the first paragraph of
Sec.
1
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original paper and datasets are all cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Z4PzT8GWcp", "reviewer": "Reviewer_cBDy", "summary": "This paper presents a method for 3D human body keypoint estimation from binocular images. Different from traditional multi-view settings, such methods only take two views as input, which suffer from larger uncertainty at detph wise. To alleviate the depth ambiguity, the paper describes a diffusion-based framework. The forward diffusion process is used to simulate the noisy 2D poses from GT 2D poses. The reverse denoising process refines the 3D poses obtained by triangulation of noisy binocular 2D poses. The denoised 3D poses are then projected back to 2D poses to make the loop. The proposed method has been added to previous SOTA method, RSB-Pose, and improves the performance on MHAD and Human3.6M in binocular settings.", "questions": "1. Could you provide more evidence to further verify the generalization ablity of the proposed refinement method, like equipping more SOTA method?\n2. The in-depth discussion about how the proposed method models the 3D keypoint uncertainty. Is there any evidence to prove this?\n\nIn rebuttal, these questions have been well addressed by the feedbacks.", "limitations": "No.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of using diffusion process to simulate noisy 2D poses and to refine the 3D poses via denising is interesting. The insights behind these designs have been clearly explained and sound reasonable. \n2. Experiments of adding the proposed method to previous method in Tab. 1 and 2 verify the effectiveness of the proposed method. The detailed ablation study futher shows the rationality of the architecture design. \n3. The paper is easy to understand and clearly written.", "weaknesses": "1. Relatively limited experiments. The proposed method is mainly added to one previous method, RSB-Pose. More such experiments will make the paper be more convincing. \n2. Lacking discussion about whether the proposed method can effectively model the 3D keypoint uncertainty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer cBDy,\n\nFirst and foremost, we extend our deepest gratitude for your thorough review and insightful feedback. Your recognition of our insights, experiments, and writing is appreciated. The suggestions to add more experiments and discuss 3D uncertainty modeling are particularly valuable. We have addressed these suggestions by conducting additional experiments. The detailed response is provided below:\n\n**1. More Experiments to verify the generalization of the Dual-Diffusion.**\n\nIn our manuscript, we added the Dual-Diffusion approach to two prior 2D pose detectors, RSB-Pose* and ResNet. To further verify the generalization of our method, **we extended the evaluation to include two additional methods, Epipolar_Tri and AdaFuse**, across both MHAD and H36M datasets. Additionally, **we added a transformer-based 2D pose detector, ViTPose**. The results are as follows:\n\n|Dataset|Method | 2D Pose Detector | Scale|MPJPE (mm) | BL (mm) | Sym (mm) | JDR (%)|\n|:-|:-|:--:|:--:|:--:|:--:|:--:|:--:|\n||Tri-ViTPose|ViTPose|256|70.84|42.55|48.43|95.83|\n||Dual-Diffusion-ViT|ViTPose|256|**61.02**|**37.90**|**30.09**|**95.88**|\n|MHAD|Epipolar_Tri|Epipolar_Tri*|256|90.73|33.67|34.21|-|\n||Dual-Diffusion-Epi|Epipolar_Tri*|256|**76.42**|**27.02**|**26.42**|-|\n||AdaFuse|AdaFuse*|384|70.27|36.07|30.08|83.46|\n||Dual-Diffusion-Ada|AdaFuse*|384|**53.77**|**24.59**|**23.19**|**95.37**|\n|||||||||\n||Tri-ViTPose|ViTPose|256|41.49|18.09|20.75|93.33|\n||Dual-Diffusion-ViT|ViTPose|256|**35.20**|**16.02**|**19.66**|**95.77**|\n|H36M|Epipolar_Tri|Epipolar_Tri*|256|41.22|20.39|20.18|-|\n||Dual-Diffusion-Epi|Epipolar_Tri*|256|**37.03**|**16.90**|**18.08**|-|\n||AdaFuse|AdaFuse*|384|30.27|15.23|14.36|94.25|\n||Dual-Diffusion-Ada|AdaFuse*|384|**29.17**|**13.85**|**13.57**|**96.06**|\n\nThe supplemented experimental results further demonstrate that Dual-Diffusion is capable of generating more accurate 2D and 3D results regardless of the 2D pose detector and input scale. We have updated the above results to Tab. 1 and 2, which are included in the PDF of the global response.\n\n**2. Discussion about the 3D uncertainty modeling.**\n\nWe appreciate the suggestion to include an analysis and discussion of 3D uncertainty modeling, as it is crucial for strengthening the paper. We model the 3D pose uncertainty distribution by reconstructing from noisy 2D binocular poses and trained a Denoiser to refine the noisy 3D poses. **The hypothesis is that if the 3D uncertainty is well-modeled, the Denoiser's performance in refining both estimated 3D poses and 3D poses sampled from the uncertainty distribution should be similar.** \n\nHowever, we cannot define the 3D pose uncertainty with a clear formulation. To simulate the uncertainty distribution:\n>We first calculate the error between the 3D estimation and the 3D ground truth (GT) **along each axis in the MHAD training set, storing this as the noise set**. Then, **we add noise sampled randomly from this set** to the 3D GT along each axis. Finally, we use the Denoiser to refine both the **\"GT + noise\" and \"estimated\" 3D poses**, comparing the MPJPE results to assess the similarity and effectiveness of the uncertainty modeling.\n\nThe results are illustrated as:\n\n|2D Pose Detector| dataset|3D Initial Pose|MPJPE (mm)||2D Pose Detector| dataset|3D Initial Pose|MPJPE (mm)|\n|:-|:--:|:--:|:--:|:-|:-|:--:|:--:|:--:|\n||training|estimted|15.93|||training|estimated|10.96|\n|ResNet152|training|GT+noise|17.07||RSB152|training|GT+noise|11.95|\n||testing|estimated|43.57|||testing|estimated|27.76|\n||testing|GT+noise|17.51|||testing|GT+noise|12.46|\n\nRegardless of the 2d pose detector, **the accuracy of denoised GT+noise in both training and testing sets is all close to the denoised estimated 3D poses in the training set**. The MPJPE of denoised estimated 3D poses in the testing set is different from that in the training set. This is primarily attributed to the different 3D uncertainty distribution across these two sets. Hence, we draw the conclusion that the 3D uncertainty distribution is well-modeled.\n\nThank you again for your constructive feedback. **We will incorporate the corresponding modifications and expansions into the revised paper.** In addition, the corresponding code will be open-source to ensure replication. If you have any further questions, please let us know. We'd be very happy to do anything we can that would be helpful in the time remaining. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUvFDd6GKB", "reviewer": "Reviewer_cXv4", "summary": "The authors suggest a 3D human pose estimation (HPE) in binocular (two-view) settings. In order to reduce the uncertainty of triangulation method, the authors leverage the diffusion model. They propose a method called Dual-Diffusion method, which simultaneously denoises uncertainties in both 2D and 3D to produce accurate results. Additionally, they introduce Z-embedding and pose normalization related to baseline width. Experiments validate the effectiveness of the proposed method, showing performance improvements over the triangulation method.", "questions": "Method\n- What is the meaning of the sentence in Line 176? \"However, if the uncertainty of the 3D pose follows a Gaussian distribution is still confusing.\"\n- According to Fig2 (b), the denoiser estimate $y_0$ from $y_t$, not $y_{t-1}$. Why does the denoising process needs to be repeated $K$ times? I also found that only $K=1$ is used for the experiments.\n\nExperiments\n- What is the performance of the proposed method in 3-View or 4-View settings?", "limitations": "Limitations\n- The method can be used only for binocular settings.\n\nSuggestions\n- Extending the framework to multi-view 3D HPE and validating the superiority over other multi-view 3D HPE methods would strengthen the paper.\n\nTypos\n- Figure 2. Initial estimated -> initially estimated\n- Line 149. Inherently -> inherent\n- Line 182. $0, 1$ -> ${0,1}$\n- Line 320, by $P_v$ -> by multiplying $P_v$", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method achieves state-of-the-art results in binocular 3D HPE settings.", "weaknesses": "- The concept of applying diffusion models to 3D HPE is similar to DiffPose[8].\n- While the method can be applied to any multi-view setting, the authors limited their experiments to binocular settings, which restricts the applicability of the proposed method. Additionally, the performance improvements due to the dual diffusion over triangulation are marginal. Since accurate 2D pose detection reduces uncertainty in the triangulation method, it is challenging to assess the significance of the dual-diffusion models. It would be more convincing to show how the proposed method performs better with noisy 2D inputs.\n- The clarity and readability of the paper need to be enhanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer cXv4,\n\nWe appreciate the detailed feedback and hope our clarifications address your concerns. We apologize if there was any misunderstanding or misinterpretation of our paper. We're eager to highlight the significance and potential of our work.\n\n**1. The difference from DiffPose.**\n\n**Methodology Novelty.** We acknowledge that both our Dual-Diffusion and DiffPose use diffusion models to denoise the initial estimated 3D pose. **However, two methods differ in how the uncertainty distribution of 3D initial pose is modeled in the forward diffusion process.** DiffPose uses statistical methods, specifically a histogram described in its Sec. 4.1 (notably, in the code on GitHub, the Context Encoder is only applied during inference). **In contrast, Dual-Diffusion models 3D uncertainty by reconstructing from noisy 2D poses using Triangulation.** Since 2D uncertainty is well-defined, the noisy 2D poses simulated from GT in the forward diffusion process can encompass the uncertainty distribution. **Consequently, the 3D uncertainty distribution can be captured, which is challenging for statistical methods given the limited 3D pose data.** The many-to-many correspondence between 2D and 3D poses in monocular settings restricts 3D uncertainty reconstruction to statistical methods. But, in multi-view scenarios, geometric constraints inherently link these two uncertainties, making our Dual-Diffusion approach feasible. **Dual-Diffusion can be seen as an extension of DiffPose to multi-view settings, innovating by proposing a more effective method for 3D uncertainty modeling.** \n\n**Performance Superiority.** **3D uncertainty reconstruction through 2D poses is more robust than statistics because the 2D uncertainty is more stable than 3D (illustrated in the PDF of the global rebuttal).** We have compared the 2D and 3D uncertainty obtained from ResNet152 across different depths (we separate the MHAD training set into large and small parts based on the average depth of GT).\n\n|depth |2D MPJPE | 3D MPJPE (mm)|3D MPJPE STD (mm)|\n|:--:|:--:|:--:|:--:|\n|large|5.26|40.39|1025.88|\n|small|5.33|38.17|161.88|\n\nThe 3D MPJPE and STD highlight the instability of 3D uncertainty. To further verify the superiority of our Dual-Diffusion over the statistical DiffPose, **we trained two models on large and small datasets respectively and tested in another.**\n \n||Method|3D MPJPE (mm)|||Method|3D MPJPE (mm)|\n|:--:|:-|:--:|:--:|:--:|:-|:--:|\n|training in large|Tri|38.17||training in small|Tri|40.39|\n|and|Dual-Diff|**35.23**||and|Dual-Diff|**39.11**|\n|testing in small|DiffPose|40.26||testing in large|DiffPose|54.12|\n\n**Diffpose suffers from the change of 3D poses uncertainty distribution while our Dual-Diffusion remains stable.** This stability is since Dual-Diffusion models the diffusion target using 2D uncertainty, which is more stable than 3D uncertainty.\n\n**2. Why the applicability is limited in binocular settings?**\n\nWe have conducted experiments to denoise the initial estimated 3D poses using ResNet152 in 2-view, 3-view, and 4-view H36M testing sets.  \n\n|2-view|MPJPE (mm)|\n|:--|:--:|\n|Tri|31.51|\n|Dual-Diff|**29.15**|\n\n|3-view|MPJPE (mm)|\n|:--|:--:|\n|Tri|30.13|\n|Dual-Diff|**28.69**|\n\n|4-view|MPJPE (mm)|\n|:--|:--:|\n|Tri|29.93|\n|Dual-Diff|**28.44**|\n\n**The improvement compared to Triangulation demonstrates the denoising capability of Dual-Diffusion.** However, we acknowledge that the improvement decreases as the view number increases. As we clarified in Sec. 1, the 3D uncertainty distribution is more ambiguous in binocular settings compared to multi-view setups. **The 3D uncertainty in multi-view settings is smaller making denoising unnecessary.** \n\n**3. The performance improvements due to the dual diffusion over Triangulation.**\n\nWe need to clarify that all experiments are conducted using 2D estimated poses, which are noisy data rather than accurate data. The red values in Tab. 1 and 2 highlight the improvements achieved with Dual-Diffusion over Triangulation. **Although some improvements appear small, the ratios are comparable.** For instance, even though MPJPE improvement of Dual-Diffusion over Triangulation using RSB152 in H36M is 1.87mm, it represents a 6.12% improvement. **Additionally, we have added more backbones to verify that our method is superior to Triangulation (refer to tables in the PDF of the global rebuttal).**\n\n**4. The meaning of the sentence in Line 176.**\n\nWe argue that a key characteristic of noise added in the diffusion process is the additive property. **This property allows for the skipping process in Eq. (2) and ensures that the noisy data at t=T aligns with the initial uncertainty distribution.** Consequently, the reverse process can start from this uncertainty distribution. Typical diffusion models use Gaussian noise and start from a Gaussian distribution, which itself is additive. However, in binocular 3D HPE, the 3D pose uncertainty is still complex and lacks a clear formula. **This complexity prevents us from modeling the diffusion process in the 3D pose but in the 2D pose.**\n\nWe are deeply sorry for the misunderstanding this sentence has caused you and will clarify it in the revision.\n\n**5. The meaning of K in the denoising process.**\n\nFig. 2(b) illustrates the inference of Dual-Diffusion, where t ranges from (0, T). In classical DDPM, the denoising inference is performed step-by-step from T to 0. Then DDIM is proposed to speed the inference by skipping some steps. Here, we utilize the DDIM method. K is the number of denoising steps during inference which is a hyper-parameter change from 1 to T. K=1 is finally decided by ablation study.\n\n**6. Typos**\n\nWe deeply regret the oversight. These errors will be addressed in the revised manuscript.\n\nFinally, we humbly request you to kindly reconsider our submission based on the above justification. If you have any further questions, please let us know. We’d be very happy to do anything we can that would be helpful in the time remaining! Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hz7bPxJMJI", "reviewer": "Reviewer_o1bT", "summary": "This paper presents a new method for binocular human pose estimation. The key idea is to employ diffusion model into this problem.  The authors propose a novel dual-diffusion process: jointly diffuse 2D pose uncertainty and 3D pose uncertainty. Because 3D uncertainty relates to both 2D noise (which directly relates to diffusion step t) and depth, the authors further proposed z-embedding and BaseL-norm. Results on bioncular H36M and MHAD datasets demonstrate the effectiveness of the proposed method.", "questions": "No  detailed question.", "limitations": "The authors addressed limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dual-denoising is interesting for human pose estimation. The novelty of the proposed method is clear. \n\n2. The quantitative results seem good in the Table.1 and Table.2. \n\n3. The paper is clearly written.", "weaknesses": "I have some questions about the evalution. \n\n1. The paper compares a lot with RSB-pose. However, in the orignial paper of RSB-pose, the absolute MPJPE is not reported. How do the authors compute the results of RSB-pose? Why not compare with RSB-pose with MPJPE_re (relative to pelvis) ? \n\n2. MIssing comparison with recent paper \"Triangulation Residual Loss for Data-efficient 3D Pose Estimation (NeurIPS 2023)\". It optimizes both multiview 2D and 3D with an extension of epipolar loss.  at least the authors should discuss it. \n\n3. Why not use transformer based backbones, which are the mainstreams of 2D pose estimation? Accuracy may improve with transformer backbones. The table.1 may indicate that larger backbone may mitigate the gap among different methods. The table.2 may indicate that larger scale may narrow the gap too. \n\n4. I want to know the parameter size of different methods (parameters other than the same backbone part) in table.1 and table.2. I want to make sure that the better accuracy does not simply come from large parameter capability but come from the network design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer o1bT,\n\nFirst and foremost, we'd like to express our gratitude for your comprehensive review and insightful comments. In response, we have supplemented the evaluation and provided clarifications to enhance the clarity of our experiments.\n\n**1. About the comparison with RSB-Pose.**\n\n**Given the geometric constraints inherent in two-view setups, it is entirely feasible to reconstruct the absolute position of 3D poses.** Therefore, we propose that absolute MPJPE should replace relative MPJPE as the primary metric for 3D accuracy here, not like the monocular settings. Since RSB-Pose is specifically designed for binocular 3D HPE, we compare a lot with it. We have communicated with the authors of RSB-Pose and suggested including absolute MPJPE results. They agreed to incorporate this metric in the published version and shared their code with us. They have updated the results in arXiv. We reproduced their training process and calculated the absolute MPJPE results accordingly.\n\n**2. About the comparison with \"Triangulation Residual Loss for Data-efficient 3D Pose Estimation (NeurIPS 2023)\".**\n\nConsidering that \"Triangulation Residual Loss for Data-efficient 3D Pose Estimation (NeurIPS 2023)\" (TR) is a method to optimize 2D and 3D poses simultaneously, similar to our approach, we apologize for previously overlooking it and have now included a comparison and discussion as follows:\n\n* To reduce reliance on 3D annotations, TR proposes the TR loss, which utilizes multi-view geometric constraints to optimize multi-view 2D poses. This loss essentially minimizes the sum of distances from the triangulated 3D points to all view rays originating from 2D points. Minimizing the smallest singular value of the Triangulation matrix is used for computational optimization. It is a simple yet effective method.\n\n* Similar to our approach, TR improves 2D pose estimation, which in turn enhances the 3D results by using geometric constraints during the optimization process. The overall framework is quite similar.\n\n* The key difference is that **TR optimizes individual points without considering the entire pose globally, lacking pose prior constraints**. In contrast, we introduce a diffusion model to capture the distribution of 3D poses and optimize the entire pose as a whole. **Our Dual-Diffusion approach incorporates both geometric and prior constraints**, offering a more comprehensive optimization objective.\n\nWe cannot provide a direct evaluation comparison because the model weights for TR are unavailable. However, we acknowledge the relevance and similarity of this approach to ours and **will include a discussion of it in the related work section**.\n\n\n**3. About the comparison with transformer-based backbones.**\n\nWe thank for the kind advice to compare with the transformer-based 2D pose detector. We have added the comparison using ViTPose[1] and the results are shown as:\n\n|Dataset|Method | 2D Pose Detector | Scale|MPJPE (mm) | BL (mm) | Sym (mm) | JDR (%)|\n|:-|:-|:--:|:--:|:--:|:--:|:--:|:--:|\n|MHAD|Tri-ViTPose|ViTPose|256|70.84|42.55|48.43|95.83|\n||Dual-Diffusion-ViT|ViTPose|256|**61.02**|**37.90**|**30.09**|**95.88**|\n|||||||||\n|H36M|Tri-ViTPose|ViTPose|256|41.49|18.09|20.75|93.33|\n||Dual-Diffusion-ViT|ViTPose|256|**35.20**|**16.02**|**19.66**|**95.77**|\n\nThe baseline results (ViTPose + Triangulation) are lower than expected because ViTPose has not been fine-tuned on these two datasets, limiting the effectiveness of 2D pose estimation and, consequently, impacting 3D pose estimation results. The comparison between ViTPose and ResNet152 (found in the PDF of the global rebuttal) reveals that **as 2D accuracy increases, the gap between the baseline method and Dual-Diffusion decreases.** With the same training process, more accurate 2D estimation often relies on a larger backbone and higher image scale. As clarified in the third paragraph of the introduction, **more accurate 2D poses lead to more accurate 3D pose reconstruction since their uncertainties are intrinsically linked within the geometric framework.** Consequently, a smaller refinement space remains for the 3D poses.\n\n**4. About the comparison of model parameters.**\n\nWe sincerely appreciate the thoughtful advice to include the model parameters. The results are listed below. \n\n|Methods|Params of Model other than Backbone (M)|\n|:-|:--:|\n|TPPT|9.7|\n|Epipolar_Tri|0.08|\n|Algebraic_Tri|10.88|\n|AdaFuse|1.02|\n|RSB-Pose|9.25|\n|Dual-Diffusion|**0.74**|\n\n\nOur Dual-Diffusion requires only 0.74 million parameters, significantly fewer than other methods. **This demonstrates that the effectiveness of pose refinement in our method does not rely on a large number of parameters, but rather on the dual diffusion modeling and training.** The model parameters will be **added to Table 2 in the revised manuscript** and can be previewed in the PDF of the global rebuttal.\n\nIn summary, we sincerely thank you for your valuable comments. We believe that by addressing these points, our work will be significantly improved. We hope our responses provide clarity, and we remain open to further feedback. Thanks!\n\n\n[1] Xu Y, Zhang J, Zhang Q, et al. Vitpose: Simple vision transformer baselines for human pose estimation[J]. Advances in Neural Information Processing Systems, 2022, 35: 38571-38584.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1pLQKXROvz", "reviewer": "Reviewer_9xXX", "summary": "The paper proposes a dual-diffusion model for Binocular 3D Human Pose Estimation, which models the uncertainties integrated in the binocular configuration. Since constructing the depth distribution of 3D poses is difficult, the authors use 2D poses with triangulation to initialize the 3D poses during the diffusion process. Additionally, they adopt the depth of the 3D root as an additional condition and introduce pose normalization to improve generalization ability. The experiments show that the method achieves superior performance on the MHAD and H36M datasets.", "questions": "- How does the proposed method perform in multi-view scenarios with more than 2 views?\n- The proposed framework does not require 3D supervision. Could the performance be improved with additional 3D pose supervision?\n- What is the performance gap between the proposed triangulation and common 3D initial distributions like DiffPose [8] in the 3D pose diffusion process?\n- How could the proposed diffusion framework enhance the results of direct triangulation?", "limitations": "The limitations and societal impact have been discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Using multi-view 2D poses with triangulation to construct the initial distribution for 3D poses during the diffusion process is interesting and superior to previous designs in diffusion-based 3D pose estimation.\n- The Z-embedding Condition and Baseline-width-related Pose Normalization are simple but reasonable approaches that consider the varying depth of humans and the baseline-width in the binocular setting.", "weaknesses": "- The comparison between triangulation and common 3D initial distributions like DiffPose [8] should be conducted, which is necessary for revealing the strength of the proposed dual diffusion structure.\n- The scalability of the proposed method to address more camera views should be clarified. It would be helpful if the authors could discuss how their approach handles an increased number of cameras and any potential impacts on performance.\n- 3D poses can be directly obtained by triangulating multi-view 2D poses. What is the accuracy of the 3D poses from direct triangulation? This comparison is important to assess the necessity of the proposed diffusion process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9xXX,\n\nWe greatly appreciate your thoughtful review and the time you have taken to provide insights and feedback on our submission. We are encouraged by the positive aspects you've highlighted and grateful for the critical points you've raised. In response, we have addressed the weaknesses and questions to further reveal the strength of our method.\n\n**1. The comparison between Triangulation and common 3D initial distribution.**\n\nIt is crucial for us to compare the 3D uncertainty distribution reconstructed by Triangulation and statistical methods like DiffPose.  We opt to use 2D uncertainty to reconstruct 3D uncertainty for two reasons: **the difficulty in formulating 3D uncertainty and the greater stability of 2D uncertainty.** We will now elaborate on the second reason.\n\n**The reconstruction from 2D poses using Triangulation is more robust than statistical methods because the 2D uncertainty is more stable than 3D uncertainty.** \n>We have evaluated the 2D MPJPE and 3D MPJPE estimated by ResNet50 under binocular settings with varying baseline widths. \n\n>|baseline (mm)|2D MPJPE (pixel)| 3D MPJPE (mm)|\n>|:--:|:--:|:--:|\n>|100|3.084|120.37|\n>|200|3.079|63.18|\n>|400|3.079|54.69|\n\n>We have compared the 2D and 3D uncertainty obtained from ResNet152 across different depths (we separate the MHAD training set into large and small datasets based on the average depth of GT pose).\n\n>|depth|2D MPJPE (pixel)| 3D MPJPE (mm)|3D MPJPE STD (mm)|\n>|:--:|:--:|:--:|:--:|\n>|large|5.26|40.39|1025.88|\n>|small|5.33|38.17|161.88|\n\nThe 3D MPJPE and 3D STD highlight the instability of 3D uncertainty. To further verify the superiority of our reconstruction method over the statistical method, **we trained two models on large and small datasets respectively and tested in another.**\n \n||Method|3D MPJPE (mm)|||Method|3D MPJPE (mm)|\n|:--:|:-|:--:|:--:|:--:|:-|:--:|\n|training in large-dataset|Tri-ResNet152|38.17||training in small-dataset|Tri-ResNet152|40.39|\n|and|Dual-Diff|**35.23**||and|Dual-Diff|**39.11**|\n|testing in small-dataset|DiffPose|40.26||testing in large-dataset|DiffPose|54.12|\n\nCompared to the Triangulation baseline, performance improves with Dual-Diffusion refinement but decreases with DiffPose. **DiffPose suffers from the change of 3D poses uncertainty distribution while our Dual-Diffusion remains stable.** This stability is due to the fact that Dual-Diffusion models the diffusion target using 2D uncertainty, which is significantly more stable than 3D uncertainty. \n\n**2. The scalability to multi-view settings.**\n\n**Our Dual-Diffusion module can be directly applied to multi-view settings (more than 2 views) without the need for additional fine-tuning.** Since the input to the denoiser network is the 3D pose reconstructed from 2D poses, the network structure remains unchanged as the number of views increases. Furthermore, the 3D uncertainty in 2-view settings often encompasses the uncertainty in 3-view and 4-view settings, making fine-tuning optional, though beneficial. We have tested Dual-Diffusion, trained on binocular 2D poses, in 3-view and 4-view scenarios and compared the 3D MPJPE results with those from Triangulation.\n\n|2-view|MPJPE (mm)|BL (mm)| Sym (mm)| JDR (%)|\n|:-|:--:|:--:|:--:|:--:|\n|Tri-ResNet152|31.51|14.38|16.29|95.81|\n|Dual-Diffusion-ResNet152|29.15|12.06|13.37|95.92\n\n|3-view|MPJPE (mm)|BL (mm)| Sym (mm)| JDR (%)|\n|:-|:--:|:--:|:--:|:--:|\n|Tri-ResNet152|30.13|13.26|15.75|96.46|\n|Dual-Diffusion-ResNet152|28.69|11.99|12.42|96.60|\n\n|4-view|MPJPE (mm)|BL (mm)| Sym (mm)| JDR (%)|\n|:-|:--:|:--:|:--:|:--:|\n|Tri-ResNet152|29.93|13.14|14.57|94.96|\n|Dual-Diffusion-ResNet152|28.44|12.22|12.39|95.21|\n\n**The performance improvements across four metrics validate the scalability of Dual-Diffusion.** However, it needs to be acknowledged that the gains are more modest when the baseline performance is already high.\n\n**3. The comparison between Dual-Diffusion and Triangulation.**\n\nThe baseline comparison, highlighted in light blue in Tables 1 and 2, shows the 3D poses reconstructed by Triangulation. The red values indicate the improvements achieved by Dual-Diffusion over Triangulation. To prevent any misunderstanding, we will provide a clear explanation of the baseline method in Section 4.1.\n\n**4. Adding 3D supervision.**\n\nWe appreciate the insightful suggestion. We have implemented additional experiments by using 3D supervision alone and in combination with 2D supervision to retrain Dual-Diffusion, based on the RSB152 backbone.\n\n|Loss|MPJPE (mm)|BL (mm)| Sym (mm)|\n|:-|:--:|:--:|:--:|\n|2D|27.76|7.56|9.83|\n|3D|29.38|8.62|10.57|\n|2D+3D|**27.73**|**7.43**|**9.43**|\n\nAdding 3D supervision additionally enhances 3D accuracy and plausibility. Considering that Dual-Diffusion still has room for improvement, future work will focus on addressing the limitations discussed in the paper and exploring more additional enhancement methods.\n\nIn summary, we sincerely thank you for your valuable comments. We believe that by addressing these points, our work will be significantly improved and provide a solid contribution to the community. We hope our responses provide clarity, and we remain open to further feedback. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear ACs and Reviewers,

We are very grateful for your time and effort in reviewing this manuscript. We value every helpful suggestion and comment. **The two most frequently concerned questions are: 1) comparative evaluation with more 2D pose detectors, and 2) comparative evaluation with Diffpose.** We would like to make more clarifications here.

We have included comparisons with previous methods, such as Epipolar-Tri and AdaFuse, as well as a transformer-based 2D detector, ViTPose, in Tables 1 and 2. The parameters of modules other than the backbone have also been added. **The updated tables can be previewed in the PDF.**

Our Dual-Diffusion method demonstrates superiority over DiffPose in terms of robustness to changes in 3D pose uncertainty. We have provided experimental results in the rebuttal to reviewers. **Here, we add a figure in the PDF to illustrate that the 3D uncertainty in binocular 3d HPE  is more unstable than 2D uncertainty,** which will be affected by the baseline width and 3D target depth. Therefore, diffusion modeled by 2D uncertainty is advantageous and necessary.

We hope these replies address all the concerns. If you have any further questions, please feel free to contact us. We will be happy to do anything we can that would be helpful in the time remaining! 

Best regards,

Meta-review: This paper proposes a dual-diffusion model for binocular 3D human pose estimation where the depth ambiguities are integrated in the binocular configuration. The forward diffusion process is used to simulate the noisy 2D poses from GT 2D poses. The reverse denoising process refines the 3D poses obtained by triangulation of noisy binocular 2D poses. The proposed method is evaluated on MHAD and Human3.6M, outperforming SOTAs. While the idea of levering diffusion process for human pose estimation is appreciated, the reviewers mainly raised concerns regarding difference from DiffPose, insufficient experiments, the applicability of the proposed method to 3- or 4-view setting.  In the rebuttal, the authors adequately addressed the concerns.  During the reviewer-author discussion phase, the authors further clarified the advantages of the proposed method.  The reviewers unanimously support this paper.  The paper should be accepted, accordingly.  All the discussion during the rebuttal and post-rebuttal discussion should be appropriately incorporated in the final version.
Decision: Accept (poster)