$\text{Di}^2\text{Pose}$: Discrete Diffusion Model for Occluded 3D Human Pose Estimation
Authors: Weiquan Wang
Jun Xiao
Chunping Wang
Wei Liu
Zhao Wang
Long Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Occluded 3D human pose estimation; Discrete diffusion model
Primary area: machine_vision

Abstract: Diffusion models have demonstrated their effectiveness in addressing the inherent uncertainty and indeterminacy in monocular 3D human pose estimation (HPE). 
Despite their strengths, the need for large search spaces and the corresponding demand for substantial training data make these models prone to generating biomechanically unrealistic poses. 
This challenge is particularly noticeable in occlusion scenarios, where the complexity of inferring 3D structures from 2D images intensifies. 
In response to these limitations, we introduce the **Di**screte **Di**ffusion **Pose** (**$\text{Di}^2\text{Pose}$**), a novel framework designed for occluded 3D HPE that capitalizes on the benefits of a discrete diffusion model. 
Specifically, **$\text{Di}^2\text{Pose}$** employs a two-stage process: it first converts 3D poses into a discrete representation through a pose quantization step, which is subsequently modeled in latent space through a discrete diffusion process. 
This methodological innovation restrictively confines the search space towards physically viable configurations and enhances the model’s capability to comprehend how occlusions affect human pose within the latent space. 
Extensive evaluations conducted on various benchmarks (e.g., Human3.6M, 3DPW, and 3DPW-Occ) have demonstrated its effectiveness.

Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimation
Weiquan Wang', Jun Xiao', Chunping Wang", Wei Liu Zhao Wang', Long Chen
*Zhejiang University FFinvolution Group TTencent
*Hong Kong University of Science and Technology edu edu en, wangchunpi wwwggchhuningg22aeee wnngnng com, w122230 lumoi a edu, zha o wangez ju. edu. cn, longchen@ust.
(wqwanges, junx @zju.
Abstract
Diffusion models have demonstrated their effectiveness in addressing the inherent uncertainty and indeterminacy monocular 3D human pose estimation (HPE). Despite their strengths, the need for large search spaces and the corresponding demand for substantial training data make these models prone to generating biome- chanically unrealistic poses. This challenge is particularly noticeable in occlusion scenarios, where the complexity of inferring 3D structures from 21) images in- tensifies. In response to these limitations, we introduce the Discrete Diffusion Pose (Di'Pose), a novel framework designed for occluded 3D HPE that capitalizes on the benefits of a discrete diffusion model. Specifically, Pise employs a two-stage process: it first converts 3D poses into discrete representation through a pose quantization step, which is subsequently modeled in latent space through a discrete diffusion process. This methodological innovation restrictively confines the search space towards physically viable configurations and enhances the model's 's capability comprehend how occlusions affect human pose within the latent space. Extensive evaluations conducted on various benchmarks (e.g., Human 6.M, 3DPW, and 3DPW-Oce) have demonstrated its effectiveness.
Introduction
3D Human Pose Estimation (HPE) from monocular images remains challenging yet pivotal research in the realm of computer vision, boasting a wide range applications including human- human-machine machine in- teraction, autonomous driving, and animations [57, 81, 5, 70]. Generally, the mainstream approaches, including Direct Estimation [68, 43, 47] and 2D-to-3D Lifting [87, 51, 86), aim to perform 3D HPE by either directly predicting 3D poses from 2D images lifting detected 2D poses into 3D space. These approaches aim to address the inherent 2D-3D ambiguity in 3D HPE tasks by learning mapping from training data. Despite significant advancements, accurately estimating 3D poses from monocular images remains a formidable challenge, particularly when humans are partially occluded [39]. Such occlusions introduce considerable uncertainty and indeterminacy into the estimation process.
Existing 3D HPE methods try to handle the occlusion challenges with pose priors/constraints [58, 62] or data augmentation strategies (e.g., annotations augmentation [61]. pose transformation [35], and differentiable operations [82]). However, due to the inherent discreteness of 3D poses (primarily defined by discrete anatomical landmarks). these methods tend to represent poses using coordinate vectors or heatmap embeddings, treating joints as independent units and overlooking the interdepen- dencies among body joints. Recent research [21] has introduced compositional pose representation that captures the dependencies among joints by converting a pose into multiple tokens, enabling the
"Long Chen is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

116
Diffhose
Image
DiffPose
DFf Pose (ours)
240/22 (mxz))
Propertion of Truiiip Sumples (14
(8)
(b)
Figure I: (a) Results of DiffPose [22] and Pose in Human3.6M [72] dataset (with MPJPE metric), across varying proportions of training samples. (b) Prediction results of two methods under occlusion.
use of mutual context between joints. This approach, which learns from real pose datasets, results in each learned token corresponding to physically realistic prototype. Nevertheless, Geng el al. [21] casts HPE to a classification task, where the system simply classifies tokens based on prototype poses. Unfortunately, such scheme does not account for the effects of occlusions in the estimation process, potentially leading to inaccuracies due to unresolved uncertainty and indeterminacy.
Recent studies have shown marked progress in 3D HPE via generative models [2, 84, 19, 27]. Notably, diffusion models [25] have demonstrated effectiveness in handling complex and uncertain data distributions, making them suitable for handling uncertainty and indeterminacy in 3D HPE tasks [19, 91, 66, 27, 37J. They excel at generating samples that conform to a target data distribution by iteratively removing noise through series of diffusion steps, ultimately predicting more accurate 3D poses. However, these diffusion- 3D HPE methods initialize the 3D pose from random noise at the begining of the diffusion process, where each joint can be sampled from the continuous 31) space. Since the continuous 3D space has an infinite number of points, training such diffusion- models requires a large amount of 3D pose data to achieve optimal outcomes [23, 75, This demand implies a substantial need for training data, presenting a stark contradiction to the limited availability of 3D human pose datasets. As illustrated in Figure (a)), the predictive performance of DiffPosse [22] declines more rapidly as the proportion of training data decreases. Given the scarcity of 3D pose training data, previous diffusion models may generate physically implausible configurations that do not adhere to human biomechanics, leading to inaccurate human pose estimations, particularly in occluded scenes (c.f., Figure ((b) with DiffPose).
In this paper, we propose a novel framework for 3D HPE with occlusions: Discrete Diffusion Pose (Di Pose), drawing on compositional pose representation and diffusion model to achieve the best of two worlds. Specifically, Pose employs a two-stage approach: a pose quantization step followed by discrete diffusion process. The pose quantization step leverages the discrete nature of 3D poses and represents them as quantized tokens by capturing the local interactions between joints. This step effectively confines the search space to physically plausible configurations by learning from real 3D human poses. Subsequently, the discrete diffusion process models the quantized pose tokens in the latent space through a conditional diffusion model. By integrating the forward and reverse processes, our framework adeptly simulates the transition of a 3D pose from occluded to recovered. By modeling occlusion implicitly within the latent space, Pose enhances its understanding of how occlusions affect human poses, providing valuable insights during the training phase.
For the pose quantization step. consisting of a pose encoder, a quantization process, and a pose decoder. To effectively capture local interactions between 3D joints, we introduce the Local-MLP block for both pose encoder and decoder. Within each Local-MLP block, simple Joint Shift operation integrates information from different joints. The pose encoder utilizes several Local-MLP blocks convert 3D pose into multiple rich token features, each representing a sub-structure of the overall pose. These tokens are quantized using a shared codebook, yielding corresponding discrete indices. Additionally, we implement the finite scalar quantization (FSQ) [49] to address the codebook collapse issue observed in traditional VQ-VAE methods [59, 89, 56, 42]. This strategy ensures that the generated codewords are meaningful, crucial aspect for the subsequent success of the discrete diffusion process.
we devise a pose quantization step inspired by VQ-VAE [71],

---

For the discrete diffusion process, during the training phase, we introduce occlude and replace strategies to model the quantized pose tokens. enabling the discrete diffusion model to predict occluded tokens and update potential tokens. The occluded token represents the occlusion of the corresponding sub-structure of the 3D pose. The token replacement mechanism is designed to enhance the diversity of potential sub-structures, reflecting the indeterminacy in occluded parts. During the inference phase, pose tokens are either occluded or initialized randomly. The denoising diffusion process estimates the probability density of pose tokens step-by-step based on the input 2D image until the tokens are completely reconstructed. Each step leverages contextual information from all tokens of the entire pose predicted in the previous step, facilitating the estimation of new probability density distribution and the prediction of the current step's 's tokens. This sequential approach ensures detailed and accurate reconstruction of 3D poses from occluded scenes.
We extensively evaluate our approach in 3D HPE on three challenging benchmarks (Human3.6 6M [34], 3DPW [72] and 3DPW-Occ), Pose consistently yields lower errors compared to state-of-the methods. In particular, it achieves significantly better results when evaluated on occluded scenarios, verifying its advantages of ecclusion-hanlling capability. Our contributions are threefold:
We propose the Pose framework, which integrates the inherent discreteness of 3D pose data into the diffusion model, offering new paradigm for addressing 3D HPE under occlusions.
The designed pose quantization step represents 3D poses compositional manner, effectively capturing local correlations between joints and confining search space to reasonable configurations.
The constructed discrete diffusion process simulates the complete process of a 3D pose transitioning from occluded to recovered, which introduces the impact of occlusions into pose estimation process.
2
Related Work
Monocular 3D HPE. Existing approaches can generally be classified into frame-based and video- based methodologies. Frame-based methods predict the 3D pose from a single RGB image, employing different networks in various studies [18, 20, 52, 54, 55] to directly output the human pose from the 2D image. Alternatively, a significant number of studies [48, 77, 85, 88] initially determine the 2D pose, which subsequently forms the foundation for inferring the 3D pose. In contrast, video-based methods leverage temporal relationships across video frames. Such methods predominantly [9, 11, 15, 31, 65, 73, 76] commence with the extraction of 2D pose sequences using a 2D pose detector from the video clips, aiming to harness the essential spatio-temporal data for 3D pose estimation. To validate the efficacy of our approach, we evaluate our Pose on the more challenging frame-based setting, wherein the 3D human pose is directly inferred from 2D image.
Occluded 3D HPE. Occlusions significantly challenge 3D HPE. As evidenced by research [76, 58, 62], pose priors and constraints have been proven crucial for mitigating such issue. Approaches typi- cally involve statistical models deduce occluded parts from visible cues [76, 41, 44] or pre-defined rules to constraint poses [61, 1]. Moreover, due to the scarcity of 3D pose data, data augmentation, including synthetic occlusions [7, 38, 60, 64, 13] and pose transformations [35, 82], remains vital for enhancing model robustness. Diverging from these aforementioned methods, our method innovatively introduces occlusion the latent space without extra priors or explicit augmentations, providing a deeper feature-ban based understanding of occlusion's 's effects on pose estimation.
Diffusion Models for 3D HPE. Recent advancements have shown that diffusion models are capable of managing complex and uncertain data distributions [25, 26, 17, 4, 32, 8, 10, 80, 40, 36, 74, 78], which is particularly beneficial for 3D HPE. Typically, these models predict 3D poses by progressively refining the pose distribution from high to low uncertainty [22, 14, 19, 91, 633. Other approaches
use diffusion models to generate multiple pose hypotheses from single 2D observation [66, 27]. These 3D pose estimators effectively reduce uncertainty and indeterminacy throughout the estimation process. Moreover, discrete di diffusion models have also gained attention in various domains [30, 40, 24, 33]. Inspired by these advancements, our work introduces a discrete diffusion model for occluded 3D HPE, which aligns more closely with the inherent discreteness of 3D pose data and effectively incorporates occlusion into the estimation process, providing novel perspective in the field.
3

---

Stage Pose Quantization Step
P
P
F
F5Q
T
340
k
Pose Encoder
Pose Decoder
Stage 2: Discrete Diffusion Process
kg
k
Forword
32
...
145
I61
process
Condition
Pose Denoiser
Pose Decoder
AdaLNorm
Attention
Mult-head
AdaL.Nrrm
Attention
Multi-head
LNGa Linear
Image Encoder
Feature
XM
Sequence
Figure 2: Overview of two-stage Di² Pose framework. In the stage 1, train pose quantization step that transforms a 3D pose P into multiple discrete tokens k, each token representing the indices of implied codebook C. In the stage 2, model k in the discrete space by discrete diffusion process. In the forward process, each token is probabilistically occluded with Occ token or replaced with another available token. In the reverse process, the model leverages an independent image encoder and a pose denoiser to reconstruct all the tokens based on the condition 2D image. These reconstructed tokens are finally decoded by the pose decoder, resulting the recovered 3D pose. Notably, we only update the parameters of pose denoiser, pose decoder and image encoder are frozen.
Di'Pose
Given an 2D image "xx33 the goal of 3D HPE is to predict P RJx3 which represents the 3DD coordinates of all the J joints of the human body. In this paper, construct occluded 3D) HPE task as two-stage framework including the pose quantization step and the discrete diffusion process.
As shown in Figure 2, in the training phase, Stage learns pose quantization step by VQ-VAE like structure (Sec. 3.1), which able to encode 3D pose into multiple quantized tokens. Stage models quantized pose tokens in the latent space by the forward and reverse process of conditional diffusion model (Sec. 3.2). the inference phase, we only use the reverse process of Stage and the pre- rre-trained pose decoder of Stage recover 3D pose from the 2D image. Notably, pose tokens are either occluded or initialized randomly at the beginning
the inference phase. The model reconstructs all the tokens based on the condition 2D image step-by-step. These reconstructed tokens are finally decoded by the pre-trained pose decoder, resulting in the recovered 3D pose.
3.1 Pose Quantization Step
As depicted Figure 2, pose quantization step comprises pose encoder, the quantization process, and pose decoder. Initially, for a real 3D pose P E R the pose encoder fPE(-) converts P to token features F. During the quantization process, we utilize FSQ to quantize F (f, fz, ... fN (f, P) into tokens T = (t1, ...... ,tx) (t; E RD) Finally, the quantized tokens T are decoded by the pose decoder (-n(-) reconstruct 3D pose P.
Pose Encoder. Considering the interdependencies among human body joints, our goal to represent 3D poses in compositional manner, moving away from reliance coordinates vectors heatmap

---

embeddings. The VQ-VAE architecture, incorporating MLP-Mixer blocks [69] within its encoder and decoder, has been proven effective in decomposing pose into multiple token features, each corresponding to sub-structure of the pose [21]. However, the MIP-MMxer block is designed to extract global information across all joints, which can not adequately capture the local relationships between joints within individual sub-structure.
In response aforementioned limitation, we design Local-MLP block capture the local interactions between 3D joints. The pose encoder fPE(.), comprising several Local-MLP blocks, converts P to N token features:
F f2,
[(x)= (Sm (P)),
(1)
where femb(.) embeds to P Pemb E JxD by linear layer.
As shown in Figure 3(a), a Local-MLP block
composed of a Layer Normalization layer, Joint Shift block (JS-Block), a Channel MLP. and residual connection. The JS-Block is specifically designed to capture local interac- tions among joints. It extracts features by linear projection, and the Joint Shift operation enables feature translation along joint connec- tion directions. As shown in Figure 3(b), with the input Pemb E DDX the feature is evenly divided into X segments (.X = 3 in the exam- ple), each segment being shifted incrementally by units from - [X/2] to [X/2]. The central segment remains stationary, while the segments to the left and right away from the center by up to = =|X/2] units. Zero padding is used to maintain dimensionality. Features highlighted within the dashed box are selected for further linear projection. Finally, the Channel MLP processes these features channel- wise facilitate information integration.
A
J Ja 1 J.
A
1 A
Channel MLP
LayusNorm
Proj
Au A
& Ja
&
Juint Shift
A A /a
JS-Block
Linear Proj.
fx A A
LayesVoom
A
Ja
1 0
4
symmetrically shifted
(8)
(b)
Figure 3: (a) depicts the structure of the Local- MLP block; (b) shows the Joint Shift operation, where the arrows indicate the steps, and different subscript numbers represent the features of differ- ent joints. The gray blocks indicate zero padding.
During this process, we exploit FSQ [49] to enhance the utilization of codewords. FSQ quantizes token features F as corresponding token indices:
Quantization Process.
k= K2,, .. kN) = FSQ(fpro)))))
(2)
where Sproj() (-) projects each f; R.D of q: E Rdd, and each k; of k denotes the entries of implied codebook C. For each q, FSQ employs a bounding function fond qi + L/2] tanh(q:)) constrain each channel of d. As a result, each channel in qi round( fimd(q:)) takes one of L unique values. This procedure yields q: E C, where the total number of unique codebook entries is |C| - I Ta (mapping the i-th channel to L2 values). The vectors in C can be enumerated, establishing a bijective mapping from any q to an integer within (1,..., [C]]. In addition, the corresponding codeword of k; which is denoted as t; E RD represents the quantized result of f, Thereby, using FSQ, the token features are quantized as T = (t), t2..,........ txy).
Pose Decoder. The pose decoder frp(.) designed to recover 3D pose Po from T. fPD(.) adopts a structure similar to the pose encoder but in reverse, utilizing a reduced number of Local-MIP blocks.
Loss. The pose quantization step. including the pose encoder, quantization process, and pose decoder, jointly optimized by minimizing L1 loss LPQ = ||P Pll across the training dataset.
3.2 Diserete Diffusion Process
After training the pose quantization step, acquire N quantized tokens k from the original 3D pose P. The next step in Pise pipeline is to model k in the latent space by the discrete diffusion process. the following, we first briefly introduce the diffusion models and clarify the basic principles of the discrete diffusion model. Then we explain the details of discrete diffusion
5

---

process, including the designed transition matrix and loss function. Eventually, we illustrate the architecture and training and inference process.
Discrete Diffusion Model. Our discrete diffusion model is characterized by two distinct processes:
1) Forward process: It progresses through discrete steps E {0, ..., .), gradually transforming the initial tokens kn (the quantized token k) into a noise-infused latent representation ks. 2) Reverse process: is tasked with reconstructing the original data kg from the latent ks, following a reverse temporal sequence & E {S, S - 1,..., 1,
Followed previous studies [67, 3, 28], ilk,-1 = j) E TR [[] elucidate the likelihood of transitioning from k to kg. Then the forward process for the entire sequence of tokens is expressed as:
transition probability matrix [M,y g(k,
(k,, (1--k) c (k., M,c(k,-))
where e(-) symbolizes a function capable of converting a scalar into a one-hot column vector. The distribution of k, follows a categorical distribution, determined by the vector M,c(k,-) 1). Leveraging the Markov chain property, it is feasible bypass intermediate stages, directly computing the probability of k, from ko for any given step as:
((k = c (k.) (c.k)), with M, M M
(4)
Moreover, the posterior of the reverse process, (kks- [ks, ko), can be ascertained as:
q(k, [[k,-i/k,,ke) ko) = 1.o, ko)
(c (k.)M.c(kk..)))
(K.-))M.. c(ko))
(5)
g(k,
c (K.))M,e(kk)
Occlude and Replace Transition Matrix. Notably, suitable design for transition probability matrix M, is significant to train the discrete diffusion process. As illustrated in Section 3.1, through pre- trained pose encoder and FSQ. P can be converted to k- = (k1, ,ky). each k; corresponding to sub-structure of the overall pose. With this foundation, we specifically devise the occl ude and replace scheme, which is inspired by [24], for tackling the challenges of occluded 3D HPE. In occlusion scenes, the human body is always occluded in various situations (self-ocelusions, object or people-to-person occlusions), and the typical manifestation is that some sub-structures of the pose are invisible. Consequently, we design the occ ude scheme simulating the occlusion of corresponding joints, which introduces occlusion impact in the training process. Additionally, recognizing the inherent uncertainty in occlusion scenarios where a single occluded region may correspond to multiple potential 3D human poses, we develop the replace strategy to update certain token with another available token.
In practice, each quantized token k; has probability of to transition to the Oce token. Moreover, k; also subject to probability of [C],, to be uniformly resampled across all [C] categories. Furthermore, h; retains probability of a,, 1 |C]3, 7s to remain unchanged. Then, the transition matrix M, R[((++))x(((+ +1) is defined as:
Ta + 8,
3.
0
B.
a₂ + 8
M. =
(6)
%
%
where a.. 9, E [0, 1). The prior distribution of step S can be derived as: p(ks) B3s, Ps.
7s).
where āg = -I 0; 7s = 1 adapt the linear schedule [25] as noise schedule to pre-define the value of transition matrices (ās, 3g, and 75)) Subsequently, we can calculate (k, (k) according to Eq. (4). However, when the number of categories CI and time step S is too large, it can quickly become impractical to store all of the transition matrices M, in memory, as the memory usage grows like O(C(25). Actually, it is unnecessary to store all of the transition matrices. Instead we only store all of प. and B3 in advance, since we can calculate q(k,)k (o) according to following formula (refer to Appendix for proofs):
and 3g = (1 (1-68 as - 7/. In this study, we
M,c(ko) T.C(K)) + (5, B.)c((C| + 1) + B..
Training Objectives. We train a network fo (k,-1 [ks y) to estimate q(k, [k, ko) in the reverse process. The network is trained to minimize the variational lower bound (VLB):
Lus (k2 + +
--11 (DKa ılk,, (o),, kk, v)]}.
(8)

---

In addition, we follow [50, 24] to utilize the reparameterization trick, which lets Pose predict the noiseless token distribution fe (ko k y) at each reverse step, and then compute fa(k,- (2 y) as:
fo(ks- y) = Eko 19k- fo k., y)-
(9)
Based on the Eq. (9), an auxiliary denoising objective loss is introduced, which encourages the network to predict fo (Ko k. y):
Lko = - log fo (oo y)
(10)
Our final loss function is defined as:
L= A + Lulbs
(11)
where À is hyper-paramett to control the weight of the auxiliary loss Lio-
Diffussio Architecture. As depicted in Figure 2, our discrete diffusion model consists of three main components: an image encoder, a pose denoiser, and pose decoder. The pre-trained image encoder processes the 2D image to produce a conditional feature sequence. The pose denoiser, receiving the quantized pose tokens kg and and step S, predicts the distribution of noiseless tokens fo(kg kk: y)- This component equipped with several transformer blocks, each featuring an Adal .Norm operator [6], multi-head attention blocks that combine the image feature information with k, and layer normalization and linear layers. At the end of the reverse process, all recovered tokens obtained, and the final prediction of 3D pose is decoded by the well-trained pose decoder.
Training and Inference Process. In the training process, as for step 8, we sample k, from g(k, (k)) based on Eq. (7) in the forward process. We then estimate fokk,-ikk, y) in the reverse process. The final loss will be calculated according to Eq. (11). In the inference process, all pose tokens are either masked or initialized randomly. Subsequently. we predict fe fkkk, y) step by step until the tokens are completely recovered. Finally, reconstructed tokens are decoded by the pose decoder, resulting in the recovered 3D pose. The complete algorithms are summarized in Appendix.
4 Experiments
4.1 Datasets and Evaluation Metrics
Datasets. Human3.6M [34] is the most extensive benchmark for 3D HPE, consisting of 3.6 million images. We follow [22] with same protocol, which involves training on subjects S1, S5, S6, S7, and S8, and testing on subjects $9 and $11. 3DPW [72] is the first dataset the wild that includes video footage taken from a moving phone camera. We also evaluate our method on this dataset to measure the robustness and generalization. Additionally, to further verify the
we evaluate 2 Pose on the 3DPW-Oce [83], which subset of the 3DPW.
Evaluation Metrics. For Human3.6N, and 3DPW, we follow the standard protocols. Mcan per joint position error (MPJPE) calculates the mean Euclidean distance between the root-aligned reconstructed poses and ground truth joint coordinates. PA-MPJPE employs a Procrustes alignment between the poses before calculating the MPJPE. In addition, to further evaluate the effectiveness of our method under occlusion scenes, we devise an adversarial protocol, termed 3DPW-AdvOcc, following the previous research [84]. We apply occlusion patches the input image identify the most challenging predictions. This process involves assessing the relative performance degradation on the visible joints. Similar to [84], utilize textured patches generated by randomly cropping texture maps from the DTD [16]. We employ two square patch sizes: 40 and 80 relative to 256 x 192 image, denoted as Oce@ 40 and Occ@ @0 respectively, with stride of 10.
4.2 Implementation Details
Pose Quantization Step. The pose encoder is constructed with four Local-MLP blocks, while the pose decoder incorporates single block. Within these Local-MLP blocks, the embedding dimensions D for the pose encoder and decoder are configured to 2048 and 512, respectively. For the quantization process, the projected vector q, features the channel = 5. The levels per channel, denoted as L....., La). are specified as [7, 5, 5, 5, 5). The number of quantized tokens N is set to 100.
7

---

Table 1: Results on Human3. 6M in millimeters under MPJPE. The best results are in bold, and the second-bes best ones are underlined.
Methods
Dir
Disc
Fat
Gr.
Phon.
Phot.
Pose
Pur.
Sit
SitD.
Sm.
Wait
W.D.
Walk
W.T.
Avg
Pavlakos ef al.
[54]
cv 67.4
71.9
66.7
69.1
72.0
77.0
65.0
68.3
83.7
96.5
71.7
65.8
74.9
59.1
63.2
71.9
Martinez et al.
[48] icee1
51.8
56.2
58.1
59.0
69.5
78.4
55.2
58.1
74.0
94.6
62.3
59.1
65.1
49.5
52.4
62.9
Hossain al.
[29] ROCV'A
48.4
50.7
57.2
55.2
63.1
72.6
51.7
66.1
80.9
59.0
57.3
62.4
46.6
49.6
58.3
Zhao ol.
[85] cVPR119
48.2
60.8
51.8
64.0
64.6
53.6
51.1
67.
88.7
57.7
73.2
65.6
48.9
64.8
51,9
60.8
Liu al.
[45] cCCV1s
46.3
52.2
47.3
50.7
55.5
67.1
49.2
46.0
60.4
71.1
51.5
50.1
54.5
40.3
43.7
52.4
ef [77] cVewar
45.2
49.9
47.5
50.9
54.9
66.1
48.5
46.3
59.7
71.5
51.4
48.6
53.9
39.9
44,1
51.9
Zhao [88] cvew22
45.2
50.8
48.0
50.0
54.9
65.0
48.2
47.1
60.2
70.0
51.6
48.7
54.1
39.7
43.1
51.8
Geng al. [21] cVPw22
50.8
Choi et al. [14] 1005'22
44.3
51.6
46.3
51.1
50,3
54.3
49.4
45.9
57.7
71.6
48.6
49.1
52.1
44.0
44,4
50.7
Zhang
1990022
50.2
Gong al. [22]
cVew33
42.8
49.1
45.2
48.7
52.1
63.5
46.3
45.2
58.6
66.3
50.4
47.6
52.0
37.6
40.2
49.7
Pose (Ours)
41.9
47.8
45.0
49.0
51.5
62.2
45.7
45.6
57.6
67.1
50.1
51.4
37.3
40.9
49.2
Table 2: Evaluation on 3DPW, 3DPW-Oce, and 3DPW-AdvOce The number 40 and 80 after 3DPW- AdvOce denote the occluder size. * denotes the results from our implementation. The best results are in bold, and the second-best ones are underlined.
3DPW [72]
3DPW-Occ [83]
3DPW-AdvOco@40
3DPW-AdvOco@Bl
Methods
MPJPE JPA-MPJPE IMPIPE JPA-MPJPE |MPJPE JPA-MPJPE IMPJPE JPA-MPJPE
et [9]
112.9
69.6
115.8
72.3
241.1
101.4
355.9
116.3
Pavilo
[55] cuPe
1018
106.7
67.1
221.6
99.4
334.3
112.9
Cheng et al.
[12]
AMFZI
64.2
85.7
279.4
113.2
371.4
119.8
Zheng et al. [90]
118.2
73.1
132.8
80.5
247.9
106.2
359.6
115.5
Zhang et al. [82] rraar22
91.1
54.3
94.6
56.7
142.5
73.8
251.8
103.9
Geng et [21] cvPee
83.1
53.9
82.8
53.7
127.2
71.9
192.5
92.1
Gong [22] cvPe2s
82.7
82.1
121.4
20.9
189.3
92.4
Di Pose (Ours)
79.3
50.1
79.6
50.7
108.4
59.8
153.6
78.7
Discrete Diffusion Process. For the occlude and replace transition matrix, we linearly increase B, and F. from 0 to 0.1 and 0.9, respectively, and decrease a, from 0. For the discrete diffusion model, we use off-the-shelf image encoder [79] to extract feature sequence of conditional 2D image. As for the pose denoiser, we build a 21 21-layer 16-head transformer with the dimension of 1024. We set steps S as 100 and loss weight À is set to 5e-4. Please refer to Appendix for more details.
4.3 Comparsion with State-of-the-Arrt
Human3.6M. To explore the effectiveness of Pise, we evaluate its performance in the challenging context of frame-based 3D pose estimation. Specifically, within the discrete diffusion process, context information is extracted from single input frame using the image encoder. As shown in Table 1, we benchmark Di Pose against SOTA 3D HPE methods on the Human3 49.2mm in average MPJPE, surpassing the performance of the SOTA diffusion model [22] by 0.5mm, which indicates that 2 Pose is able to enhance monocular 3D HPE in indoor scenes.
Our Di achieves
3DPW. Beyond indoor settings, we evaluate the performance of Pose on the in-the-wild 3DPW dataset. As Table 2 shows, Di achieves the SOTA performance, and outperforms the SOTA method [22] by 3.4mm in MPJPE and 3. 3.7mm PA-MPJPE. On the occlusion-centrie 3DPW-Occ. Di Pose maintains its superiority. When assessed under the 3DPW-AdvOct protocol, all methods exhibit performance drops MPJPE surges by up 129% and PA-MPJPE by up 72%. Despite this, Pose remains markedly robust, leading the SOTA by significant margins in both MPJPE and PA-MPJPE, underscoring its effectiveness in handling occlusions.
Qualitative Results. Figure presents the qualitative results of DifiPose [22] in comparison with our Pose across two datasets. It can be observed that our method yields more accurate predictions than compared diffusion model (DiffPose), especially under various occlusion scenarios (self sself-ocelusion
and object occlusion). This demonstrates the superior occlusion-robustene of our 2 Pose.

---

Image
DiffPose
Di
Image
DiffPose
Di Pose
(a) Human36M
(b) 3DPW
Figure 4: Qualitative results two datasets. The black lines represent the ground truth poses and the blue lines are prediction results.
Table 3: Ablations on Human3.6M. P-1 and P-2 represent MPJPE and PA-MPIPE, respectively.
Loc. Num.
P-1
P-2
Levels
P-1
P-2
Occ. Rate
P-1
P-2
Training Steps
50.4
39.3
Inference Steps
25
50
100
14.5
13.0
5,55
0.3
50.7
39.3
25
51.7
51.1
50.3
13.6
12.5
5,5,5,5,5
13.6
12.5
0.6
49.5
49.2
39.0
39.1
50.6
49.9
5
14.1
12.8
8,
13.8
12.7
51.0
39.5
100
49.2
(a) Different local joint (b) Different levels per (c) Different final occlude number X of Joint Shift channel [L,,-., La] of rate Ts occlude operations JS- Block.
(d) Different number of training and inference quantization process FSQ. replace transition matrix.
steps S. P-1 are reported.
4.4 Ablation Study
Effectiveness of Pose Quantization Step. Our pose quantization step, which consists of Local-MLP blocks, is designed for repre- senting 3D human pose by capturing the local interactions between 3D joints. Table 4 displays the MPJPE metrics comparing the orig- inal 3D poses with those reconstructed via various methods. The results show that our pose quantization step reconstructs 3D poses with lower errors compared to previous method [21]. which uses an MLP-Mixer for global joint information extraction. indicates that our model learns more accurate representation of 3D poses. In addition, we conducted other ablation studies to investigate different local joint numbers and levels per channel [L..... La] within pose quantization step, as shown in Table 3a and Table 3b. As to different X, note that when X 1, we only extract feature of individual joint, and when X 1, JS-Block is able to capture local interactions of different joints. Experimental results indicate that X = 3 reaches lowest reconstruct error. As for [L,, La], the best level of FSQ for pose quantization [7, 5, 5, 5, 5).
Table 4: Different representa- tion methods for 3D HPE.
Pose Repr.
MPJPE PA-MPIPE
PCT [21]
15.9
Ours
13.6
12.5
Impact of Different Transition Matrices. To demonstrate the effec- tiveness of the specifically designed occlude and replace transition matrix, we constructed three transition matrices for discrete diffusion process: occlude transition matrix, replace transition matrix, and occlude and replace transition matrix. Table 5 illustrates that the optimal results are achieved when utilizing the occlude and replace transition matrix. The suboptimal performance observed when exclu- sively employing the other two transition matrices can attributed to the following reasons: Utilizing solely the replace transition ma- trix introduces the challenge random, irrelevant sub-structures, complicating the learning of the reverse process; Conversely, relying exclusively the occlude
Table 5: Different transition matrices for discrete diffusion model.
Matrices MPJPF PA-MPIPE
Occlude Replace
51.0
39,5
50.4
39.3
Both
49.2
39,0

---

Image
GT
Owr
Image
GT
Ours
(a) Human3.60 MM
(b) 3DPW
Figure 5: Failure cases of our Pose for 3D HPE. These instances primarily occur in scenarios with severe occlusions, compared against ground truth (GT) poses. The content encircled by the dashed line indicates the parts where differences exist.
transition matrix causes the model to overly focus on the occluded portions, neglecting the contextual information from other visible parts. These clarifications can be verified in Table 3c, where investigate the impact of different Ys- When Ts = 0, the occlude and replace transition matrix can be seen the replace transition matrix, and when 7s 1, the occlude and replace transition matrix can be seen as the occlude transition matrix. The best performance is obtained when Ys = 0.9.
In addition, conducted an ablation study to investigate the impact of S on the training and inference processes, as shown in Table 3d. We observed that using larger numbers of steps during both training and inference stages improves performance but also increases time complexity. Moreover, the results indicate that performance remains satisfactory even when the number of inference steps is reduced by 75% (e.g., from 100 steps during training 25 steps during inference). This finding suggests viable strategy for enhancing generation speed without significantly compromising quality.
Limitations
Figure illustrates several results of 3D human pose estimation. When substantial occlusions cover the human obscuribg the exact pose to the extent that it confounds even human observers-the predictions made by Di Pose may deviate from GT 3D pose. This deviation primarily stems from the inherent limitation of inferring 3D poses directly from 2D images, which lack critical spatial depth information. Such limitations introduce uncertainty and indeterminacy in the predictions.
Despite these challenges, Pose manages occlusions effectively by producing physically plausible outcomes. This capability is attributed to the integration of pose quantization step within Di Pose, which constrains the model's 's search space to physically reasonable configurations. Note that the pose quantization step is trained real 3D human pose data, enhancing its reliability under severe occlusions.
Currently, Di is primarily designed for frame-based 3D HPE and does not utilize interframe data from videos. Future enhancements will focus on incorporating interframe information to refine the accuracy of 3D pose predictions further within the Pose framework.
Conclusion
This paper presents PosP novel diffusion-basee framework that tackles occluded 3D HPE in discrete space. Pose first captures the local interactions of joints and represents 3D pose by multiple quantized tokens. Then, the discrete diffusion process models the discrete tokens in latent space through conditional diffusion model, which implicitly introduces occlusion into the modeling process for more reliable 3D HPE with occlusions. Experimental results show that our method surpasses the stte-of-the-are approaches on three widely used benchmarks.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China (62337001) and the Fundamental Research Funds for the Central Universities (226-2024-00058), Long Chen is supported by HKUST Special Support for Young Faculty (F0927) and HKUST Sports Science and Technology Research Grant (SSTRG24EG04).
References
[1] I. Akhter and M. J. Black. coss-conditioned joint angle limits for 3d human pose reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1446-1455 2015.
[2] T. Alldieck, H. Xu, and C. Sminchisescu. imghum: Implicit generative models of 3d human shape and articulated pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
5461-5470, 2021.
[3] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-1799, 2021.
[4] O. Avrahami, D. Lischinski, and O. Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208-
18218, 2022.
[5] S. Azadi, A. Shah, T. Hayes, D. Parikh, and S. Gupta. Make-an-animation: Large-scale text-conditional 3d human motion generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 15039-1500 15048, 2023.
[6] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint 1rXiv:1607.0645 2016.
[7] B. Biggs, D. Novotny, S. Ehrhardt, H. Joo, B. Graham, and A. Vedaldi. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Advances in neural information processing systems, 11220496-00500 2020.
[8] E. Brempong. Kornblith, T. Chen, N. Parmar, M. Minderer, and M. Norouzi. Denoising pretraining for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4175 4186, 2022.
[9] Y. Cai. Ge. Liu. Cai, T.-J. Cham, J. Yuan, and M. Thalmann. Exploiting spatial- temporal relationships for 3d pose estimation via graph convolutional networks. In Proceedings of the IEEE/CVF international conference computer vision, pages 2272-2281 2019.
[10] S. Chen, Sun, Song, and Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 99800-1843, 2023.
[11] T. Chen, C. Fang. X. Shen, Y. Zhu, Z. Chen, and J. Luo. Anatomy- aware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology. 32(1):198-209, 2021.
[12] Y. Cheng. B. Wang. B. Yang, and R. T. Tan. Graph and temporal convolutional networks for 3d multi person pose estimation in monocular videos. In Proceedings of the AAAI Conference on Artificial Intelligence,
volume 35, pages 1157-1165, 2021.
[13] H.-g. Chi, S. Chi, S. Chan, and K. Ramani. Pose relation transformer refine occlusions for human pose estimation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 6138-6145.
IEFF, 2023.
[14] J. Choi, D. Shim, and H. J. Kim. Diffupose: Monocular 3d human pose estimation via denoising diffusion
probabilistic model. In 2023 IEEF/RSI International Conference Intelligent Robots and Systems (IROS), pages 3773-3780. IEEE, 2023.
[15] H. Ci, C. Wang, X. Ma, and Y. Wang. Optimizing network structure for 3d human pose estimation. In
Proceedings of the IEEE/CVF international conference on computer vision, pages 2262-2271. 2019.
[16] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In
Proceedings of the IEEE conference computer vision and pattern recognition, pages 3606-3600000 3606-3613, 2014. [17] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
[18] Z. Fan, J. Liu, and Y. Wang. Motion adaptive pose estimation from compressed videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11719-11728. 2021.
[19] R. Feng. Y. Gao, T. H. F. Tse, X. Ma, and H. J. Chang. Diffpose: Spatiotemporal diffusion model for video-based human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14861-14872, 2023.
[20] L. G. Foo. J. Gong. Z. Fan, and J. Liu. jystem-status-ee aware adaptive network for online streaming video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10514-10523, 2023.
[21] Z. Geng. C. Wang. Y. Wei, Z. Liu, H. Li, and H. Hu. Human pose compositional tokens. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 660-671, 2023. 2023.
11

---

[22] J. Gong, L. G. Foo, Z. Fan. Q. Ke. H. Rahmani, and J. Liu. Diffpose: Toward more reliable 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 13041-130S1, 2023.
[23] S. Gong, M. Li. J. Feng, Z. Wu, and L. Kong. Diffuseq-v2: Bridging discrete and continuous text spaces for accelerated seq2seq diffusion models. arXiv preprint arXiv:2310.0579) 2023.
2023.
[24] S. Gu, D. Chen, J. Bao, F. Wen. B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[25] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information
processing systems, 33:6840-66 161400066111666 2020.
[26] J. Ho, C. Saharia, W. Chan,
Fieet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(4)),1-33, 2022.
[27] K. Holmquist and B. Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In
Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15977-15987, 2023. [28] E. Hoogchoom, D. Nielsen, P. Jaini, P. Forré, and M. Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12466 2021.
[29] M. R. I. Hossain and J. J. Little. Exploiting temporal information for 3d human pose estimation. In
P Proceedings of the European conference on computer vision (ECCV), pages 68-84, 2018
[30] M. Hu, Y. Wang, T.-J. Cham, J. Yang, and P. N. Suganthan. Global context with discrete diffusion vector quantised modelling for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11502-1151, 2022.
[31] W. Hu, C. Zhang, F. Zhan, L. Zhang, and T.-T. Wong. Conditional directed graph convolution for 3d human pose estimation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2021.
[32] R. Huang, Z. Zhao, H. Liu, J. Liu, C. Cui, and Y. Ren. Prodiff: Progressive fast diffusion model for high-quality text-to- -oseeo.. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2595 2605, 2022.
[33] N. Inoue, K. Kikuchi, E. Simo-Serra, M. Otani, and K. Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10167-10176 2023.
[34] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-1)))
Large scale datasets and predictive
2013.
[35] W. Jiang. S. Jin, W. Liu, C. Qian, P. Luo, and S. Liu, Posetrans: A simple yet effective pose transformation augmentation for human pose estimation. In European Conference on Computer Vision, pages 643-659.
Springer, 2022.
[36] Z. Jiang, Z. Wang, and L. Chen. Combing text-based and drag-based editing for precise and flexible image editing. arXiv preprint arXiv:2410.03097, 2024.
[37] Z. Jiang, Z. Zhou, L. Li, W. Chai, C.-Y. Yang, and J.-N. Hwang. Back to optimization: Diffusion-based zero-shot 3d human pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6142 6152, 2024.
[38] H. Joo, N. Neverova, and A. Vedaldi. Exemplar fine-tuning for 3d human model fitting towards in-the-wild
3d human pose estimation. In 2021 International Conference on 3D Vision (3DV), pages 42 52. IEEE,
2021.
[39] M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black. Pare: Part attention regressor for 3d human body estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11127 11137, 2021.
[40] H. Kong, K. Gong, D. Lian, M. B. Mi, and X. Wang. Priority-centric human motion generation in discrete latent space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14806-14816, 2023.
[41] J. N. Kundu, S. Seth, M. Rahul, M. Rakesh, V. B. Radhakrishnan, and A. Chakraborty. Kinematic- stracture-preservedd representation for unsupervised 3d human pose estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11312-11319, 2020.
[42] D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
pages 11523-
[43] J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu, and C. Lu. Human pose regression with residual log- likelihood estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11025 11034, October 2021.
[44] J. Li, C. Xu, Z. Chen, S. Bian, L. Yang, and C. Lu. Hybrik: A hybrid amalytical-neure inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3383-33933 2021.
12

---

[45] K. Liu, R. Ding, Z. Zou, L. Wang. and W. Tang. A comprehensive study of weight sharing in graph networks for 3d human pose estimation. In Proceedings of the European conference on computer vision (ECCV), pages 318-334. Springer, 2020.
[46] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 171000101,,
2017.
[47] W. Mao, Y. Ge. C. Shen, Z. Tian, X. Wang, Z. Wang, and A. v. den Hengel. Poseur: Direct human pose regression with transformers. In Proceedings of the European conference on computer vision (ECCV), pages 72-88. Springer, 2022.
[48] J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective haseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision, pages 2640-2649.
2017.
[49] F. Mentzer, D. Minnen, F. Agustsson, and M. Tschannen. Finite scalar quantization: Vq-vac made simple. arXiv preprint arXiv:2309. 15505, 2023.
[50] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pages 8162-8171. PMLR, 2021.
[51] Q Nie, Z. Liu, and Liu. Lifting 2d human pose to 3d with domain adapted 3d body concept. International Journal of Computer Vision, (5): 2200- 1268, 2023.
[52] S. Park, J. Hwang, and N. Kwak. 3d human pose estimation using convolutional neural networks with 2d pose information. In Computer Vision ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 156-169. Springer. 2016.
[53] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[54] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to- torsiee volumetric prediction for single- image 3d human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7025-7034, 2017.
[55] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 3d human pose estimation in video with temporal convolutions and semi- supervised training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7753-7762, 2019.
[56] J. Peng. D. Liu, S. Xu, and H. Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10775- 10784, 2021.
[57] L. A. Petrov, R. Marin, J. Chibane, and G. Pons-Moll. Object pop-up: Can infer 3d objects and their poses from human interactions alone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4726-4736, 2023.
[58] I. Radwan, A. Dhall, and R. Goecke. Monocular image 3d human posc estimation under self-occlusion In Proceedings of the IEEE International Conference on Computer Vision, pages 1888 1895, 2013.
[59] A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[60] C. Rockwell and D. F. Fouhcy. Full-body awareness from partial observations. In European Conference on Computer Vision, pages 522-539, Springer, 2020.
[61] G. Rogez and C. Schmid. Mocap-guided data augmentation for 3d pose estimation in the wild. Advances in neural information processing systems, 29, 2016.
[62] G. Rogez, P. Weinzacpfel, and C. Schmid. Ler-net: paallization-laassiicaiooo regresssoon for human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3433-3441, 2017.
[63] C. Rommel, E. Valle, M. Chea, S. Khalfaoui, R. Marlet, M. Cord, and P. Pérez. Difthpp: Robust, coherent 3d human pose lifting with diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3220-3229, 2023.
[64] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe. How rohust is 3d human pose estimation to occlusion? arXiv preprint arXiv:1808 09316. 2018
[65] W. Shan, Z. Liu, X. Zhang. S. Wang, S. Ma, and W. Gao. P-stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation. In European Conference on Computer Vision, pages 461-478. Springer, 2022.
[66] W. Shan, Z. Liu, X. Zhang, Z. Wang, K. Han, S. Wang. S. Ma, and W. Gao. Difflsioon-based 3d human pose estimation with multi- hollii-hypohhesis aggregation. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 14761-1477) 2023.
[67] J. Sohl-Dickstein, F. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
[68] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei. Integral human pose regression. In Proceedings of the European conference computer vision (ECCV), pages 529-545, 2018
13

---

[69] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov. L. Beyer, X. Zhai. T. Unterthiner, J. Yung, A. Steiner. D. Keysers, J. Uszkoreit, al. Mip-mixer: An all-mip architecture for vision. Advances neural information processing systems, 34:24261-242777 2021.
[70] S. Tripathi, L. Müller, C.-H. P. Huang, O. Taheri. M. Black, and D. Tzionas. 3d human pose estimation via intuitive physics. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition (CVPR), pages 4713-4725, 2023.
[71] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[72] T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-611 2018.
[73] J. Wang. S. Yan, Y. Xiong, and D. Lin. Motion guided 3d pose estimation from videos. In European conference on computer vision, pages 764-780. Springer, 2020.
[74] Z. Wang, Y. Jiang, Zheng, Xiao, and L. Chen. Exstt--ustomized image generation. arXiv preprint arXiv:2410.02455 02483, 2024,
[75] E. Xie. L. Yao, H. Shi, Z. Liu, D. Zhou, Z. Liu, J. Li, and Z. Li. Difflit: Unlocking transferability of large diffuusion models via simple parameter-effiienn fine-tuning, Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4230-422 4239,, 2023.
[76] J. Xu, Z. Yu, Ni, J. Yang. X. Yang, and W. Zhang. Deep kinematies analysis for monocular 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on computer vision and Pottern recognition, pages 899-908, 2020.
1771 T. Xu and W. Takano. Graph stacked hourglass networks for 3d human posc estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6105-16114 14, 2021. [78] Y. Xu, Z. Wang. J. Xiao, W. Liu, and L. Chen. Freetuner: Any subject in any style with training-free
diffusion. arXiv preprint 24455 240.2.4251,21 2024.
[79] Y. Xu. J. Zhang, Q. Zhang, and D. Tao. Vitpose: Simple vision transformer baselines for human pose
estimation. Advances in Neural Information Processing Systems, 35:38571 --3554, 2022.
[80] D. Yang. Yu. Wang. W. Wang. C. Weng. Y. Zou, and Yu. Diffsound: Discrete diffiusion model for
text-to-sound generation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[81] A. Zanfir, M. Zanfir, A. Gorban, J. Ji, Y. Zhou, D. Anguelov, and C. Sminchisescu. Hum3dil: Semi-
supervised multi-modal 3d humanpose estimation for autonomous driving. Conference on Robot Learning, pages 1 1114-1124 PMLR, 2023.
[82] J. Zhang. K. Gong, X. Wang, and J. Feng. Learning to augment poses for 3d human pose estimation in images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[83] T. Zhang. B. Huang, and Y. Wang, Object-oceluded human shape and pose estimation from single color image. In Proceedings of the IEEE/CVF conference on compuier vision and pattern recognition, pages 7376-7385, 2020.
[84] Y. Zhang. P. Ji. A. Wang, J. Mei. A. Kortylewski, and A. Yuille. 3d-aware neural body fitting for occlusion robust 3d human posc estimation. In Proceedings of the IEEF/CVF International Conference on Computer
Vision. pages 9399-9410, 2023.
[85] L. Zhao, X. Peng, Y. Tian, M. Kapadia, and D. N. Metaxas. Semantic graph convolutional networks for
3d human pose regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3425-3435, 2019.
[86] Q Zhao, C. Zheng, M. Liu, and C. Chen. A single 2d pose with context is worth hundreds for 3d human posc estimation. Advances in Neural Information Processing Systems, 36, 2024.
[87] Q Zhao, C. Zheng, M. Liu, Wang, and C. Chen. Poseformer22: Exploring frequency domain for efficient and robust 3d human posc estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 8877-8886, 2023.
[88] Zhao, W. Wang, and Y. Tian. Graformer: Graph-oriented transformer for 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20438- 20447, 2022.
[89] C. Zheng and A. Vedaldi. Online clustered codebook. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 22798-22801 2023.
[90] C. Zheng, S. Zhu, M. Mendicta, T. Yang, C. Chen, and Z. Ding. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pages 11656-1 100-11166,,,222
[91] J. Zhou, T. Zhang, Z. Hayder, L.. Petersson, and M. Harandi. Diff3dhpe: A diffusion model for 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2092-2102, 2023.
14

---

Appendix
In this Appendix, we provide relevant preliminary knowledge, mathematical proofs, complete training and inference algorithms, additional experimental results, more implementation details about our Pose and broader impacts.
A Preliminary: Continuous Diffusion Model
The continuous diffusion model consists of two primary processes: the forward process and the reverse process. The forward process methodically corrupts the original data Do into a noisy latent variable sg. which converges stationary distribution (c.g.. a Gaussian distribution). Conversely, the reverse process aims to reconstruct the original data 2g from æs, utilizing learned parameters.
Forward Process Starting with Eu drawn from the distribution (<x),, the forward process incremen- tally corrupts 2o through a sequence of latent variables 11.5 (æ) 22; æs). where each æ, retains the same dimensionality as This transformation is modeled as a fixed Markov chain:
(x11)|10)) = II
q (1 x,,1)
(12)
where each transition /(>,//.... x., is defined by a Gaussian distribution:
((m.lass))) N(ss; / 11, 11
(13)
Here, 7/s a small positive constant that follows a predefined schedule (1) .......... ())))...ss)............. allowing the data to progressively approach isotropic Gaussian distribution, ((0II) I), as increases. The overall transition from Vy to Dg can thus be expressed as:
((x,/x0) = Naxsi / (1 (.)
(14)
where C. = 1 1)s and C, I1 Si-
Reverse Process In the reverse process, the model aims to convert the latent variable æs. which A
assumed to follow the distribution (0, I). back into the original data ag. The joint probability distribution is given by:
S
po (mo.s) = p(æs) II
po (o (xx.)
(15)
The conditional distributions involved are inferred using Bayes rule as follows:
12. 11,,, 1, (0)
q( 0( I., 20)
(16)
q<xx/xx)
To optimize the generative model po(2)) for fitting the data distribution q(zo), minimize variational upper bound on the negative log- lieeliohh:ood:
Leb = (g(()) [xilg(æs|<z) [p(=s)] ))++
Σ Eyts,, (mo) [Dzzl( 2o)l (17)
However, continuous diffusion models are not applicable in discrete spaces, such as quantized token indices k (ki, kg...., kN) where each k; assumes one of C³ discrete values. This limitation arises because Gaussian noise cannot corrupt discrete elements in a meaningful way. Thus, modeling in discrete spaces necessitates the development of discrete diffusion processes.
B Mathematical Proofs
In this section, we provide detailed mathematical proofs for Eq. (6), which can quickly calculate q(k, ko) according to Eq. (2).
15

---

Concretely, we use mathematical induction to prove Eq. (6). At first, we have following conditional information:
0:.: 33 E |00, 11. 0s 1 |c|3. 7ap
(18)
- n,,,3, = -- -
==1
Now we want to prove that M.c(ko) (.c(()) + (7. -B,)c(IC] + 1) + B.g Firstly, when 1, we have:
01
B,
kg
Mic(ko)
31.
k + ko and + C
(19)
R1
k = C7 + 1
which is clearly hold. Suppose the Eq. (6) holds at step then for s = s + 1. we have:
M M,
(20)
Now we consider three conditions:
(1) when k kn in step + 1, we have:
MatIc(Ko)(k) = 3,3 - (ast1 + ( +B.)
= -,(1C184+1 + a.+1) (.,t+ + 8.+1)
1 (B,(1 9 (1(1-))11 a.9.11 B.+1) * [C] + aot1 + B.+1
C
(21)
C [11 a. -1. 7041) [C|a.0.41 (1 -a++1 7+till + 2.41 Bet1
1
C -(+1)) -,((1 70+1 KB,+1) (0-7) a+t1] + 0st1 Bst1
1 + +Bat1-
(2) when [C] in step s + 1, we have:
Ms+1c(ko)(k) = 7, + (1 - 7.)7++1 = 7o+1) = Tot1
(22)
(3) when k + kn and + |C + in step s + 1, we have:
M Mstic(ko) (k) = (a,a--1 + 8s+1) + +3,85+1 1) + a,
Ba [C13,+] + 2,
C.. Ts (1 - 7 + a,3
[C|
(23)
1 7s+1
C 7o+1) + aB.41
[C]
The proof of Eq. (6) completed. Notably, according to Eq. (6), the computation cost of g g(k, kn) can be reduced from O((C(2)) to O(CC).
C Algorithms for Discrete Diffusion Process
In this section, we provide complete training and inference algorithms for discrete diffusion process.
C.1 Training Procedure
The discrete diffusion process aims to model quantized 3D pose tokens in discrete space. This involves utilizing a 2D image and its corresponding 3D human pose as inputs. The image serves contextual condition, while P is converted into discrete tokens for modeling.
16

---

Algorithm Training Algorithm for the discrete diffusion process.
Require:
A transition matrix M,, the number of steps S. parameters of pose denoiser 0, training epoch T. pose dataset D (including 2D image I and 3D human pose P), and the well-lcarned pose encoder (-g()).
1: for 2 = 1 to T do
2: for (7, P) in D do
=ImageEncoder(D) 4: sample s from Uniform [1, 2, ..., 1, } calculate (kk))k) based on Eq. (6);
3: kn = FSQ(FrE()))
5:
6:
estimate fa(k,-1 k. y)
7:
calculate loss according to Eq. (10);
8: update 0;
9: end for
10: end for
11: return 0.
Algorithm 2 Inference Algorithm for the discrete diffusion process.
Require:
The number of steps S, input 2D image I, the pose decoder fPD(-), parameters of pose denoiser
0, stationary distribution p(k pkss;
1: S. IImageEnnoder([]]
2: sample k, from p(ks); do
3: while
4: k, < sample from pe (k.-1 k,, 3)
5: 8 8 8 11
6: end while
7: return (PD(
Firstly, the 3D human pose P encoded by fPE(-) and subsequently quantized using the FSQ technique, resulting in multiple discrete tokens. Concurrently, tre-trained Image Encoder extracts contextual features from I, producing a conditional feature sequence y. During the forward process, sample from uniform distribution {1, 2, S S} and compute q(k ko) based on Eq. (6). In the reverse process, the pose denoiser fo k,, y) is trained to estimate q(,,- kk, kn). Finally, the overall loss is calculated according to Eq. (10), and the parameters of the pose denoiser 0 are updated accordingly.
The complete training algorithm for the discrete diffusion process is presented in Algorithm 1.
C.2 Inference Procedure
In the inference process, our objective is to recover the 3D human pose P from an input 2D image and discrete tokens.
Initially, all pose tokens are either masked or initialized randomly, which is achieved by sampling from the stationary distribution p(ks). The 2D image is encoded using the pre-trained Image Encoder. Subsequently, we predict fo(k,-1 kg, y) step by step until the pose tokens are fully recovered. Finally, the reconstructed tokens decoded using the pose decoder fre(-), yielding the recovered 3D pose P.
The complete inference algorithm for the discrete diffusion process is presented in Algorithm
D Additional Implementation Details
All experiments are carried out on one NVIDIA A PCIe GPU. The proposed Pose is completely implemented PyTorch [53]. In this section, we provide the detailed training settings for the pose quantization step and the discrete diffusion process.
17

---

Table Results on Human3. 6M in millimeters under PA-MPIPE. The best results are in bold, and the second-bes, ones are underlined.
Methods
Dir
Disc
Eat
Greet
Phone
Photo
Pose
Pur
Sit
SitD
Smoke
Wait
WalkD
Walk
WalkT]
Avg
Martinez al. [48]
JCCV11
39.5
43.2
46.4
47.0
51.0
56.0
41.4
40.6
56.5
69.4
49.2
45.0
49.5
38.0
43.1
47.7
Pavlakos et ol.
[54] cwwrr
34.7
39.8
41.8
38.6
42.5
47.5
38.0
36.6
50.7
56.8
42.6
39.6
43.9
32.1
36.5
41.8
Liu [45]
35.9
40.0
38.0
41.5
42.5
51.4
37.8
36.0
48.6
56.6
41.8
38.3
42.7
31.7
36.2
41.2
Zhang al.
39.1
Choi al. [14]
180523
36.7
41.1
37.6
42.2
40.5
44.1
37.8
36.3
47.0
60.5
39.8
38.9
42.7
33.7
35.1
40.9
Gong of al. [22] cvew22
33,9
38.2
36.0
39.2
40.2
46.5
35.8
34.8
48.0
52.5
41.2
36.5
40.9
30.3
33.8
39.2
Di' (Ours)
34.5
38.4
35.1
40.8
39.8
47.0
34,9
34.7
47.1
52.3
40.4
36.1
42.9
30.0
33.4
|39.0
For pose quantization step, we employ the AdamW [46] optimizer with 31 = 0.9 and B3 0.999,
adhering base learning rate of le-3 and weight decay parameter of 0.15. The training process is configured with a batch size of 256 across a total 20 epochs.
For the discrete diffusion process, we still utilize the the AdamW optimizer with 31 = 0.9 and 32 = 0.96, adhering to base learning rate 5.5e-4 and a weight decay parameter of 4.5e-2. The training process is configured with batch size of 64 across total of 50 epochs.
E
Additional Experimental Results
We exhibit more experimental results to verify the effectiveness of our Pose.
E.1
Quantitative Results
As shown in Table we benchmark Pose against SOTA 3D HPE methods the Human3.6M under PA-MPJPE protocol. Our Di Pose achieves 39. 990m in average PA-MPJPE, surpassing the performance of the compared SOTA 3D HPE methods, which indicates that Dise is able to enhance monocular 3D HPE in indoor scenes.
E.2 Qualitative Results
In this part, we present additional qualitative results on the Human3.6M and 3DPW datasets. As illustrated in Figure 6, our PisPose model demonstrates the ability to accurately recover 3D human poses in both indoor and in-the-wild scenarios. Particularly noteworthy is its performance under various occlusion conditions, including self-occlusion and object occlusion. Even in these challenging situations, Di Pose consistently produces reasonable 3D pose estimations, highlighting its robustness occlusions.
F Broader Impacts
This research focuses on estimating physically valid 3D human poses from monocular frames, espe- cially under occlusion scenes. Such method can be positively used for sports analysis, surveillance, healthcare, autonomous driving, etc. where clear, unobstructed views of the subject may not always be available.
also lead to malicious use cases, such as illegal surveillance and video synthesis. Thus, it is essential to deploy these algorithms with care and make sure that the extracted human poses are with consent and not misused. Moreover, compared to other CNN or GCN-based methods, causing more computational resources and energy consumption.
iiffusion-basee model has longer runtime
18

---

Image
GT
Ours
Imegge
GT
Ous
(a) Human3.6M
3DPW
Figure 6: Qualitative results on two datasets. Joints on the right side are marked in green, while other joints highlighted in blue.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer to the Abstract and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer to Sec.5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Refer to Sec.3 and Sec.B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Refer Sec.4.1, Sec.4.2 and Sec.D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We will release code upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: /tts:s////pp///eee cc/
publ / guides
des CodeSuumm ssion sssioooll Mopollyyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. cc/ cttppcccpcccccccccc pubppcplcooooo wwwp//uuisssseeeeeeeeeeee Code Submi ssissiionnn ceiionnoliinn ooelllyyyyy....o. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Refer to Sec.4.2 and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Refer to Sec.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer Sec. in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /neurips..... http/pubccc/chhhculleeee
Answer:
[Yes]
Justification:
We make sure to conduct this paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer Sec. in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All used datasets and models in this paper are explicitly mentioned and properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper raperswi swi thoode com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not involve this issue.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We use publicly available 3D human pose datasets in this paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve this issue.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "qKVJq2ardP", "reviewer": "Reviewer_Z93t", "summary": "The paper presents  novel diffusion-based framework for occluded 3D Human Pose Estimation (HPE) that operates in discrete space. Di2Pose leverages a two-stage process: a pose quantization step and a discrete diffusion process. The pose quantization step captures the local interactions between joints and represents the 3D pose as multiple quantized tokens. These tokens are then modeled in the latent space through a discrete diffusion process. This approach allows the framework to effectively manage occlusions by simulating the transition of a 3D pose from occluded to recovered, enhancing the reliability of pose estimation under occlusion conditions. Extensive evaluations on benchmarks like Human3.6M, 3DPW, and 3DPW-Occ demonstrate that Di2Pose outperforms state-of-the-art methods, particularly in occluded scenarios.", "questions": "See above", "limitations": "Limitations are not explored or discussed. It follows the same formula - compare against a benchmark and do ablation studies and state it is better than state of art.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Introduction of Di2Pose Framework\n   Di2Pose integrates the inherent discreteness of 3D pose data into the diffusion model, providing a novel paradigm for addressing 3D HPE under occlusions. This framework leverages a two-stage process involving pose quantization and discrete diffusion to confine the search space to physically plausible configurations and simulate the transition from occluded to recovered poses.\n\nPose Quantization Step:\n   The designed pose quantization step effectively captures local correlations between joints by representing 3D poses in a compositional manner. This step confines the search space to reasonable configurations by learning from real 3D human poses, ensuring that the model generates physically plausible poses even under severe occlusions.\n\nDiscrete Diffusion Process\n   The discrete diffusion process simulates the complete transition of a 3D pose from occluded to recovered, incorporating the impact of occlusions into the pose estimation process. This process models the quantized pose tokens in latent space, enhancing the model’s capability to understand and predict occluded parts of the human pose.\n\nStrengths claimed and shown include\n\nEffectiveness in Occluded Scenarios: Di2Pose demonstrates significant improvements in 3D HPE accuracy under occlusions compared to state-of-the-art methods, highlighting its superior occlusion-handling capabilities.\n  \nPhysically Plausible Pose Generation: By integrating pose quantization and discrete diffusion, Di2Pose confines the search space to physically reasonable configurations, ensuring the generation of biomechanically valid poses even in challenging occluded scenarios.\n\nComprehensive Evaluation: The framework has been extensively evaluated on multiple challenging benchmarks, consistently yielding lower errors and demonstrating its robustness and generalizability across different datasets.\n\nUse of Discrete Diffusion: The introduction of a discrete diffusion process tailored for 3D HPE provides a new perspective in the field, aligning more closely with the inherent discreteness of 3D pose data.\n\nThese contributions collectively position Di2Pose as a robust and innovative solution for 3D HPE, particularly in the presence of occlusions, advancing the state-of-the-art in this challenging domain.", "weaknesses": "Mechanistic Insights:\n   The paper does not delve deeply into the mechanistic aspects of how the discrete diffusion process works in conjunction with pose quantization. Detailed explanations of the underlying mechanisms and how they contribute to the observed improvements would enhance understanding.\n\nComparative Benchmarking:\n   While Di2Pose shows improvements over existing methods, a direct comparison with other contemporary methods such as the Pose Relation Transformer (Chi et al., ICRA 2023) and InfoGCN (Chi et al., CVPR 2022) would provide a clearer picture of its relative strengths and weaknesses.\n\nCitation of Related Works:\n   Incorporating references to other successful approaches in the field, such as the above\n   \n These citations would position Di2Pose within the broader context of ongoing research and highlight its unique contributions relative to other leading methods.\n\nAddressing these areas would strengthen the paper, providing deeper insights into the framework’s operation and situating it more firmly within the current landscape of 3D HPE research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n\n## Q1: Detailed explanations and insights about the two main parts: pose quantization and discrete diffusion.\n\nThe proposed Di²Pose is a two-stage framework designed to address the challenges of occluded 3D human pose estimation (HPE).\n\n- **In the first stage**, we train a pose quantization step that transforms a 3D pose  $\\mathbf{P}$ into $N$ discrete tokens $\\mathbf{k}$. Each token represents a sub-structure of the whole pose. This pose quantization step leverages the discrete nature of 3D poses and **represents them as quantized tokens** by capturing the local interactions between joints. This quantization step is crucial for the subsequent discrete diffusion process, as *it allows the discrete diffusion model to simulate occlusions of specific sub-structures of the 3D pose*. The quantized tokens $\\mathbf{k}$ **serve as a vital link that binds pose quantization and the discrete diffusion process together**, ensuring coherent interaction between the two stages.\n- **In the second stage**, we model tokens in the discrete space using a discrete diffusion process, which consists of a forward and a reverse process.\n    \n    **Forward Process:** During the forward process, each token of $\\mathbf{k}$ is probabilistically occluded with an Occ. token or replaced with another available token. The **occluded token** represents the *occlusion of the corresponding sub-structure of the 3D pose*. The **token replacement** mechanism is designed to *enhance the diversity of potential sub-structures*, reflecting the indeterminacy in occluded parts.\n    \n    **Reverse Process:** In the reverse process, the pose tokens are initially occluded or randomly initialized. The denoising diffusion process estimates the probability density of pose tokens step-by-step based on the input 2D image until the tokens are completely reconstructed. **Each step leverages contextual information from all tokens of the whole pose as predicted in the previous step**, facilitating the estimation of a new probability density distribution and the prediction of the current step’s tokens. This sequential approach ensures a detailed and accurate reconstruction of 3D poses from occluded scenes.\n    \n\nThis two-stage framework allows Di²Pose to effectively handle occlusions by breaking down the pose into meaningful sub-structures and reconstructing them through a probabilistic diffusion process. The integration of pose quantization with the discrete diffusion process significantly contributes to the observed improvements in handling occluded poses.\n\n## Q2: Comparison with related works\n\nWe compare our Di²Pose with the mentioned related works as follows.\n\n- **Pose Relation Transformer (PORT) [1]:** PORT is designed to address the HPE problem, mitigating the effect of occlusions inspired by the sentence completion task in NLP. The **similarity** between PORT and our Di²Pose is that *both methods recognize the importance of capturing local context between adjacent joints*. PORT leverages an attention mechanism to aggregate adjacent joint features, while Di²Pose proposes a Local-MLP to capture local relationships within a sub-structure of the 3D pose. Moreover, PORT introduces a Masked Joint Modeling (MJM) approach to reconstruct randomly masked joints, which helps refine occlusions.\n    \n    However, MJM randomly selects joint indices for masking and trains PORT to reconstruct the masked joints, **explicitly simulating occlusions** and treating the masked joints as independent. In contrast, Di²Pose uses a discrete diffusion process to **implicitly model occlusion** within the latent space, enhancing its understanding of how occlusions affect human poses. Additionally, we found that **PORT is specifically designed for 2D HPE**, which **differs from our focus on 3D HPE**. Thus, we cannot directly compare these two methods by experiments.\n    \n- **InfoGCN [2]:** InfoGCN is proposed to solve the human skeleton-based action recognition task. This novel method focuses on embedding physical constraints and intention information into the latent representations of human actions. The **similarity** between InfoGCN and Di²Pose is that *both methods aim to learn informative latent representations from raw data*. InfoGCN introduces a novel learning objective based on the information bottleneck theory, which aims to learn an efficiently compressed latent representation of an action. However, Di²Pose proposes a pose quantization step, which leverages VQ-VAE to convert a 3D pose into multiple discrete latent tokens, **compressing information in different ways**. Moreover, **InfoGCN focuses on the human skeleton-based action recognition task**, which **is different from 3D HPE**. Therefore, we also cannot directly compare our method with InforGCN by experiments.\n\nAlthough the above works focus on different tasks, we recognize the importance of discussing these methods to provide a comprehensive context for our work. To address this, **we will cite these two works in the Related Work section** and add these discussions  about their contributions and differences with our approach. Additionally, we will further investigate other relevant works to enhance the completeness and context of our study.\n\n## Q3: Limitations\n\nWe have illustrated the limitations of our method in detail in the **Appendix Sec.F**, including failure cases, physically reasonable outcomes, and frame-based limitations.\n\n[1] Pose Relation Transformer Refine Occlusions for Human Pose Estimation. In ICRA, 2023\n\n[2] InfoGCN: Representation Learning for Human Skeleton-based Action Recognition. In CVPR, 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VMMHnCQ1MH", "reviewer": "Reviewer_tz5G", "summary": "This work claims that the 3D human pose of a single frame is discrete, and learns the local pairing relationship between joint points to generate the human pose under occlusion. At the method level, VQ-VAE is used for human skeleton quantization, and then combined with the diffusion model to solve this discrete relationship. The main contributions are as follows: 1. The Di2Pose framework is proposed, which integrates the inherent discreteness of 3D pose data into the diffusion model, providing a new paradigm for 3D HPE under occlusion. 2. The designed pose quantization step represents the 3D pose in a combinatorial manner, effectively captures the local correlation between joints, and limits the search space to reasonable configurations. 3. The constructed discrete diffusion process simulates the complete process of 3D pose from occlusion to recovery, and introduces the impact of occlusion into the pose estimation process.", "questions": "I hope to see the results on new datasets and SMPL renders to images. I will consider improving my score.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The structure and writing of the article are relatively clear.\n2. The inherent discreteness of 3D pose data is integrated into the diffusion model. Although this is a downstream task of VQ-VAE+Diffusion, it is indeed a new idea for 3D HPE of monocular 2D images.\n3. The reasoning process of Di^2Pose in Chapter 3 is very clear, which helps to understand the principle of the overall method.\n3. The influence of occlusion is introduced into the pose estimation process, and the codebook that encodes the human pose is back-diffused, which is also a solution to occluded human pose.", "weaknesses": "1. From an intuitive level, the human skeleton in this work still uses the simplest H3.6M skeleton with only 17 joints, not even the version with 32 key points. The error tolerance rate is very high for pose estimation of models with such a small number of joints. There is not even any obvious joint change. The Average MPJPE value in the Human3.6M list of 17 joint skeletons has stabilized within 30. The human skeleton in this work needs to use more complex SMPL, SMPL-X, and STAR to see the effect better.\n2. The image and skeleton prediction are independent. I hope the author can render the skeleton into the image and map it in the image to better observe the subtle differences in movement. It is difficult to observe the 3D skeleton alone, especially in Figure 4. I can't even find any obvious difference in the human posture in Figure a. In addition, when estimating a single-frame 3D skeleton, it is best to give results from multiple perspectives. I think the perspectives shown in the article are too casual.\n3. Although the method is clearly written, I think Di^2Pose is just a downstream task of VQ-VAE+Diffusion. In terms of design, it is not even as clever as [1]. I think Di^2Pose is just DiffPose with the addition of VQ-VAE.\n4. The dataset used in the experiment is too outdated. I hope to see more new datasets for verification. Like EMDB[2], SLOPER4D[3], CIMI4D[4], etc.\n\n[1] Feng H, Ma W, Gao Q, et al. Stratified Avatar Generation from Sparse Observations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 153-163.\n[2] Kaufmann M, Song J, Guo C, et al. Emdb: The electromagnetic database of global 3d human pose and shape in the wild[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 14632-14643.\n[3] Yan M, Wang X, Dai Y, et al. Cimi4d: A large multimodal climbing motion dataset under human-scene interactions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 12977-12988.\n[4] Dai Y, Lin Y T, Lin X P, et al. Sloper4d: A scene-aware dataset for global 4d human pose estimation in urban environments[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 682-692.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n\n## Q1: Extended experiments on more complex datasets\n\n**Clarification**. In 3D skeleton-based HPE task, the **17-joint annotation** of the Human3.6M is a **widely-used and standard benchmark**. Most mainstream methods validate their methods on this dataset. For **fair comparisons**, we also follwed this setting in our experiments. As described in [3], this limited number of joints helps discard the smallest links associated to details for the hands and feet, going as far down the kinematic chain to only reach the wrist and the ankle joints.\" This has led subsequent works to predominantly use the 17 joints for experiments rather than the 32 joints.\n\n**Extra results on a new dataset**: We acknowledge the reviewer's concern that the 17-joint Human3.6M dataset is relatively simple. Thus, we utilize a recent **H3WB dataset** [4], an extended annotation of Human3.6M with 133 keypoints covering the body, hands, and face, significantly increasing complexity. The experimental results are shown in **Table_R 1** in pdf. Di²Pose still achieves comparable results with SOTAs in terms of “All” and “Body”, but slightly underperforms on the \"Face\" and \"Hand\". This may be because the joints within the “face\" and \"Hand\" are densely distributed and highly correlated, unlike the torso. Separate pose quantization for \"Face,\" \"Hand,\" and \"Body\" might be needed. Due to the limited rebuttal time, we did not have the opportunity to try such idea. \n\n**Di²Pose with SMPL, SMPL-X, and STAR** and **add experiments on new mesh-based datasets?** It is important to clarify the differences between skeleton-based (3D coordinates) and mesh-based (e.g., SMPL) 3D HPE tasks. Di²Pose is specifically designed for skeleton-based HPE. SMPL models, on the other hand, require estimating both pose and shape parameters, necessitating a redesign of existing Di²Pose. This level of complexity is beyond the scope of the current paper. However, we are very interested in adapting Di²Pose for mesh-based models. We will try to explore this direction in future work. \n\n## Q2: Better visualizations\n\n**Render the 3D skeleton into the image?** In 3D skeleton-based HPE, projecting the 3D skeleton onto a 2D plane can **lead to overlapping joints and visual confusion**, making it difficult to assess estimation quality. This is different from mesh-based methods, where shape parameters allow for more accurate visual alignment with human silhouettes. Consequently, skeleton-based methods usually visualize results directly in 3D space [1,2].\n\n**Comparison with GT in the same 3D space**. To solve the reviewer's concern about the difficulty in discerning differences in human poses. We visualize the predictions and ground truth in the same 3D space, as shown in **Fig._R 5** in pdf.\n\n**Visualization on multiple views**: We add visualization results on multiple viewpoints, as shown in **Fig._R 3** in pdf.\n\n**We will add these visualizations in the final version.** \n\n## Q3: Clarification for the question about “VQ-VAE+Diffusion”\n\nAlthough Di²Pose follows the “codebook learning + discrete diffusion process” pipeline, it is not as simple as “VQ-VAE + DiffPose”.\n\n- **Codebook learning:** The primary challenge in the first stage is to *learn effective and representative tokens, with each representing a sub-structure*. Simply applying vanilla VQ-VAE does not capture the relationships between adjacent joints within a sub-structure and is prone to causing codebook collapse. These issues would render the subsequent discrete diffusion process ineffective due to invalid tokens, preventing occlusion simulation and recovery. Specifically, we design the Local-MLP and exploit FSQ to ensure successful sub-structure representations.\n- **Discrete Diffusion Process:** While our second stage indeed involves a diffusion process, it differs significantly from *DiffPose, which employs a continuous diffusion model*. Di²Pose operates on a **discrete token sequence**, while DiffPose diffuses **3D poses in continuous space**. Moreover, Our process uses a **transition matrix** for state transitions, while DiffPose employs a **Gaussian Mixture Model** to model the uncertainty distribution.\n\nImportantly, **Di²Pose seamlessly integrates both stages to address occluded 3D HPE**. Quantized tokens from the first stage are processed by the discrete diffusion model to simulate occlusions and recover the full pose. This mechanism effectively incorporates occlusions into the HPE process, offering valuable insights for tackling the occluded 3D HPE task.\n\n**Comparsion with [5]:** Regarding the comment \"In terms of design, it is not even as clever as [1],\" we would like to highlight the following points:\n\n- Thanks for bring this paper to our attention. Firstly, we want to mention that this paper [5] was released to arXiv on May 30, 2024, which is after the NeurIPS submission deadline (May 22, 2024).\n- Both Di²Pose and [5] utilize a pipeline involving codebook learning and a diffusion process. However, **[5] focuses on body-motion generation**, a task involving sequential signals that ***require inter-frame information***. In contrast, Di²Pose addresses occluded 3D HPE, targeting ***single-frame setting***. The differences in the problems being solved naturally lead to different design choices and frameworks, making a direct comparison of the methods' cleverness inappropriate. We will add these discussions and comparisons to our paper.\n\n[1] Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation. In ICCV, 2023\n\n[2] GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular Video. In ICCV, 2023\n\n[3] Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. In TPAMI, 2013\n\n[4] H3WB: Human3.6M 3D WholeBody Dataset and Benchmark. In ICCV. 2023\n\n[5] Stratified Avatar Generation from Sparse Observations. In CVPR. 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0SBDku7ee", "reviewer": "Reviewer_s6UY", "summary": "The work aims to solve the occlusion in 3D human pose estimation, which is an interesting and inherent topic in this field. The authors critiqued that the current continuous diffusion-based pose estimation method requires a large amount of data in the training, while 3D pose datasets are commonly insufficient and thus, hurt the accurate pose generation, especially for occluded cases. To address the problem, the authors first designed a Local-mlp block to quantize poses into tokens, while being aware of the local connection between each keypoints. Then, a diffusion model is exploited to learn the generation of tokens in the condition of a given image. To better address the occluded case, the author designed two addition tokens named \"occ\" to simulate the occlusion. All combined, the method shows on-par results in 3D pose estimations and the state-of-art performance in the occluded dataset.", "questions": "1. For local-map, could there be a more clear explanation of how it exactly captures the interrelation between points?\n\n2. Is it possible to visualize a few quantitative samples in the training process? Especially for poses that get replaced by occ tokens or some tokens get transferred to different tokens. I suspect such a replacement is potentially to be a good method to simulate the person-to-person occlusions.\n\n3. What is the performance of directly using VQ-VAE to encode the 3D poses? Without these results, it is difficult to know the effectiveness of the proposed local-mlp.", "limitations": "No potential negative societal impact of the work is found.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The design of the \"occ\" tokens is interesting. Previous data augmentation is majorly in the image level, i.e. blocking a few key points in the image by adding noise or set to black. However, this paper does it at the token-level, which potentially leads to a more natural simulation of the occlusion.\n\n2. Combining diffusion with 3D poses is an interesting topic. Especially generate the pose at a token-level, which leverages a similar idea to latent diffusion. \n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. In the introduction, the authors criticized the previous method, which requires a large amount of 3D pose data in training, while the proposed solution is to use the quantization of 3D poses. It is unclear why intuitively using a quantized representation of poses could be a solution to data dependency. \n\n2. The target of this paper is also unclear. In my understanding of the author's argument, it is the data dependency of the discrete diffusion model that leads to inaccurate pose estimation in occlusion. However, instead of addressing the data dependency problem, the authors develop occ. token and replacement strategy to address the occlusion problem. This is confusing to me. It is like claiming there is a reason A for problem P, but instead of addressing A, the authors propose another method B completely irrelevant to A to address P from another perspective. \n\n3. Confusing definitions of \"Continuous\" and \"Discrete\". There are two pairs of \"Continuous\" and \"Discrete\" in the paper. The first pair is the continuous and discrete representation of 3D poses. The second pair is a continuous and discrete diffusion model, as current DDPM or DDIM could be considered as a discretization of a continuous SDE or ODE. It is confusing why it is called continuous in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n\n## Q1: Effectiveness of pose quantization for addressing data dependency\n\nThe pose quantization step is designed to convert a 3D pose into multiple quantized tokens, which can be modeled in the latent space by the discrete diffusion model. The keypoint of addressing data dependency is that leveraging discrete diffusion process model discrete tokens rather than exploiting continuous diffusion model diffuses 3D pose in continuous space. For clarification, **we provide the detailed analyse in the global response**. Please refer to “**A general response to common concern”**.\n\n## Q2: Clarification for the target of our paper\n\nFirst, we want to emphasize that **the primary goal** of our Di²Pose framework is to **address the occlusion problem in 3D HPE**. The framework consists of two stages, both designed to tackle occlusions effectively:\n\n- **Pose Quantization Step** transforms a 3D pose into multiple quantized tokens. Our Local-MLP and FSQ mechanisms are introduced to learn a better codebook, *ensuring that each quantized token represents an effective substructure of the pose*. This step is crucial for the subsequent discrete diffusion process, where the Occlude and Replace strategy is applied. By encoding poses into meaningful tokens, we facilitate the handling of occlusions using the discrete diffusion model.\n- During **Discrete Diffusion Process**, we use the Occlude and Replace strategy to address occlusions. Each Occ token represents an occluded sub-structure, while the replacement mechanism mitigates uncertainty. This process relies on the quantized tokens learned in Stage 1, ensuring that the model can simulate occlusions and accurately recover the complete pose.\n\nRegarding the data dependency issue, our discussion aims to highlight **an additional bonus** of using the discrete diffusion model over continuous ones. **This is beneficial but secondary to our main objective of solving the occlusion problem**.\n\n## Q3: Clarification for the definitions of \"Continuous\" and \"Discrete\"\n\n**Discrete Representation of 3D Poses:** In this paper, we focus on the discrete representation of 3D poses. As described in the Introduction, existing 3D HPE methods tend to represent poses using ***coordinate vectors*** or ***heatmap embeddings***. These methods are both considered as discrete representations of 3D poses. Our ***pose quantization step*** converts a 3D pose into multiple discrete tokens, which still belong to discrete representations.\n\n**Continuous and Discrete Diffusion Models:** For the continuous and discrete diffusion models, we distinguish them by their initialization patterns at the beginning of the reverse process:\n\n- **Continuous Diffusion Model:** Continuous diffusion models initialize the 3D pose from random noise, where each joint can be sampled from the continuous 3D space. This aligns with the general understanding of continuous diffusion processes, such as those modeled by continuous SDEs or ODEs.\n- **Discrete Diffusion Model:** The discrete diffusion model, on the other hand, initializes the 3D pose from limited quantized tokens. The range of each token index depends on the size of the codebook, which is a finite number.\n\n## Q4: Clear explanation of the Local-MLP\n\nWe have redrawed Figure 3 from the original paper with more detailed structures, as shown in **Fig._R 1** in pdf.\n\nIn Local-MLP, the key component is the JS-Block, which captures local interactions among X joints. Both Linear Proj. 1 and Linear Proj. 2 are implemented as Conv1d layers with a stride and padding of 1, which allows for information fusion along the joint dimension.\n\n1. **Linear Proj. 1** integrates features within each joint in $\\mathbf{P_{emb}^\\top} \\in \\mathbb{R}^{D \\times J}$.\n2. **Joint Shift Operation** shifts features of adjacent joints into the same channel (Figure b(1) → Figure b(2)). The central part (green box) remains stationary, while the adjacent parts (blue and orange boxes) shift in opposite directions.\n3. We extract the section indicated by the red dashed box (Figure b(2) → Figure b(3)), resulting in each green dashed box **containing features of different adjacent joints**.\n4. **Linear Proj. 2** integrates features within each green dashed box, combining adjacent joint features (Figure b(3) → Figure b(4)).\n5. **Channel MLP** further integrates information across each channel by expanding the dimension D by a factor of 4 and then mapping it back to D dimensions.\n\n## Q5: Visualization of “Replace” mechanism\n\nWe visualize the relevant results in **Fig._R 2** in pdf. First, Di²Pose uses the pose quantization step to represent a 3D pose as multiple discrete tokens, where each number represents an index of the codebook. To demonstrate the effect of the \"Replace\" operation in the discrete diffusion process, we ***replace a token at a specific position with other available tokens***, while ***keeping the tokens at other positions unchanged***. The resulting token sequence is then decoded by the Pose decoder to obtain a 3D pose. It can be seen that **replacing a certain token with other available tokens consistently changes the same sub-structure**, which is circled in red.\n\n## Q6: Ablation studies on the Local-MLP\n\nWe have addressed this issue in the **Ablation Study of our paper**.\n\nIn **Table 3(a)**, we present the ablation study results for different local joint numbers X in the Joint Shift operations. It is noteworthy that the case where ***X=1 corresponds to not using the joint shift operation at all, which is regarded as a vanilla VQ-VAE***. The results indicate that our Local-MLP, which incorporates the joint shift operation, provides better pose representation performance.\n\n## Q7: Potential negative societal impact\n\nIn **Appendix F.2**, we have discussed the potential negative societal impacts of our work, including ***the risk of malicious applications such as illegal surveillance and video synthesis***.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dhxl0qc77u", "reviewer": "Reviewer_ACgE", "summary": "This paper introduces a discrete diffusion method for 3D human pose estimation (HPE). Recent works have successfully applied diffusion models for HPE. However, they need a lot of training data and sometimes output non-anthropomorphic poses. In this work, the authors propose to use discrete diffusions, which leverage a quantized latent space and model the diffusion process as transitions between discrete states. The hope is that the result will be more constrained to the space of plausible poses as it can only take a limited number of values.\n\nThe main contributions of this paper are the following: (1) The authors propose a VQ-VAE for quantizing human pose and modeling the dependencies between the body joints; (2) A discrete diffusion model is designed to do HPE in the quantized latent space; (3) The \"Occlude and replace\" strategy is proposed to model occlusions, further improving the results.", "questions": "- Is $k$ (L185) a sequence?\n- Are there some changes between the proposed diffusion model and the model from [23]?\n- Is the relationship between the $N=100$ tokens modeled?\n- Is the plausibility of the results due to the quantization or the discrete diffusion process? I know this is linked, but for instance, would it be possible to do a latent diffusion process on the continuous latent representation of dimension $N \\times d$ (which is much smaller) and do the quantization just before decoding?\n- Were other backbones tested to quantify the role of this component in the results?\n- What are the running times (training and testing)? Is it comparable to other methods?\n- In Section 4, where are the error bars and the information about the statistical significance? Were the experiments run multiple times?\n- One of the advantages of diffusion models is that they can generate diverse outputs given a single 2D observation, which helps address the ambiguity of predicting 3D from a single 2D POV and occlusions. However, no analysis is performed on that; it seems that a single prediction is made given an image. Is there a reason for that?", "limitations": "The section on the limitations is quite limited. I believe the problem of predictions under heavy occlusion could have been solved by making multiple predictions given an image, which is straightforward with generative models such as diffusions. Limitations could also mention the running time of the diffusion-based approaches. Limitations include the fact that the experiments ran for a single time, which is particularly concerning since diffusion models introduce stochasticity in predictions.\n\nThe societal impact could include environmental considerations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The approach is new: this is the first work using discrete diffusion for human pose estimation. This approach is promising and performs well, especially in the presence of occlusions.\n\n- I found the paper quite well written. The weakness of prior works is clearly pointed out, and we understand how the authors hope to address them. \n\n- There are plenty of details about the models and their mathematical ground. Calculations are well-detailed, which makes them easy to follow.\n\n- The experiments and numerous ablations help capture the strengths of the introduced approach. The proposed approach outperforms SOTA approaches.", "weaknesses": "The method section could be improved to make it more clear:\n- Giving the dimensions of $F$ and $T$ from the beginning (L137) would help understanding the VQ-VAE.\n- (L183) I find it very surprising that the loss for training the VQ-VAE is a cross-entropy. Usually, we use reconstruction loss for training such models (here, it could have been an MSE or some MPJPE). It is unclear why this is not the case for the proposed model.\n- (L185) It is unclear if $k$ is the sequence of $N$ tokens or only one token. Given the Equation 2, I guess it is the sequence?\n- (L193) The notation $k_i$ is already used in Equation 2 (and I think this does not denote the same thing as here it is temporal).\n- (L270) Using the same variable name for 2 different values is not a good practice.\n\nEven if this paper is the first to apply discrete diffusion models to HPE, the methodology section does not bring much novelty. Section 3.2 could be considered preliminary as this directly applies [23]. From my understanding, the only difference is that the \"Mask and replace\" operation of [23] is renamed \"Occlude and replace\" (but it does the same thing). I think it is good to use prior works when designing a new method, but it should be made clear which part is new and which part is an application of prior works.\n\nThe introduction justifies this work by saying continuous diffusion models have high dimensions and that relationships between the joints are ignored. From my understanding, the latent representation of a pose is composed of $N=100$ tokens of dimension $d=5$. Each of the $d_i$ dimensions can take a certain number of values (respectively 7,5,5,5,5). That means that the codebook has 4375 tokens. The transition matrix $M$ between consecutive intermediate predictions is then of dimension $4375 \\times 4375$ if we suppose that the $N$ tokens are independent (which is not supposed to be the case if we want the pose to be globally coherent). In the end, I am not sure that the proposed model's dimension is smaller than continuous diffusion models and that the relationships between the joints are better modeled.\n\nI also have some doubts about the fairness of experiments. The proposed model uses a ViT backbone, while all others (except [20]) have CNN or GCN backbones with much fewer parameters. I believe this is a crucial component of the model and that it is impossible to evaluate the contributions if basic components differ from other approaches.\n\nThe material used for experiments is given, but this is valuable only if we have running time for experiments (which is not the case).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1: Clarifications for Method\n\n1. The dimensions of **F** and **T** are provided from the beginning (L137).\n2. We have re-examined our code and confirm that the **L1 loss function** ($L_{PQ}=||P-{\\hat{P}}||_1$) was indeed used throughout our experiments. While this was a typographical error in the manuscript, it did not affect our results or conclusions.\n3. **Different ‘k’ in the paper**.\n    - **$k_i$** (Eq.2, **italicized, not bold**) is a scalar, specifically the $i$-th entry to the codebook.\n    - **$\\mathbf{k}$** (Eq.2 and L185, **non-italicized, bold**) is a vector, denoting the token indices which are quantized from token features by FSQ.\n    - **$\\mathbf{k}_s$** (L193), where $s\\in\\{0,1,...,S\\}$, is the token indices $\\mathbf{k}$ at different discrete step $s$.\n    \n    While the same letter ‘k’ is used, the distinctions are made clear by **the use of italics, bolding, and subscripts**. Moreover, the consistent use of $\\mathbf{k}$ across two stages ensures uniformity and avoids using unnecessary new symbols.\n    \n4. **Model the relationship between 𝑁=100 tokens?** We only model the relationship between adjacent joints within a sub-structure, as shown in Fig._R 1 in pdf. However, we do not further model the relationship between different sub-structures.\n\n**We will update all these typos and further polish our presentation.**\n\n## Q2: Differences between Di²Pose and [1]\n\nAdmittedly, Di²Pose and [1] follow a mainstream scheme that utilizes a quantization step combined with a diffusion process, explored in various domains [2-4]. However, Di²Pose addresses a different problem with specific motivations and objectives.\n\n- **Distinct Motivation and Problem Domain:** Di²Pose aims to solve **occluded 3D HPE** by leveraging the discrete nature of 3D poses and the strengths of diffusion models in ***handling uncertainty and indeterminacy***. Our goal is to enhance the robustness and accuracy of 3D HPE under occlusion.\n    \n    In contrast, [1] focuses on **text-to-image generation task**, specifically ***addressing the weaknesses of Autoregressive models*** like DALLE.\n    \n- **Distinct Mechanism:** While the \"Mask and replace\" in [1] and our \"Occlude and replace\" share a similar transition matrix implementation, **their purposes are fundamentally different**. The strategy in [1] is designed to ***mitigate unidirectional bias and accumulated prediction errors***. Our pose quantization step produces a token sequence representing pose substructures, enabling the \"Occlude and Replace\" mechanism. **\"Occlude\" simulates the occlusion of a substructure**, and **\"Replace\" solves the uncertainty under occlusions**, where a single occluded region may correspond to multiple potential 3D poses. This effectively simulates the transition from occluded to recovered, integrating occlusion impacts into the estimation process.\n- **Contribution to the Field:** Di²Pose provides ***a new paradigm for tackling occluded 3D HPE***. By designing a specific pose quantization step and leveraging discrete diffusion tailored for this task, we contribute a novel approach for addressing occlusions effectively.\n\nWe acknowledge the importance of distinguishing our contributions from prior work and ***will clarify and discuss these differences in the Method section***.\n\n## Q3: Dimensions of search space\n\nIn the introduction, we mention that continuous diffusion models need a larger search space to achieve optimal generative outcomes, whereas discrete diffusion models do not. It is noteworthy that our emphasis is on the size of the **search space** rather than the size of the **model's dimension**. For clarification, please **refer to the detailed analysis in “A general response to common concern” in the *global response***.\n\n## Q4: Extra experiments\n\n- **Replace backbone**: We add experiments replacing the ViT backbone with a CNN-based backbone [6] and evaluate on the 3DPW. As shown in **Table_R 2** in pdf, it shows that Di²Pose maintains its superiority even with a CNN backbone.\n- **Repeated experiments**: We repeat experiments three times and evaluate on the 3DPW. As shown in **Table_R 3** in pdf, we report the “Mean $\\pm$ Std” of MPJPE and PA-MPJPE across multiple runs. It shows robustness of Di²Pose, and the mean results still achieves SOTA.\n- **Multiple inference results**: In our paper, we focused on single inference outputs for practical 3D HPE applications. However, we acknowledge the diversity of outputs possible with diffusion models due to different initializations. Thus, we add experiments with multiple inferences from different initializations. The “Mean $\\pm$ Std” is reported in **Table_R 3** in pdf, which shows that Di²Pose produces relatively stable results.\n    \n    Additionally, we **visualize** the diverse outputs of Di²Pose from a single input image in **Fig._R 4** in pdf, demonstrating the model's ability to generate varied predictions under occlusion.\n    \n- **Running times**: The training time and inference speed are shown in **Table_R 4** in pdf.\n\n## Q5: Continuous latent representation + Quantization before decoding?\n\nThe idea is indeed very interesting. However, this specific approach is beyond the scope of our current paper. We appreciate your comments and will consider exploring this idea in the future. \n\n## Q6: Limitation\n\n**Environmental considerations**: We add potential negative environmental impacts as follow. The diffusion-based model has a longer runtime compared to other CNN or GCN-based methods, causing more computational resources and energy consumption.\n\n[1] Vector Quantized Diffusion Model for Text-to-Image Synthesis. CVPR 2022\n\n[2] Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation. ICCV 2022\n\n[3] Priority-centric human motion generation in discrete latent space. ICCV 2023\n\n[4] Layoutdm: Discrete diffusion model for controllable layout generation. CVPR 2023\n\n[5] Deep High-Resolution Representation Learning for Human Pose Estimation. CVPR 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for recognizing our paper well-written (Reviewers tz5G, s6UY, ACgE), easy to follow (Reviewers s6UY, ACgE),  and with novel ideas/methods (all Reviewers).

We appreciate their careful reviews and constructive comments. We have revised our paper according to all comments. The major changes are summarized as follows.

- **According to Reviewer Z93t’s comments**:
    - Detailed explanations and insights about our method. We detailedly illustrate the designs and insights of two main parts: pose quantization and discrete diffusion.
    - Comparison with related works. We compare our Di²Pose with two related works mentioned by the reviewer.  We will cite these works in the Related Work section and further enhance the completeness of our study.
- **According to Reviewer tz5G’s comments**:
    - Extended experiments. We clarify the differences between skeleton-based 3D HPE (our purpose) and mesh-based 3D HPE. Moreover, we add experiments on a new dataset (H3WB) with more keypoints to increase complexity (cf **Table_R 1** in **pdf**).
    - Better visualizations. We visualize the predictions and ground truth in the same 3D space (cf **Fig._R 5** in **pdf**) and we also add visualization results from multiple viewpoints (cf **Fig._R 3** in **pdf**).
    - Clear clarification. We illustrate the specific design of our two-stage model for occluded 3D HPE task and compare our Di²Pose to the simple “VQ-VAE + DiffPose”. In addition, we make acomparisons with the mentioned paper for clear distinction.
- **According to Reviewer s6UY’s comments**:
    - Clear clarifications. We make clearer clarifications to address the following concerns.
        - Effectiveness of pose quantization for addressing data dependency. We illustrate this by comparing the search space between continuous and discrete diffusion models.
        - Target of our paper. We emphasize that the primary goal of our Di²Pose framework is to address the occlusion problem in 3D HPE. An additional bonus of using the discrete diffusion model is its ability to alleviate data dependency issues.
        - Clarification for the definitions of "Continuous" and "Discrete". We provide detailed explanations of both definitions mentioned in our paper for better understanding.
    - Further explanation of the Local-MLP. We have redrawed Figure 3 from the original paper with more detailed structures of Local-MLP (cf **Fig._R 1** in **pdf**).
    - Visualizations. We add new visualizations about the “Replace” mechanism in the discrete diffusion process (cf **Fig._R 2** in **pdf**).
- **According to Reviewer ACgE’s comments**:
    - Clear clarifications. We provide clearer clarifications to address the following concerns.
        - Method section. We correct relevant typos and further polish our presentation.
        - Differences between Di²Pose and the mentioned method. We distinguish both methods from the perspectives of motivations, problem domains, and mechanisms. We also highlight the contribution of Di²Pose to the occluded 3D HPE field.
        - Search space comparisons. We emphasize that the strength of Di²Pose lies in the size of the search space rather than the model's dimensionality. We further compare the search space between continuous and discrete diffusion models.
    - Experiments. We add various experiments to enhance the completeness of our study.
        - Replace backbone: We add experiments replacing the ViT backbone with a CNN-based backbone (cf **Table_R 2** in **pdf**).
        - Repeated experiments: We repeat experiments for the entire training and inference process (cf **Table_R 3** in **pdf**).
        - Multiple inference results: We add experiments with multiple inferences from different initializations (cf **Table_R 3** in **pdf**). In addition, we visualize the diverse outputs of Di²Pose from a single input image (cf **Fig._R 4** in **pdf**).
        - Running times: We provid the training time and inference speed of our model (cf **Table_R 4** in **pdf**).
    - Limitation. We add the potential negative environmental impacts in the Limitation section.

## A general response to common concerns raised by Reviewer s6UY and Reviewer ACgE

We appreciate the detailed feedback from both reviewers. In the Introduction section, we illustrated that continuous diffusion models need a large search space to achieve optimal generative outcomes. This statement raised concerns from both reviewers regarding **whether the pose quantization of Di²Pose effectively addresses data dependency** and **whether Di²Pose has a smaller search space**. To address both questions, we would like to compare the search spaces of continuous diffusion models and the discrete diffusion model as follows:

**Search Space Comparison:** To understand this, let's consider the reverse process of the diffusion model. For continuous diffusion models, the initialization of the 3D pose is sampled randomly from the continuous 3D space, which means the theoretical search space is **{continuous 3D space}^{joint number}**. Since ***the continuous 3D space has an infinite number of points***, training such a continuous diffusion model requires a large amount of 3D pose data to achieve optimal outcomes.

In contrast, for the discrete diffusion model, we initialize a limited number of quantized tokens. For each token, the number of initialization choices is {codebook size+1}, where 1 represents the Occ token. Thus, the theoretical search space for the discrete diffusion model is **{codebook size+1}^{quantized token numbers}**. This finite search space significantly reduces the amount of 3D pose data required for training. 

We hope the above analysis will address the reviewers' concerns.

Meta-review: VQ-VAE followed by Diffusion has been applied to various tasks, but its use in 3D Human Pose Estimation is novel. As noted by Reviewers tz5G and z93t, this application introduces some ideas around using the Reverse discrete diffusion process to enhance robustness against occlusion and to ensure contextual consistency.

Reviewer tz5G raised a concern that the H3.6M dataset is not sufficiently challenging for demonstrating improved results on complex poses. Although the authors provided results on an extended annotated version of H3.6M in their rebuttal, this does not fully address the underlying simplicity of the data. Additionally, Reviewer z93t rightly points out the lack of mechanistic insights in the pose estimation process.

Nonetheless, the authors have given detailed responses in their rebuttals, leading some reviewers, such as ACgE, to upgrade their scores after considering the authors' clarifications, due to which I recommend an accept.
Decision: Accept (poster)