Title: $\text{Di}^2\text{Pose}$: Discrete Diffusion Model for Occluded 3D Human Pose Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Occluded 3D human pose estimation; Discrete diffusion model
Abstract: Diffusion models have demonstrated their effectiveness in addressing the inherent uncertainty and indeterminacy in monocular 3D human pose estimation (HPE). 
Despite their strengths, the need for large search spaces and the corresponding demand for substantial training data make these models prone to generating biomechanically unrealistic poses. 
This challenge is particularly noticeable in occlusion scenarios, where the complexity of inferring 3D structures from 2D images intensifies. 
In response to these limitations, we introduce the **Di**screte **Di**ffusion **Pose** (**$\text{Di}^2\text{Pose}$**), a novel framework designed for occluded 3D HPE that capitalizes on the benefits of a discrete diffusion model. 
Specifically, **$\text{Di}^2\text{Pose}$** employs a two-stage process: it first converts 3D poses into a discrete representation through a pose quantization step, which is subsequently modeled in latent space through a discrete diffusion process. 
This methodological innovation restrictively confines the search space towards physically viable configurations and enhances the model’s capability to comprehend how occlusions affect human pose within the latent space. 
Extensive evaluations conducted on various benchmarks (e.g., Human3.6M, 3DPW, and 3DPW-Occ) have demonstrated its effectiveness.

Di 2 Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimation

Abstract

Diffusion models have demonstrated their effectiveness in addressing the inherent uncertainty and indeterminacy in monocular 3D human pose estimation (HPE). Despite their strengths, the need for large search spaces and the corresponding demand for substantial training data make these models prone to generating biomechanically unrealistic poses. This challenge is particularly noticeable in occlusion scenarios, where the complexity of inferring 3D structures from 2D images intensifies. In response to these limitations, we introduce the Di screte Di ffusion Pose ( Di 2 Pose ), a novel framework designed for occluded 3D HPE that capitalizes on the benefits of a discrete diffusion model. Specifically, Di 2 Pose employs a two-stage process: it first converts 3D poses into a discrete representation through a pose quantization step , which is subsequently modeled in latent space through a discrete diffusion process . This methodological innovation restrictively confines the search space towards physically viable configurations and enhances the model's capability to comprehend how occlusions affect human pose within the latent space. Extensive evaluations conducted on various benchmarks ( e.g. , Human3.6M, 3DPW, and 3DPW-Occ) have demonstrated its effectiveness.

1 Introduction

3D Human Pose Estimation (HPE) from monocular images remains a challenging yet pivotal research in the realm of computer vision, boasting a wide range of applications including human-machine interaction, autonomous driving, and animations [57, 81, 5, 70]. Generally, the mainstream approaches, including Direct Estimation [68, 43, 47] and 2D-to-3D Lifting [87, 51, 86], aim to perform 3D HPE by either directly predicting 3D poses from 2D images or lifting detected 2D poses into 3D space. These approaches aim to address the inherent 2D-3D ambiguity in 3D HPE tasks by learning mapping from training data. Despite significant advancements, accurately estimating 3D poses from monocular images remains a formidable challenge, particularly when humans are partially occluded [39]. Such occlusions introduce considerable uncertainty and indeterminacy into the estimation process.

Existing 3D HPE methods try to handle the occlusion challenges with pose priors/constraints [58, 62] or data augmentation strategies ( e.g. , annotations augmentation [61], pose transformation [35], and differentiable operations [82]). However, due to the inherent discreteness of 3D poses (primarily defined by discrete anatomical landmarks), these methods tend to represent poses using coordinate vectors or heatmap embeddings, treating joints as independent units and overlooking the interdependencies among body joints. Recent research [21] has introduced a compositional pose representation that captures the dependencies among joints by converting a pose into multiple tokens, enabling the

GT

Image

DiffPose

Di

2

Pose (ours)

(b)

(a)

110

DiffPose

105

Di²Pose(ours)

100

95

90

85

80

75

20

30

40

50

60

70

80

06

100

Proportion of Training Samples(%)

use of mutual context between joints. This approach, which learns from real pose datasets, results in each learned token corresponding to a physically realistic prototype. Nevertheless, Geng et al. [21] casts HPE to a classification task, where the system simply classifies tokens based on prototype poses. Unfortunately, such scheme does not account for the effects of occlusions in the estimation process, potentially leading to inaccuracies due to unresolved uncertainty and indeterminacy.

Recent studies have shown marked progress in 3D HPE via generative models [2, 84, 19, 27]. Notably, diffusion models [25] have demonstrated effectiveness in handling complex and uncertain data distributions, making them suitable for handling uncertainty and indeterminacy in 3D HPE tasks [19, 91, 66, 27, 37]. They excel at generating samples that conform to a target data distribution by iteratively removing noise through a series of diffusion steps, ultimately predicting more accurate 3D poses. However, these diffusion-based 3D HPE methods initialize the 3D pose from random noise at the begining of the diffusion process, where each joint can be sampled from the continuous 3D space. Since the continuous 3D space has an infinite number of points, training such diffusion-based models requires a large amount of 3D pose data to achieve optimal outcomes [23, 75, 3]. This demand implies a substantial need for training data, presenting a stark contradiction to the limited availability of 3D human pose datasets. As illustrated in Figure 1(a), the predictive performance of DiffPose [22] declines more rapidly as the proportion of training data decreases. Given the scarcity of 3D pose training data, previous diffusion models may generate physically implausible configurations that do not adhere to human biomechanics, leading to inaccurate human pose estimations, particularly in occluded scenes ( c.f. , Figure 1(b) with DiffPose).

In this paper, we propose a novel framework for 3D HPE with occlusions: Di screte Di ffusion Pose ( Di 2 Pose ), drawing on compositional pose representation and diffusion model to achieve the best of two worlds. Specifically, Di 2 Pose employs a two-stage approach: a pose quantization step followed by a discrete diffusion process . The pose quantization step leverages the discrete nature of 3D poses and represents them as quantized tokens by capturing the local interactions between joints. This step effectively confines the search space to physically plausible configurations by learning from real 3D human poses. Subsequently, the discrete diffusion process models the quantized pose tokens in the latent space through a conditional diffusion model. By integrating the forward and reverse processes, our framework adeptly simulates the transition of a 3D pose from occluded to recovered. By modeling occlusion implicitly within the latent space, Di 2 Pose enhances its understanding of how occlusions affect human poses, providing valuable insights during the training phase.

For the pose quantization step , we devise a pose quantization step inspired by VQ-VAE [71], consisting of a pose encoder, a quantization process, and a pose decoder. To effectively capture local interactions between 3D joints, we introduce the Local-MLP block for both pose encoder and decoder. Within each Local-MLP block, a simple Joint Shift operation integrates information from different joints. The pose encoder utilizes several Local-MLP blocks to convert a 3D pose into multiple rich token features, each representing a sub-structure of the overall pose. These tokens are quantized using a shared codebook, yielding corresponding discrete indices. Additionally, we implement the finite scalar quantization (FSQ) [49] to address the codebook collapse issue observed in traditional VQ-VAE methods [59, 89, 56, 42]. This strategy ensures that the generated codewords are meaningful, a crucial aspect for the subsequent success of the discrete diffusion process.

For the discrete diffusion process , during the training phase, we introduce occlude and replace strategies to model the quantized pose tokens, enabling the discrete diffusion model to predict occluded tokens and update potential tokens. The occluded token represents the occlusion of the corresponding sub-structure of the 3D pose. The token replacement mechanism is designed to enhance the diversity of potential sub-structures, reflecting the indeterminacy in occluded parts. During the inference phase, pose tokens are either occluded or initialized randomly. The denoising diffusion process estimates the probability density of pose tokens step-by-step based on the input 2D image until the tokens are completely reconstructed. Each step leverages contextual information from all tokens of the entire pose as predicted in the previous step, facilitating the estimation of a new probability density distribution and the prediction of the current step's tokens. This sequential approach ensures a detailed and accurate reconstruction of 3D poses from occluded scenes.

We extensively evaluate our approach in 3D HPE on three challenging benchmarks (Human3.6M [34], 3DPW [72] and 3DPW-Occ). Di 2 Pose consistently yields lower errors compared to state-of-the-art methods. In particular, it achieves significantly better results when evaluated on occluded scenarios, verifying its advantages of occlusion-handling capability. Our contributions are threefold:

We propose the Di 2 Pose framework, which integrates the inherent discreteness of 3D pose data into the diffusion model, offering a new paradigm for addressing 3D HPE under occlusions.

The designed pose quantization step represents 3D poses in a compositional manner, effectively capturing local correlations between joints and confining search space to reasonable configurations.

The constructed discrete diffusion process simulates the complete process of a 3D pose transitioning from occluded to recovered, which introduces the impact of occlusions into pose estimation process.

2 Related Work

Monocular 3D HPE. Existing approaches can generally be classified into frame-based and videobased methodologies. Frame-based methods predict the 3D pose from a single RGB image, employing different networks in various studies [18, 20, 52, 54, 55] to directly output the human pose from the 2D image. Alternatively, a significant number of studies [48, 77, 85, 88] initially determine the 2D pose, which subsequently forms the foundation for inferring the 3D pose. In contrast, video-based methods leverage temporal relationships across video frames. Such methods predominantly [9, 11, 15, 31, 65, 73, 76] commence with the extraction of 2D pose sequences using a 2D pose detector from the video clips, aiming to harness the essential spatio-temporal data for 3D pose estimation. To validate the efficacy of our approach, we evaluate our Di 2 Pose on the more challenging frame-based setting, wherein the 3D human pose is directly inferred from a 2D image.

Occluded 3D HPE. Occlusions significantly challenge 3D HPE. As evidenced by research [76, 58, 62], pose priors and constraints have been proven crucial for mitigating such issue. Approaches typically involve statistical models to deduce occluded parts from visible cues [76, 41, 44] or pre-defined rules to constraint poses [61, 1]. Moreover, due to the scarcity of 3D pose data, data augmentation, including synthetic occlusions [7, 38, 60, 64, 13] and pose transformations [35, 82], remains vital for enhancing model robustness. Diverging from these aforementioned methods, our method innovatively introduces occlusion in the latent space without extra priors or explicit augmentations, providing a deeper feature-based understanding of occlusion's effects on pose estimation.

Diffusion Models for 3D HPE. Recent advancements have shown that diffusion models are capable of managing complex and uncertain data distributions [25, 26, 17, 4, 32, 8, 10, 80, 40, 36, 74, 78], which is particularly beneficial for 3D HPE. Typically, these models predict 3D poses by progressively refining the pose distribution from high to low uncertainty [22, 14, 19, 91, 63]. Other approaches use diffusion models to generate multiple pose hypotheses from a single 2D observation [66, 27]. These 3D pose estimators effectively reduce uncertainty and indeterminacy throughout the estimation process. Moreover, discrete diffusion models have also gained attention in various domains [30, 40, 24, 33]. Inspired by these advancements, our work introduces a discrete diffusion model for occluded 3D HPE, which aligns more closely with the inherent discreteness of 3D pose data and effectively incorporates occlusion into the estimation process, providing a novel perspective in the field.

…

32

7

145

61

…

…

32

61

…

32

51

Occ

61

…

32

7

145

61

…

…

Forward process

Reverse process

𝐤𝑠

𝐤𝑠-1

𝐤𝑆

𝐤0

Image

Encoder

Pose

Decoder

Condition

q𝑖

ෝ

q𝑖

FSQ

Step

×M

AdaLNorm

AdaLNorm

Multi-head

Attention

Multi-head

Attention

LN

&Linear

Stage 1: Pose Quantization Step

Stage 2: Discrete Diffusion Process

𝐏

෡

𝐏

Local-MLP

Local-MLP

Local-MLP

Local-MLP

𝐅

𝐓

𝐤

Occ

Occ

Occ

Occ

Occ

Occ

𝑠

Feature

Sequence

𝐶

𝐼

Local-MLP

y

x

V

Z

X

y

Z

X

3 Di 2 Pose

Given an 2D image I ∈ R H × W × 3 , the goal of 3D HPE is to predict ˆ P ∈ R J × 3 , which represents the 3D coordinates of all the J joints of the human body. In this paper, we construct occluded 3D HPE task as a two-stage framework including the pose quantization step and the discrete diffusion process.

As shown in Figure 2, in the training phase , Stage 1 learns a pose quantization step by a VQ-VAE like structure (Sec. 3.1), which is able to encode a 3D pose into multiple quantized tokens. Stage 2 models quantized pose tokens in the latent space by the forward and reverse process of a conditional diffusion model (Sec. 3.2). In the inference phase , we only use the reverse process of Stage 2 and the pre-trained pose decoder of Stage 1 to recover 3D pose from the 2D image. Notably, pose tokens are either occluded or initialized randomly at the beginning of the inference phase. The model reconstructs all the tokens based on the condition 2D image step-by-step. These reconstructed tokens are finally decoded by the pre-trained pose decoder, resulting in the recovered 3D pose.

3.1 Pose Quantization Step

As depicted in Figure 2, a pose quantization step comprises a pose encoder, the quantization process, and a pose decoder. Initially, for a real 3D pose P ∈ R J × 3 , the pose encoder f PE ( · ) converts P to token features F . During the quantization process, we utilize FSQ to quantize F = ( f 1 , f 2 , · · · , f N ) ( f i ∈ R D ) into tokens T = ( t 1 , t 2 , · · · , t N ) ( t i ∈ R D ) . Finally, the quantized tokens T are decoded by the pose decoder f PD ( · ) to reconstruct 3D pose ˆ P .

Pose Encoder. Considering the interdependencies among human body joints, our goal is to represent 3D poses in a compositional manner, moving away from reliance on coordinates vectors or heatmap

embeddings. The VQ-VAE architecture, incorporating MLP-Mixer blocks [69] within its encoder and decoder, has been proven effective in decomposing a pose into multiple token features, each corresponding to a sub-structure of the pose [21]. However, the MLP-Mixer block is designed to extract global information across all joints, which can not adequately capture the local relationships between joints within individual sub-structure.

In response to aforementioned limitation, we design Local-MLP block to capture the local interactions between 3D joints. The pose encoder f PE ( · ) , comprising several Local-MLP blocks, converts P to N token features:

where f emb ( · ) embeds P to P emb ∈ R J × D by a linear layer.

As shown in Figure 3(a), a Local-MLP block is composed of a Layer Normalization layer, a Joint Shift block (JS-Block), a Channel MLP, and a residual connection. The JS-Block is specifically designed to capture local interactions among X joints. It extracts features by linear projection, and the Joint Shift operation enables feature translation along joint connection directions. As shown in Figure 3(b), with the input P ⊤ emb ∈ R D × J , the feature is evenly divided into X segments ( X = 3 in the example), each segment being shifted incrementally by units from -⌊ X/ 2 ⌋ to ⌊ X/ 2 ⌋ . The central segment remains stationary, while the segments to the left and right are symmetrically shifted away from the center by up to ±⌊ X/ 2 ⌋ units. Zero padding is used to maintain dimensionality. Features highlighted within the dashed box are selected for further linear projection. Finally, the Channel MLP processes these features channelwise to facilitate information integration.

Quantization Process.

During this process, we exploit FSQ [49] to enhance the utilization of codewords. FSQ quantizes token features F as corresponding token indices:

where f proj ( · ) projects each f i ∈ R D of F to q i ∈ R d , and each k i of k denotes the entries of implied codebook C . For each q i , FSQ employs a bounding function f bnd : q i ↦→⌊ L/ 2 ⌋ · tanh ( q i ) to constrain each channel of d . As a result, each channel in ˆ q i = round( f bnd ( q i )) takes one of L unique values. This procedure yields ˆ q i ∈ C , where the total number of unique codebook entries is |C| = ∏ d i =1 L i (mapping the i -th channel to L i values). The vectors in C can be enumerated, establishing a bijective mapping from any ˆ q i to an integer within { 1 , . . . , |C| }. In addition, the corresponding codeword of k i , which is denoted as t i ∈ R D , represents the quantized result of f i . Thereby, using FSQ, the token features F are quantized as T = ( t 1 , t 2 , · · · , t N ) .

Pose Decoder. The pose decoder f PD ( · ) is designed to recover 3D pose ˆ P from T . f PD ( · ) adopts a structure similar to the pose encoder but in reverse, utilizing a reduced number of Local-MLP blocks.

Loss. The pose quantization step, including the pose encoder, quantization process, and pose decoder, is jointly optimized by minimizing L1 loss L PQ = || P -ˆ P || 1 across the training dataset.

3.2 Discrete Diffusion Process

After training the pose quantization step, we can acquire N quantized tokens k from the original 3D pose P . The next step in Di 2 Pose pipeline is to model k in the latent space by the discrete diffusion process. In the following, we first briefly introduce the diffusion models and clarify the basic principles of the discrete diffusion model. Then we explain the details of discrete diffusion

D

J

𝐽 1

𝐽 1

𝐽 1

𝐽 2

𝐽 2

𝐽 2

𝐽 3

𝐽 3

𝐽 3

𝐽 4

𝐽 4

𝐽 4

𝐽 2

𝐽 1

𝐽 3

𝐽 2

𝐽 1

𝐽 4

𝐽 3

𝐽 2

𝐽 4

𝐽 3

𝐽 1

𝐽 4

𝐽 2

𝐽 1

𝐽 3

𝐽 2

𝐽 1

𝐽 4

𝐽 3

𝐽 2

𝐽 4

𝐽 3

(a)

(b)

Linear Proj.

Joint Shift

Linear Proj.

Channel MLP

LayerNorm

JS-Block

LayerNorm

process, including the designed transition matrix and loss function. Eventually, we illustrate the architecture and training and inference process.

Discrete Diffusion Model. Our discrete diffusion model is characterized by two distinct processes: 1) Forward process : It progresses through discrete steps s ∈ { 0 , 1 , 2 , ..., S } , gradually transforming the initial tokens k 0 (the quantized token k ) into a noise-infused latent representation k S . 2) Reverse process : It is tasked with reconstructing the original data k 0 from the latent k S , following a reverse temporal sequence s ∈ { S, S -1 , ..., 1 , 0 } .

Followed previous studies [67, 3, 28], we use a transition probability matrix [ M s ] ij = q ( k s = i | k s -1 = j ) ∈ R |C|×|C| elucidate the likelihood of transitioning from k s -1 to k s . Then the forward process for the entire sequence of tokens is expressed as:

where c ( · ) symbolizes a function capable of converting a scalar into a one-hot column vector. The distribution of k s follows a categorical distribution, determined by the vector M s c ( k s -1 ) . Leveraging the Markov chain property, it is feasible to bypass intermediate stages, directly computing the probability of k s from k 0 for any given step as:

Moreover, the posterior of the reverse process, q ( k s -1 | k s , k 0 ) , can be ascertained as:

Occlude and Replace Transition Matrix. Notably, a suitable design for transition probability matrix M s is significant to train the discrete diffusion process. As illustrated in Section 3.1, through pretrained pose encoder and FSQ, P can be converted to k = ( k 1 , k 2 , · · · , k N ) , each k i corresponding to a sub-structure of the overall pose. With this foundation, we specifically devise the occlude and replace scheme, which is inspired by [24], for tackling the challenges of occluded 3D HPE. In occlusion scenes, the human body is always occluded in various situations (self-occlusions, object or people-to-person occlusions), and the typical manifestation is that some sub-structures of the pose are invisible. Consequently, we design the occlude scheme simulating the occlusion of corresponding joints, which introduces occlusion impact in the training process. Additionally, recognizing the inherent uncertainty in occlusion scenarios where a single occluded region may correspond to multiple potential 3D human poses, we develop the replace strategy to update certain token with another available token.

In practice, each quantized token k i has a probability of γ s to transition to the Occ token. Moreover, k i is also subject to a probability of |C| β s to be uniformly resampled across all |C| categories. Furthermore, k i retains a probability of α s = 1 - |C| β s -γ s to remain unchanged. Then, the transition matrix M s ∈ R ( |C| +1) × ( |C| +1) is defined as:

where α s , β s ∈ [0 , 1] . The prior distribution of step S can be derived as: p ( k S ) = [ β S , β S , · · · , γ S ] , where α S = ∏ S i =1 α i , γ S = 1 -∏ S i =1 (1 -γ i ) and β S = (1 -α S -γ S ) / |C| . In this study, we adapt the linear schedule [25] as noise schedule to pre-define the value of transition matrices ( α S , β S , and γ S ). Subsequently, we can calculate q ( k s | k 0 ) according to Eq. (4). However, when the number of categories |C| and time step S is too large, it can quickly become impractical to store all of the transition matrices M s in memory, as the memory usage grows like O ( |C| 2 S ) . Actually, it is unnecessary to store all of the transition matrices. Instead we only store all of α s and β s in advance, since we can calculate q ( k s | k 0 ) according to following formula (refer to Appendix for proofs):

Training Objectives. We train a network f θ ( k s -1 | k s , y ) to estimate q ( k s -1 | k s , k 0 ) in the reverse process. The network is trained to minimize the variational lower bound (VLB):

In addition, we follow [50, 24] to utilize the reparameterization trick, which lets Di 2 Pose predict the noiseless token distribution f θ ( ˆ k 0 | k s , y ) at each reverse step, and then compute f θ ( k s -1 | k s , y ) as:

Based on the Eq. (9), an auxiliary denoising objective loss is introduced, which encourages the network to predict f θ ( ˆ k 0 | k s , y ) :

where λ is a hyper-parameter to control the weight of the auxiliary loss L k 0 .

Diffusion Architecture. As depicted in Figure 2, our discrete diffusion model consists of three main components: an image encoder, a pose denoiser, and a pose decoder. The pre-trained image encoder processes the 2D image to produce a conditional feature sequence. The pose denoiser, receiving the quantized pose tokens k s and and step S , predicts the distribution of noiseless tokens f θ ( ˆ k 0 | k s , y ) . This component is equipped with several transformer blocks, each featuring an AdaLNorm operator [6], multi-head attention blocks that combine the image feature information with k s , and layer normalization and linear layers. At the end of the reverse process, all recovered tokens are obtained, and the final prediction of 3D pose is decoded by the well-trained pose decoder.

Training and Inference Process. In the training process , as for step s , we sample k s from q ( k s | k 0 ) based on Eq. (7) in the forward process. We then estimate f θ ( k s -1 | k s , y ) in the reverse process. The final loss will be calculated according to Eq. (11). In the inference process , all pose tokens are either masked or initialized randomly. Subsequently, we predict f θ ( k s -1 | k s , y ) step by step until the tokens are completely recovered. Finally, reconstructed tokens are decoded by the pose decoder, resulting in the recovered 3D pose. The complete algorithms are summarized in Appendix.

4 Experiments

4.1 Datasets and Evaluation Metrics

Datasets. Human3.6M [34] is the most extensive benchmark for 3D HPE, consisting of 3.6 million images. We follow [22] with same protocol, which involves training on subjects S1, S5, S6, S7, and S8, and testing on subjects S9 and S11. 3DPW [72] is the first dataset in the wild that includes video footage taken from a moving phone camera. We also evaluate our method on this dataset to measure the robustness and generalization. Additionally, to further verify the occlusion-robustness, we evaluate Di 2 Pose on the 3DPW-Occ [83], which is a subset of the 3DPW.

Evaluation Metrics. For Human3.6M and 3DPW, we follow the standard protocols. Mean per joint position error ( MPJPE ) calculates the mean Euclidean distance between the root-aligned reconstructed poses and ground truth joint coordinates. PA-MPJPE employs a Procrustes alignment between the poses before calculating the MPJPE. In addition, to further evaluate the effectiveness of our method under occlusion scenes, we devise an adversarial protocol, termed 3DPW-AdvOcc , following the previous research [84]. We apply occlusion patches to the input image to identify the most challenging predictions. This process involves assessing the relative performance degradation on the visible joints. Similar to [84], we utilize textured patches generated by randomly cropping texture maps from the DTD [16]. We employ two square patch sizes: 40 and 80 relative to a 256 × 192 image, denoted as Occ@40 and Occ@80 respectively, with a stride of 10.

4.2 Implementation Details

Pose Quantization Step. The pose encoder is constructed with four Local-MLP blocks, while the pose decoder incorporates a single block. Within these Local-MLP blocks, the embedding dimensions D for the pose encoder and decoder are configured to 2048 and 512, respectively. For the quantization process, the projected vector q i features the channel d = 5 . The levels per channel, denoted as [ L 1 , · · · , L d ] , are specified as [7 , 5 , 5 , 5 , 5] . The number of quantized tokens N is set to 100.

Our final loss function is defined as:

Discrete Diffusion Process. For the occlude and replace transition matrix, we linearly increase β s and γ s from 0 to 0.1 and 0.9, respectively, and decrease α s from 1 to 0. For the discrete diffusion model, we use off-the-shelf image encoder [79] to extract feature sequence of conditional 2D image. As for the pose denoiser, we build a 21-layer 16-head transformer with the dimension of 1024. We set steps S as 100 and loss weight λ is set to 5e-4. Please refer to Appendix for more details.

4.3 Comparsion with State-of-the-Arts

Human3.6M. To explore the effectiveness of Di 2 Pose, we evaluate its performance in the challenging context of frame-based 3D pose estimation. Specifically, within the discrete diffusion process, context information is extracted from a single input frame using the image encoder. As shown in Table 1, we benchmark Di 2 Pose against SOTA 3D HPE methods on the Human3.6M. Our Di 2 Pose achieves 49.2mm in average MPJPE, surpassing the performance of the SOTA diffusion model [22] by 0.5mm, which indicates that Di 2 Pose is able to enhance monocular 3D HPE in indoor scenes.

3DPW. Beyond indoor settings, we evaluate the performance of Di 2 Pose on the in-the-wild 3DPW dataset. As Table 2 shows, Di 2 Pose achieves the SOTA performance, and outperforms the SOTA method [22] by 3.4mm in MPJPE and 3.7mm in PA-MPJPE. On the occlusion-centric 3DPW-Occ , Di 2 Pose maintains its superiority. When assessed under the 3DPW-AdvOcc protocol, all methods exhibit performance drops-MPJPE surges by up to 129% and PA-MPJPE by up to 72%. Despite this, Di 2 Pose remains markedly robust, leading the SOTA by significant margins in both MPJPE and PA-MPJPE, underscoring its effectiveness in handling occlusions.

Qualitative Results. Figure 4 presents the qualitative results of DiffPose [22] in comparison with our Di 2 Pose across two datasets. It can be observed that our method yields more accurate predictions than compared diffusion model (DiffPose), especially under various occlusion scenarios (self-occlusion and object occlusion). This demonstrates the superior occlusion-robustness of our Di 2 Pose.

(b) 3DPW

(a) Human36M

Image

DiffPose

Di 2

Pose

Image

DiffPose

Di 2

Pose

(a) Different local joint number X of Joint Shift operations in JS-Block. (b) Different levels per channel [ L 1 , · · · , L d ] of quantization process FSQ. (c) Different final occlude rate γ S for the occlude and replace transition matrix. (d) Different number of training and inference steps S . P-1 are reported.

4.4 Ablation Study

Effectiveness of Pose Quantization Step. Our pose quantization step, which consists of Local-MLP blocks, is designed for representing 3D human pose by capturing the local interactions between 3D joints. Table 4 displays the MPJPE metrics comparing the original 3D poses with those reconstructed via various methods. The results show that our pose quantization step reconstructs 3D poses with lower errors compared to previous method [21], which uses an MLP-Mixer for global joint information extraction. It indicates

that our model learns a more accurate representation of 3D poses. In addition, we conducted other ablation studies to investigate different local joint numbers X and levels per channel [ L 1 , · · · , L d ] within pose quantization step, as shown in Table 3a and Table 3b. As to different X , note that when X = 1 , we only extract feature of individual joint, and when X > 1 , JS-Block is able to capture local interactions of different joints. Experimental results indicate that X = 3 reaches lowest reconstruct error. As for [ L 1 , · · · , L d ] , the best level of FSQ for pose quantization is [7 , 5 , 5 , 5 , 5] .

Impact of Different Transition Matrices. To demonstrate the effectiveness of the specifically designed occlude and replace transition matrix, we constructed three transition matrices for discrete diffusion process: occlude transition matrix, replace transition matrix, and occlude and replace transition matrix. Table 5 illustrates that the optimal results are achieved when utilizing the occlude and replace transition matrix. The suboptimal performance observed when exclusively employing the other two transition matrices can be attributed to the following reasons: Utilizing solely the replace transition matrix introduces the challenge of random, irrelevant sub-structures,

complicating the learning of the reverse process; Conversely, relying exclusively on the occlude

(b) 3DPW

Image

GT

Ours

Image

GT

Ours

(a) Human3.6M

transition matrix causes the model to overly focus on the occluded portions, neglecting the contextual information from other visible parts. These clarifications can be verified in Table 3c, where we investigate the impact of different γ S . When γ S = 0 , the occlude and replace transition matrix can be seen as the replace transition matrix, and when γ S = 1 , the occlude and replace transition matrix can be seen as the occlude transition matrix. The best performance is obtained when γ S = 0 . 9 .

In addition, we conducted an ablation study to investigate the impact of S on the training and inference processes, as shown in Table 3d. We observed that using larger numbers of steps during both training and inference stages improves performance but also increases time complexity. Moreover, the results indicate that performance remains satisfactory even when the number of inference steps is reduced by 75% (e.g., from 100 steps during training to 25 steps during inference). This finding suggests a viable strategy for enhancing generation speed without significantly compromising quality.

5 Limitations

Figure 5 illustrates several results of 3D human pose estimation. When substantial occlusions cover the human body-obscuring the exact pose to the extent that it confounds even human observers-the predictions made by Di 2 Pose may deviate from GT 3D pose. This deviation primarily stems from the inherent limitation of inferring 3D poses directly from 2D images, which lack critical spatial depth information. Such limitations introduce uncertainty and indeterminacy in the predictions.

Despite these challenges, Di 2 Pose manages occlusions effectively by producing physically plausible outcomes. This capability is attributed to the integration of a pose quantization step within Di 2 Pose, which constrains the model's search space to physically reasonable configurations. Note that the pose quantization step is trained on real 3D human pose data, enhancing its reliability under severe occlusions.

Currently, Di 2 Pose is primarily designed for frame-based 3D HPE and does not utilize interframe data from videos. Future enhancements will focus on incorporating interframe information to refine the accuracy of 3D pose predictions further within the Di 2 Pose framework.

6 Conclusion

This paper presents Di 2 Pose, a novel diffusion-based framework that tackles occluded 3D HPE in discrete space. Di 2 Pose first captures the local interactions of joints and represents a 3D pose by multiple quantized tokens. Then, the discrete diffusion process models the discrete tokens in latent space through a conditional diffusion model, which implicitly introduces occlusion into the modeling process for more reliable 3D HPE with occlusions. Experimental results show that our method surpasses the state-of-the-art approaches on three widely used benchmarks.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62337001) and the Fundamental Research Funds for the Central Universities (226-2024-00058). Long Chen is supported by HKUST Special Support for Young Faculty (F0927) and HKUST Sports Science and Technology Research Grant (SSTRG24EG04).

References

I. Akhter and M. J. Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1446-1455, 2015.

T. Alldieck, H. Xu, and C. Sminchisescu. imghum: Implicit generative models of 3d human shape and articulated pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5461-5470, 2021.

J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981-17993, 2021.

O. Avrahami, D. Lischinski, and O. Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1820818218, 2022.

S. Azadi, A. Shah, T. Hayes, D. Parikh, and S. Gupta. Make-an-animation: Large-scale text-conditional 3d human motion generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15039-15048, 2023.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

B. Biggs, D. Novotny, S. Ehrhardt, H. Joo, B. Graham, and A. Vedaldi. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Advances in neural information processing systems , 33:20496-20507, 2020.

E. A. Brempong, S. Kornblith, T. Chen, N. Parmar, M. Minderer, and M. Norouzi. Denoising pretraining for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4175-4186, 2022.

Y. Cai, L. Ge, J. Liu, J. Cai, T.-J. Cham, J. Yuan, and N. M. Thalmann. Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2272-2281, 2019.

S. Chen, P. Sun, Y. Song, and P. Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19830-19843, 2023.

T. Chen, C. Fang, X. Shen, Y. Zhu, Z. Chen, and J. Luo. Anatomy-aware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology , 32(1):198-209, 2021.

Y. Cheng, B. Wang, B. Yang, and R. T. Tan. Graph and temporal convolutional networks for 3d multi-person pose estimation in monocular videos. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 1157-1165, 2021.

H.-g. Chi, S. Chi, S. Chan, and K. Ramani. Pose relation transformer refine occlusions for human pose estimation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 6138-6145. IEEE, 2023.

J. Choi, D. Shim, and H. J. Kim. Diffupose: Monocular 3d human pose estimation via denoising diffusion probabilistic model. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3773-3780. IEEE, 2023.

H. Ci, C. Wang, X. Ma, and Y. Wang. Optimizing network structure for 3d human pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2262-2271, 2019.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Z. Fan, J. Liu, and Y . Wang. Motion adaptive pose estimation from compressed videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11719-11728, 2021.

R. Feng, Y. Gao, T. H. E. Tse, X. Ma, and H. J. Chang. Diffpose: Spatiotemporal diffusion model for video-based human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14861-14872, 2023.

L. G. Foo, J. Gong, Z. Fan, and J. Liu. System-status-aware adaptive network for online streaming video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10514-10523, 2023.

Z. Geng, C. Wang, Y. Wei, Z. Liu, H. Li, and H. Hu. Human pose as compositional tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 660-671, 2023.

J. Gong, L. G. Foo, Z. Fan, Q. Ke, H. Rahmani, and J. Liu. Diffpose: Toward more reliable 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13041-13051, 2023.

S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong. Diffuseq-v2: Bridging discrete and continuous text spaces for accelerated seq2seq diffusion models. arXiv preprint arXiv:2310.05793 , 2023.

S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research , 23(47):1-33, 2022.

K. Holmquist and B. Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15977-15987, 2023.

E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems , 34:12454-12465, 2021.

M. R. I. Hossain and J. J. Little. Exploiting temporal information for 3d human pose estimation. In Proceedings of the European conference on computer vision (ECCV) , pages 68-84, 2018.

M. Hu, Y. Wang, T.-J. Cham, J. Yang, and P. N. Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11502-11511, 2022.

W. Hu, C. Zhang, F. Zhan, L. Zhang, and T.-T. Wong. Conditional directed graph convolution for 3d human pose estimation. In Proceedings of the 29th ACM International Conference on Multimedia , pages 602-611, 2021.

R. Huang, Z. Zhao, H. Liu, J. Liu, C. Cui, and Y. Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia , pages 2595-2605, 2022.

N. Inoue, K. Kikuchi, E. Simo-Serra, M. Otani, and K. Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10167-10176, 2023.

C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence , 36(7):1325-1339, 2013.

W. Jiang, S. Jin, W. Liu, C. Qian, P. Luo, and S. Liu. Posetrans: A simple yet effective pose transformation augmentation for human pose estimation. In European Conference on Computer Vision , pages 643-659. Springer, 2022.

Z. Jiang, Z. Wang, and L. Chen. Combing text-based and drag-based editing for precise and flexible image editing. arXiv preprint arXiv:2410.03097 , 2024.

Z. Jiang, Z. Zhou, L. Li, W. Chai, C.-Y. Yang, and J.-N. Hwang. Back to optimization: Diffusion-based zero-shot 3d human pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6142-6152, 2024.

H. Joo, N. Neverova, and A. Vedaldi. Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation. In 2021 International Conference on 3D Vision (3DV) , pages 42-52. IEEE, 2021.

M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black. Pare: Part attention regressor for 3d human body estimation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11127-11137, 2021.

H. Kong, K. Gong, D. Lian, M. B. Mi, and X. Wang. Priority-centric human motion generation in discrete latent space. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14806-14816, 2023.

J. N. Kundu, S. Seth, M. Rahul, M. Rakesh, V. B. Radhakrishnan, and A. Chakraborty. Kinematicstructure-preserved representation for unsupervised 3d human pose estimation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 11312-11319, 2020.

D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523-11532, 2022.

J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu, and C. Lu. Human pose regression with residual log-likelihood estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 11025-11034, October 2021.

J. Li, C. Xu, Z. Chen, S. Bian, L. Yang, and C. Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3383-3393, 2021.

K. Liu, R. Ding, Z. Zou, L. Wang, and W. Tang. A comprehensive study of weight sharing in graph networks for 3d human pose estimation. In Proceedings of the European conference on computer vision (ECCV) , pages 318-334. Springer, 2020.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

W. Mao, Y. Ge, C. Shen, Z. Tian, X. Wang, Z. Wang, and A. v. den Hengel. Poseur: Direct human pose regression with transformers. In Proceedings of the European conference on computer vision (ECCV) , pages 72-88. Springer, 2022.

J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision , pages 2640-2649, 2017.

F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505 , 2023.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

Q. Nie, Z. Liu, and Y. Liu. Lifting 2d human pose to 3d with domain adapted 3d body concept. International Journal of Computer Vision , 131(5):1250-1268, 2023.

S. Park, J. Hwang, and N. Kwak. 3d human pose estimation using convolutional neural networks with 2d pose information. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14 , pages 156-169. Springer, 2016.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-fine volumetric prediction for singleimage 3d human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7025-7034, 2017.

D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7753-7762, 2019.

J. Peng, D. Liu, S. Xu, and H. Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10775-10784, 2021.

I. A. Petrov, R. Marin, J. Chibane, and G. Pons-Moll. Object pop-up: Can we infer 3d objects and their poses from human interactions alone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4726-4736, 2023.

I. Radwan, A. Dhall, and R. Goecke. Monocular image 3d human pose estimation under self-occlusion. In Proceedings of the IEEE International Conference on Computer Vision , pages 1888-1895, 2013.

A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019.

C. Rockwell and D. F. Fouhey. Full-body awareness from partial observations. In European Conference on Computer Vision , pages 522-539. Springer, 2020.

G. Rogez and C. Schmid. Mocap-guided data augmentation for 3d pose estimation in the wild. Advances in neural information processing systems , 29, 2016.

G. Rogez, P. Weinzaepfel, and C. Schmid. Lcr-net: Localization-classification-regression for human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3433-3441, 2017.

C. Rommel, E. Valle, M. Chen, S. Khalfaoui, R. Marlet, M. Cord, and P. Pérez. Diffhpe: Robust, coherent 3d human pose lifting with diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3220-3229, 2023.

I. Sárándi, T. Linder, K. O. Arras, and B. Leibe. How robust is 3d human pose estimation to occlusion? arXiv preprint arXiv:1808.09316 , 2018.

W. Shan, Z. Liu, X. Zhang, S. Wang, S. Ma, and W. Gao. P-stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation. In European Conference on Computer Vision , pages 461-478. Springer, 2022.

W. Shan, Z. Liu, X. Zhang, Z. Wang, K. Han, S. Wang, S. Ma, and W. Gao. Diffusion-based 3d human pose estimation with multi-hypothesis aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14761-14771, 2023.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV) , pages 529-545, 2018.

I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34:24261-24272, 2021.

S. Tripathi, L. Müller, C.-H. P. Huang, O. Taheri, M. J. Black, and D. Tzionas. 3d human pose estimation via intuitive physics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 4713-4725, 2023.

A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV) , pages 601-617, 2018.

J. Wang, S. Yan, Y. Xiong, and D. Lin. Motion guided 3d pose estimation from videos. In European conference on computer vision , pages 764-780. Springer, 2020.

Z. Wang, Y. Jiang, D. Zheng, J. Xiao, and L. Chen. Event-customized image generation. arXiv preprint arXiv:2410.02483 , 2024.

E. Xie, L. Yao, H. Shi, Z. Liu, D. Zhou, Z. Liu, J. Li, and Z. Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4230-4239, 2023.

J. Xu, Z. Yu, B. Ni, J. Yang, X. Yang, and W. Zhang. Deep kinematics analysis for monocular 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on computer vision and Pattern recognition , pages 899-908, 2020.

T. Xu and W. Takano. Graph stacked hourglass networks for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16105-16114, 2021.

Y. Xu, Z. Wang, J. Xiao, W. Liu, and L. Chen. Freetuner: Any subject in any style with training-free diffusion. arXiv preprint arXiv:2405.14201 , 2024.

Y. Xu, J. Zhang, Q. Zhang, and D. Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems , 35:38571-38584, 2022.

D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y. Zou, and D. Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2023.

A. Zanfir, M. Zanfir, A. Gorban, J. Ji, Y. Zhou, D. Anguelov, and C. Sminchisescu. Hum3dil: Semisupervised multi-modal 3d humanpose estimation for autonomous driving. In Conference on Robot Learning , pages 1114-1124. PMLR, 2023.

J. Zhang, K. Gong, X. Wang, and J. Feng. Learning to augment poses for 3d human pose estimation in images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

T. Zhang, B. Huang, and Y. Wang. Object-occluded human shape and pose estimation from a single color image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7376-7385, 2020.

Y. Zhang, P. Ji, A. Wang, J. Mei, A. Kortylewski, and A. Yuille. 3d-aware neural body fitting for occlusion robust 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9399-9410, 2023.

L. Zhao, X. Peng, Y. Tian, M. Kapadia, and D. N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3425-3435, 2019.

Q. Zhao, C. Zheng, M. Liu, and C. Chen. A single 2d pose with context is worth hundreds for 3d human pose estimation. Advances in Neural Information Processing Systems , 36, 2024.

Q. Zhao, C. Zheng, M. Liu, P. Wang, and C. Chen. Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8877-8886, 2023.

W. Zhao, W. Wang, and Y. Tian. Graformer: Graph-oriented transformer for 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2043820447, 2022.

C. Zheng and A. Vedaldi. Online clustered codebook. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22798-22807, 2023.

C. Zheng, S. Zhu, M. Mendieta, T. Yang, C. Chen, and Z. Ding. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11656-11665, 2021.

J. Zhou, T. Zhang, Z. Hayder, L. Petersson, and M. Harandi. Diff3dhpe: A diffusion model for 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2092-2102, 2023.

Appendix

In this Appendix, we provide relevant preliminary knowledge, mathematical proofs, complete training and inference algorithms, additional experimental results, more implementation details about our Di 2 Pose and broader impacts.

A Preliminary: Continuous Diffusion Model

The continuous diffusion model consists of two primary processes: the forward process and the reverse process . The forward process methodically corrupts the original data x 0 into a noisy latent variable x S , which converges to a stationary distribution (e.g., a Gaussian distribution). Conversely, the reverse process aims to reconstruct the original data x 0 from x S , utilizing learned parameters.

Forward Process Starting with x 0 drawn from the distribution q ( x 0 ) , the forward process incrementally corrupts x 0 through a sequence of latent variables x 1: S = ( x 1 , x 2 , . . . , x S ) , where each x s retains the same dimensionality as x 0 . This transformation is modeled as a fixed Markov chain:

where each transition q ( x s | x s -1 ) is defined by a Gaussian distribution:

Here, η s is a small positive constant that follows a predefined schedule ( η 1 , η 2 , . . . , η S ) , allowing the data to progressively approach an isotropic Gaussian distribution, N ( 0 , I ) , as s increases. The overall transition from x 0 to x s can thus be expressed as:

where ζ s = 1 -η s and ζ s = ∏ s i =1 ζ i .

Reverse Process In the reverse process, the model aims to convert the latent variable x S , which is assumed to follow the distribution N ( 0 , I ) , back into the original data x 0 . The joint probability distribution is given by:

The conditional distributions involved are inferred using Bayes rule as follows:

To optimize the generative model p θ ( x 0 ) for fitting the data distribution q ( x 0 ) , we minimize a variational upper bound on the negative log-likelihood:

However, continuous diffusion models are not applicable in discrete spaces, such as quantized token indices k = ( k 1 , k 2 , . . . , k N ) where each k i assumes one of |C| discrete values. This limitation arises because Gaussian noise cannot corrupt discrete elements in a meaningful way. Thus, modeling in discrete spaces necessitates the development of discrete diffusion processes.

B Mathematical Proofs

In this section, we provide a detailed mathematical proofs for Eq. (6), which can quickly calculate q ( k s | k 0 ) according to Eq. (2).

Concretely, we use mathematical induction to prove Eq. (6). At first, we have following conditional information:

Now we want to prove that M s c ( k 0 ) = α s c ( k 0 ) + ( γ s -β s ) c ( |C| +1) + β s . Firstly, when s = 1 , we have:

̸

̸

which is clearly hold. Suppose the Eq. (6) holds at step s , then for s = s +1 , we have:

Now we consider three conditions:

(1) when k = k 0 in step s +1 , we have:

(2) when k = |C| +1 in step s +1 , we have:

̸

(3) when k = k 0 and k = |C| +1 in step s +1 , we have:

̸

The proof of Eq. (6) is completed. Notably, according to Eq. (6), the computation cost of q ( k s | k 0 ) can be reduced from O ( |C| 2 S ) to O ( |C| ) .

C Algorithms for Discrete Diffusion Process

In this section, we provide complete training and inference algorithms for discrete diffusion process.

C.1 Training Procedure

The discrete diffusion process aims to model quantized 3D pose tokens in a discrete space. This involves utilizing a 2D image I and its corresponding 3D human pose P as inputs. The image I serves as a contextual condition, while P is converted into discrete tokens for modeling.

Algorithm 1 Training Algorithm for the discrete diffusion process.

Require:

A transition matrix M s , the number of steps S , parameters of pose denoiser θ , training epoch T , pose dataset D (including 2D image I and 3D human pose P ), and the well-learned pose encoder f ( · ) .

Algorithm 2 Inference Algorithm for the discrete diffusion process.

Require:

The number of steps S , input 2D image I , the pose decoder f PD ( · ) , parameters of pose denoiser θ , stationary distribution p ( k ) ;

Firstly, the 3D human pose P is encoded by f PE ( · ) and subsequently quantized using the FSQ technique, resulting in multiple discrete tokens. Concurrently, a pre-trained Image Encoder extracts contextual features from I , producing a conditional feature sequence y . During the forward process, we sample s from a uniform distribution { 1 , 2 , ..., S -1 , S } and compute q ( k s | k 0 ) based on Eq. (6). In the reverse process, the pose denoiser f θ ( k s -1 | k s , y ) is trained to estimate q ( k s -1 | k s , k 0 ) . Finally, the overall loss is calculated according to Eq. (10), and the parameters of the pose denoiser θ are updated accordingly.

The complete training algorithm for the discrete diffusion process is presented in Algorithm 1.

C.2 Inference Procedure

In the inference process, our objective is to recover the 3D human pose ˆ P from an input 2D image and discrete tokens.

Initially, all pose tokens are either masked or initialized randomly, which is achieved by sampling from the stationary distribution p ( k S ) . The 2D image I is encoded using the pre-trained Image Encoder. Subsequently, we predict f θ ( k s -1 | k s , y ) step by step until the pose tokens are fully recovered. Finally, the reconstructed tokens are decoded using the pose decoder f PE ( · ) , yielding the recovered 3D pose ˆ P .

The complete inference algorithm for the discrete diffusion process is presented in Algorithm 2.

D Additional Implementation Details

All experiments are carried out on one NVIDIA A100 PCIe GPU. The proposed Di 2 Pose is completely implemented in PyTorch [53]. In this section, we provide the detailed training settings for the pose quantization step and the discrete diffusion process.

For the pose quantization step, we employ the AdamW [46] optimizer with β 1 = 0 . 9 and β 2 = 0 . 999 , adhering to a base learning rate of 1e-3 and a weight decay parameter of 0.15. The training process is configured with a batch size of 256 across a total of 20 epochs.

For the discrete diffusion process, we still utilize the the AdamW optimizer with β 1 = 0 . 9 and β 2 = 0 . 96 , adhering to a base learning rate of 5.5e-4 and a weight decay parameter of 4.5e-2. The training process is configured with a batch size of 64 across a total of 50 epochs.

E Additional Experimental Results

We exhibit more experimental results to verify the effectiveness of our Di 2 Pose.

E.1 Quantitative Results

As shown in Table 6, we benchmark Di 2 Pose against SOTA 3D HPE methods on the Human3.6M under PA-MPJPE protocol. Our Di 2 Pose achieves 39.0mm in average PA-MPJPE, surpassing the performance of the compared SOTA 3D HPE methods, which indicates that Di 2 Pose is able to enhance monocular 3D HPE in indoor scenes.

E.2 Qualitative Results

In this part, we present additional qualitative results on the Human3.6M and 3DPW datasets. As illustrated in Figure 6, our Di 2 Pose model demonstrates the ability to accurately recover 3D human poses in both indoor and in-the-wild scenarios. Particularly noteworthy is its performance under various occlusion conditions, including self-occlusion and object occlusion. Even in these challenging situations, Di 2 Pose consistently produces reasonable 3D pose estimations, highlighting its robustness to occlusions.

F Broader Impacts

This research focuses on estimating physically valid 3D human poses from monocular frames, especially under occlusion scenes. Such a method can be positively used for sports analysis, surveillance, healthcare, autonomous driving, etc. where clear, unobstructed views of the subject may not always be available. It can also lead to malicious use cases, such as illegal surveillance and video synthesis. Thus, it is essential to deploy these algorithms with care and make sure that the extracted human poses are with consent and not misused. Moreover, the diffusion-based model has a longer runtime compared to other CNN or GCN-based methods, causing more computational resources and energy consumption.

(a) Human3.6M

(b) 3DPW

Image

GT

Ours

Image

GT

Ours

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Refer to the Abstract and Sec.1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Refer to Sec.5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Refer to Sec.3 and Sec.B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Refer to Sec.4.1, Sec.4.2 and Sec.D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release code upon paper acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Refer to Sec.4.2 and Sec.D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Refer to Sec.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Refer to Sec.D in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We make sure to conduct this paper conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Refer to Sec.F in the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All used datasets and models in this paper are explicitly mentioned and properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not involve this issue.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We use publicly available 3D human pose datasets in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve this issue.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a discrete diffusion method for 3D human pose estimation (HPE). Recent works have successfully applied diffusion models for HPE. However, they need a lot of training data and sometimes output non-anthropomorphic poses. In this work, the authors propose to use discrete diffusions, which leverage a quantized latent space and model the diffusion process as transitions between discrete states. The hope is that the result will be more constrained to the space of plausible poses as it can only take a limited number of values.\n\nThe main contributions of this paper are the following: (1) The authors propose a VQ-VAE for quantizing human pose and modeling the dependencies between the body joints; (2) A discrete diffusion model is designed to do HPE in the quantized latent space; (3) The \"Occlude and replace\" strategy is proposed to model occlusions, further improving the results.\n\nSTRENGTHS:\n- The approach is new: this is the first work using discrete diffusion for human pose estimation. This approach is promising and performs well, especially in the presence of occlusions.\n\n- I found the paper quite well written. The weakness of prior works is clearly pointed out, and we understand how the authors hope to address them. \n\n- There are plenty of details about the models and their mathematical ground. Calculations are well-detailed, which makes them easy to follow.\n\n- The experiments and numerous ablations help capture the strengths of the introduced approach. The proposed approach outperforms SOTA approaches.\n\nWEAKNESSES:\nThe method section could be improved to make it more clear:\n- Giving the dimensions of $F$ and $T$ from the beginning (L137) would help understanding the VQ-VAE.\n- (L183) I find it very surprising that the loss for training the VQ-VAE is a cross-entropy. Usually, we use reconstruction loss for training such models (here, it could have been an MSE or some MPJPE). It is unclear why this is not the case for the proposed model.\n- (L185) It is unclear if $k$ is the sequence of $N$ tokens or only one token. Given the Equation 2, I guess it is the sequence?\n- (L193) The notation $k_i$ is already used in Equation 2 (and I think this does not denote the same thing as here it is temporal).\n- (L270) Using the same variable name for 2 different values is not a good practice.\n\nEven if this paper is the first to apply discrete diffusion models to HPE, the methodology section does not bring much novelty. Section 3.2 could be considered preliminary as this directly applies [23]. From my understanding, the only difference is that the \"Mask and replace\" operation of [23] is renamed \"Occlude and replace\" (but it does the same thing). I think it is good to use prior works when designing a new method, but it should be made clear which part is new and which part is an application of prior works.\n\nThe introduction justifies this work by saying continuous diffusion models have high dimensions and that relationships between the joints are ignored. From my understanding, the latent representation of a pose is composed of $N=100$ tokens of dimension $d=5$. Each of the $d_i$ dimensions can take a certain number of values (respectively 7,5,5,5,5). That means that the codebook has 4375 tokens. The transition matrix $M$ between consecutive intermediate predictions is then of dimension $4375 \\times 4375$ if we suppose that the $N$ tokens are independent (which is not supposed to be the case if we want the pose to be globally coherent). In the end, I am not sure that the proposed model's dimension is smaller than continuous diffusion models and that the relationships between the joints are better modeled.\n\nI also have some doubts about the fairness of experiments. The proposed model uses a ViT backbone, while all others (except [20]) have CNN or GCN backbones with much fewer parameters. I believe this is a crucial component of the model and that it is impossible to evaluate the contributions if basic components differ from other approaches.\n\nThe material used for experiments is given, but this is valuable only if we have running time for experiments (which is not the case).\n\nQUESTIONS:\n- Is $k$ (L185) a sequence?\n- Are there some changes between the proposed diffusion model and the model from [23]?\n- Is the relationship between the $N=100$ tokens modeled?\n- Is the plausibility of the results due to the quantization or the discrete diffusion process? I know this is linked, but for instance, would it be possible to do a latent diffusion process on the continuous latent representation of dimension $N \\times d$ (which is much smaller) and do the quantization just before decoding?\n- Were other backbones tested to quantify the role of this component in the results?\n- What are the running times (training and testing)? Is it comparable to other methods?\n- In Section 4, where are the error bars and the information about the statistical significance? Were the experiments run multiple times?\n- One of the advantages of diffusion models is that they can generate diverse outputs given a single 2D observation, which helps address the ambiguity of predicting 3D from a single 2D POV and occlusions. However, no analysis is performed on that; it seems that a single prediction is made given an image. Is there a reason for that?\n\nLIMITATIONS:\nThe section on the limitations is quite limited. I believe the problem of predictions under heavy occlusion could have been solved by making multiple predictions given an image, which is straightforward with generative models such as diffusions. Limitations could also mention the running time of the diffusion-based approaches. Limitations include the fact that the experiments ran for a single time, which is particularly concerning since diffusion models introduce stochasticity in predictions.\n\nThe societal impact could include environmental considerations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work aims to solve the occlusion in 3D human pose estimation, which is an interesting and inherent topic in this field. The authors critiqued that the current continuous diffusion-based pose estimation method requires a large amount of data in the training, while 3D pose datasets are commonly insufficient and thus, hurt the accurate pose generation, especially for occluded cases. To address the problem, the authors first designed a Local-mlp block to quantize poses into tokens, while being aware of the local connection between each keypoints. Then, a diffusion model is exploited to learn the generation of tokens in the condition of a given image. To better address the occluded case, the author designed two addition tokens named \"occ\" to simulate the occlusion. All combined, the method shows on-par results in 3D pose estimations and the state-of-art performance in the occluded dataset.\n\nSTRENGTHS:\n1. The design of the \"occ\" tokens is interesting. Previous data augmentation is majorly in the image level, i.e. blocking a few key points in the image by adding noise or set to black. However, this paper does it at the token-level, which potentially leads to a more natural simulation of the occlusion.\n\n2. Combining diffusion with 3D poses is an interesting topic. Especially generate the pose at a token-level, which leverages a similar idea to latent diffusion. \n\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. In the introduction, the authors criticized the previous method, which requires a large amount of 3D pose data in training, while the proposed solution is to use the quantization of 3D poses. It is unclear why intuitively using a quantized representation of poses could be a solution to data dependency. \n\n2. The target of this paper is also unclear. In my understanding of the author's argument, it is the data dependency of the discrete diffusion model that leads to inaccurate pose estimation in occlusion. However, instead of addressing the data dependency problem, the authors develop occ. token and replacement strategy to address the occlusion problem. This is confusing to me. It is like claiming there is a reason A for problem P, but instead of addressing A, the authors propose another method B completely irrelevant to A to address P from another perspective. \n\n3. Confusing definitions of \"Continuous\" and \"Discrete\". There are two pairs of \"Continuous\" and \"Discrete\" in the paper. The first pair is the continuous and discrete representation of 3D poses. The second pair is a continuous and discrete diffusion model, as current DDPM or DDIM could be considered as a discretization of a continuous SDE or ODE. It is confusing why it is called continuous in the paper.\n\nQUESTIONS:\n1. For local-map, could there be a more clear explanation of how it exactly captures the interrelation between points?\n\n2. Is it possible to visualize a few quantitative samples in the training process? Especially for poses that get replaced by occ tokens or some tokens get transferred to different tokens. I suspect such a replacement is potentially to be a good method to simulate the person-to-person occlusions.\n\n3. What is the performance of directly using VQ-VAE to encode the 3D poses? Without these results, it is difficult to know the effectiveness of the proposed local-mlp.\n\nLIMITATIONS:\nNo potential negative societal impact of the work is found.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work claims that the 3D human pose of a single frame is discrete, and learns the local pairing relationship between joint points to generate the human pose under occlusion. At the method level, VQ-VAE is used for human skeleton quantization, and then combined with the diffusion model to solve this discrete relationship. The main contributions are as follows: 1. The Di2Pose framework is proposed, which integrates the inherent discreteness of 3D pose data into the diffusion model, providing a new paradigm for 3D HPE under occlusion. 2. The designed pose quantization step represents the 3D pose in a combinatorial manner, effectively captures the local correlation between joints, and limits the search space to reasonable configurations. 3. The constructed discrete diffusion process simulates the complete process of 3D pose from occlusion to recovery, and introduces the impact of occlusion into the pose estimation process.\n\nSTRENGTHS:\n1. The structure and writing of the article are relatively clear.\n2. The inherent discreteness of 3D pose data is integrated into the diffusion model. Although this is a downstream task of VQ-VAE+Diffusion, it is indeed a new idea for 3D HPE of monocular 2D images.\n3. The reasoning process of Di^2Pose in Chapter 3 is very clear, which helps to understand the principle of the overall method.\n3. The influence of occlusion is introduced into the pose estimation process, and the codebook that encodes the human pose is back-diffused, which is also a solution to occluded human pose.\n\nWEAKNESSES:\n1. From an intuitive level, the human skeleton in this work still uses the simplest H3.6M skeleton with only 17 joints, not even the version with 32 key points. The error tolerance rate is very high for pose estimation of models with such a small number of joints. There is not even any obvious joint change. The Average MPJPE value in the Human3.6M list of 17 joint skeletons has stabilized within 30. The human skeleton in this work needs to use more complex SMPL, SMPL-X, and STAR to see the effect better.\n2. The image and skeleton prediction are independent. I hope the author can render the skeleton into the image and map it in the image to better observe the subtle differences in movement. It is difficult to observe the 3D skeleton alone, especially in Figure 4. I can't even find any obvious difference in the human posture in Figure a. In addition, when estimating a single-frame 3D skeleton, it is best to give results from multiple perspectives. I think the perspectives shown in the article are too casual.\n3. Although the method is clearly written, I think Di^2Pose is just a downstream task of VQ-VAE+Diffusion. In terms of design, it is not even as clever as [1]. I think Di^2Pose is just DiffPose with the addition of VQ-VAE.\n4. The dataset used in the experiment is too outdated. I hope to see more new datasets for verification. Like EMDB[2], SLOPER4D[3], CIMI4D[4], etc.\n\n[1] Feng H, Ma W, Gao Q, et al. Stratified Avatar Generation from Sparse Observations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 153-163.\n[2] Kaufmann M, Song J, Guo C, et al. Emdb: The electromagnetic database of global 3d human pose and shape in the wild[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 14632-14643.\n[3] Yan M, Wang X, Dai Y, et al. Cimi4d: A large multimodal climbing motion dataset under human-scene interactions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 12977-12988.\n[4] Dai Y, Lin Y T, Lin X P, et al. Sloper4d: A scene-aware dataset for global 4d human pose estimation in urban environments[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 682-692.\n\nQUESTIONS:\nI hope to see the results on new datasets and SMPL renders to images. I will consider improving my score.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents  novel diffusion-based framework for occluded 3D Human Pose Estimation (HPE) that operates in discrete space. Di2Pose leverages a two-stage process: a pose quantization step and a discrete diffusion process. The pose quantization step captures the local interactions between joints and represents the 3D pose as multiple quantized tokens. These tokens are then modeled in the latent space through a discrete diffusion process. This approach allows the framework to effectively manage occlusions by simulating the transition of a 3D pose from occluded to recovered, enhancing the reliability of pose estimation under occlusion conditions. Extensive evaluations on benchmarks like Human3.6M, 3DPW, and 3DPW-Occ demonstrate that Di2Pose outperforms state-of-the-art methods, particularly in occluded scenarios.\n\nSTRENGTHS:\nIntroduction of Di2Pose Framework\n   Di2Pose integrates the inherent discreteness of 3D pose data into the diffusion model, providing a novel paradigm for addressing 3D HPE under occlusions. This framework leverages a two-stage process involving pose quantization and discrete diffusion to confine the search space to physically plausible configurations and simulate the transition from occluded to recovered poses.\n\nPose Quantization Step:\n   The designed pose quantization step effectively captures local correlations between joints by representing 3D poses in a compositional manner. This step confines the search space to reasonable configurations by learning from real 3D human poses, ensuring that the model generates physically plausible poses even under severe occlusions.\n\nDiscrete Diffusion Process\n   The discrete diffusion process simulates the complete transition of a 3D pose from occluded to recovered, incorporating the impact of occlusions into the pose estimation process. This process models the quantized pose tokens in latent space, enhancing the model’s capability to understand and predict occluded parts of the human pose.\n\nStrengths claimed and shown include\n\nEffectiveness in Occluded Scenarios: Di2Pose demonstrates significant improvements in 3D HPE accuracy under occlusions compared to state-of-the-art methods, highlighting its superior occlusion-handling capabilities.\n  \nPhysically Plausible Pose Generation: By integrating pose quantization and discrete diffusion, Di2Pose confines the search space to physically reasonable configurations, ensuring the generation of biomechanically valid poses even in challenging occluded scenarios.\n\nComprehensive Evaluation: The framework has been extensively evaluated on multiple challenging benchmarks, consistently yielding lower errors and demonstrating its robustness and generalizability across different datasets.\n\nUse of Discrete Diffusion: The introduction of a discrete diffusion process tailored for 3D HPE provides a new perspective in the field, aligning more closely with the inherent discreteness of 3D pose data.\n\nThese contributions collectively position Di2Pose as a robust and innovative solution for 3D HPE, particularly in the presence of occlusions, advancing the state-of-the-art in this challenging domain.\n\nWEAKNESSES:\nMechanistic Insights:\n   The paper does not delve deeply into the mechanistic aspects of how the discrete diffusion process works in conjunction with pose quantization. Detailed explanations of the underlying mechanisms and how they contribute to the observed improvements would enhance understanding.\n\nComparative Benchmarking:\n   While Di2Pose shows improvements over existing methods, a direct comparison with other contemporary methods such as the Pose Relation Transformer (Chi et al., ICRA 2023) and InfoGCN (Chi et al., CVPR 2022) would provide a clearer picture of its relative strengths and weaknesses.\n\nCitation of Related Works:\n   Incorporating references to other successful approaches in the field, such as the above\n   \n These citations would position Di2Pose within the broader context of ongoing research and highlight its unique contributions relative to other leading methods.\n\nAddressing these areas would strengthen the paper, providing deeper insights into the framework’s operation and situating it more firmly within the current landscape of 3D HPE research.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nLimitations are not explored or discussed. It follows the same formula - compare against a benchmark and do ablation studies and state it is better than state of art.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}