Geometry-Guided Diffusion Model with Masked Transformer for Robust Multi-View 3D Human Pose Estimation

Authors: xinyi  zhang
qinpeng  cui
qiqi  bao
wenming  yang
qingmin  liao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:40:41

Abstract: Recent research on Diffusion Models and Transformers has brought significant advancements to 3D Human Pose Estimation (HPE). Nonetheless, existing methods often fail to concurrently address the issues of accuracy and generalization. In this paper, we propose a **G**eometry-guided D**if**fusion Model with Masked Trans**former** (Masked Gifformer) for robust multi-view 3D HPE. Within the framework of the diffusion model, a hierarchical multi-view transformer-based denoiser is exploited to fit the 3D pose distribution by systematically integrating joint and view information. To address the long-standing problem of poor generalization, we introduce a fully random mask mechanism without any additional learnable modules or parameters. Furthermore, we incorporate geometric guidance into the diffusion model to enhance the accuracy of the model. This is achieved by optimizing the sampling process to minimize reprojection errors through modeling a conditional guidance distribution. Extensive experiments on two benchmarks demonstrate that Masked Gifformer effectively achieves a trade-off between accuracy and generalization. Specifically, our method outperforms other probabilistic methods by $\textgreater 40\\%$ and achieves comparable results with state-of-the-art deterministic methods. In addition, our method exhibits robustness to varying camera numbers, spatial arrangements, and datasets.

Main paper: 
 introduction: 
 3D Human Pose Estimation (HPE) aims to accurately localize sparse joints on the human skeleton in 3D space through analysing the multimedia data such as images and videos. The obtained geometric and positional information of the human body can be beneficial for various downstream tasks [57] , including action recognition [40, 51, 59] , action prediction [3, 44, 54] , action correction and online coaching [8, 9, 42] , animation [10, 25, 47] , and augmented reality/virtual reality [46, 50, 52] .Depending on the number of views, 3D HPE can be categorized into single-view and multi-view methods. Single-view methods [23, 24, 33, 53, 56] estimate 3D human pose from monocular images or videos, which may suffer from depth ambiguity. Such challenge can be alleviated to some extent when using multiple synchronized and calibrated cameras. Therefore, more attention has been paid to multi-view methods. Multi-view 3D HPE methods primarily follow a two-stage paradigm: (1) Utilizing off-the-shelf 2D pose detectors [4, 49] to locate 2D positions of joints in each view; (2) Mapping 2D positions to 3D poses and fusing multi-view information selectively. In this work, we focus on the latter step, known as the 2D-to-3D lifting process, following recent processes [18, 29, 34, 58] .2D-to-3D lifting methods accurately predict 3D poses from multiview 2D poses by designing effective neural networks. Many approaches [13, 27, 28, 34] leverage the powerful long-range dependency and interaction modeling capabilities of Transformers [41] to fuse multi-view geometric information, thereby minimizing reprojection errors. However, these methods still face several challenges. Firstly, strictly constrained by camera numbers and spatial arrangements, it is difficult to generalize well to an arbitrary number of unseen views. Moreover, these methods are generally deterministic, inferring the most probable single pose directly from the given input, which may fall into local optima in complex scenarios.Probabilistic methods [1] can alleviate uncertainty by generating multiple plausible pose hypotheses, leading to more accurate pose estimation. They have been extensively studied in single-view 3D HPE [2, 11, 15, 24, 33] , which is an ill-posed problem due to depth ambiguity brought on by single-view data. However, uncertainties persist in multi-view scenarios from severe occlusions, selfocclusions and noisy 2D detections. Therefore, probabilistic models for multi-view are worthy of further investigation. Existing methods directly utilize Diffusion Models (DMs) [14, 36, 37] to generate multiple hypotheses. Due to the lack of geometric guidance, the high-accuracy potential remains underexplored.To address the above issues, we propose a Geometry-guided Diffusion Model with Masked Transformer (Masked Gifformer), which unifies the geometry-guided conditional diffusion generative process with the masked hierarchical multi-view transformer, facilitating accurate distribution forecasting for 3D poses while enhancing generalization capabilities. As illustrated in Figure 1 , we use the framework of DDPM [14] . Initially, noise of varying scales is added to the ground truth 3D poses. Subsequently, a transformerbased denoiser is exploited to approximate the probability distribution 𝑝 (𝑋 3𝐷 | 𝑋 2𝐷 ) of 3D coordinates 𝑋 3𝐷 given the detected 2D coordinates 𝑋 2𝐷 . During the inference, Masked Gifformer employs its geometry-guided diffusion process to iteratively refine pose predictions.Motivated by [53] , we use a Hierarchical Multi-view Fusion method which aggregates joint and view information separately. More specifically, a Joint Transformer Block (JTB) is employed to learn physical constraints among joints within each view. Then, a Multi-view Transformer Block (MTB) is used to allow each joint to independently learn the corresponding information from other views. These two transformer blocks are trained alternately, facilitating comprehensive multi-view information fusion. To address the problem of poor generalization, we introduce a fully random mask mechanism into MTB without any additional learnable modules or parameters. By randomly masking a portion of views during training, it is possible to avoid the learned features that are highly correlated with camera numbers and positions. Different from [34] , our model even discards information from the current view. This strategy promotes a more effective integration of features from alternative views instead of overfitting to features from the source dataset, thus exhibiting robustness across different datasets. The efficiency of self-attention computations in Transformers can also be improved due to introducing sparsity.During the inference, we sample multiple noise instances from a Gaussian distribution as initial 3D pose hypotheses. These pose hypotheses are then fed into the transformer-based denoiser conditioned on multi-view 2D keypoints. To further enhance the accuracy, we incorporate geometric knowledge into the sampling process. Inspired by [7, 43] , we model the 𝑋 3𝐷 -conditioned guidance distribution 𝑝 (𝑋 2𝐷 | 𝑋 3𝐷 ) as a negative exponential form of reprojection errors and use its gradient ∇ 𝑋 3𝐷 log 𝑝 𝑋 2𝐷 | 𝑋 3𝐷 to perturb the pre-trained predictions. This strategy guides the sampling process towards minimizing reprojection errors, thereby further enhancing the accuracy of pose estimation. Finally, a single accurate and robust 3D pose is generated by aggregating all of hypotheses for practice use.In summary, our main contributions are as follows:• We propose a Geometry-guided Diffusion Model with Masked Transformer (Masked Gifformer) for robust multi-view 3D HPE. This method combines the capabilities of Transformers for high-quality multi-view information fusion and the probabilistic nature of Diffusion Models for generating multiple robust 3D pose hypotheses. • We introduce a fully random mask mechanism into the multiview transformer block, enhancing its strong robustness to camera numbers, spatial arrangements and datasets without adding any learnable modules or parameters.  
related works: 
 Transformer-based 3D HPE. With the superior capability of Transformers [41] in modeling long-range dependencies, numerous Transformer-based single-view [22, 23, 53] and multi-view [13, 27, 28, 34] 3D HPE methods have emerged. In multi-view settings, researchers have employed epipolar geometry for information fusion [13, 27] to minimize reprojection errors. Specifically, He et al. [13] extracted multiple features along the epipolar line in the source view and then fused them with features from the reference view. To fully exploit the semantic information in the source view, Ma et al. [27] proposed the concept of epipolar field, merging more information near the epipolar line while maintaining epipolar constraints. However, these methods cannot generalize well to an arbitrary number of unseen views as constrained by camera numbers and spatial arrangements. To address this issue, Shuai et al. [34] introduced a random mask mechanism within Transformers, enabling them to adaptively handle varying numbers of views and video lengths. Nevertheless, this mechanism does not guarantee generalization to different datasets. Jiang et al. [18] improved crossdataset generalization by modelling camera distributions. However, this method employed 2D heatmaps as feature inputs and used computationally intensive voxel networks during pose reconstruction, making it impractical for real-time applications. Diffusion-based 3D HPE. The aforementioned Transformerbased methods often infer the most probable single pose from the given input, falling under deterministic methods. They may be trapped in local optima, especially in complex scenarios. Probabilistic methods achieve more accurate pose estimation by generating multiple plausible pose hypotheses. Due to the inherent probabilistic nature of DMs, numerous diffusion-based single-view 3D HPE methods have emerged [2, 11, 15, 24, 33] to mitigate the uncertainty brought by depth ambiguity. Shan et al. [33] directly applied DMs to single-view 3D HPE. Cai et al. [2] introduced a disentanglement strategy in the forward process of the diffusion model to integrate the explicit human body prior. However, high-precision potential Summary. Both Transformers and DMs demonstrate significant potential in 3D HPE. To benefit from high-quality multi-view information fusion and multiple high-precision 3D pose hypotheses generation, we explore an approach that integrates Transformers and DMs. We introduce a fully random mask mechanism to enhance the generalization capabilities across varying camera numbers, spatial arrangements, and cross-datasets in a lightweight manner. Additionally, we incorporate geometric information into the sampling process of the diffusion model to guide it towards minimizing reprojection errors. 
method: 
 We consider the problem of estimating 3D positions of the human body when multi-view cameras are precisely calibrated.Formally, given 2D joints𝑋 2𝐷 = 𝑥 2𝐷 𝑖 ∈ R 𝐽 ×2 | 𝑖 = 1 . . . 𝑁 , our goal is to predict 3D positions 𝑋 3𝐷 = 𝑥 3𝐷 𝑖 ∈ R 𝐽 ×3 | 𝑖 = 1 .. . 𝑁 in the world coordinate system. 𝑁 and 𝐽 represent the number of cameras and joints, respectively. When cameras are calibrated with precisely known intrinsic and extrinsic parameters 𝑌 , deterministic methods leverage Transformers [13, 27, 28] denoted as 𝐹 to fuse multi-view geometric information, resulting in 𝑋 3𝐷 = 𝐹 (𝑋 2𝐷 , 𝑌 ).However, due to occlusions or self-occlusions, some joints may not be visible in the given multiple views, resulting in non-uniqueness of the human pose. The noisy 2D joints further exacerbates the uncertainty of HPE. Therefore, it is necessary to use probabilistic methods to generate multiple (𝐻 ) hypotheses of human poses {𝑋 3𝐷 𝑖 } 𝐻 𝑖=1 and find the most probable one. Combining the probabilistic nature of DMs with multi-view fusion capabilities of Transformers, we propose Masked Gifformer for robust multi-view 3D HPE. As shown in Figure 1 , we train a transformer-based denoiser to fit the probability distribution 𝑝 (𝑋 3𝐷 | 𝑋 2𝐷 ) within the framework of diffusion models. With the proposed fully random mask strategy, our model is robust to varying camera numbers, spatial arrangements and datasets. During the inference, by modeling the conditional guidance distribution 𝑝 (𝑋 2𝐷 | 𝑋 3𝐷 ), multiple 3D pose hypotheses are sampled towards minimizing reprojection errors. Finally, a single accurate and robust 3D pose is obtained by aggregating all of hypotheses.DMs [6, 14, 36, 38, 39] have been demonstrated to possess promising capabilities in modelling complex data distributions. A Denoising Diffusion Probabilistic Model (DDPM) [14] is one of them, which encompasses forward and reverse processes.Specifically, in the forward process, DDPM incrementally introduces Gaussian noise within 𝑇 steps to perturb the input {𝑥 𝑡 } 𝑇 𝑡 =0 . Such noising process forms a Markov chain and satisfies:EQUATIONHere, {𝛽 𝑡 } 𝑇 𝑡 =1 ∈ (0, 1) is a monotonically increasing set of noise scales with diffusion steps 𝑇 . I represents the identity matrix, and N (𝑥; 𝜇, 𝜎) denotes the Gaussian distribution generating 𝑥 with mean 𝜇 and covariance 𝜎. Let 𝛼 𝑡 = 1 -𝛽 𝑡 and ᾱ𝑡 = 𝑡 𝑖=1 𝛼 𝑖 , then noisy samples 𝑥 𝑡 can be obtained in a single step:EQUATIONIf ᾱ𝑡 → 0, the distribution of 𝑥 𝑇 should closely approximate the standard Gaussian distribution, i.e., 𝑞 (𝑥 𝑇 ) = N (0, I).In the reverse process, DDPM learns the mapping from a simple normal distribution to the true data distribution. If {𝛽 𝑡 } 𝑇 𝑡 =1 is sufficiently small, 𝑝 𝜃 (𝑥 𝑡 -1 | 𝑥 𝑡 ) still follows a Gaussian distribution. Fixing the covariance as a constant, a neural network D 𝜃 (𝑥 𝑡 , 𝑡) is trained to predict the mean of the Gaussian distribution, that is:EQUATIONwhere 𝜃 represents learnable parameters. Starting from the random pose 𝑥 𝑇 ∼ N (0, I), x0 conforming to the true distribution can be generated through iterative sampling:𝑥 𝑡 -1 ∼ N 𝑥 𝑡 -1 ; √ ᾱ𝑡 -1 D 𝜃 (𝑥 𝑡 , 𝑡) , (1 -ᾱ𝑡 -1 ) I . (4)In this section, we utilize DMs to model the probability distribution of 3D human poses across multiple views, and generate multiple pose hypotheses leveraging its probabilistic nature. Based on the diffusion framework [14] discussed in Sec.3.1, we first add noise to the ground truth 3D poses, and then train a denoiser D 𝜃 on a training set to fit the true data distribution. Note that given the 2D joint coordinates 𝑋 2𝐷 , we model the conditional probability distribution 𝑝 𝜃 (𝑋 3𝐷 |𝑋 2𝐷 ). Therefore, unlike the forward process (Eq.( 1)) which is independent of 𝑋 2𝐷 , the reverse process depends on the input 2D joint coordinates 𝑋 2𝐷 , i.e.,𝑝 𝜃 𝑋 3𝐷 𝑡-1 |𝑋 3𝐷 𝑡 , 𝑋 2𝐷 =N 𝑋 3𝐷 𝑡 -1 ; √ 𝛼 𝑡 D 𝜃 𝑋 3𝐷 𝑡 , 𝑡, 𝑋 2𝐷 , (1-𝛼 𝑡 )I .(5) Denoiser D 𝜃 . Due to the promising information interaction and global aggregation capabilities of Transformers [41] , we implement the denoiser D 𝜃 using Transformers. As illustrated in Figure 2 , previous work [13, 27, 28, 34, 55] proposed hybrid multi-view fusion methods, using a single type of transformer block to integrate joint information across all views. Inspired by [53] , we employ a hierarchical multi-view fusion method, using two distinct types of transformer blocks to focus on physical constraints among joints and the correlation across different views. Specifically, we first use a Joint Transformer Block (JTB) to learn the physical correlations between joints within each view. Subsequently, a Multi-view Transformer Block (MTB) is employed to enable each joint to individually learn corresponding information from other views. These two types of transformer blocks are alternately trained, achieving comprehensive multi-view information fusion.Fully random mask mechanism. However, the aforementioned naive architecture imposes strict limitations on camera numbers and spatial arrangements, making it challenging to generalize well to an arbitrary number of unseen views. To address this issue, [34] proposed a random mask mechanism, which randomly masked the attention matrix except for the elements on the diagonal at the rate of 𝑀. However, experimental results in Table 6 demonstrate that such random mask mechanism exhibits poor generalization across datasets. Therefore, we propose a fully random mask mechanism as depicted in Figure 3(b) , which even randomly discards information from the current viewpoint. This fully random mask mechanism offers the following advantages:• By randomly masking a portion of views during training, it is possible to avoid the learned features that are highly correlated with camera numbers and positions. Therefore, our model demonstrates robust generalization to varying camera numbers and spatial arrangements. • By discarding information from the current view, a more effective integration of features from alternative views is learned instead of overfitting to features from the source dataset, thus enhancing robustness across different datasets. • By introducing sparsity, it can improve the efficiency of the self-attention computations in Transformers. Incorporating the proposed fully random mask mechanism into the MTB, the detailed structure of the denoiser is showed in Figure 4 . The Masked Multi-head Self-attention (MMSA) is calculated by:EQUATIONwhere the query Q, the key K, and the value V are obtained from the linear projections of the input X. M is a fully random mask. 𝐷 is the dimension of feature. EQUATIONwhere the expectation is taken over all diffusion time 𝑡, corresponding to diffused samples 𝑋 3𝐷 𝑡 ∼ 𝑞 𝑋 3𝐷 𝑡 | 𝑋 3𝐷 0 , 𝑋 2𝐷 . Multi-Hypothesis Generation and Aggregation. During the inference, given a new set of 2D joints coordinates from multi-view, Eq.( 4) is rewritten as:EQUATION)Starting from 𝐻 random poses sampled from the Gaussian distribution, samples { X 3𝐷 0𝑖 } 𝐻 𝑖=1 conforming to the true distribution can be generated by the iterative sampling shown in Eq. (8) . Following [33] , the reverse process iterates 𝐾 times to enhance the differences between hypotheses.Since a single 3D pose is still required for practical use, we use the joint-wise reprojection-based multi-hypothesis aggregation method [33] , which selects the optimal hypothesis at the joint level based on reprojection errors (3D → 2D).As discussed in Sec.3.2, DMs can directly map 2D poses to the 3D poses via the supervised denoiser. However, deep networks often exhibit poor performance in precise quantity regression and are prone to overfitting [21, 43] . Inspired by [43] , we incorporate geometric information into the sampling process, significantly enhancing the accuracy of Masked Gifformer while maintaining its generalization capability. Focusing on the consistency between 2D screen coordinates and 3D world coordinates, we use reprojection errors to guide DDPM sampling iterations (Eq.( 8)).Reprojection errors. Given the 2D pose 𝑋 2𝐷 and the predicted 3D pose 𝑋 3𝐷 , we compute errors between the projection of the predicted 3D world coordinates onto the 2D image plane and their true projection [12] , i.e., Reprojection-guided sampling. Following the classifier diffusion guidance [7] , we guide the sampling towards minimizing reprojection errors, thereby satisfying the geometric consistency constraint from 3D to 2D.EQUATIONSpecifically, we model 𝑋 3𝐷 -conditioned guidance distribution 𝑝 (𝑋 2𝐷 | 𝑋 3𝐷 ) as a negative exponential form of reprojection errors:EQUATION)During each sampling iteration, classifier-guidance perturbs the predicted mean with a gradient of the conditioned guidance distribution 𝑝 (𝑋 2𝐷 | 𝑋 3𝐷 ), i.e., D𝜃 𝑋 3𝐷 , 𝑡, 𝑋 2𝐷 = D 𝜃 𝑋 3𝐷 , 𝑡, 𝑋 2𝐷 + 𝑠∇ 𝑋 3𝐷 log 𝑝 𝑋 2𝐷 | 𝑋 3𝐷 (11) where 𝑠 controls the strength of the guidance. Finally, the perturbed mean D𝜃 𝑋 3𝐷 , 𝑡, 𝑋 2𝐷 replaces D 𝜃 𝑋 3𝐷 , 𝑡, 𝑋 2𝐷 in Eq.( 5) and Eq.( 8), thereby achieving the reprojection-guided sampling. 
experiments 4.1 datasets and metrics: 
 Experiments are conducted on widely used 3D HPE benchmark datasets, including Human 3.6M [16] and CMU Panoptic [19, 20, 35] .Human3.6M is one of the largest motion capture datasets. It consists of videos captured from 11 professional actors and actresses performing 15 distinct activities (e.g., walking), with 4 calibrated and synchronized high-speed motion cameras placed at the corners of a rectangular room. Human3.6M dataset not only provides calibrated camera intrinsic and extrinsic parameters but also accurate 3D joint positions. Following [18, 33, 34] , our model is trained on 5 subjects (S1, S5, S6, S7, and S8) and tested on S9 and S11.CMU Panoptic stands out as a large-scale dataset that utilizes a larger number of cameras. It consists of 480 VGA cameras and 31 HD cameras installed on a hemispherical structure, capturing various activities performed by single or multiple individuals, such as dancing. Given our focus on single-person HPE, we only use HD videos from individual subjects, following [1, 48] . In accordance with [1, 18] , we partition the dataset into four subsets (CMU1 ∼ CMU4), with each subset varying in camera numbers and arrangements. CMU Panoptic also provides calibrated camera parameters and 3D joint coordinates.Metrics. Mean Per Joint Position Error (MPJPE) is used to evaluate the accuracy of 3D HPE. A lower MPJPE indicates higher accuracy in predicted joint positions relative to the ground truth. To ensure fair comparison with previous studies, we use CPN [4] to extract 2D poses for the Human3.6M dataset and a more powerful 2D pose detector [49] for the CMU Panoptic dataset. The proposed method is implemented in PyTorch [30] using the AdamW optimizer [26] , with momentum parameters 𝛽 1 = 0.9 and 𝛽 2 = 0.999, and weight decay of 0.1. Our model is trained for 100 epochs with an initial learning rate of 6𝑒 -5 , decayed by a factor of 0.993 after the 10th epoch. During training, the number of hypotheses and iterations are set to 𝐻 = 1 and 𝐾 = 1, respectively. During inference, 𝐻 = 20 and 𝐾 = 10. The maximum diffusion time step 𝑇 is 1000. To avoid spurious local minima, we adopt geometry-guided sampling starting from the second diffusion step. The guidance strength 𝑠 in Eq.( 11) can be set manually. In our paper, 𝑠 = 1.0. All experiments were conducted on a single NVIDIA GeForce RTX 3090 GPU.Results on Human3.6M. As shown in Table 1 (Top), we compare Masked Gifformer with the state-of-the-art deterministic multiview 3D HPE methods on Human3.6M. Following [1], the table only displays methods trained and tested on Human3.6M without additional training data (thus excluding [17] ). For fairness, we set 𝐻 = 1 to generate a single prediction. Although Masked Gifformer is designed to generate multiple 3D pose hypotheses, it achieves 24.7mm under the single hypothesis scenario, outperforming most deterministic methods and reaching performance comparable to the state-of-the-art [28] . Additionally, we compare Masked Gifformer with multi-view probabilistic methods, as shown in Table 1 (Bottom). When the number of hypotheses 𝐻 increases to 20, our method reduces the MPJPE from 24.7mm to 17.4mm. This demonstrates that generating multiple plausible pose hypotheses can alleviate the uncertainty still present in multi-view scenarios, thereby achieving more accurate pose estimation. Our results outperform [1] (𝐻 = 200) by a large margin with 11.7mm, even when using fewer hypotheses. To further validate the generality of our proposed Geometry-guided Sampling (GS), we apply it to the single-view probabilistic method [33] . As shown in Table 1 (Middle), with the help of GS, MPJPE of [33] decreased by 1.7mm. This indicates that incorporating geometric information during the sampling process can further enhance the accuracy of generated hypotheses. Results on CMU Panoptic. Table 2 compares our method with others on CMU Panoptic. Under a single hypothesis scenario (𝐻 = 1), Masked Gifformer surpasses the majority of deterministic methods. When 𝐻 increases to 20, MPJPE of our method drops to 11.2mm, also outperforming [1]. This is attributed to the incomplete views across most CMU Panoptic cameras, resulting in significant occlusions and missing body parts. Our method can provide more feasible poses, thus being more robust to occluded scenarios.In addition to strong multi-view pose fitting capability, another ability of Masked Gifformer is that it can generalize well to varying cameras numbers, spatial arrangements, and even different datasets, which has been a primary limitation of previous methods.Generalization performance on different camera numbers and spatial placements. The generalization results of Masked Gifformer and competitive methods on Human3.6M are shown in Table 3 . Masked Gifformer is trained with 2 or 4 views and tested with varying numbers of views. From a horizontal analysis perspective, our method can successfully generalize to any number of cameras with minimal generalization loss (< 7.4𝑚𝑚). We observe that the more cameras used during training, the smaller the generalization gap. Specifically, the maximum generalization gap reduces to 2.6𝑚𝑚 when trained with 4 views. Moreover, our method can generalize to different spatial arrangements (Table 3 (Left)). From a vertical analysis perspective, our method exhibits stronger generalization than [18] and [34] without sacrificing fitting abilities.Generalization performance on different datasets. To test the cross-dataset generalization capability of Masked Gifformer, we follow [1, 18] to divide the CMU Panoptic dataset into four subsets, denoted as CMU1 ∼ CMU4. Table 4 presents generalization results from four CMU Panoptic subsets to Human3.6M. From a horizontal comparison, our method successfully achieves cross-dataset generalization regardless of the training set. From a vertical comparison, our method demonstrates stronger generalization capabilities, surpassing [1, 17, 18] . For instance, when trained on CMU3 but tested on Human3.6M, MPJPE of [17] , [1] and [18] are 34.0𝑚𝑚, 31.0𝑚𝑚, and 29.2𝑚𝑚, respectively. In contrast, our method achieves 23.6𝑚𝑚, indicating a notable improvement in cross-dataset generalization. Effectiveness of each component. We evaluate the effectiveness of our method by incrementally adding its components. for the cross dataset. These results also show the synergistic effect of the fully random mask and geometry-guided sampling.Analysis on fully random mask mechanism. Fully random mask is designed to enhance the generalization capability. To validate its effectiveness, we train all models on Human3.6M with 4 views and set mask rates 𝑀 to 0, 0.2, 0.5, 0.8 and 1.0. As 𝑀 increases, more features from different views are discarded during training. 𝑀 = 0 indicates that features from all views are involved in the fusion process. During inference, we evaluate Masked Gifformer under vaying mask rates by using test samples with different numbers of views, as shown in Table 6 (3rd column). Masked Gifformer generalizes well to any number of views when 𝑀 ∈ (0, 1), achieving Table 6 : Generalization capability of different attention masks with varying mask rate 𝑀. Geometry-guided sampling is removed for a fair comparison. In the third column, all the models are trained on Human3.6M with 4 views and tested with 2,3 and 4 views. In the last column, all the models are trained on CMU3 and tested on Human3.6M. Best in bold. optimal performance with 𝑀 = 0.5. Furthermore, we compare our fully random mask with the random mask in [34] . On one hand, our method exhibits a lower generalization gap when adapting to different numbers of views. When 𝑀 = 0.5, although both methods achieve 22.7𝑚𝑚 when testing with 4 views, MPJPE of our method for 2 and 3 views is only 26.0𝑚𝑚 and 24.6𝑚𝑚, respectively, which is 0.4𝑚𝑚 and 0.3𝑚𝑚 lower than [34] . The greater advantage of the fully random mask is manifested in its cross-dataset generalization capability. We train all models on CMU3 and test them on Hu-man3.6M. Trained on CMU3 and tested on Human3.6M, our method improves from 36.9𝑚𝑚 to 29.1𝑚𝑚, surpassing [34] by 8.1𝑚𝑚 (Table 6 , last column). Therefore, with the proposed fully random mask mechanism, our model demonstrates strong robustness to varying camera numbers, spatial arrangements and cross-datasets. Analysis on geometry-guided sampling process. We further analyze the proposed geometry-guided sampling by exploring alternative methods such as uniform sampling and gaussian sampling, as shown in Table 7 . Compared to unconditional sampling, both gaussian-guided sampling and uniform-guided sampling reduce the MPJPE to some extent, but not as effectively as our proposed geometry-guided sampling (22.7𝑚𝑚 → 17.4𝑚𝑚).Analysis on JTB and MTB. JTB learns the spatial structure of joints while MTB captures multi-view information. In JTB, each joint is treated as a token, outputting 𝑋 ∈ R 𝐽 ×𝑁 ×𝐶 (𝐶 is the number of hidden dimensions). Then, MTB integrates view information into the learned representations. Unlike previous methods, we separate different joints along the view dimension, treating the views of each joint as a single token 𝑝 ∈ R 1×𝑁 ×𝐶 . We model different joints in parallel before concatenating them. This can reduce dimensions from 𝐽 × 𝐶 to 𝐶, enabling to handle more views. We conduct an ablation study in Table 8 . The difference between the last two columns is that our method alternately learns JTB and MTB. These results clearly demonstrate the effectiveness of JTB and MTB. Qualitative results are shown in Figure 5 . Consistent with quantitative results, Figure 5 (a) demonstrates superior qualitative performance of our model compared to [33] and [1] on Human3.6M.  
conclusion: 
 This paper presents Masked Gifformer, a novel diffusion model based on transformers for robust multi-view 3D HPE. During training, a transformer-based denoiser is supervised to fit 3D pose distribution by fusing joint and view information hierarchically. With the proposed fully random mask mechanism, Masked Gifformer is robust to varying camera numbers, spatial arrangements and datasets. During the inference, by modeling the conditional guidance distribution, multiple 3D pose hypotheses are sampled towards minimizing reprojection errors. Experimental results on two benchmarks show that our method effectively achieves a trade-off between accuracy and generalization. 
 
 Appendix:  


Decision: After considering all reviews, the rebuttal, and the subsequent discussion, the consensus is to accept the paper.

Reviews: review: paper_summary: paper_summary: The authors aim to improve the accuracy and generalization of 3D human pose estimation from multi-view images. The authors introduce a Geometry-guided Diffusion Model with a Masked Transformer (Masked Gifformer) that integrates joint and view information using a hierarchical multi-view transformer-based denoiser. To enhance generalization, the model incorporates a fully random mask mechanism, which avoids overfitting to specific camera numbers and positions. Additionally, geometric guidance is used in the sampling process to minimize reprojection errors, further improving accuracy. Extensive experiments verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: - In general, this paper is written well. The method is well explained with sufficient illustrative diagrams (such as Fig.1,2). 
- The ablation studies show the effectiveness of the proposed method.
limitations: limitations: - Incremental Contribution: This paper heavily relies on previous diffusion model-based 3D HPE methods that use 2D key points, such as [1]. The proposed geometric-guided sampling and random mask mechanisms are minor improvements and less impressive, falling short of the high standards required for acceptance in ACM-MM.

- The paper seems to lack a detailed demonstration of the Joint Transformer Block (JTB) and the Multi-view Transformer Block (MTB) in the "Denoiser" module. These components are not thoroughly discussed in the text or included in the ablation studies, leaving uncertainty about their contributions. This requires clarification.

- As shown in Table 6, the generalization capability gained by the proposed mask mechanism is not significantly better compared to the method in [34]. The improvement is marginal, which raises concerns about the effectiveness of the proposed approach.

[1] Diff3DHPE: A Diffusion Model for 3D Human Pose Estimation. CVPR'2023
paper_summary: paper_summary: This paper introduces Masked Gifformer, a Geometry-guided Diffusion Model with a Masked Transformer, for improved multi-view 3D HPE. The model uses a hierarchical multi-view transformer-based denoiser within a diffusion framework to integrate joint and view information for accurate 3D pose estimation. To enhance generalization, it employs a fully random mask mechanism without extra learnable modules. Geometric guidance is incorporated to optimize accuracy by minimizing reprojection errors.
strength_weakness: strength_weakness: 1. The paper demonstrates the strong performance of diffusion-based models for 3D HPE, marking the early study to evaluate and report the effectiveness of diffusion models in this domain, highlighting their potential for accurately capturing the complex 3D pose distributions.

2. The introduction of a fully random mask mechanism within the multi-view transformer block significantly enhances the model's robustness to variations in camera numbers, spatial arrangements, and different datasets. This mechanism achieves this without the need for any additional learnable modules or parameters, maintaining model simplicity and efficiency.

3. Integrating geometric information into the sampling process of diffusion models improves the accuracy of the 3D pose estimation. This is achieved by optimizing the diffusion process to minimize reprojection errors, effectively guiding the model to generate more precise and geometrically consistent poses.
limitations: limitations: 1. More analysis is needed to clearly demonstrate the advantages of probabilistic methods over deterministic ones. This could include a detailed comparison of performance metrics, robustness to noise, and ability to capture uncertainty in pose estimation.

2. Although the performance improvements in multi-view settings are not very strong, a deeper exploration into other metrics where deterministic methods may underperform, such as calibration accuracy and handling of occlusions, could provide a more comprehensive evaluation of the model's benefits.

3. The paper would benefit from further analysis of the Geometry-guided sampling process. Exploring alternative approaches, such as uniform sampling or implementing a learnable or tunable sampling prior, could offer insights into optimizing the sampling process and potentially enhance the model's accuracy and flexibility.
paper_summary: paper_summary: This paper introduces Masked Gifformer, a geometry-guided diffusion model designed for robust multiview 3D human pose estimation. The model comprises a hierarchical multiview transformer-based denoiser and incorporates a random mask mechanism. Furthermore, it integrates geometry guidance to enhance accuracy by modeling a conditional guidance distribution. Experimental results demonstrate its effectiveness across various datasets, outperforming other probabilistic methods. It achieves competitive results with state-of-the-art methods while maintaining good robustness.
strength_weakness: strength_weakness: 1. The idea presented in this paper is novel. It introduces the concept of building distributions from geometry to serve as guidance for diffusion models, thereby enhancing their accuracy.
2. The experimental results are compelling. They demonstrate a significant improvement in the accuracy of generated methods, particularly notable given the challenge of stability during generation. Moreover, the model exhibits robustness, a critical factor in the field of pose estimation.
3. The paper is well-written. Its motivation is clear, and the methods are presented in an easy-to-follow manner. Overall, it is highly readable.
limitations: limitations: 1. Expand Comparison on Robustness: In Table 3, only two multi-view methods are provided. Comparing with additional methods would significantly enhance the value of the paper, as robustness is a critical aspect of Human Pose Estimation (HPE).
2. Detailed Explanation of Experimental Results: Provide a thorough explanation of the experimental results. For example, clarify why the best results are achieved when the mask rate M is 0.5.
paper_summary: paper_summary: The paper proposes a Geometry-guided Diffusion Model with Masked Transformer (Masked Gifformer) for robust multi-view 3D HPE.
strength_weakness: strength_weakness: To address the long-standing problem of poor generalization, the paper introduces a fully random mask mechanism without any additional learnable modules or parameters; Authors incorporate geometric guidance into the diffusion model to enhance the accuracy of
the model.  The experiments on two popular 3D HPE benchmarks demonstrate the effectiveness of the proposed model.
limitations: limitations: (1) There are still some abbreviations and symbols are not clearly indicated and are not explained in detail, that make confusion, for example：a, line506-507：SO(3)，b. \lambda in equ(9), c. line 140 and equ (11): ∇_X, etc.
(2)Table 5 needs to be remade, as it is difficult to understand.
(3)There are some obvious formatting errors in the text. e,g., Highlight the vocabulary errors in bold, etc.
(4) Figure 3 should declare the meaning of X, Y axis and the meaning of different color in mask.
(5) Should provide the code, which helps to read the detail of the model.
(6) lack of novelty, the paper just combines the diffusion model with the transformer model, but the shortage of these model need more computing resource, AND the structure of JTB, MTB is just traditional transformer structure, Can these joint position information be effectively captured.