Title: Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images
Venue: NeurIPS 2024
Accepted: yes
Keywords: human reconstruction

point cloud diffusion

multi-hypothesis
Abstract: 3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. While implicit function methods capture detailed clothed shapes, they require aligned shape priors and or are weak at inpainting occluded regions given an image input. Parametric models i.e. SMPL, instead offer whole body shapes, however, are often misaligned with images. In this work, we propose a novel pipeline composed of a probabilistic SMPL model and point cloud diffusion for pixel-aligned detailed 3D human reconstruction under occlusion. Multiple hypotheses generated by the probabilistic SMPL method are conditioned via continuous 3D shape representations. Point cloud diffusion refines the distribution of 3D points fitted to both the multi-hypothesis shape condition and pixel-aligned image features, offering detailed clothed shapes and inpainting occluded parts of human bodies. In the experiments using the CAPE, MultiHuman and Hi4D datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code is publicly available at https://donghwankim0101.github.io/projects/mhcdiff.

Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images

Abstract

3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. Parametric models i.e. SMPL(X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clothed human shapes. Implicitfunction-based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such as clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a novel pipeline, MHCDIFF, M ultih ypotheses C onditioned Point Cloud Diff usion, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function-based methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The core of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code is publicly available at https://donghwankim0101.github.io/projects/mhcdiff .

1 Introduction

Realistic virtual humans play a significant role in various industries, such as metaverse, tele-presence, and game modeling. However, conventional methods require expensive artist efforts and complex scanning equipments, so they are not readily applicable. A more practical approach is to reconstruct high-fidelity 3D humans from 2D images taken in the wild. This is still an ongoing research task due to its challenges; people wear a wide variety of clothing styles and adopt diverse poses. Furthermore, human-object and human-human interaction, fundamental aspects of daily social life, make it more challenging due to severe occlusions.

Existing 3D human reconstruction methods cannot predict the pixel-aligned 3D shapes of humans robustly from occluded images. The parametric body models [27, 48, 67, 93, 75] have been widely used to reconstruct 3D human shapes. Several methods [11, 28, 34, 83, 26, 15, 10, 44, 82] predict the parameters of the statistical models and are robust to occlusion because they can be trained on large scale datasets [23, 56] and parametric models are well regularized with human body priors. However, the parametric models lack geometric details like clothing and hair, so these approaches cannot align the results to the subjects with loose clothing. More recently, 3D clothed human reconstruction methods [76, 77, 106, 92, 7, 91, 90, 95, 96], which are based on implicit functions and integrate the human body prior from the 3D body models, i.e SMPL [48, 67], present pixel-aligned detail shapes. Despite the impressive advances of the previous methods, they are not robust to occlusion because (1) small misalignment of estimated parametric models ruins the final shapes, (2) the implicit function takes features independently and cannot inpaint the invisible regions with missing image features, and (3) datasets [74, 87, 66] usually consists of segmented full-body images.

To address the aforementioned limitations, we propose MHCDIFF (Multi-hypotheses Conditioned Point Cloud Diffusion). (1) Several existing methods [6, 60, 73, 78, 79, 35, 8, 61, 81, 14] predict multiple SMPL meshes to model uncertainty due to occlusions. The sampled distribution is also important prior knowledge of human motions, but none of the existing work utilizes the distribution for pixel-aligned 3D human reconstruction. We leverage the multi-hypotheses to be robust on the misalignment of each sample. (2) We adopt denoising diffusion probabilistic models (DDPMs) [20] to take global consistent features and generate the invisible regions. Diffusion based methods generate 3D shapes by denoising point clouds [50, 108, 63, 22], latent [101, 62, 36], neural fields [69], 3D Gaussian [84] or meshes [46]. We adopt the unstructured point clouds to project pixel-aligned image features at each diffusion step. (3) Additionally, we synthesize partial body images by random masking [107], augmenting the limited datasets.

Specifically, our goal is pixel-aligned and detailed 3D human reconstruction in a robust manner to occlusion in images. Given a single occluded RGB image, we extract 2D features and generate multiple plausible SMPL hypotheses using an off-the-shelf method [4, 14]. The proposed method, MHCDIFF, Multi-hypotheses Conditioned Point Cloud Diffusion, performs the diffusion process to denoise a randomly-sampled point cloud into a target human shape. To reconstruct a pixel-aligned 3D shape and leverage the human body prior, the diffusion process is conditioned on the projected image feature (Sec. 3) and local features extracted from SMPL (Sec. 4.2). The key of MHCDIFF is a novel conditional diffusion process with multiple hypotheses (Sec. 4.3), which is not sensitive to misaligned SMPL estimation. Given global 2D features and the distribution of hypotheses, the denoising diffusion model can generate the occluded parts (Sec. 4.4)

We train MHCDIFF on randomly masked THuman2.0 dataset [85]. Our experiments on CAPE dataset [53, 68] with synthesized occlusion and MultiHuman dataset [105] with real-world interaction demonstrate that MHCDIFF reconstructs pixel-aligned 3D human shapes robustly to various occlusion ratios and achieves state-of-the-art performance. Our main contributions are as follows:

We introduce a novel multi-hypotheses conditioning mechanism that effectively captures the distribution of multiple plausible SMPL meshes. It is robust to the noise of each SMPL estimation due to the occlusion of given images. To the best of our knowledge, MHCDIFF is the first work that extends the multi-hypotheses SMPL estimation to pixel-aligned 3D human reconstruction.

We adopt point cloud diffusion model to capture the global consistent features and inpaint the invisible parts. Unlike the previous implicit function, the misaligned SMPL estimation can be corrected during the denoising process. The point cloud diffusion model also offers detailed human meshes.

MHCDIFF, trained on synthesized partial body images, outperforms previous methods on occluded and even full-body images.

2 Related Work

2.1 Diffusion models for point clouds

Over the past years, denoising diffusion probabilistic models (DDPMs) [20] have been applied to point clouds. For unconditional generation, Luo et al. [50], Zhou et al. [108] and LION [101] use PointNet [70], Point-Voxel-CNN [47] and latent space, respectively. PointInfinity [22] tackles the quadratic complexity of transformer [89], and generates high-resolution point clouds with a fixed-size latent vector. Otherwise, Point-E [63] is a text-conditioned generation model using CLIP [72] and PDR [51] is a point cloud completion method from partial point clouds. PC 2 [57], which is the baseline of MHCDIFF, reconstructs the point cloud conditioned on projected image features (please refer to Sec. 3 for more details).

2.2 Explicit-shape-based human reconstruction

Parametric models [27, 48, 67, 93, 75] have been primary representations for 3D human reconstruction. Due to the strength that they capture the statistics across a large corpus of human shapes, a lot of work [11, 28, 34, 83, 26, 15, 10, 44, 82] reconstructs 3D body meshes from an RGB image. To reduce the gaps between the image and parameter space of the statistical models and improve image alignment, they propose intermediate representations or additional supervisions, such as semantic segmentation [64, 94, 33, 100] and keypoints [9, 41]. To model the uncertainty due to occlusions or depth ambiguities, some work proposes multi-hypotheses [6], heatmaps [60], probability density functions [73, 78, 79, 35] or diffusion models [8, 61, 102, 81, 39]. ProPose [14] adopts the matrix Fisher distribution [13, 30] over SO (3) for the joint rotation conditioned on the von Mises-Fisher distribution [55] for the unit directions of bones, which is not only mathematically correct but also learning friendly (please refer to Sec. 3 for more details). However, these methods are limited to recovering minimally-clothed humans and lack the ability to capture geometric details such as clothing and hair.

Several works aim at modeling geometric details in explicit shapes such as meshes, voxels, depth maps and point clouds. Mesh-based methods [1, 2, 3, 37, 109, 5, 25] model 3D offsets on the vertices of SMPL [48], but they do not generalize on loose clothing such as skirts and dresses. Voxel-based methods [24, 88, 17, 86] reconstruct 3D human shapes in fine-grained voxel representations. However, free-form 3D reconstruction is challenging without prior, and they need high computation costs to output high-resolution 3D shapes. Point-cloud-based methods [52, 99, 54, 19, 85] model point clouds of clothing humans. Han et al. [19] estimate depth maps based on different body parts, and convert the depth maps into point clouds. Tang et al. [85], the most related work, reconstruct 3D humans with point cloud diffusion from an RGB image. First, they convert the estimated SMPL mesh and depth map from the RGB image to point clouds. Conditioned on this point cloud, the conditional diffusion model refines the point cloud. However, they only handle complete images without occlusion and are not robust to misaligned SMPL estimation.

2.3 Implicit-function-based human reconstruction

Implicit-function-based methods regress occupancy fields [58] or signed distance fields (SDF) [65] utilizing Multi-Layer Perceptron (MLP) decoders as implicit functions (IF). PIFu [76] and PIFuHD [77], which are pioneering works, extract pixel-aligned image features for clothed 3D human reconstruction. Later works [106, 92, 7, 91, 90, 40, 95, 103, 104, 96] leverage parametric models or body keypoints as prior information on the human body. They extract global features from voxelized SMPL meshes with a 3D encoder [106, 90] or local features such as signed distances and normals from SMPL meshes [92, 95, 103, 104] or both [7, 96]. The use of global features helps regularize global shapes and ensure consistency and local features help reconstruct local details. However, the global encoder is sensitive to global pose changes of SMPL and decreases the performance given misaligned SMPL estimation due to occlusion. The local features do not contain the global consistent features and cannot inpaint the occluded parts. Wang et al. [90] aim to reconstruct complete 3D

shapes from occluded images by primarily using the generative global encoder with a discriminator, but only assuming the accurate SMPL meshes.

3 Preliminary

PC 2 [57]. The projection-conditioned point cloud diffusion model is proposed for single-view 3D shape reconstruction. Denoising diffusion probabilistic model [20], which is the foundation of this framework, learns to recurrently transform noise X T ∼ N (0 , I ) into a sample from the target data distribution X 0 ∼ q ( X 0 ) over a series of steps. In order to learn this denoising process, a neural network is trained F θ ( X t -1 | X t ) ≈ q ( X t -1 | X t ) . To reconstruct geometrically consistent 3D point clouds from single RGB images I ∈ R H × W × 3 , 2D feature map E ( I ) ∈ R h × w × c is projected onto the partially denoised points at each step in the diffusion process. Therefore, F θ ( · ) : R (3+ c ) N → R 3 N is a function that predicts the noise ϵ ∈ R 3 N from the point cloud X t ∈ R 3 N and the projected features X proj t ∈ R cN , where c is the number of feature channels.

ProPose [14]. Recovering accurate body meshes and 3D joint rotations from single images remains a challenging problem, particularly in cases of severe occlusion, including self-occlusion and occlusion from other subjects or objects. ProPose [14] addresses this limitation by modeling the probability distributions for human mesh recovery. Since the pose parameters θ ∈ R 72 of SMPL [48] represent the 3D rotation of each joint and the root orientation, they adopt the matrix Fisher distribution [13, 30] over SO (3) . Due to the gaps between the RGB images and the rotation representations, the neural network cannot easily model the distribution. ProPose [14] also introduces 3D unit vectors for bone directions as the corresponding observation on the previous matrix Fisher distribution as the prior. Leveraging Bayesian inference, they model the posterior distribution of the joint rotations from the prior distribution and observation.

4 MHCDIFF: Multi-hypotheses Conditioned Point Cloud Diffusion

4.1 Overview

Our work aims at reconstructing pixel-aligned 3D human shape as a point cloud given a single occluded RGB image via conditional point cloud diffusion, as shown in Fig. 2. Formally, the diffusion model F θ ( · ) learns the conditional distribution q ( X 0 | I ) of 3D human shapes given the RGB images I ∈ R H × W × 3 . Following PC 2 , we extract the 2D feature map E ( I ) ∈ R h × w × c using ViT [12], to capture the details in the images. The image features are projected onto the partially denoised points: X proj t = Π( E ( I ) , X t ) , where Π is the projection function. This helps obtain pixel-aligned detailed body shapes. Additionally, the diffusion model is conditioned on the local features X SMPL t from SMPL mesh S to exploit statistical human body priors to complete 3D shapes from occluded body parts (Sec. 4.2). However, the SMPL estimation from single occluded RGB images has a high probability of large errors. To tackle this, we propose a novel multi-hypotheses conditioned diffusion model that considers the distribution of multiple plausible SMPL meshes { S i } i ∈{ 1 ,...,s } (Sec. 4.3). Given the partially denoised point cloud X t , the projected image features X proj t , and the local features from SMPL X SMPL t , MHCDIFF predicts the noise ϵ :

We also discuss how MHCDIFF takes the generative property and the global consistent features to reconstruct occluded parts (Sec. 4.4).

4.2 Local features from SMPL

Given the SMPL (or SMPL-X) mesh S and the partially denoised point cloud X t at t -th diffusion step, we extract the local features X SMPL t as:

Overall framework

E(1)

8

FeatureProjection

1.9

Pose &Shape Estimation

Extract LocalFeatures

(Sec. 3.)

(Sec. 4.2 ~ 4.3)

{Sijie(1...s)

F

XT

XT-1

OX

ConditionedPoint CloudDiffusion Model (Sec 4.4)

-Local features

Multi-hypotheses

where d ( · ) : R 3 → R and n ( · ) : R 3 → R 3 are the signed distance and normal obtained from the closest surface of SMPL mesh respectively. In order to map scalar values to a higher dimensional space, we adopt an encoding inspired by the positional encoding in NeRF [59]:

The local features X SMPL t ∈ R (2 L +3) N , which contain the signed distance and normal vector from SMPL, are used to predict the noise ϵ of the point cloud X t . The local property, which is independent of global pose, helps MHCDIFF to generalize well in diverse SMPL estimation due to occlusion and capture local details.

4.3 Multi-hypotheses condition

The local features are robust to noisy SMPL estimation, but cannot correct the SMPL estimation errors. Following previous multi-hypotheses human pose estimation [6, 43, 18, 21, 8], MHCDIFF takes multi-hypotheses SMPL meshes from estimated distributions and predicts the most plausible outputs. We modify Eq. 2 to handle multiple sampled SMPL meshes { S i } i ∈{ 1 ,...,s } using ProPose [14] as an off-the-shelf method:

where ¯ i = argmin i ∈{ 1 ,...,s } | d ( X t | S i ) | , which semantically means that each point follows the closest SMPL mesh S ¯ i to consider all plausible samples in denoising steps. However, each point gets conditions from only one sample and cannot leverage off-the-shelf probability distributions. In addition to the local features, we also adopt occupancy values:

where o ( · ) : R 3 → { 0 , 1 } is the occupancy function of the given SMPL mesh, which is a binary signal while the signed distance is continuous. With the mean occupancy and closest signed distance, MHCDIFF can assume all distributions with their respective probabilities. The proposed multihypotheses conditioning can take an arbitrary number of SMPL, SMPL-X, and their combined.

4.4 Conditioned point cloud diffusion model

Finally, F θ ( · ) : R (3+ c +4 L +3) N → R 3 N predicts the noise ϵ ∈ R 3 N given the concatenation of partially denoised point cloud X t ∈ R 3 N , projected image features X proj t ∈ R cN , and local features from SMPL X SMPL t ∈ R (4 L +3) N (Eq. 1). Notably, we do not need any learnable parameters to extract the local features from SMPL and aggregate the features of multiple SMPL meshes. We freeze the pre-trained 2D image encoder, so it is straightforward to train the diffusion model without additional training strategies.

The point cloud diffusion model of MHCDIFF takes the role of the decoder of previous implicitfunction-based methods. Given the encoded features from RGB images or SMPL meshes, the decoder predicts 3D shapes such as point clouds, occupancy fields, or signed distance fields. The implicitfunction-based methods need to sample the query points randomly, so the decoder has been primarily Multi-Layer Perceptron (MLP), which takes the input points independently. MHCDIFF consists of the point cloud diffusion model instead of MLP because (1) the point cloud model considers the global consistent features, (2) the diffusion model has the generative properties, and (3) the denoising process approximates correcting the misaligned SMPL estimation. Given the globally encoded image features X proj t and the local features from SMPL X SMPL t , MHCDIFF can inpaint or restore invisible body parts and is robust to noisy SMPL estimation due to occlusion.

Algorithm 1 Pseudocode of learning pipeline of MHCDIFF

Require: α 1: T : diffusion noise scheduling

1: repeat

2: Sample X 0 from q ( X 0 )

3: Load the corresponding image I and ground truth SMPL-X S

4: t ∼ Uniform ( { 1 , ..., T } )

5: ϵ ∼ N (0 , I )

6: X t = √ ¯ α t X 0 + √ 1 -¯ α t ϵ

7: X proj t = Π( E ( I ) , X t )

9: Take gradient descent step on

10: until converged

5 Experiments

Implementation. We use the Pytorch3D library [71] for image feature projection (Sec. 3) and the kaolin library [16] to extract local features from SMPL (Sec. 4.2). MHCDIFF is trained with batch size 8 in 100,000 steps. We use MSN [4] as the image feature encoder. We use AdamW [31] with β = (0 . 9 , 0 . 999) and a learning rate which is decayed linearly from 0 . 0002 to 0 . For diffusion noise schedule, we use linear scheduling from 1 · 10 -5 to 8 · 10 -3 with warmup. For inference, we denoise the point cloud for 1 , 000 steps. The training process takes approximately 1 day on a single 24GB NVIDIA RTX 4090 GPU with 28M learnable parameters.

Learning. We synthesize the THuman2.0 dataset [98], which contains 526 high-fidelity textured scans with corresponding SMPL-X fits. We use 500 subjects for training and the others for validation.

8: X SMPL t = [ γ ( o ( X t | S )) , γ ( d ( X t | S )) , n ( X t | S )]

▷ Extract local features from SMPL (Sec. 4.2

▷ Point cloud diffusion model (Sec. 4.4)

▷ Project image features (Sec. 3)

We render each human subject from 36 multiple viewpoints and randomly mask the images, resulting in partially occluded body images. We use the farthest point sampling operation to sample 16,384 points from each GT scan. During the training, local features X SMPL t are extracted from a single corresponding GT SMPL-X. The learning pipeline is presented in Algorithm 1.

Inference. First, we use the CAPE dataset [53, 68] with 150 textured scans. Similar to the training stage, we render each subject from 3 multiple viewpoints and randomly mask the images. During the inference, local features X SMPL t are extracted from multiple sampled SMPL or single estimated SMPL-X. We sample 10 SMPL meshes for our experiments. To further show the generalizability on the real-world interaction, we also evaluate MHCDIFF on the MultiHuman [105] and Hi4D [97] dataset. MultiHuman, which includes the diverse interaction with objects and people, provides 3D textured scans, so we render each subject from 3 multiple viewpoints. We evaluate the performance of MHCDIFF qualitatively on Hi4D, which includes close human-human interaction with high-fidelity meshes. The inference pipeline is presented in Algorithm 2.

Baseline models. We compare MHCDIFF with parametric models and pixel-aligned reconstruction methods. For parametric models, which are robust for occlusion, we select ProPose [14] as SMPL estimator and PIXIE [15] as SMPL-X estimator. For pixel-aligned reconstruction methods, which can capture geometric details, we select PaMIR [106] for global features, ICON [92] for local features, and HiLo [96] and SIFU [104] for both. For the fair comparison, we primarily condition with the mean of SMPL distribution estimated via ProPose, and PIXIE is also used for ICON, which supports SMPL-X. We use pre-trained weights and evaluate under our test setting.

Evaluation metrics. We employ Point-to-Surface distance and Chamfer Distance as evaluation metrics. MHCDIFF outputs a point cloud, so Chamfer Distance is the average L2 distance from the reconstructed point cloud to vertices of ground-truth scans and vice versa, and Point-to-Surface distance is the average point-to-surface from the reconstructed point cloud to ground-truth scans. The outputs of implicit-function-based methods can be converted meshes via the Marching Cubes algorithm [49]. For fair comparison, we sample the same number of points from the reconstructed meshes uniformly.

5.1 Comparison with state-of-the-art methods

MHCDIFF outperforms prior implicit-function-based methods and SMPL estimation methods on occluded and even full-body images. Fig. 3 presents the robustness of 3D human reconstruction to the occlusion ratio. PaMIR and HiLo cannot handle the occlusions because the global feature encoder is sensitive to misaligned SMPL estimation. SIFU does not use the 3D encoder, but the cross-attention from the normal map of SMPL takes global features and is sensitive to occlusion

Reconstructionfrom occluded CAPEdataset

14

MHCDiff

14

MHCDiff

ProPose

ProPose

12

PaMIR

12

PaMIR

(cm)

ICON

(cm)

ICON

Distance

10

HiLo

SIFU

Point-to-Surface

10

HiLo

SIFU

8

8

Chamfer

6

6

4

4

2

Z

0

10

20

OE

40

0

10

20

30

40

Occlusion ratio (%)

Occlusionratio (%)

Image

GT

Ours

ProPosePaMIRICON

SIFU

HiLo

and misaligned SMPL estimation. ICON shows comparable robustness due to its locality, but worse quality than ProPose estimation used to condition as the occlusion ratio increases. On the contrary, MHCDIFF is as robust as the statistical models, showing the most accurate results for all occlusion ratios. The results of 40% occlusion ratio are also displayed in numbers in Tab. 1. Tab. 2 presents the performance on real-world interaction scenarios with MultiHuman dataset. The dataset is divided into 5 categories by the level of occlusions: "occluded single" and "two closely-inter" show the most severe occlusion, and "single" and "three" show the least occlusion. We compare the performance in each category and similar to randomly masked settings, MHCDIFF achieves state-of-the-art on severe occluded images, and comparable performance on full-body images. The major improvements of MHCDIFF are (1) correcting the misaligned SMPL estimation as shown in Tab. 1, and (2) inpainting the invisible regions as shown in Fig. 3. The qualitative results on CAPE dataset are shown in Fig. 4, and MultiHuman and Hi4D datasets are shown in the appendices Sec. E.

5.2 Ablation study

We conduct an ablation on MHCDIFF to validate the effectiveness of each component. In Tab. 3-B, we condition the diffusion model with single SMPL-X (PIXIE) or SMPL (ProPose) estimation.

We improve the performance with multi-hypotheses condition (Sec. 4.3). In Tab. 4, we show the correlation between the number of SMPL sampled and the reconstruction quality. More SMPL hypotheses may include more accurate samples and improve the quality (15 samples), as well as extreme samples and decrease the quality (20 samples). From PC 2 [57], which only takes image condition, we also validate the local features from SMPL in Tab. 3-A. All of these features improve the performance, especially the signed distance. In Tab. 3-C, MHCDIFF is trained without random masking or by conditioning the distribution estimated by ProPose [14] instead of GT SMPL-X.

II

indu

Segmented images

文文文文复

PaMIR

ICON

HiLo

6 Conclusion

In this paper, We present MHCDIFF, which robustly reconstructs pixel-aligned and detailed 3D humans from single occluded images. Rather than implicit-function-based methods, we choose the point cloud diffusion model to generate invisible regions capturing the features globally. Our multi-hypotheses conditioning mechanism extracts local features from multiple SMPL estimations and integrates them without learnable parameters, so MHCDIFF is robust to a single erroneous SMPL due to occlusion. We augment the limited training data by random masking to synthesize occlusion by diverse interaction. The experiments demonstrate that our proposed method outperforms state-of-the-art methods from various levels of occlusion and interaction. In the future, the point cloud of human shapes can be applied to intermediate stages for implicit function [58] and human body deformation [45] or motion flow [38].

References

Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single RGB camera. In CVPR , 2019.

Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Detailed human avatars from monocular video. In 3DV , 2018.

Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from a single image. In ICCV , 2019.

Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV , 2022.

Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multigarment net: Learning to dress 3d people from images. In ICCV , 2019.

Benjamin Biggs, Sébastien Ehrhart, Hanbyul Joo, Benjamin Graham, Andrea Vedaldi, and David Novotny. 3D multibodies: Fitting sets of plausible 3D models to ambiguous image data. In NeurIPS , 2020.

Yukang Cao, Kai Han, and Kwan-Yee K. Wong. Sesdf: Self-evolved signed distance field for implicit 3d clothed human reconstruction. In CVPR , 2023.

Hanbyel Cho and Junmo Kim. Generative approach for probabilistic human mesh recovery using diffusion models. In ICCV Workshop , 2023.

Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In ECCV , 2020.

Hongsuk Choi, Gyeongsik Moon, JoonKyu Park, and Kyoung Mu Lee. Learning to estimate robust 3d human mesh from in-the-wild crowded scenes. In CVPR , 2022.

Vasileios Choutas, Lea Muller, Chun-Hao P. Huang, Siyu Tang, Dimitris Tzionas, and Michael J. Black. Accurate 3d body shape regression using metric and semantic attribute. In CVPR , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Thomas D Downs. Orientation statistics. In Biometrika , volume 59, pages 665-676, 1972.

Qi Fang, Kang Chen, Yinghui Fan, Qing Shuai, Jiefeng Li, and Weidong Zhang. Learning analytical posterior probability for human mesh recovery. In CVPR , 2023.

Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collaborative regression of expressive bodies using moderation. In 3DV , 2021.

Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: A pytorch library for accelerating 3d deep learning research. https://github.com/NVIDIAGameWorks/kaolin , 2022.

Andrew Gilbert, Marco Volino, John Collomosse, and Adrian Hilton. Volumetric performance capture from minimal camera viewpoints. In ECCV , 2018.

Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. Diffpose: Toward more reliable 3d pose estimation. In CVPR , 2023.

Sang-Hun Han, Min-Gyu Park, Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park, and Hae-Gon Jeon. High-fidelity 3d human digitization from single 2k resolution images. In CVPR , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In ICCV , 2023.

Zixuan Huang, Justin Johnson, Shoubhik Debnath, James M Rehg, and Chao-Yuan Wu. Pointinfinity: Resolution-invariant point diffusion models. CVPR , 2024.

Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI , 2014.

Aaron S Jackson, Chris Manafas, and Georgios Tzimiropoulos. 3d human body reconstruction from a single image via volumetric regression. In ECCV Workshop , 2018.

Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. Bcnet: Learning body and cloth shape from a single image. In ECCV , 2020.

Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar fine-tuning for 3d human pose fitting towards in-the-wild 3d human pose estimation. In 3DV , 2020.

Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces. In CVPR , 2018.

Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR , 2018.

Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM TOG , 2013.

CG Khatri and Kanti V Mardia. The von mises-fisher matrix distribution in orientation statistics. In Journal of the Royal Statistical Society: Series B (Methodological) , volume 39, pages 95-106, 1977.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2017.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In ICCV , 2021.

Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In ICCV , 2019.

Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling for human mesh recovery. In ICCV , 2021.

Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. SALAD: Part-level latent diffusion for 3d shape generation and manipulation. In ICCV , 2023.

Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-degree textures of people in clothing from a single image. In 3DV , 2019.

Jihyun Lee, Junbong Jang, Donghwan Kim, Minhyuk Sung, and Tae-Kyun Kim. Fourierhandflow: Neural 4d hand representation using fourier query flow. In NeurIPS , 2023.

Jihyun Lee, Shunsuke Saito, Giljoo Nam, Minhyuk Sung, and Tae-Kyun Kim. Interhandgen: Two-hand interaction generation via cascaded reverse diffusion. In CVPR , 2024.

Jihyun Lee, Minhyuk Sung, Honggyu Choi, and Tae-Kyun Kim. Im2hands: Learning attentive implicit representation of interacting two-hand shapes. In CVPR , 2023.

Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In CVPR , pages 3383-3393, 2021.

Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. PAMI , 2020.

Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Luc Van Gool. Mhformer: Multihypothesis transformer for 3d human pose estimation. In CVPR , 2022.

Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In ECCV , 2022.

Minghua Liu, Minhyuk Sung, Radomír Mˇ ech, and Hao Su. Deepmetahandles: Learning deformation meta-handles of 3d meshes with biharmonic coordinates. In CVPR , 2021.

Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In ICLR , 2023.

Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. In NeurIPS , 2019.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM TOG , 2015.

William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. SIGGRAPH Comput. Graph. , 1987.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR , 2021.

Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion-refinement paradigm for 3d point cloud completion. In ICLR , 2022.

Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, and Michael J. Black. SCALE: Modeling clothed humans with a surface codec of articulated local elements. In CVPR , 2021.

Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to Dress 3D People in Generative Clothing. In CVPR , 2020.

Qianli Ma, Jinlong Yang, Siyu Tang, and Michael J. Black. The power of points for modeling humans in clothing. In ICCV , 2021.

Kanti V Mardia, Peter E Jupp, and KV Mardia. Directional statistics. In Whiley Online Library , volume 2, 2000.

Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 3DV , 2017.

Luke Melas-Kyriazi, Christian Rupprecht, and Andrea Vedaldi. Pc2: Projection-conditioned point cloud diffusion for single-image 3d reconstruction. In CVPR , 2023.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR , 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In ECCV , 2020.

Lea Muller, Vickie Ye, Georgios Pavlakos, Michael J. Black, and Angjoo Kanazawa. Generative proxemics: A prior for 3D social interaction from images. CVPR , 2023.

Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 , 2022.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 , 2022.

Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler, and Bernt Schiele. Neural body fitting: Unifying deep learning and model-based human pose and shape estimation. In 3DV , 2018.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR , 2019.

Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael J. Black. AGORA: Avatars in geography optimized for regression analysis. In CVPR , 2021.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR , 2019.

Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black. Clothcap: Seamless 4d clothing capture and retargeting. ACM TOG , 2017.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR , 2023.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501 , 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 , 2021.

Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas. Humor: 3d human motion model for robust pose estimation. In ICCV , 2021.

RenderPeople. Renderpeople. renderpeople.com , 2018.

Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM TOG , 2017.

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV , 2019.

Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixelaligned implicit function for high-resolution 3d human digitization. In CVPR , 2020.

Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild. In ICCV , 2021.

Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Probabilistic 3D Human Shape and Pose Estimation from Multiple Unconstrained Images in the Wild. In ICCV , 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Anastasis Stathopoulos, Ligong Han, and Dimitris Metaxas. Score-guided diffusion for 3d human recovery. In CVPR , 2024.

Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV , 2021.

Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, and Michael J. Black. Putting people in their place: Monocular regression of 3D people in depth. In CVPR , 2022.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR , 2024.

Yingzhi Tang, Qijian Zhang, Junhui Hou, and Yebin Liu. Human as points: Explicit pointbased 3d human reconstruction from single-view rgb images. arXiv preprint arXiv:2311.02892 , 2023.

Matthew Trumble, Andrew Gilbert, Adrian Hilton, and John Collomosse. Deep autoencoder for combined human pose estimation and body model upscaling. In ECCV , 2018.

Twindom. Twindom. twindom.com , 2018.

Gül Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. BodyNet: Volumetric inference of 3D human body shapes. In ECCV , 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Junying Wang, Jae Shin Yoon, Tuanfeng Y. Wang, Krishna Kumar Singh, and Ulrich Neumann. Complete 3d human reconstruction from a single incomplete image. In CVPR , 2023.

Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In CVPR , 2023.

Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: Implicit Clothed humans Obtained from Normals. In CVPR , 2022.

Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, Bill Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR , 2020.

Yuanlu Xu, Song-Chun Zhu, and Tony Tung. Denserac: Joint 3d pose and shape estimation by dense render-andcompare. In ICCV , 2019.

Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang, Hao Xu, and Zhaoxin Fan. D-IF: Uncertainty-aware Human Digitization via Implicit Distribution Field. In ICCV , 2023.

Yifan Yang, Dong Liu, Shuhai Zhang, Zeshuai Deng, Zixiong Huang, and Mingkui Tan. Hilo: Detailed and robust 3d clothed human reconstruction with high-and low-frequency information of parametric models. In CVPR , 2024.

Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Zarate, Jie Song, and Otmar Hilliges. Hi4d: 4d instance segmentation of close human interaction. In CVPR , 2023.

Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In CVPR , 2021.

Ilya Zakharkin, Kirill Mazur, Artur Grigorev, and Victor Lempitsky. Point-based modeling of human clothing. In ICCV , 2021.

Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Zanfir, Bill Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Neural descent for visual 3d human pose and shape. In CVPR , 2021.

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In NeurIPS , 2022.

Siwei Zhang, Qianli Ma, Yan Zhang, Sadegh Aliakbarian, Darren Cosker, and Siyu Tang. Probabilistic human mesh recovery in 3d scenes from egocentric views. In ICCV , 2023.

Zechuan Zhang, Li Sun, Zongxin Yang, Ling Chen, and Yi Yang. Global-correlated 3ddecoupling transformer for clothed avatar reconstruction. In NeurIPS , 2023.

Zechuan Zhang, Zongxin Yang, and Yi Yang. Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction. In CVPR , 2024.

Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu. Deepmulticap: Performance capture of multiple characters using sparse multiview cameras. In ICCV , 2021.

Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. PAMI , 2021.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI , 2020.

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In ICCV , 2021.

Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hierarchical mesh deformation. In CVPR , 2019.

A Broader impact

Our method can be potentially used for AR/VR applications. The real-world interaction can be captured and modeled in virtual scenes, which can be extended to reinforcement learning. However, there are potential risks associated with falsifying human avatars, which could inadvertently compromise personal privacy. Consequently, there is a pressing need to establish regulations that clarify the fair use of such technology.

B Limitations

Our method, based on DDPM [20] sampling with 1 , 000 steps, has limitation on efficiency. The training time is reasonable because we do not need query point sampling, which yields CPU bottleneck to learn implicit-function. However, evaluation on CAPE dataset takes about 12 hours, while other implicit-function-based methods take about 30 minutes. We can apply DDIM [80] sampling with fewer steps to shorten the inference time.

C Pointcloud to mesh

Following previous work [85, 19], we try to convert our reconstructed point cloud to mesh with Screened Poisson surface reconstruction [29]. However, the process takes about 10 hours per sample with 16 , 384 points. The implicit function [58] converts the point clouds to occupancy fields by encoding features with a PointNet [70]. This two-stage pipeline can generate occluded regions and capture details. We will try this pipeline in our future work.

D Statistical significance

We evaluate MHCDIFF on CAPE dataset [53, 68] with 10 different random seeds. The random seeds effect on random noise in the diffusion process and SMPL sampling from the estimated distribution vis ProPose [14]. The Chamfer Distance and Point-to-Surface are 1 . 872( ± 0 . 008) and 1 . 810( ± 0 . 008) with 1-sigma error bars, respectively.

E Qualitative results

For the real-world interaction, we evaluate MHCDIFF on MultiHuman [105] and Hi4D [97] datasets. We render the textured scans with Pytorch3D library [71] for MultiHuman dataset, and segment each subject with pre-trained network [42] for Hi4D dataset. Our proposed method is robust not only to the occlusion but also to noise in full images or segmentation process.

Image

GT

Ours

ProPose PaMIRICON

SIFU

HiLo

Input image

Segmented images

Ours

ProPose

PaMIR

ICON

SIFU

HiLo

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Abstract and Introduction Sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Sec. B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper is based on experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Implementation paragraph in Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be public after cleansing.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to Sec. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Implementation paragraph in Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conform the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to Sec. A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the creators of the assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We submit the code as a zip file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing, and we only use public datasets for human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing, and we only use public datasets for human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
Authors propose MHCDIFF (Multi-hypotheses Conditioned Point Cloud Diffusion) to reconstruct 3D human point could from a single image under occlusion. The key idea behind MHCDIFF is the smart use of image projection features and features from multiple SMPL hypothesis, coupled with point cloud diffusion.
The method achieves favourable reconstruction performance on CAPE, MultiHuman, and Hi4D datasets.

STRENGTHS:
+ The proposed idea to leverage information from multiple SMPL hypothesis is simple and intuitive.
+ The ablation studies are detailed and show the significance of each component of MHCDIFF.

WEAKNESSES:
Some key details are unclear:
- L222: How are SMPL meshes sampled? 
- Eq. 5: Why use mean occupancy and not mean distance? Is there an advantage of using a combination of distance and occupancy?

Experiments:
- Authors mostly evaluate their method on in door datasets with controlled settings. What about more “in the wild” data like 3DPW, MS COCO.
- Is there any correlation between the SMPL hypothesis and the reconstruction accuracy? How accurate do the hypothesis have to be? Does number of SMPL sampled matter? Does the global translation of the SMPL matter?

Minor:
- L180: shouldn’t it be argmin if we want to compute distance to closest SMPL mesh?

QUESTIONS:
Overall I'm positive about the work. It is interesting to see that adding basic SMPL features (average occupancy and distance to closest SMPL point) to the PC2 formulation is a useful combination. The quantitative results also look good. I'm a bit unclear on some experimental and design details as stated above.

LIMITATIONS:
Authors discuss limitations and broader impact in thier manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe target of this paper is to reconstruct 3D clothed human shape. The conditional point clouds diffusion model is adopted as the main structure of the proposed reconstruction model. This work focuses on designing effective conditioning features, especially for overcoming the occlusions. The conditioning features include the similar projected image features as PC2[55], the local features [signed distance, normal vector, occupancy] extracted from multiple estimated SMPL or SMPL-X human 3D parametric models. The performance of the proposed model is evaluated on synthetic and real datasets, and compared with different SOTA methods. The experimental results show good de-occlusion ability in 3D clothed human shape reconstruction.\n\nSTRENGTHS:\n1.\tThe paper is well written and easy to understand.\n2.\tIt is the first work that extends the multi-hypotheses SMPL estimation to pixel-aligned 3D clothed human reconstruction.\n3.\tThe experimental results show good de-occlusion ability when reconstructing 3D clothed human 3D shape.\n\nWEAKNESSES:\n1.\tLow reconstruction quality of details, such as hands and feet.\n2.\tThe analysis and description of the experimental results are not clear and in-depth enough. For specific details, please refer to the following questions.\n\nQUESTIONS:\n1.\tMulti-hypotheses was proposed in [8] to model the uncertainty caused by occlusions or depth ambiguities in 3D shape reconstruction. What are the differences between your work and [8] on multi-hypotheses? \n2.\tIf one person wears different loose clothes with the same pose, can the proposed model reconstruct their 3D shapes effectively? It should be helpful if the related experimental examples are presented. \n3.\tTable 1 shows that the Chamfer Distances of HiLo [92] and SIFU [100] are 13.711cm and 13.397cm, while the result of the proposed method is 1.872cm, a significant improvement. My question is, both HiLo and SIFU also use SMPL model, which can provide good human 3D shape and pose prior, what is the reason for their poor results? \n4.\tTable 3 shows the results of ablation study. It indicates the Chamfer Distance decreases from 3.640cm (baseline PC2[55]) to 1.872cm (proposed MHCDIFF), totally 17.68mm improvement. It seems that the improvement should be contributed by the added conditioning features extracted from multiple SMPL. But from group A, the results show that each conditioning feature has small contributes, such as ‘w/o occupancy’, the value is 1.893cm vs. 1.872cm, only 0.21mm improvement. I am a bit confusing. A clearer explanation would be better. \n5.\tFrom Table 3, group B, it seems that using SMPL model (conditioned on single ProPose estimation) is better than using SMPL-X model (conditioned on PIXIE estimation). If it is true, please analyze the reasons. \n6.\tFrom the Qualitative results (Figure 3, Figure 5), it can be seen that the reconstruction qualities of some parts are not good, such as hands, feet, and so on. Please analyze the reasons.\n7.\tThe multi-hypotheses conditioning can take an arbitrary number of SMPL, SMPL-X, and their combined. What is the optimal recommendation?\n\nLIMITATIONS:\nAs the paper states that proposed method has low efficiency caused by diffusion model, which has limited applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces method for reconstructing detailed 3D human shapes from single occluded RGB images. The key contributions are:\n- A point cloud diffusion model conditioned on projected 2D image features and local features from multiple SMPL mesh hypotheses.\n- A multi-hypotheses conditioning mechanism that extracts and aggregates local features from multiple plausible SMPL meshes to handle uncertainty due to occlusions.\n- Training on synthetically occluded images to improve robustness to real-world occlusions.\n\nThe method is evaluated on the CAPE dataset with synthetic occlusions and MultiHuman dataset with real interactions, demonstrating state-of-the-art performance in reconstructing detailed 3D human shapes from occluded views.\n\nSTRENGTHS:\n- **Novel approach**: The combination of point cloud diffusion with multi-hypotheses SMPL conditioning (from ProPose) is an innovative solution to the problem of 3D human reconstruction from occluded images.\n- **Robust to occlusions**: Results demonstrate significant improvement over prior work on occluded inputs.\n- **Detailed reconstructions**: The approach can recover fine geometric details like clothing directly on point clouds, going beyond parametric body models.\n- **Thorough evaluation**: Experiments on both synthetic (CAPE) and real (MultiHuman) datasets with varying levels of occlusion provide a comprehensive assessment.\n\nWEAKNESSES:\n- **Limited Dataset Diversity**: While results on CAPE and MultiHuman are strong, evaluation on additional datasets would further demonstrate generalization. In particular, testing on datasets with more diverse clothing styles and body shapes would be valuable. Suggested datasets include:\n   - THuman2.0: Contains 2500 high-quality human scans with various poses and clothing styles.\n   - AGORA: A synthetic dataset with high realism, featuring multiple people per image.\n   - DeepFashion-MultiModal: Includes high-resolution images with rich annotations for clothing shapes and textures.\n\n- **Limited practical applicability**: The method's output is a point cloud, which may not be directly usable in many real-world applications that require mesh representations. While the authors mention the possibility of converting the point cloud to a mesh using Poisson surface reconstruction, this process is computationally expensive (taking about 10 hours per sample) and is not integrated into the main pipeline. This limitation significantly reduces the method's immediate practical utility in applications requiring real-time or near-real-time 3D human reconstruction.\n\n- **Texture and color reconstruction and real world usage**: Unlike methods like PHORHUM, this approach does not address texture or color reconstruction, which limits its applicability in photorealistic avatar creation.\n\nQUESTIONS:\n- How does MHCDIFF compare to recent methods like PIFuHD and PHORHUM in terms of reconstruction quality and handling of occlusions?\n- Have the authors explored more efficient sampling techniques like DDIM to reduce inference time? What speedups might be achievable?\n- How well does the method generalize to different clothing styles or body shapes not seen during training?\n- Could the approach be extended to include texture and color reconstruction, similar to PHORHUM?\n- How does the method perform on real-world, in-the-wild images with multiple humans and complex interactions? Have you considered evaluating on datasets like AGORA or DeepFashion-MultiModal?\n\nLIMITATIONS:\nYes. The authors do acknowledge the computational cost as a key limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper investigates the problem of improving the robustness of occluded 3D human reconstruction from a single image. The idea is to achieve better stability under occlusion via two steps: 1. refining pixel-aligned local image feature extraction part with the help of multi-hypothesis human pose and shape estimation; 2. inpainting the invisible body part via a point cloud diffusion process. Experiments on CAPE and MultiHuman datasets show the superior of the proposed method.

STRENGTHS:
1. The ablation study in Tab. 3 reveals the effectiveness of each designs. 
2. Using point cloud diffusion process to inpainting the invisible part seems reasonable.

WEAKNESSES:
1. Composing the multi-hypothesis human pose and shap with point cloud diffusion seems incremental. The paper lacks solid insight to inspire the readers. 
2. The quantitative results on CAPE and MultiHuman don't show obvious improvement, compared with previous methods. The CAPE dataset is processed to "randomly mask the images about 40% in average". While on MultiHuman without any masking, the performance is quite similar to the baselone model.  
3. More qualitative results would be helpful to determine the superior of the proposed method.

QUESTIONS:
What is the key insight behind the two designs presented in this paper?
The experiment results on CAPE and MultiHuman may need further explanation.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13