Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images
Authors: Donghwan Kim
Tae-Kyun Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: human reconstruction
point cloud diffusion
multi-hypothesis
Primary area: diffusion_based_models

Abstract: 3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. While implicit function methods capture detailed clothed shapes, they require aligned shape priors and or are weak at inpainting occluded regions given an image input. Parametric models i.e. SMPL, instead offer whole body shapes, however, are often misaligned with images. In this work, we propose a novel pipeline composed of a probabilistic SMPL model and point cloud diffusion for pixel-aligned detailed 3D human reconstruction under occlusion. Multiple hypotheses generated by the probabilistic SMPL method are conditioned via continuous 3D shape representations. Point cloud diffusion refines the distribution of 3D points fitted to both the multi-hypothesis shape condition and pixel-aligned image features, offering detailed clothed shapes and inpainting occluded parts of human bodies. In the experiments using the CAPE, MultiHuman and Hi4D datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code is publicly available at https://donghwankim0101.github.io/projects/mhcdiff.

Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images
Donghwan Kim Tae-Kyun Kim
KAIST
(kdoh2522, kintaekyyun@kkisttacc..
Abstract
3D human shape reconstruction under severe occlusion due to human-object human- interaction is a challenging problem. Parametric models i.c. SMPL(- X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clotie function- based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a hypohhesss Conditioned Point Cloud Diffusion,
human shapes. Implicit-
novel pipeline, MHCDIFF, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function funttto--nuntt methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code publicly available at https::/ bttps:://dnghwankkin0l1. wwt.:000...iittht github........... wtabb..o/rrroo/ wttpprooectss/mcdi....
Input image
Segmented images
3D reconstruction as point cloud
Figure 1: Image 3D shape. From the segmented images, containing occlusion due interaction, MHCDIFF reconstructs 3D human shapes as point clouds.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
75 have been widely predict the Realistic virtual humans play a significant role in various industries, such as metaverse, seeepresence, and game modeling. However, conventional methods require expensive artist efforts and complex scanning equipments, so they are not readily applicable. A more practical approach is to reconstruct high-fidelity 3D humans from 2D images taken in the wild. This is still an ongoing research task due to its challenges; people wear a wide variety of clothing styles and adopt diverse poses. Furthermore, human-object and human-human interaction, fundamental aspects of daily social life, make it more challenging due to severe occlusions.
Existing 3D human reconstruction methods cannot predict the pixel-aligned 3D shapes of humans robustly from occluded images. The parametric body models (27
48
67
93
used to reconstruct 3D human shapes. Several methods 11
28
34
26
15
10
82
parameters of the statistical models and are robust to occlusion because they can be trained on large scale datasets 23 56]] and parametric models are well regularized with human body priors. However, the parametric models lack geometric details like clothing and hair, so these approaches cannot align the results to the subjects with loose clothing. More recently, 3D clothed human reconstruction 91 90 95| 96), which are based on implicit functions and integrate the human body prior from the 3D body models, i.e SMPL [48 67], present pixel-aligned detail shapes. Despite the impressive advances of the previous methods, they are not robust to occlusion because (1) small misalignment of estimated parametric models ruins the final shapes, (2) the implicit function takes features independently and cannot inpaint the invisible regions with missing image features, usually consists of segmented full-body images.
methods (76 77 106 92 7
and (3) datasets [74 87 66
To address the aforementioned limitations, we propose MHCDIFF Multi-hypothhen Point Cloud Diffusion).
Conditioned
(1) Several existing methods 6 60 73 78 79 35, 8 61 81 14 predict multiple SMPL meshes to model uncertainty due to occlusions. The sampled distribution is also important prior knowledge of human motions, but none of the existing work utilizes the distribution for pixel-aligned 3D human reconstruction. We leverage the multi-hypotheses to be robust on the misalignment of cach sample. (2) We adopt denoising diffusion probabilistic models (DDPMs) [20] to take global consistent features and generate the invisible regions. Diffusion based methods generate 3D shapes by denoising point clouds [50] 63 22], latent [00 62] 36), neural fields (69] 3D Gaussian [84] or meshes [46). We adopt the unstructured point clouds to project pixel-aligned image features at each diffusion step. (3) Additionally, synthesize partial body images by random masking [07]. augmenting the limited datasets.
Specifically, our goal is pixel-aligned and detailed 3D human reconstruction in robus! manner to occlusion images. Given a single occluded RGB image, we extract 2D features and generate multiple plausible SMPI. hypotheses using an off-the-shelf method 4 [4]. The proposed method, MHCDIFF, Multi-hypothesse Conditioned Point Cloud Diffusion, performs the diffusion process to denoise a randomly sunpee point cloud into a target human shape. To reconstruct pixel-aligned 3D shape and leverage the human body prior, the diffusion process is conditioned on the projected image feature (Sec. 3) and local features extracted from SMPL (Sec. 4.2). The key of MHCDIFF is a novel conditional diffusion process with multiple hypotheses (See. 4.3) which is not sensitive
to misaligned SMPL estimation. Given global 2D features and the distribution of hypotheses, the denoising diffusion model can generate the occluded parts (Sec. 4.4)
We train MHCDIFF on randomly masked THuman2.0 dataset 85
Our experiments on CAPE
dataset [53 68 with synthesized occlusion and MultiHuman dataset 105) with real-world interaction demonstrate that MHCDIFF reconstructs pixel-aligned 3D human shapes robustly to various occlusion ratios and achieves state-of-the-art performance. Our main contributions are as follows:
We introduce a novel multi-hypotheses conditioning mechanism that effectively captures the distribution of multiple plausible SMPL meshes. It is robust to the noise of each SMPL estimation due to the occlusion of given images. To the best of our knowledge, MHCDIFF is the first work that extends the multi-hypotheses SMPL estimation to pixel-aligned 3D human reconstruction.
We adopt point cloud diffusion model to capture the global consistent features and inpaint the invisible parts. Unlike the previous implicit function, the misaligned SMPL estimation can be corrected during the denoising process. The point cloud diffusion model also offers detailed human meshes.
2

---

MHCDIFF, trained on synthesized partial body images, outperforms previous methods on occluded and even full-body images.
2 Related Work
2.1 Diffusion models for point clouds
Over the past years, denoising diffusion probabilistic models (DDPMs) 200 have been applied to
point clouds. For unconditional generation, Luo et al. [50), Zhou et al. [08 and LION [10] use PointNet [70), Voxel-Vxel- [47 and latent space, respectively. PointInfinity [22] tackles the
quadratic complexity of transformer (9), and generates high-resolution point clouds with fixed-size
latent vector. Otherwise, Point-E [33 a text-conditioned generation model using CLIP [22 and
PDR [51] is point cloud completion method from partial point clouds. PC [57], which is the
baseline of MHCDIFF, reconstructs the point cloud conditioned on projected image features (please refer to Sec.
g for more details).
2.2 Explicit-shape-based human reconstruction
have been primary representations for 3D) human reconstruction. Due to the strength that they capture the statisties across large corpus of human shapes, lot of
Parametric models 27
48
67
93
75
reconstructs 3D body meshes from an RGB image. To reduce the gaps between the image and parameter space of the statistical models and improve image alignment, they propose intermediate representations or additional supervisions, such as semantic segmentation 664 94
work [1] 28 34
83
26
15
10
44 82)
To model the uncertainty due to occlusions or depth ambiguities, some work proposes muuli-hypothesee (6. heatmaps [60], probability density functions (73 78|
100 and keypoints
81 [9). ProPose [4] adopts the matrix Fisher distribution 13 30 over SO(3) for the joint rotation conditioned on the von Mises-Fisher distribution (55) for the unit directions of bones, which is not only mathematically correct but also learning friendly (please refer to Sec. 3 for more details). However, these methods are limited to recovering minimally-clothed humans and lack the ability to capture geometric details such as clothing and hair.
79
35 or diffusion models & 61] [02
Several works aim at modeling geometric details in explicit shapes such as meshes, voxels, depth maps and point clouds. Mesh-based methods [ 3
5 25 model 3D offsets on the vertices of SMPL [48, but they do not generalize on loose clothing such as skirts and dresses. Voxel-based 88 17 86 reconstruct 3D human shapes in fine- fin-grained voxel representations. However, free-form 3D reconstruction is challenging without prior, and they need high computation costs to output high-resolution 3D shapes. Point-cloud-basee methods [52 99 54
37
109
methods [24
model point clouds of clothing humans. Han al. [19 estimate depth maps based on different body parts, and convert the depth maps into point clouds. Tang et al. [85]. the most related work, reconstruct 3D humans with point cloud diffusion from an RGB image. First, they convert the estimated SMPL mesh and depth map from the RGB image to point clouds. Conditioned on this point cloud, the conditional diffusion model refines the point cloud. However, they only handle complete images without occlusion and are not robust to misaligned SMPL estimation.
19 85
2.3 Implicit-function-ball human reconstruction
Implicit-function--assd methods regress occupancy fields 588 or signed distance fields (SDF)
665 utilizing Multi-Layer Perceptron (MLP) decoders implicit functions (IF). PIFu 776 and
PIFuHD [77]. which are pioneering works, extract pixel- exxellalgged image features for clothed 3D human reconstruction. Later works [106]
40 95 [03] 104 [96] leverage parametric models or body keypoints as prior information on the human body. They extract global features from voxelized
92 7
91
90
SMPL meshes with 3D encoder [06 90] or local features such as signed distances and normals
04 or both 16). The use of global features helps regularize global shapes and ensure consistency and local features help reconstruct local details. However, the global encoder is sensitive to global pose changes of SMPL and decreases the performance given misaligned SMPL. estimation due occlusion. The local features do not contain the global consistent features and cannot inpaint the occluded parts. Wang et al. [90] aim to reconstruct complete 3D
from SMPL meshes [92 95 103
3

---

shapes from occluded images by primarily using the generative global encoder with discriminator, but only assuming the accurate SMPL meshes.
3
Preliminary
The projection-condiiione point cloud diffusion model is proposed for single-view 3D shape reconstruction. Denoising diffusion probabilistic model [20], which the foundation of this framework, learns to recurrently transform noise XT ~ NO, I) into a sample from the target data distribution Xg ~ (Xp) over a series of steps. In order to learn this denoising process, neural network is trained Fo (o( = x To reconstruct geometrically consistent 3D point clouds from single RGB images / E IR #xxxxx3 x3 2D feature map E()) E Axx is projected onto the partially denoised points at each step in the diffusion process. Therefore, Fo(.): : R(3+))N - R is a function that predicts the noise e E R&N from the point cloud X, E R and the projected features X ReN where is the number of feature channels.
PC 57
ProPose [4]. Recovering accurate body meshes and 3D joint rotations from single images remains a challenging problem, particularly in cases of severe occlusion, including self-occlusion and occlu- sion from other subjects objects. ProPose [14] addresses this limitation by modeling the probability distributions for human mesh recovery. Since the pose parameters 0 E T2?2 of SMPI. [48] represent the 3D rotation of each joint and the root orientation, they adopt the matrix Fisher distribution [13 30 over SO(3). Due to the gaps between the RGB images and the rotation representations, the neural network cannot easily model the distribution. ProPose [4] also introduces 3D unit vectors for bone directions as the corresponding observation on the previous matrix Fisher distribution as the prior. Leveraging Bayesian inference, they model the posterior distribution of the joint rotations from the prior distribution and observation.
4 MHCDIFF: Multi-hypotheses Conditioned Point Cloud Diffusion
4.1
Overview
Our work aims at reconstructing pixel-aligned 3D human shape point cloud given a single occluded RGB image via conditional point cloud diffusion, as shown Fig. 2 Formally, the diffusion model Fo(.) learns the conditional distribution (Xool 7) of 3D human shapes given the RGB images / E R # W x3 Following PC², we extract the 2D feature map E() E Rhxuuxe using ViT [12. capture the details in the images. The image features are projected onto the partially denoised points: X II(E(I),, where II is the projection function. This helps obtain pixel-aligned detailed body shapes. Additionally, the diffusion model is conditioned on the local features X X,SMPL PL from SMPL mesh 5 to exploit statistical human body priors to complete 3D shapes from occluded body parts (Sec. 4.2). However, the SMPL estimation from single occluded RGB images has high probability of large errors. To tackle this, we propose a novel mylti-hypotheses conditioned diffusion model that considers the distribution of multiple plausible SMPL meshes {Si
(Sec. 4.3p. Given the partially denoised point cloud X,, the projected image features X,Pro,, and the local features from SMPL SSMPL MHCDIFF predicts the noise
Fo Xproj XSMPL) = E.
We also discuss how MHCDIFF takes the generative property and the global consistent features to reconstruct occluded parts (Sec. 4.4).
4.2 Local features from SMPL
Given the SMPL (or SMPL-X) mesh S and the partially denoised point cloud X; at t-th diffusion step, we extract the local features X SSMPL as:
x SMMPL (xd(X, (x)
(2)

---

Overall framework
Local features
3
E(1)
Feature Projection (.)
Pose & Shape Estimation: (Sec. 3.)
Extract Local Features (Sec. 4.2 4.3)
Multi-hy pothesess
X
X(T
Conditioned Point Cloud Diffission Model (See 4.4)
Figure 2: (Lefft) Overview of MHCDIFF. Given an occluded image T, MHCDIFF reconstructs 3D human shape point cloud. First, we extract the 2D feature map {T) and hypothesize pose and shape parameters of multiple plausible SMPL meshes {S: (L...e) Our method consists of the conditioned point cloud diffusion model (Sec. 4.4). We project the 2D image features capture details of the image (Sec. 3 and extract local features from multiple hypothesized SMPL meshes to leverage human body priors (Sec. (4.)) (Upper Right) The details of local features (Sec. 4.2) The signed distance field is visualized positive and negative regions. The arrows indicate normal vectors n. (Lower Right) The details multi- multi--hypothess (Sec. 4.3 We can consider the whole distribution during denoising process with the argmin i, and the denoising
be approximated by red arrows. However, is sensitive to extreme samples of the distribution, so we condition the mean of occupancy values, which is visualized by transparency, and the denoising can be approximated by blue arrows.
where d(.) 3 R and n(-)
E + 3 are the signed distance and normal obtained from the closest surface of SMPL mesh respectively. In order to map scalar values to a higher dimensional space, adopt encoding inspired by the positional encoding in NeRF [59]:
(d) (sin(2" rd),
cos(22 md), sin(s" md), cos(s=-11mm)) Td)).
The local features X XSMPL e IR (2L+3)N which contain the signed distance and normal vector from SMPL, are used to predict the noise of the point cloud X,. The local property, which is independent of global pose, helps MHCDIFF to generalize well in diverse SMPL estimation due to occlusion and capture local details.
4.3 Multi-hypotheses condition
The local features are robust to noisy SMPL estimation, but cannot correct the SMPL estimation errors. Following previous multi-hypotheses human pose estimation 66 43 18 21 &. MHCDIFF takes multi-hypotheses SMPL meshes from estimated distributions and predicts the most plausible outputs. We modify Eq. to handle multiple sampled SMPL meshes [S:..e using ProPose 14 as an off-the--heelf method:
X SMML = (d((,)))) n (X)
where = (r...
which semantically means that each point follows the closest SMPL mesh S; to consider all plausible samples in denoising steps. However, each point gets conditions from only one sample and cannot leverage off-the-shelf probability distributions. In addition to the local features, also adopt occupancy values:
5

---

X SMPL
(5)
where o(.) I23
{0. 1 is the occupancy function of the given SMPL mesh, which binary signal while the signed distance is continuous. With the mean occupancy and closest signed distance, MHCDIFF can assume all distributions with their respective probabilities. The proposed multi- hypotheses conditioning can take an arbitrary number of SMPL, SMPL-X, and their combined.
4.4 Conditioned point cloud diffusion model
Finally, Fo(+) R(3+++1L+3)N R&N predicts the noise R&N given the concatenation of partially denoised point cloud X2 E 3R projected image features X ReN and local features from SMPL X..MPI. () (Eq. D. Notably, we do not need any learnable parameters to extract the local features from SMPL and aggregate the features of multiple SMPL meshes, We freeze the pre-trained 2D image encoder, so it straightforward train the diffusion model without additional training strategies.
The point cloud diffusion model of MHCDIFF takes the role of the decoder of previous implicit- function- methods. Given the encoded features from RGB images or SMPL meshes, the decoder predicts 3D shapes such point clouds, occupancy fields, signed distance fields. The implicit- function-based methods need to sample the query points randomly, so the decoder has been primarily Multi-Layer Perceptron (MLP), which takes the input points independently. MHCDIFF consists of the point cloud diffusion model instead of MLP because (1) the point cloud model considers the global consistent features, (2) the diffusion model has the generative properties, and (3) the denoising process approximates correcting the misaligned SMPL estimation. Given the globally encoded image features X and the local features from SMPL XSMPL MHCDIFF can inpaint restore invisible body parts and robust to noisy SMPL estimation due to occlusion.
Algorithm 1 Pseudocode of learning pipeline of MHCDIFF
Require: 01:7: diffusion noise scheduling
1: repeat
2: Sample Xy from (Xxo)
3: Load the corresponding image and ground truth SMPL-X S
4:
Uniform([L. T})
5:
6 ~ (0,
6:
X, N0 + VI ōne
7:
x.roj = (E), Xt)
> Project image features (Sec.
8:
X XSMPL (((((()))) (d((x())) n(X,|S)
Extract local features from SMPL (Sec. 4.2
9: Take gradient descent step on
V E - FX(, proj x XSMPL)
Point cloud diffusion model (Sec. 4.4
10: until converged
Experiments
Implementation. We use the Pytorch3D library 71 for image feature projection (Sec. and the kaolin library [16] to extract local features from SMPL. (Sec. 4.2p. MHCDIFF is trained with batch size 8 in 100,000 steps. We use MSN [4] the image feature encoder. We use AdamW [31] with = (0.9, 0.999) and a learning rate which is decayed linearly from 0.0002 to O. For diffusion noise schedule, we use linear scheduling from 1 10 to 8 10-3 with warmup. For inference, we denoise the point cloud for 1, 000 steps. The training process takes approximately day on a single 24GB NVIDIA RTX 4090 GPU with 28M learnable parameters.
Learning. We synthesize the THuman2.0 dataset (98), which contains 526 high- fidelity textured scans with corresponding SMPL-X fits. We use 500 subjects for training and the others for validation.

---

Algorithm 2 Pseudocode of inference pipeline of MHCDIFF
Require: Input image
1: Sample Xy from from.((O) (0.
2: Estimate single
multi SMPL(-X) meshes {Si (Ei))E((.....)))
3: for all from T to do
4:
I) if (0, 5: X/ = (T),
else
b Project image features (Sec.
6: for all from 1 to do
7: Compute (X),)S.), (S))) and (x(()))
Can be accelerated by kaolin
8:
end for
9:
+ wrgmin (1).....) [d( (())
10:
X-MPL = Ei= 1 ((( n (S)
> Multi-hypotheses conditioning (Sec.
4
ét Fo (X,, Xproj ; XSMPL)
12:
X,-1 t - von
(X, 1-aa
+ 0
DDPM 20 sampling
vi-ai
13:
end for
14: return Xo
We render each human subject from 36 multiple viewpoints and randomly mask the images, resulting in partially occluded body images. We use the farthest point sampling operation to sample 16,384 points from each GT scan. During the training, local features X; X/MPL extracted from a single corresponding GT SMPL-X. The learning pipeline presented in Algorithm [
Inference.
First, we use the CAPE dataset 53
68
with 150 textured scans. Similar to the training
stage, render each subject from 3 multiple viewpoints and randomly mask the images. During the inference, local features X SSML are extracted from multiple sampled SMPL or single estimated SMPL-X. We sample 10 SMPL meshes for experiments. To further show the generalizability on the real-world interaction, we also evaluate MHCDIFF on the MultiHuman [005] and Hi4D [97 dataset. MultiHuman, which includes the diverse interaction with objects and people, provides 3D textured seans, so we render each subject from multiple viewpoints. We evaluate the performance of MHCDIFF qualitatively Hi4D, which includes close human-humas interaction with high-fidelity meshes. The inference pipeline is presented in Algorithm 2
Baseline models. We compare MHCDIFF with parametric models and pixel-aligned reconstruction methods. For parametric models, which are robust for occlusion, select ProPose [14] as SMPL estimator and PIXIE [5) as SMPL-X estimator. For pixel-aligned reconstruction methods, which can capture geometric details, select PaMIR [106] for global features, ICON [92] for local features, and Hil lo [96] and SIFU [T04 for both. For the fair comparison, we primarily condition with the mean SMPL distribution estimated via ProPose, and PIXIE is also used for ICON, which supports SMPL-X. We use pre-trained weights and evaluate under our test setting.
Evaluation metrics. We employ Point-to-Surface distance and Chamfer Distance as evaluation metrics. MHCDIFF outputs point cloud, so Chamfer Distance the average L2 distance from the reconstructed point cloud to vertices of ground-truth scans and vice versa, and Point-to-Surfat distance is the average point-to-surface from the reconstructed point cloud ground-truth scans. methods can be converted meshes via the Marching Cubes
The outputs of implicit-function-aal algorithm 44). For fair comparison, we sample the same number of points from the reconstructed meshes uniformly.
5.1 Comparison with state-of-the-art methods
MHCDIFF outperforms prior implicit- implicit-funnction-aael methods and SMPL estimation methods on occluded and even full-body images. Fig. presents robustness of 3D human reconstruction to the occlusion ratio. PaMIR and HiLo cannot handle the occlusions because the global feature encoder is sensitive misaligned SMPL. estimation. SIFU does not use the 3D encoder, but the cross-attention from the normal map of SMPL takes global features and is sensitive occlusion
7

---

Methods
Chamfer Distance (em)
Point-to Seatttto-Sufffcee (cm)
A
PaMIR T06
12.912
12.619
ICON 92
2,896
2.789
ICON (PIXIE estimation)
3.329
3.212
SIFU [04]
14.397
14.087
HiLo 196]
13.711
13.405
B
PIXIE (SMPL-X) 11
2.705
2.662
ProPose (SMPL) 14
2.370
2.307
Ours
MHCDIFF
1.872
1.810
Table 1:
Quantitative evaluation on CAPE dataset. We report the average Chamfer Distance
(cm) and Point-to-Surfiet distance (cm) on CAPE dataset. We randomly mask the images about 40% in average. We compare the performance with respect to (A) implicit-function seniiit-unntioo-baseeee based methods; and (B) SMPL estimation methods used to condition MHCDIFF and (A). Best in bold, second-best underlined.
Methods
single
occluded single
two natural-inter
two closcly-inter
three
PaMIR T06
0.690
2.349
5.154
3.752
4.714
ICON
0.555
0.549
0.563
0.786
0.669
SIFU 104
0.644
3.335
4.796
3.503
3,264
HiLo 996
0.606
2.808
4.139
3,346
4.398
B
PIXIE (SMPL-XX
15
0.868
0.813
0.755
0.951
0.809
ProPose (SMPL) [4
0.675
0.567
0.574
0.766
0,688
Ours
MHCDIFF
0.591
0.491
0.536
0.703
0.673
Table 2:
Quantitative evaluation MultiHuman dataset. We report the average Chamfer
Distance (cm) for each category, compare the performance similar to Tab.
Chamler Distance (cm)
Point-to-Surface (cm)
full
MHCDIFF
1.872
1.810
w/o occupancy w/o signed distance
1.893
1.831
2,016
1.949
w/o normal w/o encoding
1.888
1.827
1.928
1,863
PC2 [57]
3.640
3.533
conditioned on PIXIE estimation
2.314
2,237
conditioned on single ProPose estimation
1.939
1.869
trained with ProPose estimation
2.708
2.624
w/o random masking
1.940
1.868
Table Ablation study on CAPE dataset. We validate the effectiveness of (A) each component;
(B) conditioning strategies; and (C) training strategies.
Evaluation time
The number of SMPL sampled
Chamfer Distance (cm)
Point-to-Surface (cm)
CAPE dataset (hours)
1.939
1.869
5
1.882
1.817
8
10
1.872
1.810
12
15
1.833
1.773
16
20
1.836
1.777
20
Table 4: The correlation between the number of SMPL sampled and the reconstruction quality.
We report the average Chamfer Distance (cm), Point-to-Sune Surface distance (cm) and evaluation time of the various number of SMPL. sampled.

---

Reconstruction from ercluded CAPE dataset
WHEOSP
M-C
Chunter 3stano
Cccasion
Occasion
test. This figure shows the performance of different models from the images of various occlusion ratios. From the whole-body images, which is 0% occlusion, we randomly mask the images from 10% to 40%. MHCDIFF is robust to the occlusion ratio, showing the best performance.
Figure 3: cumulative
ProPose PAMIR
SIFU
Figure 4: Qualitative results CAPE dataset. We evaluate method with SMPL estimation method and pmplicit-ffucttion cannot generate the occluded regions. They cannot also handle the misaligned SMPL mesh the arms, creating incomplete bodies. ProPose predicts the full-body shape, but cannot capture the details like the blazer of the lower image. However, MHCDIFF is robust the occlusion and misalignment, and can capture pixel-aligned details.
methods. Given the upper image, PaMIR, ICON, and HiLo
and misaligned SMPL estimation. ICON shows comparable robustness due to its locality, but worse quality than ProPose estimation used condition as the occlusion ratio increases. On the contrary, MHCDIFF is as robust as the statistical models, showing the most accurate results for all occlusion presents the
ratios. The results of 40% occlusion ratio are also displayed in numbers in Tab. [ Tab. 2 performance real-world interaction scenarios with MultiHuman dataset. The dataset is divided into 5 categories by the level of occlusions: "occluded single" and "two closely-inter" show the most severe occlusion, and "single" and "three" show the least occlusion. We compare the performance in each category and similar to randomly masked settings, MHCDIFF achieves state-of -ote----rttttttt- on severe occluded images, and comparable performance on full-body images. The major improvements of MHCDIFF are (1) correcting the misaligned SMPI. estimation as shown in Tab. I and (2) inpainting the invisible regions as shown in Fig. The qualitative results CAPE dataset shown in Fig. 4 and MultiHuman and Hi4D datasets are shown in the appendices Sec.
5.2 Ablation study
conduct an ablation on MHCDIFF to validate the effectiveness of each component. In Tab. B, condition the diffusion model with single SMPL-X (PIXIE) or SMPL (ProPose) estimation.
3

---

We improve the performance with multi-hypotheses condition (Sec. 4.3 In Tab. we show the correlation between the number of SMPL sampled and the reconstruction quality. More SMPL hypotheses may include more accurate samples and improve the quality (15 samples), as well as extreme samples and decrease the quality (20 samples). From PC [57,, which only takes image condition, we also validate the local features from SMPL. in Tab. 3LAA. All of these features improve the performance, especially the signed distance. Tab. BJC. MHCDIFF is trained without random masking by conditioning the distribution estimated by ProPose [4] instead of GT SMPL-X.
ng ay
ZIDIA
Figure 5: Qualitative results on in-the-wild images. Two images on the left show occlusions due interactions, and the rightmost image shows loose clothes. From internet photos, we use 32] to segment images.
Conclusion
In this paper, We present MHCDIFF, which robustly reconstructs pixel-aligned and detailed 3D humans from single occluded images. Rather than implicit-function-bal methods, we choose the point cloud diffusion model generate invisible regions capturing the features globally. Our multi-hyyotheess conditioning mechanism extracts local features from multiple SMPI. estimations and integrates them without learnable parameters, so MHCDIFF is robust SMPL due to occlusion. We augment the limited training data by random masking to synthesize occlusion by diverse interaction. The experiments demonstrate that our proposed method outperforms of-the-art methods from various levels of occlusion and interaction. In the future, the point
single erroncous
cloud of human shapes can be applied intermediate stages for implicit function [58] and human body deformation [5] or motion flow (38].
10

---

7 Acknowledgements
This work was supported by NST grant (CRC 21011, MSIT), ITTP grant (RS-2023-0022299% RS-2024-00459749, MSIT) and KOCCA grant (RS-2024-00442338,, MCST).
References
[1] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single RGB camera. In CVPR, 2019.
[2] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Detailed human avatars from monocular video. In 3DV, 2018.
[3] Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from single image. In ICCV, 2019.
[4] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for lahel-efficient learning. In ECCV, 2022.
[5] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll, Multi- garment net: Learning to dress 3d people from images. In ICCV, 2019.
[6] Benjamin Biggs, Sébastien Ehrhart, Hanbyul Joo, Benjamin Graham, Andrea Vedaldi, and David Novotny. 3D multibodies: Fitting sets of plausible 3D models ambiguous image data. In NeurIPS, 2020.
[7] Yukang Cao, Kai Han, and Kwan-Yee K. Wong. Sesdf: Self-evolved signed distance field for implicit 3d clothed human reconstruction. In CVPR, 2023.
[8] Hanbyel Cho and Junmo Kim. Generative approach for probabilistic human mesh recovery using diffusion models. In ICCV Workshop, 2023.
[9] Hongsuk Choi, Gycongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In ECCV, 2020.
[10] Hongsuk Choi, Gyeongsik Moon, JoonKyu Park, and Kyoung Mu Lee. Learning estimate robust 3d human mesh from in-the-wild crowded scenes. In CVPR, 2022.
[1]] Vasileios Choutas, Lea Muller, Chun-Hao P. Huang, Siyu Tang, Dimitris Tzionas, and Michael J. Black. Accurate 3d body shape regression using metric and semantic attribute. In CVPR. 2022.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[13] Thomas Downs. Orientation statisties. In Biometrika, volume 59, pages 665-676, 1972.
[14] Qi Fang, Kang Chen, Yinghui Fan, Qing Shuai, Jiefeng Li, and Weidong Zhang. Learning analytical posterior probability for human mesh recovery. In CVPR, 2023.
[15] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collab- orative regression of expressive bodies using moderation. In 3DV, 2021.
[16] Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. Kaolin: A pytorch library for accelerating 3d deep learning research. /tctts://github... www.///.../....... http:///DDL.aaeeeeorss/aallllll 2022.
[17] Andrew Gilbert, Marco Volino, John Collomosse, and Adrian Hilton. Volumetric performance capture from minimal camera viewpoints. In ECCV, 2018.
[18] Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. Diffpose: Toward more reliable 3d pose estimation. In CVPR, 2023.

---

[19] Sang-Hun Han, Min- Park, Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park. and Hae-Gon Jeon. High-fidelity 3d human digitization from single resolution images. In CVPR, 2023.
[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[21] Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In JCCV, 2023.
[22] Zixuan Huang. Justin Johnson, Shoubhik Debnath, James M Rehg, and Chao-Yuan Wu. Pointinfinity: Resolution-invarian point diffusion models. CVPR, 2024.
[23] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI, 2014.
[24] Aaron S Jackson, Chris Manafas, and Georgios Tzimiropoulos. 3d human body reconstruction from a single image via volumetric regression. In ECCV Workshop, 2018.
[25] Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. Benet: Learning body and cloth shape from single image. ECCV, 2020.
[26] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar fine-tuning for 3d human pose fitting towards in in-the-wild 3d human pose estimation. In 3DV, 2020.
[27] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces. In CVPR, 2018.
[28] Angjoo Kanazawa, Michael Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
[29] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM TOG,
2013.
[30] CG Khatri and Kanti V Mardia. The von mises-fisher matrix distribution in orientation statistics. In Journal of the Royal Statistical Society: Series B (Methodological), volume 39, pages 95-106, 1977.
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2017.
[32] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yes Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint arXiv:2334. 022444, 2023.
[33] Muhammed Kocabas, Chun-Hao Huang, Otmar Hilliges, and Michael Black. PARE: Part attention regressor for 3D human body estimation. In ICCV, 2021.
[34] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting the loop. ICCV, 2019.
[35] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling for human mesh recovery. In ICCV, 2021.
[36] Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. SALAD: Part-level latent diffusion for 3d shape generation and manipulation. In ICCV, 2023.
[37] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-degree textures of people in clothing from a single image. In 3DV, 2019.
[38] Jihyun Lee, Junbong Jang, Donghwan Kim, Minhyuk Sung, and Tae-Kyun Kim. Fourierhand- flow: Neural 4d hand representation using fourier query flow. In NeurlPS, 2023.
[39] Jihyun Lee, Shunsuke Saito, Giljoo Nam, Minhyuk Sung, and Tae-Kyun Kim. Interhandgen: Two-hand interaction generation via cascaded reverse diffusion. CVPR, 2024.
[40] Jihyun Lee, Minhyuk Sung, Honggyu Choi, and Tae-Kyun Kim. Im2hands: Learning attentive implicit representation of interacting two-hand shapes. In CVPR, 2023.
[41] Jiefeng Li,, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In CVPR, pages 3383-3393 2021.
12

---

[42] Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. PAMI, 2020.
[43] Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Luc Van Gool.
Mhformer: Multi-
hypothesis transformer for 3d human pose estimation. In CVPR, 2022.
[44] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In ECCV, 2022.
[45] Minghua Liu, Minhyuk Sung. Radomír Měch, and Hao Su. Deepmetahandles: Learning deformation meta- meeta-handles of 3d meshes with biharmonic coordinates. CVPR, 2021.
[46] Zhen Liu, Yao Feng, Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In ICLR, 2023.
[47] Zhijian Liu, Haotian Tang. Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. In NeurlPS, 2019.
[48] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi- person linear model. ACM TOG, 2015.
[49] William F. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. SIGGRAPH Comput. Graph., 1987.
[50] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
[51] Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion-reffnemmen paradigm for 3d point cloud completion. In ICLR, 2022.
[52] Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, and Michael J. Black. SCALE: Modeling clothed humans with surface codec of articulated local elements. In CVPR, 2021.
[53] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to Dress 3D People in Generative Clothing. In CVPR, 2020.
[54] Qianli Ma, Jinlong Yang, Siyu Tang, and Michael J. Black. The power of points for modeling humans in clothing, In ICCV, 2021.
[55] Kanti V Mardia, Peter E. Jupp, and KV Mardia. Directional statistics. In Whiley Online Library, volume 2, 2000.
[56] Dushyant Mchta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Somychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 3DV, 2017.
[57] Luke delas-Kyriaze Christian Rupprecht, and Andrea Vedaldi. Pc2: Projection-conditione point cloud diffusion for single-image 3d reconstruction. In CVPR, 2023.
[58] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In CVPR, 2019,
[59] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[60] Gyeongsik Moon and Kyoung Mu Lee. 121-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In ECCV, 2020.
[61] Lea Muller, Vickie Ye, Georgios Pavlakos, Michael J. Black, and Angjoo Kanazawa. Genera- tive proxemics: A prior for 3D social interaction from images. CVPR, 2023.
[62] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-Idm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint 2022.
[63] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint 11X15:2212.08751, 2022.
[64] Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler, and Bernt Schiele. Neural body fitting: Unifying deep learning and model-hased human pose and shape estimation.
In 3DV, 2018
13

---

[65] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR. 2019.
[66] Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael J. Black, AGORA: Avatars in geography optimized for regression analysis. In CVPR, 2021.
[67] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from single image. In CVPR, 2019.
[68] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black. Clothcap: Seamless 4d clothing capture and retargeting. ACM TOG, 2017.
[69] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. In ICLR, 2023.
[70] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017.
[71] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Accelerating 3d deep learning with pytorch3d. arXiv preprint (0Xix:000700800) 2020.
[72] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh. Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0080 200000000000 2021.
[73] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas J. Guíbas. Humor: 3d human motion model for robust pose estimation. In ICCV, 2021.
[74]
RenderPeople. Renderpeople. |renderpeople. com 2018.
[75] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM TOG, 2017.
[76] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
[77] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-
aligned implicit function for high-resolution 3d human digitization. In CVPR, 2020.
[78] Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Hicrarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild. In ICCV, 2021.
[79] Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Probabilistic 3D Human Shape and Pose Estimation from Multiple Unconstrained Images in the Wild. In ICCV, 2021.
[80] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[81] Anastasis Stathopoulos, Ligong Han, and Dimitris Metaxas. Score-guided diffusion for 3d human recovery. In CVPR, 2024.
[82] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV, 2021.
[83] Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, and Michael J. Black. Putting people in their place: Monocular regression of 3D people in depth. In CVPR, 2022.
[84] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
[85] Yingzhi Tang, Qijian Zhang, Junhui Hou, and Yebin Liu. Human points: Explicit point- based 3d human reconstruction from single-view rgb images. arXiv preprint arXiv:2311.0222 1122211102922 2023.
14

---

[86] Matthew Trumble, Andrew Gilbert, Adrian Hilton, and John Collomosse. Deep autoencoder for combined human pose estimation and body model upscaling. In ECCV, 2018.
[87]
Twindom. Twindom. twindom. com 2018.
[88] Gül Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. BodyNet: Volumetric inference 3D human body shapes. In ECCV, 2018.
[89] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[90] Junying Wang, Jae Shin Yoon, Tuanfeng Y. Wang, Krishna Kumar Singh, and Ulrich Neumann. Complete 3d human reconstruction from a single incomplete image. In CVPR, 2023.
[91] Yuliang Xiu, Jinlong Yang. Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In CVPR, 2023.
[92] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael Black. ICON: Implicit Clothed humans Obtained from Normals. In CVPR, 2022.
[93] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, Bill Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape and articulated pose models. In CVPR, 2020.
[94] Yuanlu Xu, Song-Chun Chun Zhu, and Tony Tung. Denserac: Joint 3d pose and shape estimation by dense wender-andcompane In ICCV, 2019.
[95] Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang. Hao Xu, and Zhaoxin Fan. D-IF: Uncertainty-aware Human Digitization via Implicit Distribution Field. In ICCV, 2023.
[96] Yifan Yang, Dong Liu, Shuhai Zhang. Zeshuai Deng, Zixiong Huang, and Mingkui Tan. Hilo: Detailed and robust 3d clothed human reconstruction with high-and low-frequency information of parametric models. In CVPR, 2024.
[97] Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Zarate, Jie Song. and Otmar Hilliges. Hi4d: 4d instance segmentation of close human interaction. In CVPR, 2023.
[98] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu, Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In CVPR, 2021.
[99] Ilya Zakharkin, Kirill Mazur, Artur Grigorev, and Victor Lempitsky. Point-based modeling of human clothing. In ICCV, 2021.
[100] Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Zanfir, Bill Freeman, Rahul Sukthankar, and Cristian Sminchiseseu. Neural descent for visual 3d human pose and shape. In CVPR, 2021.
[101] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In NeurlPS, 2022.
[102] Siwei Zhang, Qianli Ma, Yan Zhang, Sadegh Aliakbarian, Darren Cosker, and Siyu Tang. Probabilistic human mesh recovery in 3d scenes from egocentric views. In ICVV, 2023.
[103] Zechuan Zhang, Li Sun, Zongxin Yang, Ling Chen, and Yi Yang. Global-corrllaed 3d- decoupling transformer for clothed avatar reconstruction. In NeurlPS, 2023.
[104] Zechuan Zhang, Zongxin Yang, and Yi Yang. Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction. In CVPR, 2024.
[105] Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin
Liu. Deepmulticap: Performance capture of multiple characters using sparse multiview cameras. In /CCV, 2021.
[106] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric conell-onditioned implicit representation for basge human reconstruction. PAMI, 2021.
[107] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020.
[108] Lingi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point- voxel diffusion. In ICCV, 2021.
[109] Hao Zhu, Xinxin Zuo Sen Wang, Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hierarchical mesh deformation. In CVPR, 2019.
15

---

A Broader impact
Our method can be potentially used for AR/VR applications. The real-world interaction can be cap- tured and modeled in virtual scenes, which can be extended to reinforcement learning. However, there are potential risks associated with falsifying human avatars, which could inadvertently compromise personal privacy. Consequently, there is pressing need to establish regulations that clarify the fair use of such technology.
B Limitations
Our method, based DDPM 20 sampling with 1, 100 steps, has limitation efficiency. The training time is reasonable because we not need query point sampling, which yields CPU bottleneck to learn implicit-function However, evaluation on CAPE dataset takes about 12 hours, while other implicit-functi seniii-ffunctoo-baeee methods take about 30 minutes. We can apply DDIM 80 sampling with fewer steps to shorten the inference time.
C Pointcloud to mesh
Following previous work [85 19 we try to convert our reconstructed point cloud to mesh with Screened Poisson surface reconstruction [29]. However, the process takes about 10 hours per sample with 16, 384 points. The implicit function 58) converts the point clouds to occupancy fields by encoding features with PointNet 770. This two-stage pipeline can generate occluded regions and capture details. We will try this pipeline in our future work.
D Statistical significance
with 10 different random seeds. The random seeds effect random noise in the diffusion process and SMPL sampling from the estimated distribution vis ProPose [4]. The Chamfer Distance and Shintto-SSurfcee are 1.872(=0.008) and 1.810(+0.008)
evaluate MHCDIFF on CAPE dataset 53 63
with I-sigma error bars, respectively.
E Qualitative results
For the real-world interaction, we evaluate MHCDIFF on MultiHuman [055 and Hi4D 997 datasets. We render the textured scans with Pytorch3D library 7 for MultiHuman dataset, and segment each subject with pre-trained network (42) for Hi4D dataset. Our proposed method is robust not only to the occlusion but also noise in full images or segmentation process.
16

---

Image
GT
Ours
ProPoss
PsMIR
SIFU
HiLo
Figure 6: Qualitative results MultiHuman dataset.
Segnested
Oun
PAMIR
ICON
SIUU
HLs
Figure 7: Qualitative results on Hi4D dataset.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Abstract and Introduction Sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper is based on experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Implemennation paragraph in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
The code will be public after cleansing.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
coiiooooollyyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwpc//ggue////eeeeeoooc CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Sec.
D
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Implementation paragraph in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: wwwpourrps.cpppuuuuuee http://p............tttcccseeeea
Answer: [Yes]
Justification: The paper conform the NeurlPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Sec. A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the creators of the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [Yes]
Justification:
We submit the code as a zip file.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing, and only public datasets for human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing, and we only use public datasets for human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "QEgDgPKh4h", "reviewer": "Reviewer_w3VR", "summary": "This paper investigates the problem of improving the robustness of occluded 3D human reconstruction from a single image. The idea is to achieve better stability under occlusion via two steps: 1. refining pixel-aligned local image feature extraction part with the help of multi-hypothesis human pose and shape estimation; 2. inpainting the invisible body part via a point cloud diffusion process. Experiments on CAPE and MultiHuman datasets show the superior of the proposed method.", "questions": "What is the key insight behind the two designs presented in this paper?\nThe experiment results on CAPE and MultiHuman may need further explanation.", "limitations": "No.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The ablation study in Tab. 3 reveals the effectiveness of each designs. \n2. Using point cloud diffusion process to inpainting the invisible part seems reasonable.", "weaknesses": "1. Composing the multi-hypothesis human pose and shap with point cloud diffusion seems incremental. The paper lacks solid insight to inspire the readers. \n2. The quantitative results on CAPE and MultiHuman don't show obvious improvement, compared with previous methods. The CAPE dataset is processed to \"randomly mask the images about 40% in average\". While on MultiHuman without any masking, the performance is quite similar to the baselone model.  \n3. More qualitative results would be helpful to determine the superior of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1.** Composing the multi-hypothesis human pose and shape with point cloud diffusion seems incremental. The paper lacks solid insight to inspire the readers.\n> \n\n> **Question 1-1.** What is the key insight behind the two designs presented in this paper?\n> \n\n**Reply:** We design MHCDIFF with the two steps with the insight in Section 1 Introduction (lines 41-51). (1) We adopt multi-hypothesis human pose and shape to model uncertainty (lines 43), and be **robust on the misalignment due to occlusion** (lines 45-46). (2) We choose the point cloud diffusion model to **generate the invisible regions** by taking global consistent features (lines 46-47). Among 3d representations, point clouds are effective to project pixel-aligned image features at each diffusion step (lines 49-50).\n\n> **Weakness 2.** The quantitative results on CAPE and MultiHuman don't show obvious improvement, compared with previous methods. The CAPE dataset is processed to \"randomly mask the images about 40% in average\". While on MultiHuman without any masking, the performance is quite similar to the baseline model.\n> \n\n> **Question 1-2.** The experiment results on CAPE and MultiHuman may need further explanation.\n> \n\n**Reply:** MultiHuman dataset is divided into 5 categories by the level of occlusions and the number of people. In Tab. 2 in the paper, “occluded single” and “two closely-inter” show the most severe occlusion, and “single” and “three” show the least occlusion. For “occluded single” and “two closely-inter”, MHCDIFF achieves state-of-the-art performance, and the results have a similar tendency to CAPE dataset with 10~20% occlusion ratios in Fig. 4 in the paper. The major improvement of MHCDIFF is **correcting the misaligned SMPL estimation** as shown in Tab. 1, group B and Tab3, group C in the paper, and **inpainting the invisible regions** as shown in Fig. 4 in the paper and Fig. R2 (Right) in PDF. We will clarify this point in the revision.\n\n> **Weakness 3.** More qualitative results would be helpful to determine the superior of the proposed method.\n> \n\n**Reply:** In Fig. R1 in PDF, We show qualitative results on in-the-wild images with occlusion and loose clothes. We will contain more results in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wefpxexn3U", "reviewer": "Reviewer_1vtm", "summary": "This paper introduces method for reconstructing detailed 3D human shapes from single occluded RGB images. The key contributions are:\n- A point cloud diffusion model conditioned on projected 2D image features and local features from multiple SMPL mesh hypotheses.\n- A multi-hypotheses conditioning mechanism that extracts and aggregates local features from multiple plausible SMPL meshes to handle uncertainty due to occlusions.\n- Training on synthetically occluded images to improve robustness to real-world occlusions.\n\nThe method is evaluated on the CAPE dataset with synthetic occlusions and MultiHuman dataset with real interactions, demonstrating state-of-the-art performance in reconstructing detailed 3D human shapes from occluded views.", "questions": "- How does MHCDIFF compare to recent methods like PIFuHD and PHORHUM in terms of reconstruction quality and handling of occlusions?\n- Have the authors explored more efficient sampling techniques like DDIM to reduce inference time? What speedups might be achievable?\n- How well does the method generalize to different clothing styles or body shapes not seen during training?\n- Could the approach be extended to include texture and color reconstruction, similar to PHORHUM?\n- How does the method perform on real-world, in-the-wild images with multiple humans and complex interactions? Have you considered evaluating on datasets like AGORA or DeepFashion-MultiModal?", "limitations": "Yes. The authors do acknowledge the computational cost as a key limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Novel approach**: The combination of point cloud diffusion with multi-hypotheses SMPL conditioning (from ProPose) is an innovative solution to the problem of 3D human reconstruction from occluded images.\n- **Robust to occlusions**: Results demonstrate significant improvement over prior work on occluded inputs.\n- **Detailed reconstructions**: The approach can recover fine geometric details like clothing directly on point clouds, going beyond parametric body models.\n- **Thorough evaluation**: Experiments on both synthetic (CAPE) and real (MultiHuman) datasets with varying levels of occlusion provide a comprehensive assessment.", "weaknesses": "- **Limited Dataset Diversity**: While results on CAPE and MultiHuman are strong, evaluation on additional datasets would further demonstrate generalization. In particular, testing on datasets with more diverse clothing styles and body shapes would be valuable. Suggested datasets include:\n   - THuman2.0: Contains 2500 high-quality human scans with various poses and clothing styles.\n   - AGORA: A synthetic dataset with high realism, featuring multiple people per image.\n   - DeepFashion-MultiModal: Includes high-resolution images with rich annotations for clothing shapes and textures.\n\n- **Limited practical applicability**: The method's output is a point cloud, which may not be directly usable in many real-world applications that require mesh representations. While the authors mention the possibility of converting the point cloud to a mesh using Poisson surface reconstruction, this process is computationally expensive (taking about 10 hours per sample) and is not integrated into the main pipeline. This limitation significantly reduces the method's immediate practical utility in applications requiring real-time or near-real-time 3D human reconstruction.\n\n- **Texture and color reconstruction and real world usage**: Unlike methods like PHORHUM, this approach does not address texture or color reconstruction, which limits its applicability in photorealistic avatar creation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1.** In particular, testing on datasets with more diverse clothing styles and body shapes would be valuable.\n> \n\n> **Question 3.** How well does the method generalize to different clothing styles or body shapes not seen during training?\n> \n\n> **Question 5.** How does the method perform on real-world, in-the-wild images with multiple humans and complex interactions?\n> \n\n**Reply:** We use THuman2.0 as training dataset. Unfortunately, AGORA is not free and public dataset and DeepFashion-MultiModal does not have ground truth 3D shapes. HiLo [92] and SIFU [100] use CAPE and THuman2.0 datasets, and ICON [88] use AGORA and CAPE datasets for evaluation. In Fig. R1 in PDF, We show qualitative results on in-the-wild images with occlusion and loose clothes. We will contain more results in the revision.\n\n> **Weakness 2.** The method's output is a point cloud, which may not be directly usable in many real-world applications that require mesh representations.\n> \n\n**Reply:** Even though point clouds may not be directly usable in real-world applications, several methods [24, 48, 49, 55, 61, 97, 104] adopt point clouds as 3d representations. Following Poisson surface reconstruction, more recent methods study surface reconstruction from point clouds [R1]. Instead of Poisson surface reconstruction, other methods or surface extraction using the marching cube [47] proposed in PointInfinity [24] may be the solution, but these are not our main scope. We will consider this aspect in our future work.\n\n> **Weakness 3.** Unlike methods like PHORHUM, this approach does not address texture or color reconstruction, which limits its applicability in photorealistic avatar creation.\n> \n\n> **Question 4.** Could the approach be extended to include texture and color reconstruction, similar to PHORHUM?\n> \n\n**Reply:** We use PC2 [55] as the baseline of MHCDIFF, and they predict the color of each point using separate models with equivalent architecture. Therefore, we can extend to include color reconstruction, but we only focus on shape reconstruction in this paper.\n\n> **Question 1.** How does MHCDIFF compare to recent methods like PIFuHD and PHORHUM in terms of reconstruction quality and handling of occlusions?\n> \n\n**Reply:** Similar to pixel-aligned reconstruction methods, PaMIR [102], ICON [88], HiLo [92], SIFU [100] (lines 231-232), PIFuHD and PHORHUM use implicit functions, which cannot inpaint the invisible regions (lines 34-40). Including PIFuHD and PHORHUM, they can capture geometric details of visible parts, but cannot handle occlusions at all.\n\n> **Question 2.** Have the authors explored more efficient sampling techniques like DDIM to reduce inference time? What speedups might be achievable?\n> \n\n**Reply:** We have explored DDIM to reduce inference time, but the model cannot fully denoise and produce blurred point cloud results. We can use lower-resolution point clouds to reduce inference time while losing some details.\n\n[R1] Huang et al., Surface Reconstruction from Point Clouds: A Survey and a Benchmark, ArXiv 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZePf0yqHA", "reviewer": "Reviewer_V2cx", "summary": "The target of this paper is to reconstruct 3D clothed human shape. The conditional point clouds diffusion model is adopted as the main structure of the proposed reconstruction model. This work focuses on designing effective conditioning features, especially for overcoming the occlusions. The conditioning features include the similar projected image features as PC2[55], the local features [signed distance, normal vector, occupancy] extracted from multiple estimated SMPL or SMPL-X human 3D parametric models. The performance of the proposed model is evaluated on synthetic and real datasets, and compared with different SOTA methods. The experimental results show good de-occlusion ability in 3D clothed human shape reconstruction.", "questions": "1.\tMulti-hypotheses was proposed in [8] to model the uncertainty caused by occlusions or depth ambiguities in 3D shape reconstruction. What are the differences between your work and [8] on multi-hypotheses? \n2.\tIf one person wears different loose clothes with the same pose, can the proposed model reconstruct their 3D shapes effectively? It should be helpful if the related experimental examples are presented. \n3.\tTable 1 shows that the Chamfer Distances of HiLo [92] and SIFU [100] are 13.711cm and 13.397cm, while the result of the proposed method is 1.872cm, a significant improvement. My question is, both HiLo and SIFU also use SMPL model, which can provide good human 3D shape and pose prior, what is the reason for their poor results? \n4.\tTable 3 shows the results of ablation study. It indicates the Chamfer Distance decreases from 3.640cm (baseline PC2[55]) to 1.872cm (proposed MHCDIFF), totally 17.68mm improvement. It seems that the improvement should be contributed by the added conditioning features extracted from multiple SMPL. But from group A, the results show that each conditioning feature has small contributes, such as ‘w/o occupancy’, the value is 1.893cm vs. 1.872cm, only 0.21mm improvement. I am a bit confusing. A clearer explanation would be better. \n5.\tFrom Table 3, group B, it seems that using SMPL model (conditioned on single ProPose estimation) is better than using SMPL-X model (conditioned on PIXIE estimation). If it is true, please analyze the reasons. \n6.\tFrom the Qualitative results (Figure 3, Figure 5), it can be seen that the reconstruction qualities of some parts are not good, such as hands, feet, and so on. Please analyze the reasons.\n7.\tThe multi-hypotheses conditioning can take an arbitrary number of SMPL, SMPL-X, and their combined. What is the optimal recommendation?", "limitations": "As the paper states that proposed method has low efficiency caused by diffusion model, which has limited applications.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well written and easy to understand.\n2.\tIt is the first work that extends the multi-hypotheses SMPL estimation to pixel-aligned 3D clothed human reconstruction.\n3.\tThe experimental results show good de-occlusion ability when reconstructing 3D clothed human 3D shape.", "weaknesses": "1.\tLow reconstruction quality of details, such as hands and feet.\n2.\tThe analysis and description of the experimental results are not clear and in-depth enough. For specific details, please refer to the following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Question 1.** Multi-hypotheses was proposed in [8] to model the uncertainty caused by occlusions or depth ambiguities in 3D shape reconstruction. What are the differences between your work and [8] on multi-hypotheses?\n> \n\n**Reply:** 3D Multi-bodies [8] learns a multi-hypothesis neural network regressor to predict different SMPL hypotheses to model the uncertainty. We use ProPose [16] as multi-hypothesis SMPL estimator, which predicts the distribution parameter of the matrix Fisher distribution to solve the same problem. We can adopt 3D Multibodies [8] to estimate multi SMPL hypotheses. The main contributions of this work lie in **the use of multi hypotheses** to reconstruct full detailed body shapes.\n\n> **Question 2.** If one person wears different loose clothes with the same pose, can the proposed model reconstruct their 3D shapes effectively?\n> \n\n**Reply:** If we know different images contain one person with the same pose, we can use the same SMPL predictions, but MHCDIFF does not consider this circumstance. We show qualitative results of in-the-wild images with loose clothes. In Fig. R1 in PDF, We show qualitative results on in-the-wild images with occlusion and loose clothes.\n\n> **Question 3.** Table 1 shows that the Chamfer Distances of HiLo [92] and SIFU [100] are 13.711cm and 13.397cm, while the result of the proposed method is 1.872cm, a significant improvement. My question is, both HiLo and SIFU also use SMPL model, which can provide good human 3D shape and pose prior, what is the reason for their poor results?\n> \n\n**Reply:** HiLo [92] and SIFU [100] use implicit functions, which cannot inpaint the invisible regions (lines 34-40). Compared to ICON [88], HiLo [92] use **the global feature encoder** and SIFU [100] use **cross-attention** from the normal map of SMPL (lines 245-248). The global features are **sensitive to misaligned SMPL estimation**, so they show poor results (lines 121-123). In Fig. 3, Fig. 5 and Fig. 6 in the paper, HiLo [92] and SIFU [100] show poor qualitative results.\n\n> **Question 4.** Table 3 shows the results of ablation study. It indicates the Chamfer Distance decreases from 3.640cm (baseline PC2[55]) to 1.872cm (proposed MHCDIFF), totally 17.68mm improvement. It seems that the improvement should be contributed by the added conditioning features extracted from multiple SMPL. But from group A, the results show that each conditioning feature has small contributes, such as ‘w/o occupancy’, the value is 1.893cm vs. 1.872cm, only 0.21mm improvement. I am a bit confusing. A clearer explanation would be better.\n> \n\n**Reply:** For example, MHCDIFF ‘w/o occupancy’ is conditioned on ‘signed distance’, ‘normal’ and ‘encoding’. Among them, ‘signed distance’ is the most important component, but all of them include human 3D shape and pose prior well. However, our major contributions are **correcting the misaligned SMPL estimation** as shown in Tab. 1, group B and Tab3, group C in the paper, and **inpainting the invisible regions** as shown in Fig. 4 in the paper and Fig. R2 (Right) in PDF. We will clarify this point in the revision.\n\n> **Question 5.** From Table 3, group B, it seems that using SMPL model (conditioned on single ProPose estimation) is better than using SMPL-X model (conditioned on PIXIE estimation). If it is true, please analyze the reasons.\n> \n\n**Reply:** We select PIXIE [17] (3DV, 2021) as SMPL-X estimator following the previous pixel-aligned reconstruction methods [88, 92, 100, 102]. We could not find multi-hypothesis SMPL-X estimator, so we use ProPose [16] (CVPR, 2023). More recent SMPL-X estimators [R1, R2, R3] may show better results, but we mainly focus on multi-hypothesis estimators in this paper.\n\n> **Weakness 1.** Low reconstruction quality of details, such as hands and feet.\n> \n\n> **Question 6.** From the Qualitative results (Figure 3, Figure 5), it can be seen that the reconstruction qualities of some parts are not good, such as hands, feet, and so on. Please analyze the reasons.\n> \n\n**Reply:** We observe that the multi-hypotheses SMPL estimation via ProPose [16] shows larger variance on the hands and feet as shown in Fig. 2 (Left) in the paper. In this paper, we mainly focus on the uncertainty of bigger articulations such as arms and legs. We are currently working on this limitation and the progress is shown in Fig. R2 (Left) in PDF.\n\n> **Question 7.** The multi-hypotheses conditioning can take an arbitrary number of SMPL, SMPL-X, and their combined. What is the optimal recommendation?\n> \n\n**Reply:** As previously discussed, we focus on the uncertainty of bigger articulations, not detailed facial expressions or finger articulation. Additionally, there are few research on multi-hypotheses SMPL-X estimation, so the current optimal recommendation is using about 10 SMPL samples, as shown in Tab. R1 in global response. However, multi SMPL-X hypotheses can also be adopted without any modification.\n\n[R1] Lin et al., One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer, CVPR 2023\n\n[R2] Cai et al., SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation, NeurIPS 2023\n\n[R3] Baradel et al., Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot, ECCV 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fyMwIQgfaD", "reviewer": "Reviewer_vFH7", "summary": "Authors propose MHCDIFF (Multi-hypotheses Conditioned Point Cloud Diffusion) to reconstruct 3D human point could from a single image under occlusion. The key idea behind MHCDIFF is the smart use of image projection features and features from multiple SMPL hypothesis, coupled with point cloud diffusion.\nThe method achieves favourable reconstruction performance on CAPE, MultiHuman, and Hi4D datasets.", "questions": "Overall I'm positive about the work. It is interesting to see that adding basic SMPL features (average occupancy and distance to closest SMPL point) to the PC2 formulation is a useful combination. The quantitative results also look good. I'm a bit unclear on some experimental and design details as stated above.", "limitations": "Authors discuss limitations and broader impact in thier manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed idea to leverage information from multiple SMPL hypothesis is simple and intuitive.\n+ The ablation studies are detailed and show the significance of each component of MHCDIFF.", "weaknesses": "Some key details are unclear:\n- L222: How are SMPL meshes sampled? \n- Eq. 5: Why use mean occupancy and not mean distance? Is there an advantage of using a combination of distance and occupancy?\n\nExperiments:\n- Authors mostly evaluate their method on in door datasets with controlled settings. What about more “in the wild” data like 3DPW, MS COCO.\n- Is there any correlation between the SMPL hypothesis and the reconstruction accuracy? How accurate do the hypothesis have to be? Does number of SMPL sampled matter? Does the global translation of the SMPL matter?\n\nMinor:\n- L180: shouldn’t it be argmin if we want to compute distance to closest SMPL mesh?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1-1.** L222: How are SMPL meshes sampled?\n> \n\n**Reply:** We sample SMPL meshes via ProPose [16] (lines 178-179). ProPose [16] predicts the distribution parameter of the matrix Fisher distribution as mentioned in Section 3 Preliminary, ProPose [16] in the paper. We sample SMPL pose parameters $\\theta$ from the matrix Fisher distribution.\n\n> **Weakness 1-2.** Eq. 5: Why use mean occupancy and not mean distance? Is there an advantage of using a combination of distance and occupancy?\n> \n\n**Reply:** Distances are in wider range, so mean distance is **sensitive to extreme samples**. We use mean occupancy as **probability density function of SMPL meshes**. With the combination of distance and occupancy, MHCDIFF can assume all distributions with their respective probabilities (lines 185-186).\n\n> **Weakness 2-1.** Authors mostly evaluate their method on in door datasets with controlled settings. What about more “in the wild” data like 3DPW, MS COCO.\n> \n\n**Reply:** 3DPW and MS COCO do not have ground truth 3D shapes. Multi-view systems are necessary to capture 3D detailed scans, so there is no existing in-the-wild dataset with corresponding 3D scans. HiLo [92] and SIFU [100] use CAPE and THuman2.0 datasets, and ICON [88] use AGORA and CAPE datasets for evaluation. In Fig. R1 in PDF, We show qualitative results on in-the-wild images with occlusion and loose clothes. We will contain more results in the revision.\n\n> **Weakness 2-2.** Is there any correlation between the SMPL hypothesis and the reconstruction accuracy? How accurate do the hypothesis have to be? Does number of SMPL sampled matter? Does the global translation of the SMPL matter?\n> \n\n**Reply:** (1) The accuracy of SMPL or SMPL-X affects the reconstruction accuracy. In Tab. 1 in the paper, ProPose [16] shows better results than PIXIE [17], and the reconstruction accuracy follows the tendency in Tab. 3, group B in the paper. As shown in Fig. 4 in the paper, the accuracy of MHCDIFF follows the one of ProPose [16]. (2) In Tab. R1 in global response, we show the correlation between the number of SMPL sampled and the reconstruction quality. More SMPL hypotheses may include more accurate samples and improve the quality with 15 samples but may include extreme samples and decrease the performance with 20 samples. (3) We normalize the image with bounding box and the reconstructed results follow the global translation of SMPL estimation. In our experiment setting, sampled SMPL meshes via ProPose [16] have equal global translations. We will clarify this point in the revision.\n\n> **Weakness 3-1.** L180: shouldn’t it be argmin if we want to compute distance to closest SMPL mesh?\n> \n\n**Reply:** Thank you for pointing this out. We use the signed distance, where inside points have positive and outside points have negative values (L170), so we will change to $\\bar{i}=argmin_{i\\in\\{1,...,s\\}}|d(X_t|S_i)|$ in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback and finding that the proposed method is “simple and intuitive” [vFH7] and shows “significant improvement” [1vtm, V2cx]. We also appreciate [V2cx] from finding that “paper is well written and easy to understand.”

|The number of SMPL sampled|CD (cm)|P2S (cm)|Evaluation time on CAPE dataset (hours)
|---|---|---|---|
|1|1.939|1.869|4|
|5|1.882|1.817|8|
|10|1.872|1.810|12|
|15|1.833|1.773|16|
|20|1.836|1.777|20|
[Table R1: The correlation between the number of SMPL sampled and the reconstruction quality. CD and P2s denote Chamfer Distance and Point-to-Surface distance, respectively.]

 We provide the figures referred to in our author response in the PDF file below.

Meta-review: 3 out of 4 reviewers recommended acceptance. the reviewer who recommended reject is mainly concerned about the main ideas: Multi-hypotheses poses and diffusion-based framework, are incremental. These are viewed by other reviewers as simple, effective and beneficial. This AC follows the majority views on this paper to be accepted.
Decision: Accept (poster)