Robust-PIFu: Robust Pixel-aligned Implicit Function for 3D Human Digitalization from a Single Image
Authors: Kennard Chan
Fayao Liu
Guosheng Lin
Chuan-Sheng Foo
Weisi Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Human Digitalization
3D Human Avatar
Pixel-aligned Implicit Models
Latent Diffusion Models
Occlusion
Robustness
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Using Transfer Learning and Latent Diffusion Models to elevate the robustness of pixel-aligned implicit models in terms of dealing with occlusions and also to achieve SOTA performance.

Abstract: Existing methods for 3D clothed human digitalization perform well when the input image is captured in ideal conditions that assume the lack of any occlusion. However, in reality, images may often have occlusion problems such as incomplete observation of the human subject's full body, self-occlusion by the human subject, and non-frontal body pose. When given such input images, these existing methods fail to perform adequately. Thus, we propose Robust-PIFu, a pixel-aligned implicit model that capitalized on large-scale, pretrained latent diffusion models to address the challenge of digitalizing human subjects from non-ideal images that suffer from occlusions.

Robust-PIfu offers four new contributions. Firstly, we propose a 'disentangling' latent diffusion model. This diffusion model, pretrained on billions of images, takes in any input image and removes external occlusions, such as inter-person occlusions, from that image. Secondly, Robust-PIFu addresses internal occlusions like self-occlusion by introducing a `penetrating' latent diffusion model. This diffusion model outputs multi-layered normal maps that by-pass occlusions caused by the human subject's own limbs or other body parts (i.e. self-occlusion). Thirdly, in order to incorporate such multi-layered normal maps into a pixel-aligned implicit model, we introduce our Layered-Normals Pixel-aligned Implicit Model, which improves the structural accuracy of predicted clothed human meshes. Lastly, Robust-PIFu proposes an optional super-resolution mechanism for the multi-layered normal maps. This addresses scenarios where the input image is of low or inadequate resolution. Though not strictly related to occlusion, this is still an important subproblem. Our experiments show that Robust-PIFu outperforms current SOTA methods both qualitatively and quantitatively. Our code will be released to the public.

3D Human Digitalization is the problem of reconstructing or generating 3D model (c.g. a mesh) of a human subject from inputs such as single image of that human subject. Recently, WWang et al. (2023b) showed that existing 3D Human Digitalization methods, such as ICON (Xiu et al.) 2022) and PIFuHD (Saito et al., 2020), face severe difficulties when given input images that have incomplete observation (i.e. missing pixels) of the human subject's full body (like in Fig. D- d). These missing pixels often occur when there are multiple human subjects the same input image (the subjects occlude one another). WWang et al. (2023b) suggested method to address this incomplete observation problem, but they assumed that a groundtruth SMPL-X mesh (Pavlakos al. 2019 is available and provided. A groundtruth SMPL-X mesh clothless and hairless Published as a conference paper at ICLR 2025
ROBUST-PIFU: ROBUST PIXEL-ALIGNED IMPLICIT FUNCTION FOR 3D HUMAN DIGITALIZATION FROM A
SINGLE IMAGE
Kennard Yanting Chan Nanyang Technological University
Fayao Liu Institute for Infocomm Research, A*STAR
Guosheng Lin
Chuan Sheng Foo
Nanyang Technological University
Centre for Frontier Al Research, S
Weisi Lin
Nanyang Technological University
ABSTRACT
Existing methods for 3D clothed human digitalization perform well when the in- put image is captured in ideal conditions that assume the lack of any occlusion. However, in reality, images may often have occlusion problems such as incom- plete observation of the human subject's full body, self-occlusion by the human subject, and non-frontal body pose. When given such input images, these exist- ing methods fail perform adequately. Thus, we propose Robust-PlFu, pixel- aligned implicit model that capitalized on large-scale, pretrained latent diffusion models to address the challenge of digitalizing human subjects from non-ideal images that suffer from occlusions.
Robust-Plfu offers four new contributions. Firstly, propose 'disentangling' latent diffusion model. This diffusion model, pretrained on billions images, takes in any input image and removes external occlusions, such as inter-person occlusions, from that image. Secondly, Robust-PIFu addresses internal occlu- sions like self-occlusion by introducing 'penetrating' latent diffusion model. This diffusion model outputs multi-layered normal maps that by-pass occlusions caused by the human subject's limbs other body parts (i.e. self-occlusion). Thirdly, in order to incorporate such multi-layered normal maps into a pixel- aligned implicit model, we introduce our Layered-Normall Pixel-aligned Im- plicit Model, which improves the structural accuracy of predicted clothed human meshes. Lastly, Robust-PIFi proposes an optional super-resolution mechanism for the multi-layered normal maps. This addresses scenarios where the input im- age of low or inadequate resolution. Though not strictly related to occlusion, this is still an important subproblem. Our experiments show that Robust-PIFu out- performs current SOTA methods both qualitatively and quantitatively. Our code will be released to the public.
INTRODUCTION

---

Published as a conference paper at ICLR 2025
(a) Groundtruth
(b) Wang et al. (c) SeSDF
(d) ECON
(e) Ours
Figure I: Results of SOTA and RobustPIFu on input images (in color) that have external occlusions.
a) Groundtruth
(b) Wang et al.)
(c) SeSDF
(d) ECON
(c) Ours
Figure 2: Results on input images (in color) that have no occlusion (Ist row), internal occlusion due to non-frontal pose (2nd row), and internal occlusion due to self-occlusion (3rd row)
3D model that has both the actual 3D body pose and 3D body shape (excluding clothes and hair) of human subject. Assuming that groundtruth SMPL-X mesh is available greatly simplifies the problem of incomplete observation, but it is practically unrealistic. In most situations, the only input that is available the input image itself.
Thus, we decided to address this problem. To this end, we introduce Robust-PlFu, pixel-aligned implicit model that capitalized on large-scale, pretrained latent diffusion models to circumvent wide- ranging occlusion problems (including incomplete observation) that may or may not occur in an input image.
Our work is inspired by the observations documented in a recent work, Zero-I-to- (Liu et al. 2023b). Zero-1-to-3 found that large-scale, pretrained latent diffusion model, when given an image of object input, be controlled to synthesize novel view (image) of that object from any specified viewpoint. This is significant as the latent diffusion model has demonstrated a 3D conceptual understanding of the object by freely generating new images of the same object from different viewpoints. To us, this is an intriguing finding as novel view is. in all cases under this context, view that contains unseen regions of an object. This bears similarities problem (i.e. occlusion in input images), which pertains to uncovering unseen regions that have been occluded. Hence, we would like to find the pretrained latent diffusion model but instead of synthesizing novel views, we want the model to synthesize views that uncover and resolve any external and internal occlusions.
our own
we can design similar mechanism to control
We classify occlusion in an input image into external and internal occlusions. We define exter- occlusions as occlusions that arise due external factors unrelated the pictured human subject. Examples include an object occluding the human subject, another human occluding the human subject that we are interested in (also called inter-person occlusion). On the other hand, we define internal occlusions occlusions caused by factors that can be controlled by the human subject. Ex- amples include self-occlusion (subject uses his/her limbs to cover a part of his/her body) or occlusion caused by the human subject not directly facing the camera (i.e. non-frontal body pose).
Firstly, to address external occlusions, our RobustPIFu introduces a pretrained latent diffusion model that is controlled to 'disentangle' human subject(s) from given input image. This diffusion model
2

---

Published as a conference paper at ICLR 2025
isolates the person of interest from any extraneous objects or other humans, thereby resolving exter- nal occlusions. This diffusion model is the first contribution of Robust-PlFu.
Secondly, to tackle internal occlusions, propose another pretrained latent diffusion model that is controlled to predict multi-layered normal maps at different camera angles. Regardless of whether the human subject tried to self-occlude or face away from the camera, these multi-layered normal maps will show the full geometry of the human subject's body. This is the second contribution of Robust-PIFu.
Thirdly, in order to incorporate such multi-layered normal maps into a pixel-aligned implicit model, introduce our Layered-Normals Pixel-aligned Implicit Model. Layered-Normal Pixel-aligned Implicit Model uses two different mechanisms (introduced later) to incorporate the multi-layered normal maps and serves improve the structural accuracy of generated clothed human meshes.
Lastly, besides external and internal occlusions, another common problem in input images is the low or inadequate resolution of the pictured human subject. This can occurs when the human subject is far away from the camera or when the camera is set to capture more than one human subject. In such cases, existing methods will not able to reconstruct a clothed human mesh with high-resolution details. In order to overcome this, we designed an optional mechanism to control a pretrained latent diffusion model for normal map super- super-resolution. This allows Robust-PIFu to create meshes with high-resolution details despite a low-resolution input image.
2 RELATED WORKS
2.1 SINGLE-VIEW CLOTHED HUMAN DIGITALIZATION
Single-view clothed human digitalization refers to the task of retrieving reconstructing 3D model (e.g. mesh) of clothed human body from a single RGB image. An important class of deep learning methods that have consistently attained state-of-the-art results for single- clothed human digitalization is the pixel-aligned implicit models. These models, which include ECON (Xiu) al. 2023), ICON (Xiu et al. 2022). IntegratedPIFu (Chan et al.) 2022), and PIFuHD (Saito al.) 2020 work by learning an implicit function that represents the enclosed surface of a clothed human body. During testing, the learned implicit function is sampled using a 3D grid of evenly-spaced sample points. For each sample point, the implicit function returns/predicts an occupancy label, which indicates if the sample point is considered 'inside' or 'outside' of clothed human body's surface. Once a 3D grid of predicted occupancy labels is computed, mesh that resembles clothed human body can be extracted from this grid by using the Marching Cubes algorithm (Lorensen & Cline 1987). For more information pixel-aligned implicit models, refer to Annex A.16.
2.2 ROBUSTNESS IN PIXEL-ALIGNED IMPLICIT MODELS
Pixel-aligned implicit models' robustness to occlusions is an important topic that has garnered at- tention from researchers. For example, ECON (Xiu et al. 2023), which is a pixel-aligned implicit model. has proposed mechanisms to improve robustness with respect to inter-person occlusions. One of the mechanisms proposed by ECON is train a shape completion module with a parametric human body model (i.c. SMPL-X mesh) and a set of randomly masked depth maps. In addition, the work by Wang al. (2023b) proposes a new pixel-aligned implicit model that also aims to be robust to mnter-person occlusions in a given input image. The proposed pixel-aligned implicit model infers the occluded or missing information of a human subject's body by incorporating a SMPL-X mesh 2D GAN, and 3D GAN into its pipeline.
However, there are number of problems with how these existing models address robustness ques- tions. Firstly, both ECON and the work by Wang al. assume that a groundtruth or very accurate SMPL-X mesh is available as an input. In situations where there are occlusions or missing pixels in given input image, it is unlikely that very accurate SMPL-X mesh can be predicted from that input image. Thus, these models may not be practical in real-life use cases.
Secondly, even if we assume groundtruth SMPL-X meshes given, the clothed human meshes reconstructed by ECON and the work by Wang et al. often suffer from poor structural accuracy and imprecise details when there are inter-person occlusions in the given input image (See Fig. (p.d).
3

---

Published as a conference paper at ICLR 2025
Lastly, other than inter-person occlusions (which is a type of external occlusion), there are other types of occlusions like internal occlusions. Internal occlusions, such as self-occlusion, are not well-addressed by existing methods yet.
Hence, we propose Robust-PIFu in order to address both external and internal occlusions effectively.
2.3 LARGE-SCALE LATENT DIFFUSION MODELS
Recently, large-scale 2D diffusion models, such as DALL-E (Ramesh al. 2021). Imagen (Saharia et al., (022), and Stable Diffusion (Rombach et al., (022). have showcased their capacity to grasp an extensive range of visual concepts from internetssall image datasets. Consequently, an increasing number of researchers are harnessing these models' rich priors about our 3D world and utilizing
them in 3D tasks (e.g. (Poole et al. 2022 Lin et al. 2023 Liu et al. 2023b a Chen et al. 2024 Lee
et al. 2024 Ho et al. 2024)).
One notable example is Zero-I-to-3 (Liu et al. 2023b). Zero-1-to-3 illustrated that latent dif- fusion model, pretrained on extensive internet image data, can be manipulated to produce fresh perspective of an object from any specified viewpoint. To achieve this, the pretrained model first undergoes fine-tuning with input RGB view of the object, a relative camera transformation, and a transformed RGB view of the same object. Once fine-tuned, the model, when provided with an input view of the object, can generate numerous novel views of that object. These new views then utilized to reconstruct 3D mesh of the object using Score Jacobian Chaining (SJC) (Wang al. 2023a). Zero-1-to-3 demonstrated superior performance compared state-of-the-an (SOTA) results in 3D object reconstruction. This finding not only shows that vast internet image data can be leveraged on to solve 3D reconstruction task, but it also demonstrates that a pre-trained latent diffusion model is able to gain 3D conceptual understanding of an object just by processing a 2D) image of that object. (For more information on Zero-1-to-3, refer to Annex Annex(A.16.2 (A.16.2)
We wish to harness and capitalize on the 3D conceptual understanding attained by pre- traitre latent diffusion model as well. We hypothesize that a 3D conceptual understanding a human will be useful in identifying and removing occlusions that are present in 2D image of that human. Just as the 1-to3- uses a pre-trained latent diffusion model to uncover an unseen view (i.e. a novel view) of an object, we wish to use pre-trained latent diffusion model uncover unseen regions (i.e. occluded regions) of a human subject. In other words, we aim to use the latent diffusion model to resolve any external and internal occlusions that may be present in given input image.
METHOD
We show an overview of Robust-PlFu in Fig. B First, Robust-PIFu is given input image that may or may not have any occlusion. This given input image is passed to Disentangling Diffuser (D), which is latent diffusion model that will remove any external occlusion (e.g. inter-person occlusions) from the input image. If there are inter-person occlusions, this means there are multiple subjects in the input image, and D will 'disentangle' the human subjects into separate images (i.e. disentangled images). Each disentangled image, now free of external occlusions, can then be easily used predict a SMPL-X mesh (if the groundtruth SMPL-X mesh not given).
Next, each of the disentangled images are separately fed into our second latent diffusion model (i.e. Penetrating Diffuser or P). For each disentangled image, will generate 8 normal maps (also called multi-layered normal maps or Nc), which correspond to the different layers of the human subject at different camera angles. These 8 normal maps allow us to bypass both self-occlusion and non-frontal pose of the human subject in the given input image. In other words, P resolves internal occlusions.
Finally, Nc and the disentangled image passed to Layered-Norman Pixcl-alignel Implicit Model, which is a pixel-aligned implicit model fitted with two new mechanisms. These mechanisms, which will be introduced later, enable multi-layered normal maps to be incorporated into pixel- aligned implicit model. Layered-Norme mesh which is then refined by differential optimizer using No-
Pixel-aligned Implicit Model produces a clothed human
In order to incorporate a SMPL-X mesh into our Robust-PIFuy we first extract multi mulli--lyyred normal maps from the SMPL-X mesh (Ns). Nş is similar to No except that Nş pertains to SMPL--X body,
4

---

Published as a conference paper at ICLR 2025
Disentangling Diffuser (D)
Input Image
Disentangled Image
Disenlangled Image 2
Disentangled Image
Penetrating Diffuser (P)
SMPL-X Multi-layered Normal Maps maps). 3
Mult-llayere Normal Maps maps), Ne
Layered-Normals Pixel-aligned Implicit Model
Differential Optimizer
(Marching Cubes Algorithm)
Ciothed Human Mesh
Figure 3: Overview of our Robust-PIFu. D first 'disentangled" the human subjects into separate images. Then, P predicts multi-layered normal maps from each disentangled image. The maps are used by our -oyemaddNoomals Pixel-aligned Implicit Model to construct clothed human mesh.
and Ne pertains clothed human body. Ns is then fed into both P and the Layered-Normal Pixel-aligned Implicit Model.
Also, Robust-PIFu also offers an optional diffuser our Refining Diffuser (R). Our R will super- resolve Ne, allowing higher-resolution Ne to be used in the differential optimizer. However, the tradeoff of using R is increased inference time.
Now, we will elaborate on the four contributions of Robust-PlFu: 1. Disentangling Diffuser (Sect.
3.1
2. Penetrating Diffuser (Sect. 3.2 3. Layered-Normals Pixel-aligned Implicit Model (Sect.
3.3
4. Refining Diffuser (Sect. 3.4
1. Pre-Trrining:
(Gives Supervised Signals)
Disentangling Diffuser (D)
Billions Internet images
(Learss Reconstruct)
2. Finetuning:
CLIP Image Embedder
Crooss-atiention)
Disentangilig Diffuser (D)
Denoising U-Net
Encoder
Encoding (64x6x44)
Disentangled Image 1
Input Image, I
Gaussian Naise (or Noised Encoding)
Concal
Human Subject Selection Map
Figure 4: The Pre-training and Finetuning stages of Disentangling Diffuser (D).
5

---

Published as a conference paper at ICLR 2025
3.1 DISENTANGLING DIFFUSER (D)
As explained in Sect. cxisting methods, like ECON and the work by Wang ct al. (2023b). do not address external occlusion like inter-person occlusion well because these models assume that very accurate SMPL-X mesh is available despite the occlusion. Moreover, even when this assumption true, these models often fail produce clothed meshes that have accurate structure and precise details.
To address external occlusion, propose Disentangling Diffuser (D). Our D is shown in Fig. and it capitalizes on the extensive visual concepts and knowledge learned by large-scale latent diffusion model that has been pre-trained on billions of images. Similar Zero-I-to-3 (Liu al. 2023b), our D injects the pre-trained latent diffusion model with architectural modifications before a round of finetuning. Specifically, of a human subject. As shown in Fig. 4 the given input image (I) to D is an image of at most used as a conditional prior to via two streams. First, we obtain CLIP image embedding of before using it for cross-attention in the denoising U-Net of the latent diffusion model. Second, / is encoded into an encoding and then channel-comcatenated with noised encoding before it is fed to the denoising U-Net. Also, in order control which of the three human subjects to extract out, we introduce a Human Subject Selection Map, which a binary map that has out of three possible patterns (see Annex for more info). The denoising U-Net will output denoised encoding that is then decoded into a "disentangled" image. (Aside: D can deal with input images that contain more than three human subjects well, see Annex A.2 A formal formulation of D is given in Eqn.
aims to denoise a noised encoding that represents an image
three human subjects.
D iEZ, O<isn
(1)
where 1dd is the disentangled image of the i human subject, I is the input image, H; is the ith Human Subject Selection Map, and n the maximum number of human subjects allowed in input image to D. In our work, we set n to be 3.
To address a wide range of external occlusions, D is finetuned with I that afflicted with different types of external occlusions, such inter-person occlusion, object-caused occlusion, and poor light- ing conditions (see Figs. and 12). After finetuning, our D will be able to remove these various forms of external occlusions from any given input image as also shown in Figs. [T and 12]
The training or finetuning process of D (as well as P and R) follows a typical latent diffusion model whereby the model learns to predict the noise added the latent encodings each time step. and the MSE between the predicted and actual noises is minimized. More training details in Annex A,14
Disentangled Image
Penetrating Diffuser (P)
Innor
Normal Map
Normal Map
Normal Map
Normal Map
SMPL-X Maps maps N
Front
Front
Normal Map
Normal Map
Normal
Normal Map
(next 4 maps from N
Penetrating Diffuser (P)
Right Outer Normal Map
Right Inner Normal Map
Lefi Outer Normal Map
Left Inner Normal Map
Figure 5: Penetrating Diffuser (P). In the 1st run (blue arrows), 4 normal maps from the front and back camera directions are produced. In the 2nd run (magenta arrows), a further normal maps from the right and left camera directions are produced. These 8 normal maps constitute our Nc.

---

Published as a conference paper at ICLR 2025
Mult-layared Normal Maps (SMPL-X), N,
Mult-loyanod Normal Maps (Ctalhed), N.
H 24
3D CNN
Nuliiidiimensional Layered Normals Grid
Differential Optimizer
Concatonation
MLP
(Marching Cubes Aigorthm)
Stacked Hourglass
Disentangled image
Clatted Human Miohh
Figure 6: The final part of Robust-PIFu: Layered-Normals Pixel-aligned Implicit Model.
.0mm
3.2 PENETRATING DIFFUSER (P)
The disentangled image produced by D may still be affected by internal occlusions, such self- occlusion and occlusion caused by non-frontal pose the human subject. For example, a human subject may put his arm across this torso, obscuring the topology of his jacket. Similarly, a human subject that is not facing the camera directly may conceal the topology of his facial features. Inter- nal occlusions like these will prevent pixel-aligned implicit model from reconstructing clothed human mesh that has an accurate and detailed topology. In order to bypass internal occlusions, we propose our Penetrating Diffuser (P), which predicts multi-layered normal maps clothed hu- man body. These maps are also known as Nc. and they describe the topology of a human subject at different layers and from distinct camera directions (illustrated and explained in Annex[A.4]
To design P. we again tap on the visual knowledge acquired by a large-scale pre-trained latent diffusion model. From the pre-trained model, we extend it by adding two new conditional priors (See Fig. [5). The first conditional prior is the disentangled image produced by D. This disentangled image is incorporated into the same way incorporated into (i.e. via cross-attention and channel concatenation). The second conditional prior is Ns. which refers to multi-layered normal maps that are extracted from SMPL-X mesh. Ns is incorporated into P via channel concatenation only. Unlike D, P does not output RGB image in each run. Instead, P outputs set of 4 normal maps in each run. In total, as shown in Fig. 5 P will run twice. In the first run, 4 normal maps from the front and back camera directions are produced. In the second run, these normal maps used as conditional priors to P, and a further normal maps from the right and left camera directions will then be produced. These 8 normal maps form our No i.c. the multi-layers normal maps of the clothed human body. These maps reveal the topology of the human subject from different sides and different layers. This resolves internal occlusions like self-occlusion or non-frontal body pose. A formal formulation of P is given in Eqn.
NV = P(. tE {1,2}
(2)
where I is the disentangled image of the tht human subject. When t= 1, No and N represent the first 4 maps from No and Ng respectively. When / N2 and N2S represent the next 4 maps from No and Ns respectively. No is a set of zero-filled maps.
3.3 LAYERED-NORMALS PIXEL-ALIGNED IMPLICIT MODEL
At this stage, we have obtained multi-layered normal maps (both Ns and Nc) that contain infor- mation of the human subject from different sides and layers. But pixel-aligned implicit model is not designed to take these layered normal inputs. Thus, we propose a new form of pixel-aligned implicit model called Layered-Normals Pixel-aligned Implicit Model, which is shown in Fig. 6
Layered-Normall Pixel-aligned Implicit Model incorporates multi-layered normal maps via two mechanisms or streams. In the first stream, concatenate Ng, Nc. and the disentangled im- age together. The concatenated product is then fed into the encoder (i.e. stacked hourglass network) of our pixel-aligned implicit model.

---

Published as a conference paper at ICLR 2025
In the second stream, we use Nc to form a Multi-dimensional Layered Normals Grid. This 3D grid is formed by aligning the eight 2D normal maps of Nç inside a 3D space, as shown in Fig. [5] the grid is 24-length vector, which consists of normal vector (3-length
Each clement/position vector) taken from each of the 8 normal maps. The grid contains not only detailed information on the topology of the human subject, but also the structural information of the human subject in the form of a visual hull (shown in bottom right of Fig. (5). This visual hull narrows down the possible shape of the human subject's body the surface of the human subject cannot lie outside of the visual hull. In other words, the visual hull acts like an initial, template 3D shape for the pixel-aligned implicit model to work on before predicting final clothed human shape/mesh. To incorporate the Multi-dimensional Layered Normals Grid into the pixel-aligned implicit model, we process this multi-dimensional grid using a 3D-CNN (see Fig. (). The features produced by the 3D-CNN are then fed into the MLP. In addition, we also allow for skip connection that directly feeds the values from the grid into the MLP.
Via these two mechanisms, we allow information from both our Ns and Nc to be incorporated into a pixel-aligned implicit model a principled and sensible way. Like a typical pixel-aligned implicit model, our Layered-Normal. Pixel-aligned Implicit Model is trained by minimizing the MSE between the predicted and actual occupancy labels of the sample points (Saito et al.) 2019). After training, our pixel-aligned implicit model model will produce a clothed human mesh using the Marching Cubes's ' algorithm. In addition, we provide a formal formulation of our Layered-Normale Pixel-aligned Implicit Model in the following equation:
F(I, Nc) = s(x, y, v)= 5(x,8,2) E [0, 1]
(3)
where 1 is the disentangled image of the ; human subject, and s(x, 2) denotes the predicted implicit function that represents a 3D clothed human body.
The clothed human mesh is then refined by differential optimizer, as is done in ICON (Xiu al. 2022). The differential optimizer works by computing an error (e.g. LI loss) between a predicted normal map and the surface normals of the clothed human mesh. The mesh then deformed or adjusted such that the error is minimized. More than one predicted normal map can be used in this refining process. For Robust-PIFu our No (i.e. maps) is used as the predicted normal maps.
3.4 REFINING DIFFUSER (R)
Our differential optimizer uses Ne refine the meshes. In order improve the refinement process, we can super-resolve No using our optional Refining Diffuser (R). The differential optimizer can then use the super-resolved No. rather than the original Nc. during the refinement process. R is latent diffusion model that takes in NC as a conditional prior and outputs a super-resolved Nc. as shown in Fig. [6 addresses scenarios where the given input image of low resolution (resulting in No having low resolution as well) or when an application requires clothed human meshes that are of a higher than usual resolution. However, the use of will also result in a noticeably increased inference time. There is thus 'inference time resolution' trade-off. More details on our R can be found in Annex A.7 In reality, inference time may be important to different applications. Thus, to ensure fairness, did not use R to produce our results, but we demonstrate the usefulness of R
in Fig. 17
EXPERIMENTS
4.1
DATASETS
To train pixel-aligned implicit models (both ours and the existing models), 3D scans/data of human subjects are required. For our experiments, we use the THuman2.0 dataset (Yu et al. 2021) to train both our models and the existing models. The THuman2.0 dataset contains 526 3D full- scans (i.e. meshes) of real-life human subjects. We use a 80-20 train-test split. For each 3D sean, we render RGB images at 10 degree intervals (azimuth rotation) using weak- beakk-ersppectve camera.
Prior to training, our D, P. and R are already pre- pre-trained with the same Internet-scale image datasets (Schuhmann al. 2022) (>2 billion images) used by Stable Diffusion (Rombach et al. (022).
8

---

Published as a conference paper at ICLR 2025
Groundtruth
(b) Wang et al."
(c) SeSDF
(d) ECON
(e) Ours
Figure 7: Results on input images (in color) that have no occlusion (Ist row), internal occlusions
due to self-occlusion (2nd row) and non-fronte troffrottll pose (3rd row), and external occlusion (4th row)
Table 1: Quantitative Evaluation between SOTA and our Robust-PIFu.
Tuummm2.0
BUFF
Multil Hurman
Methods
CD (10
P2S (10*)
CD (10-)
P2S (10-)
CD (10+)
P2S (10*)
ICON
9.947
9.014
8.659
7,892
15.76
13.39
ECON
11.17
9.090
9.064
9.750
14.69
13.13
D-IF
10.24
9.137
9.136
7.644
16.71
13.78
ScSDF
8.941
8.441
6.582
6.947
14,00
12.30
Wang al.
8.944
8.596
9.072
9.881
21.01
21.64
Pobust-PIFu (Ours)
8.426
8.000
6.525
13.38
11.66
In addition, we use BUFF dataset (Zhang al.) (017) and MultiHuman dataset (Zheng et al. 2021
to evaluate the models. None of the models is trained using these two datasets. For BUFF dataset, we followed IntegratedPIIFu (Chan et al., 2022) and performed systematic sampling (based on sequence number) on the dataset. This gives us a total of 101 human meshes that we use for evaluation. The purpose of using systematic sampling is avoid meshes that have both the same human subject and the same pose. For MultiHuman dataset, all single human scans are used (see more in Annex A.11
4.2 COMPARISON WITH STATE-OF-THE-AAE
We compared Robust-PIFu against other existing models on single-view clothed human reconstruc- tion. The models we compared with include ICON (Xiu et al. 2022), ECON (Xiu al. 2023), D-IF Yang et al. 2023), SeSDF (Single-view version) (Cao al. 2023) and the work by Wang et al. (023b). These models are selected not only because they achieved SOTA performances but also because they are designed to be robust occlusions (e.g. mnter-person occlusion, missing pixels in input image) and other various challenging conditions like complex human poses. In our quantitative evaluation, followed (Cao al.) 2023) and used Chamfer Distance (CD) and Point-to-Surface (P2S) as metrics. CD is the bidirectional point-to-surface distances between GT and reconstructed meshes. P2S is the unidirectional variant of CD.
Qualitative Evaluation We evaluate the models qualitatively in Fig. Fig. 2 and Fig. 7 Fig.
T shows how the models react external occlusions, while Fig. pertains to internal occlusions. Fig. shows more examples both with and without occlusion. From the figures, we observe that the existing models, unlike our Robust-PIFu, struggle address external and internal occlusions and fail to produce meshes that have both an accurate structure and precise details, In Fig.
10
we show
our results on real Internet images from Adobe Stock. More results in Annex A.8
A.9
and
A.10

---

Published as a conference paper at ICLR 2025
(a) Groundtruth
(b) Without D
(c) With D)
(a) Groundtruth
(b) Without
(c) With
Figure 8: Ablation of using and not using D.
Figure 9: Ablation of using and not using P.
Table Ablation study of Layered-Normahe Pixel-aligned Implicit Model.
THuman2.0
BUFF
MultiHuman
Methods
CD (10")
P2S (10-5)
CD (10")
P2S (10")
CD (10 P2S (10-9)
Robust- PlFu w/o Concat wo Multi-D Grdd
8.928
8.433
6,574
6.939
13.98
12.26
Robust-PIFa w/o Multi-D Grid
8.551
8.132
6.183
6.656
13.60
11.85
Robust-PIFu
8.426
8,000
6.127
6.525
13,38
11,66
Quantitative Evaluation We show our quantitative results in Tab. [ and shows that Robust- PIFu is able to outperform the existing models in all metrics for all three datasets.
4.3 ABLATION STUDIES
Evaluation of our Disentangling Diffuser (D) and its Effectiveness against External Occlusions Our Disentangling Diffuser (D) is designed deal with external occlusions. In order to evaluate its effectiveness,
given imput images that have external occlusions and compare the 3D clothed human meshes that are produced by our RobustPIFu when is used and when D not used in Fig. The results clearly show how D enables our RobustPIFu produce unbroken, human-like meshes.
Evaluation of our Penetrating Diffuser (P) and its Effectiveness against Internal Occlusions Our Penetrating Diffuser (P) is designed deal with internal occlusions. To evaluate its effective- ness, we use given imput images that have internal occlusions and compare the 3D clothed human meshes that are produced by our RobustPIFu when P is used and when P is not used in Fig. 9 The results show that is pivotal for constructing regions that are occluded or unseen in input images. See Annex A.4 more ablation studies pertaining to P.
Evaluation of our Layered-Normals Pixel-aligned Implicit Model and its two Mechanisms In order to incorporate layered normal maps, we proposed using two mechanisms that modify the architecture of a pixel-aligned implicit model: 1. Concatenation of Ns, Nc. and the disentangled image 2. malti-dimensional Layered Normal Grid. Here, we evaluate the impact of these two mechanisms in Tab. The table shows both mechanisms considerably improved the structural accuracy of the clothed meshes.
5 LIMITATION AND CONCLUSION
Limitation In Annex 12 we address concerns pertaining to the computation time required by Robust-PIFu. We do by comparing Robust-PIFu with ECON (Xiu al., 2023) in terms of computation time. See Annex A.13 for other limitations.
Conclusion We have proposed Robust-PIFu, pixel-aligned implicit model that capitalized on large-scale pretrained latent diffusion models to tackle external and internal occlusions in given input images. Firstly, Robust-PIFu addresses external occlusion by proposing Disentangling Diffuser that eliminates a range of external occlusions from an input image. Secondly, we intro- duce Penetrating Diffuser that generates multi-layered normal maps tackle internal occlusions. Thirdly, Robust-PiFu proposes Layered- Normall Pixel-aligned implicit model that incorporates multi-layered normal maps into a pixel-aligned implicit model via two distinct streams. Finally, Robust-PIFu introduced an optional Refining Diffuser that performs super-resolution of normal maps, thereby allowing high-resolution clothed human mesh to be produced even when given a low-resolution input image.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research work supported by the Agency for Science, Technology and Research (A (A*STAR) under its MTC Programmatic Funds (Grant No. M23L760021). It is also supported by the MoE AcRF Tier 2 grant (MOE-T2EP0223-001),
REFERENCES
Yukang Cao, Kai Han, and Kwan- Yee Wong. Sesdf: Self-evolved signed distance field for implicit 3d clothed human reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4647-4657, 2023.
Kennard Yanting Chan, Guosheng Lin, Haiyu Zhao, and Weisi Lin. Integratedpifu: Integrated pixel aligned implicit function for single-view human reconstruction. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 2022, Proceedings. Part I1, pp. 328- 344. Springer, 2022.
Mingjin Chen, Junhao Chen, Xiaojun Ye, Huan-ang Gao, Xiaoxue Chen, Zhaoxin Fan, and Hao Zhao. Ultraman: Single image 3d human reconstruction with ultra speed and detail. arXiv preprint 1rXiv:2403.12028, 2024.
I Ho, Jie Song, Otmar Hilliges, al. Sith: Single-view textured human reconstruction with image- conditioned diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 538- 338-549,10 2024.
Jungeun Lee, Sanghun Kim, Hansol Lee, Tserendorj Adiya, and Hwasup Lim. Pidiffu: Pixel-aligned diffusion model for high-fidelity clothed human reconstruction. In Proceedings the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 5172-5181, 2024.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In Advances in Neural Information Processing Systems, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9298-9309, 2023b.
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv: 2017.
Alejandro Newell, Kaiyu Yang, and Deng. Stacked hourglass networks for human pose estima- tion. In European conference computer vision, pp. 483-499. Springer. 2016.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dim- itrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from single image. Proceedings IEEE Conf. on Computer Vision and Patlern Recognition (CVPR), pp. 10975-10985, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 2rXiv:2209.14988 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp.
PMLR, 2021.
11

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-364990 2022.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp.
2019.
Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 84-93, 2020.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-25200 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12619-12629, 2023a.
Junying Wang, Jae Shin Yoon, Tuanfeng Y Wang, Krishna Kumar Singh, and UIrich Neumann. Complete 3d human reconstruction from single incomplete image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8748-88888 -8758 2023b.
Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael Black. Icon: implicit clothed hu- mans obtained from normals. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13286- 13286-13296. IEEE, 2022.
Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J Black. Econ: Explicit clothed humans optimized via normal integration. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 512-523, 2023.
Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang, Hao Xu, and Zhaoxin Fan. D-if: Uncertainty- aware human digitization via implicit distribution field. Proceedings of the IEEF/CVE Inter- national Conference Computer Vision, pp. 9122-9132, 2023.
Tao Yu, Zerong Zheng. Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rghd sensors. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR2021) I), June 2021.
Chao Zhang, Sergi Pujades, Michael Black, and Gerard Pons-Moll. Detailed, accurate, human shape estimation from clothed 3d scan sequences. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu. Deepmulticap: Performance capture of multiple characters using sparse multiview cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6239-6249, 2021.
12

---

Published as a conference paper at ICLR 2025
A
APPENDIX
roulinage
Ouruus rnsonstucceedd
VS
(Enlrgge)
IN) Inpul Image
Cutouta Disenanginng Diluser
[:4] Dssernnng Oubada From ECON
VS
VS
BOOWs
VS
Inpul
Dtsentarging Urtuser
Uheentargled Duguuti trom ECUN
(60 Our fial 3D mosh
(68) ECONs
180 Dupuu Einertang DXEinw
(ei) Disemtangcss giod Curpurs
(4) Dariiin 30 meath
(CR] EOONO
The first four rows pertain our results on real images sourced from Adobe Stock. The last row pertains to our results on an image that is rendered from group of humans that are physically in the same location during the scanning process.
Figure 10:
A.1 PERFORMANCE OF DISENTANGLING DIFFUSER (D) FOR DIFFERENT TYPES OF EXTERNAL OCCLUSION
In Fig. 11 and Fig. 12
show the outputs of under different types of external occlusions.
A.2 HOw DISENTANGLING DIFFUSER (D) DEALS WITH INPUT IMAGES WITH LESS OR MORE THAN THREE HUMAN SUBJECTS
The Human Subject Selection Map is a 64x64x1 binary map that consists zeros and ones. has three possible patterns. In pattern 1, it is filled with all zeros except for the first (leftmost) 21 columns of the map. In pattern 2, it is filled with zeros except for the columnss between the 21st and the 42nd row. pattern 3, it is filled with zeros except for the last 22 columns. Pattern indicates to D that the leftmost human subject in the given input image is to be extracted. Pattern 2 indicates the middle human subject, and Pattern indicates the rightmost human subject. is important to point out that the leftmost human subject does not need to occupy the leftmost 1/3 of the given input image. The subject can positioned anywhere on image, so long as it is the leftmost person in the given input image. This applies to the middle and rightmost human subjects as well.
we feed in Pattern 1, Pattern 2, and Pattern 3 into separately, then we will obtain different output images. However, there may be situations where there are less than three subjects in the given input image. As shown Fig. [2] when there are less than three subjects the given input image, then more of the output images from D will be blank. Specifically, if there only two subjects, then the output image generated using Pattern will be blank. there are only subject, then both output images generated using Pattern 2 and Pattern 3 will be blank.
On the other hand, if there are more than three human subjects (e.g. 5) in the given input image, off-the the- human instance segmentation model to detect the rough positions of the
13

---

Published as a conference paper at ICLR 2025
Input Image
Output from
Groundtruth
Figure 11: Results of in different types of External Occlusion
Input Image
Output #1 from D
Output #2 from D
Output #3 from D
Figure 12: Results of D in Inter-person Occlusions.
subjects and then split the image into smaller images that contain and subjects respectively. The two smaller images are then separately passed into our proposed model pipeline (starting with
D). An actual example is given in the first two rows of Fig. 20
14

---

Published as a conference paper at ICLR 2025
A.3 How D DEALS WITH VERTICALLY OCCLUDED HUMANS
Our paper, like other SOTA papers like ECON et al. ) and the work by Wang et al. (002bb) show primarily "horizontally occluded human(s)" instead of "vertically occluded human(s)" Verti- cally occluded humans refer to images where the top or bottom (rather than left or right) of a human subject occluded by another human subject or an object.
It is not true that our D cannot deal with vertically occluded human(s). During training, we can simply give pattern (Human Subject Selection Map) to our D and ask produce the topmost human subject in the image of vertically occluded human(s) during training, and it would work well during testing because latent diffusion models do not fail just because the objects of interest in an image is arranged from top-to-bottom instead of from left-to-right.
The paragraph above assumes that an image of "vertically occluded human(s)" features humans that are perfectly aligned in a vertical line. Because if the humans are not aligned exactly as such, then our proposed method will already be able to address that image without additional training. In reality, we find this possibility of perfect alignment to be very rare. Most papers do not consider images of such perfectly aligned "vertically occluded human(s)". For example, both ECON and the work by Wang et al. do not feature humans that are perfectly aligned in a vertical line. In fact, both of them use only images with "horizontally occluded human(s)", just as we did in our paper.
A.4 THE 8 NORMAL MAPS (MULTI-LAYERED NORMAL MAPS) GENERATED BY PENETRATING DIFFUSER (P) AND MORE RESULTS FROM P
The multi-layered normal maps consist of 8 normal maps. We illustrate how these 8 normal maps are rendered in Fig. [3 The groundtruth 3D clothed human mesh rendered from different camera directions (front, back, right, and left). For each camera direction, in the first normal vector encountered by a camera ray (horizontal arrow in the figure) but also the second normal vector well. The first normal vectors are placed in the 'Outer' normal map while the second normal vectors are placed in the 'Inner' normal map. In total, we will obtain normal maps, and these maps reveal the topology of the human subject different layers (i.e. inner and outer) and at different camera angles.
are interested not only
In Fig. 23 we show more examples of the multi-layered normal maps produced by our P. This figure also demonstrates the effectiveness of P in tackling internal occlusion such as self-occlusion and non-frontal pose of the human subject. Specifically, if we look at the left example in Fig. [33] we notice that the human subject has occluded his face with his left arm (i.c. self-occlusion) in the input image. In order to resolve the self-occlusion, our generates the 'Back Inner' normal map. This normal map, as shown in the figure, gives us predicted information of the human subject's face. This information will be passed to the Layered-Normal will use the information to reconstruct the geometry of the human subject's 's face. If we consider different scenario where the human subject occludes the right side of his face with his right arm, then the 'Left Inner' normal map will be pivotal for reconstructing the right side of this face (e.g. his right ear). Our P is designed to generate eight different normal maps because there are many different possible body poses that may result in self-occlusion at different view directions, and the eight normal maps ensure robustmess to a wide range of body poses. In addition, besides self- occlusion, the eight normal maps also address non-frontal pose problems. Given an input image that show a human subject at a view direction v, we can expect the generated human avatar to appear to be well-constructed when we view (the avatar) at the view direction v. However, the avatar may not be celt-constructed at other view directions, and this is demonstrated by ECON's results in the 1st row of Fig. [ and the 1st row of Fig. 7 The right and left views of ECON is not well-constructed when ECON is given the front view of human subject. This is because ECON (Xiu et al. 2023) only predicts and uses the front and back normal maps, and not the left and right normal maps. For this reason, our P generates front, back, left, and right normal maps to ensure that a 360 degree view of our generated avatar will be well-constructed. The effectiveness of this can be observed in our results in the same 1st row of Fig. 2 and the 1st row of Fig. 7
Pixel-aligned Implicit Model, which
As aforementioned, SMPL-X normal maps are fed into P. For P to predict the Front Outer, Front Inner, Back Outer, and Back Inner normal maps of clothed body. we provide P with the Front Outer, Front Inner, Back Outer, and Back Inner normal maps of a SMPL-X mesh to use as inputs. In other words, there is correspondence between which SMPL-XX normal map is given to P and
15

---

Published as a conference paper at ICLR 2025
which clothed body normal map is returned by P. In Fig. 14 show what would sometimes happen we did not give P one of the four SMPL-X normal maps. To put into words, when P trained without the correct, corresponding SMPL-X normal maps, will sometimes produce incorrect normal map as one of its outputs. For example, Fig. [hows such a P producing a Front Outer normal map when it is actually supposed to produce Back Inner normal map instead.
Clothed Mesh
Clothed Mesh
Clothed Mash
Clothed Mosh
Front Outer
Back Outer
Right Outer +
Left Outer
+
Front Inner
Back Inner
Right Inner
Left Inner
Figure 13: The multi- layer-layered normal maps of clothed human body (i.e. Nc) consists of normal maps: Front Outer, Front Inner, Back Outer, Back Inner, Right Outer, Right Inner, Left Outer, and Left Inner. Each camera ray (horizontal arrow) aligned according
camera direction. Normal vectors are shown as short orange arrows. The first normal vector captured by the ray goes to the 'Outer' map, and the second normal vector captured goes to the 'Inner' map.
Incorrect Rear Inner Map
Correct Rear Inner Map
VS
It is a Front Outer Map instead
Output Penetrating Diffuser Without using the Corresponding SMPL-X Normal Map
(b) Output Penetrating Diffuser P With the Corresponding SMPL-X Noraa Map
Figure 14: Ablation of our Penetrating Diffusor's 's use of SMPL-X normal maps
16

---

Published as a conference paper at ICLR 2025
A.5
How MULTI-DIMENSIONAL LAYERED NORMALS GRID IS OBTAINED.
We illustrate how our Multi-dimensional Layered Normals Grid is obtained in Fig.
107
(Replicale timesi its direction)
Front Ouler Normal Map
Normal Map
2:1
164
E21
181
2.8
(Rotata to algnn with is camera direction)
(Raplicata timas along its camera direccion)
Right Outer Normal Map
Right Normal Map
Lastly; put the replicated together 3D Grid:
817
(Logical-AND of non empty grid elements)
3.43
9.08
Multidimennionall Layered Normals Grid
3D Vieual Hull
Figure 15: The multi-layered normal maps of clothed human body (8 maps) are aligned in 3D grid to form the Multi-dimensional Layered Normals Grid. In the illustration above, we only show how the Front Outer Normal Map and Right Outer Normal Map are used to form the grid, but in reality all maps are used. Only Front Outer Normal Map (shown above) and the Front Inner Normal Map (not shown) do not need be rotated to align with their camera direction in the 3D grid because the 3D grid is aligned with the camera direction of these front-facing normal maps. 128. In addition, each grid element
In work, Normals Grid is a 24-Iength vector, so the grid has a shape of (24,128, 128, 128). we apply a logical-AND operation along the 24 channels to extract the non-empty grid elements (or filter out the empty grid elements), then we obtain a 3D Visual Hull of clothed human subject. In practice, we do not manually perform the Logical-AND AND operation to form a visual hull. Instead, we leave to the neural network to learn this piece of computlc information itself.
the lulti-dimeesional Layered
A.6 MORE ON OUR DIFFERENTIAL OPTIMIZER
our case, however, we cannot use ICON's differential optimizer directly as we have layered normal maps rather than regular normal maps. Thus, we extend its design and compute the loss between our layered normal maps and the layered surface normals of the clothed human mesh. In addition, the design is also extended such layered normal maps from different camera angles allowed to be included during the deformation or adjustment process of the mesh
17

---

Published as a conference paper at ICLR 2025
Normal Map (512 x 512)
Noised Normal Map (512 512)
Refining Diffuser (R)
Disentangled x 512)
Refined Normal Map (1024 1024)
SMPL-X X Normal Maps with Masking (512x512)
Figure 16: Our Refining Diffuser (R). The 'SMPL-X Normal Map with Masking" is our Ng but out of maps are masked with zeros. In other words, only its maps unmasked, and this 1 unmasked map corresponds to the normal map that will be super-resolved.
A.7 MORE ON OUR REFINING DIFFUSER (R)
Besides external and internal occlusions, another common problem in in-the-wild images is the low resolution of the given input image. This can occur when the human subject is far away from the camera or when the camera set to capture more than one human subject. The low-resolution input image may cause existing methods and our Robust-PIFu to produce a clothed human mesh that lacks high-resolution details. For our Robust-PlFu, a low-resolution input image may lead to a clothed human mesh with low-resolution details because the low-resolution input image shares the same resolution as our Nç- For example, if the resolution of the given input image only 256x256, then the resolution of No will also be 256x256. A low-resolution Ne limits the ability of our differential optimizer to refine a clothed human mesh. This is because the differential optimizer directly relies on Ne to guide the refinement process.
To overcome this, we propose our Refining Diffuser (R). The purpose of R is to perform super- resolution No that differential optimizer can add high-resolution details to clothed human mesh despite a low-resolution input image.
To perform normal map super-resolution, which is under-constrained problem, again turn to a pretrained large-scale latent diffusion model that has already acquired knowledge from billions of images. Starting from the pretrained model, inject it with three architectural modifications so that we can include three different conditional priors: 1. A disentangled image 2. A noised normal map SMPL-X normal maps with Masking. Refer to Fig. [6 for illustration. The disen- tangled image conditional prior is incorporated using cross-attention (after transforming into CLIP image embedding) and channel concatenation. The other two conditional priors are incorporated via channel concatenation only. With these conditional priors, the task of our R is to denoise latent embedding and then decode that denoised latent embedding into a high-resolution normal map. can be used super-resolve any of the 8 normal maps in our Nc-
The 'noised normal map' conditional prior obtained by adding Gaussian noise to a normal map from Nc. The reason for adding the noise is because, during testing of Rt the normal map from Ne will be prediction of our P and is thus not perfectly accurate. The noise nudges R away from fully depending the predicted normal map before producing the super-resolved normal map. This reduces propagation error that may derive from P.
Each run of R will super-resolve normal map from Nc. Since No consists of 8 normal maps, will need to run R eight times to super-resolve the entire Nc.
In each run, need to indicate to R which of the 8 normal maps should R produce. This is done using the 'SMPL-X normal maps with Masking" conditional prior. 'SMPL-X normal maps with Masking is our Nş but 7 out of its maps are masked with zeros. In other words, only of its
18

---

Published as a conference paper at ICLR 2025
maps is unmasked, and this unmasked map corresponds the normal map from Ne that will be super-resolved.
After 8 runs of R, will obtain a super-resolved Nc. which will be used in differential op- timizer for adding fine, high-resolution details to a clothed human mesh. By doing so, we ensure that our Robust-PIFa is able to produce clothed human mesh with high-resolution details despite a low-resolution input image.
In summary, a formal formulation of R is represented in the following equation:
Nº R(N / Ng)
(4)
where 1 is the disentangled image of the human subject, and N' refers Ne added with Gaussian noise.
In Fig. 24 show examples of the high-resolution normal maps produced by R. This figure also demonstrates the effectiveness of It in super-resolving the normal maps in No.
(a) Input Image Figure 17: Ablation of using and not using it.
(b) Without R
(c) With R
Evaluation and Ablation of our Refining Diffuser (R). To evaluate our Refining Diffuser (R), compare the 3D clothed human meshes produced by RobustPIFu when R is not used and when R is used Fig. 17 It is clear that when is used, the refined clothed human mesh has also conducted of using and not using the noised normal map in R. As shown in Fig. [8] there is a clear benefit using the noised normal map. The reason for this is that, by noising the input normal map, we give R the freedom to fix any flaws that may be present in the given input normal map. In other words, without noising, if the input normal map is flawed, then will reproduce and magnify the flaw in higher-resolution On the other hand, noising the given normal map informs R that there are inherent noises in the input normal map, and the diffuser is required to remove these noises. In the process of doing so, the diffuser actually fixes any flaw in the input normal map as well.
higher resolution details. In addition,
ablation study investigate the impact
A.8 RESULTS ON INTERNET IMAGES FROM SHUTTERSTOCK
In Fig. 19
we show the results of our RobustPIFu and ECON on Internet images from Shutterstock. Like all our other qualitative results, this figure shows that our RobustPIFu excels at constructing re- gions that are abscured/unseen from the given input images. Specifically, unlike ECON, RobustPIFu able to reconstruct realistic surfaces the rear/back of the clothed human meshes.
A.9 RESULTS ON IMAGE WITH MORE THAN THREE HUMANS AND ON IMAGES FROM THE MULTIHUMAN AND BUFF DATASETS
In Fig. 20 show the results of our RobustPIFu and ECON real image with than three human subjects, a rendered image from the MultiHuman dataset, and a rendered image from the BUFF dataset.
19

---

Published as a conference paper at ICLR 2025
VS
(a) Output Refining Diffuser Without using Noised Normal Map
Output Refining Diffuser R With Noised Normal Map
Figure 18: Ablation of our Refining Diffusor's use of Noised Normal Map
Ro
(a) Input Image
(b) ECON
(c) Ours
Figure 19; Results on Internet images from Shutterstock.
A. 360 DEGREE VIEW OF OUR RESULTS
In place of video, we show 360 degree view of our results using series of images in Fig. 21
A.11
WHY MULTIIPERSON AND RELATED SCANS FROM THE MULTIHUMAN DATASET CANNOT BE USED FOR EVALUATION.
As mentioned earlier, only single human scans from the MultiHuman dataset is used for evaluation. The multi-human scans are excluded. The reason for this is because the multi-human scans con- tain gaping holes that made them impossible to use for evaluation (the CD and P2S scores will be inaccurate and misleading). This problem is illustrated in Fig. 22 This "gaping hole" problem is also true for 3D scans with human-object interactions in the MuliHuman Dataset. Hence, the only way the MultiHuma Dataset in experiment is to use only the 3D scans that have
20

---

Published as a conference paper at ICLR 2025
(See next row)
Image (Nfrow Than Furmon
101 Autonciccllyy di-do imggo inta Flages coarse sufficctt
Disentardling Dificer
to obeinn indivduel images the 9r human subjoct.
VS
ECON:s Humrnn Butgect
Human
ECON's Human SUijNE
ECON's Humon Subject
Buugout
Susjec
Budject 93
VS
Vs
Siweringing
Our
160
VS
ONI Irpue imagge
100 Curpurs of Oisentanging Dfllser :0t
Disentangled from ECON
(dl that
BOONS
vs
Image
(S Irpur trraga
(N) Darr fres 3D maKe
O40 ECONN
Figure 20: Additional Results. The first two rows illustrate how a real image (from Adobe Stock) with more than three humans will be handled by our RobustPIFu. The third and fourth rows show results from the MultiHuman dataset, and the fifth and sixth rows show results from the BUFF dataset.
Figure 21: 360 degree view results from our RobustPIFu. Results using two different input images shown.
human-object and Human-huuman interactions, and these are the "single human scans" that we used our paper.
A.12 COMPUTATION TIME REQUIRED BY OUR ROBUST-PIFU
Short inference time is not a claim made by our paper. However, we acknowledge it is important that our Robust-PIFy does not have much longer inference time compared to existing models.
21

---

Published as a conference paper at ICLR 2025
(/)TT Mesh Exangle 1 (Togethor) (al) OT Mosh Bianple ((ndMoouas))
(0) Mesh Barsle 2 (Tagether)
(e)) GT Mesh Exmmpi (inhiius
Figure 22: Gaping holes observed in the multi-human scans from the MultiHuman dataset. These holes made infeasible to use multi-human scans from the dataset for evaluation purposes.
Thus, in order to reassure readers, we compare our Robust-PIFu with ECON (Xiu et al. 2023) in terms of the average time required to produce a clothed human mesh during testing. From this experiment, we observe that Robust-PlFu takes 126.42 seconds, while ECON takes 138.1 11 seconds. The 126.42 seconds include the time required for our and P generate their outputs, and the time required to construct the Multi-dimensional Layered Normals Grid. During the experiment, the same hardware (1 NVIDIA RTX A5000 GPU) is used to test both Robust-PIFu and ECON.
A.13 OTHER LIMITATIONS
Our proposed method uses generative models, which may result in hallucinations (e.g. predict clothes wrinkles that are not present at the back of a human subject). We show few examples of this in Sect. A In addition, we did not use the RenderHuman dataset for training or evaluation because is a commercial dataset that extremely costly to access and thus unavailable us.
A.14 IMPLEMENTATION DETAILS
Our D, P. and R are initialized using the same pretrained latent diffusion model that was employed in Zero- 1-to-3 (Liu al. 20236). This latent diffusion model is pretrained on the LAION datasets
(Schuhmann et al. 2022).
The pretrained latent diffusion model finetuned with THuman2.0 images, and this is done with an AdamW (Loshchilov & Hutter 2017) optimizer that has learning rate of 10 The batch size used 72. The training images have resolution of 5 512x512, and we use a latent dimension of 64x64x4. This means that our D aims to denoise 64x64x4 noised encoding that represents an image of human subject. In addition, this also implies that the input image I has be encoded into a 64x64x4 encoding. Also, the denoised encoding generated by the denoising U-Net will be decoded into 512x512 "disentangled' image.
During inference, we use 200 DDIM sampling steps.
As for our Hayered-Normal, Pixel- aeel-aligned Implicit Model. it is trained using a RMSprop optimizer that has an initial learning rate of 10-3 Its encoder is a stacked hourglass network (Newell et al., 2016) with 4 stacks. Its MI. has layers of the following dimensions: (257+64+24, 1024, 512, 256, 128, 1). The 3rd, 4th, and 5th layers have skip connections. The '+64' is for the features generated by the 3D-CNN that processes the Multi-dimensional Layered Normals Grid, and the '+24' for the skip connection that directly feeds values from the Multi-dimensional Layered Normals Grid into the MLP. The design of the MLP follows the original PIFu model (Saito et al.] (2019), except that its input dimension has been increased by total of 88 to accommodate the features generated by the 3D-CNN and the values fed by the skip connection.
Every component of Robust-PIFu is finetuned trained using NVIDIA RTX A5000 GPUs. More implementation details can be found in our source code, which will be made publicly available.
A.15 ADDITIONAL ABLATION STUDY ON LAVERED-NORMALE PIXEL-ALIGNED IMPLICIT MODEL.
In this subsection, we discuss the inclusion of Ns in our Layered-Normall Pixel-aligned Implicit Model. We observe in Tab. 3 that it was beneficial to include Ns instead of excluding it. We believe that it was due to the fact that Nşs does not contain pixels that pertain to clothes or hair. The combination of Nş and Ne thus gives our pixel-aligned implicit model information of which pixels belong to clothes and hair. Clothes and hairs in a 3D mesh are often thin rather than thick, and this information is helpful for guiding the pixel-aligned implicit model in constructing the 3D mesh.
22

---

Published as a conference paper at ICLR 2025
Table 3: Additional Ablation study on Layered-Normals Pixel-aligned Implicit Model.
THuman2.0
Methods
CD (10*)
P2S (10*)
CcobustPPIFy w/ Multi-D Gndd (No Nş)
8.643
8.227
Robust-PlFu w/o Multi-D Grid (With Nş)
8.551
8.132
Input Image
Output from P
Groundtruth
Input Image
Output
Groundtruth
Front Outer
Front Outer
Front Inner
Front Inner
Outer
Back Outer
Back Inner
Back Inner
Right Outer
Right Outer
Right Inner
Right Inner
Left Outer
Left Outer
Left Inner
Left Inner
Figure 23: Additional Results on the Outputs produced by our P
23

---

Published as a conference paper at ICLR 2025
Input Image
Output from
Groundtruth
Input Image
Output from
Groundtruth
Front Outer
Front Outer
Front Inner
Front Inner
Back Outer
Back Outer
Back Inner
Back Inner
Right Outer
Right Outer
Right Inner
Righr
Left Outer
Left Outer
Left Inner
Left Inner
Figure 24: Additional Results on the Outputs produced by our R. The input images shown above 512x512 in resolution while the normal maps ('Output from R' and 'Groundtruth") are 1024x 1024 in resolution.
24

---

Published as a conference paper at ICLR 2025
Query Point (P)
Indexiing using x and y coordinates of
coordinate
insidel
outside
Concal
Multilayer Perceptron (MIP)
Encoder Network
Feature Vector
Input Image
Set of Feature Maps
Figure 25: Pipeline and Architecture of generic Pixel-aligned implicit model. An input image is processed by an encoder network into a set feature maps. Query points are used to index the set of feature maps retrieve feature vectors. The feature vectors then fed into a MLP that will predict the occupancy of the query points.
A.16 DETAILED EXPLANATIONS ON RELATED WORKS AND CONCEPTS
A.16.1 DETAILED EXPLANATIONS ON PIXEL-ALIGNER IMPLICIT MODELS
In this section, we will explain how a vanilla pixel-aligned implicit model works. The concepts explained in here derived from the PIFu paper (Saito al., 2019), which is one of the earliest works on pixel-aligned implicit model.
show the pipeline of generic pixel-aligned implicit model. At the start of the pipeline, an input image is fed into an encoder network, which can be any 2D Convolutional Neural Network. Usually, stacked hourglass network used the encoder network. The encoder outputs a set of feature maps. The set of feature maps, as whole, has the shape of (C, H. W), where C is the number of channels, H is the height of a feature map, and is the width of a feature map.
In Fig. 25
Spatially, we say that the set of feature maps is pixel-aligned with the input image because each pixel on the input image can be spatially mapped other words, the set of feature maps corresponds to the 3D camera space of the input image.
feature vector on the set of feature maps. In
To train the pixel-aligned implicit model, 3D points are sampled within the 3D camera space of the input image, and these 3D points are called query points. For each query point, its x and coordinates are used to index the of feature maps. After indexing, each query point will retrieve corresponding feature vector. The Z coordinate of the query point is then concatenated with that feature vector before being used as input in a Multilayer Perceptron (MLP).
Finally, the MLP will output the estimated occupancy of that query point. The occupancy of query point refers to a binary prediction of whether the query point located inside or outside a groundtruth 3D human mesh that has been transformed into the camera space of the input image. In practice, the occupancy predicted by the ML.P is continuous value that has a range between 0 and 1. The value of 0.5 is interpreted where the mesh surface is located at. Also, any value between 0.5 and 1 is interpreted as being inside the mesh and any value between O and 0.5 is interpreted as being outside of the mesh.
During inference, a 3D grid of query points will be fed into the pixel-aligned implicit model's pipeline, and 3D grid of occupany predictions will be obtained. Then, the Marching Cubes algo- rithm Lorensen & Cline (1987) will be applied on the 3D grid of occupancy predictions in order to extract predicted 3D human mesh.
A.16.2 DETAILED EXPLANATIONS ON THE ZERO- 1-T0-3 MODEL.
Zero-1-to-3 (Liu et al. 2023b) is a work that proposes the use of large-scale diffusion models in the problem of Single-view 3D Object Reconstruction. In their work, the authors of Zero-1-to-3 demonstrated that a large, pre-trained latent diffusion model can manipulated to produce a novel view of an object from any specified viewpoint. This can be done by finetuning the pretrained model with an input RGB view of the object, a relative camera transformation, and a transformed RGB view of the same object. After being finetuned, the model, when provided with input view of the object, can generate numerous novel views of that object. These new views are then utilized to reconstruct 3D mesh of the object using Score Jacobian Chaining (SJC) (Wang al. 2023a).
25

---

Published as a conference paper at ICLR 2025
Input View
Zero-1-to-3 Model
T)
Novel View
Gaussian Noise
Figure 26: Pipeline Zero-1-to-3. An input view and a camera extrinsics (R,T) are used by the Zero-I-to- model to predict novel view of an object. The Gaussian noise is required Zero-1- 1- to-3 uses latent diffusion model.
The Zero-I-to- model can be condensed into the following equation:
(5) Where मस, denotes the synthesized image at novel viewpoint, x denotes the given input image. and T refers respectively to the relative camera rotation and translation between and RR, The aim of Zero- Zero-I-to-3 is to predict a JR, that is perceptually similar to the groundtruth novel view TRT In addition, provided an illustration of Zero-1-to-3 in Fig. 26
f RT), Re R
TE
To train Zero- 1 the authors prepared a dataset of paired images and their relative camera extrin- sics ie. {I,,TR,T R. }. Zero-1-to-3 used latent diffusion architecture with an encoder (denoted as E), denoising U-Net (denoted as 60), and decoder. The diffusion time step ranges from to 1000. 7ero-1-to-3 aims solve the following objective function during the finetuning process:
mjn Ex,
c0(z, c(2, R, T))
After the finetuning process, Zero-1-Be 1-to-3 (or f) will generate an image by performing iterative de- noising from Gaussian noise image by using the conditional prior c(x, Rt T", which is an embed- ding of the input image and the relative camera extrinsics.
A.17 ANALYSIS OF FAILURE CASES
In the first row of Fig. 27 we observe that the woman's occluded left arm is being generatively filled up by Disentangling Diffuser (D). But if inspect the input image closely, we
Input Image
Output #1
Output #2
Figure 27: Analysis of Failure Cases. The red circled regions indicate the erroneous
26

---

Published as a conference paper at ICLR 2025
that fingers from the woman's 's left hand actually grabbing the man's waist. Thus, the left arm predicted by our D has an incorrect position i.c. the left arm should have been straightened, and the fingers should be spread out.
In addition, in the second row of Fig. 27 there is line of shadow in the output generated by our D. This shadow is hallucinated and an error of D.
These errors are due to insufficient training data. In-the wild test images have challenging body poses and novel objects that cause occlusion. Consequently, the training data used must be large and diverse enough so as to ensure D can cope with such test images. However, such training data be challenging obtain because there is lack of existing datasets that address such occlusion problems.
A.18 FUTURE WORKS
For our future works, we intend to address the failure cases that we raised above by building large- scale dataset that addresses external and internal occlusion problems. We believe that such a dataset will allow us to train ours and related models more effectively and an important contribution to this field.
Moreover, we will also be exploring if the occlusion removal can be given a finer degree of control. Specifically, rather than just ensuring that the occlusion is removed, we want to specify how it will be removed. For example, if there an occluded hand in the input image, we wish to control the hand pose of the hand that we generated replace the occluded hand.
27


Reviews:
[{"review_id": "vh4Zdl3ELf", "reviewer": "Reviewer_BNoQ", "summary": "This paper proposes a robust method for single-image human reconstruction by building upon the PIFu framework and addressing its limitations. The improvements focus on four main aspects. First, a diffusion model is employed to remove external occlusions from the image, ensuring a clearer view of the target human. Second, another diffusion model is used to predict multi-layer normal maps to mitigate the impact of self-occlusion. Third, the pixel-aligned implicit model is modified to use both the original image and the multi-layer normal maps as inputs for predicting 3D occupancy labels. Finally, an optional refinement diffusion model is introduced to enhance the resolution of the normal maps if finer details are required.", "questions": "Please justify the design of the Penetrating Diffuser (see Weakness 1 and 2). Please justify why inner and outer normal maps are needed, and why separate models for front-back and left-right are needed.\n\nPlease also add more explanations to the architecture of the Pixel-Aligned Implicit Model (see Weakness 3).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach significantly improves the robustness of PIFu by addressing specific limitations, such as handling occlusions and introducing normal maps. The modifications are logical and well-justified, leading to more accurate reconstructions.\n2. The method demonstrates better performance on the BUFF and MultiHuman datasets compared to existing approaches, underscoring its effectiveness in general human reconstruction tasks.", "weaknesses": "1. The necessity of eight distinct normal maps (both outer and inner maps for each direction) in the Penetrating Diffuser is not fully intuitive. It is unclear why both outer and inner maps are required for each direction, as they may simply represent flipped orientations. Further explanation on the additional information provided by the inner normal maps would enhance understanding.\n\n2. The need to run the Penetrating Diffuser separately for the front-back and left-right orientations introduces significant computational overhead. The left-right normal map prediction is conditioned on the front-back prediction; however, simultaneous prediction of all eight maps could potentially enhance the conditioning across all pairs. This two-stage execution doubles the computational load, requiring two diffusion models and multiple sampling runs, which could be optimized.\n\n3. Clarification on MLP in Pixel-Aligned Implicit Model (Section 3.3): Additional clarification on the structure of the MLP in the Pixel-Aligned Implicit Model would be beneficial. Specifically, it is unclear if this MLP follows the structure of the original PIFu model, and whether all normal maps and features are concatenated directly as inputs to the network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iUP6st0F7Y", "reviewer": "Reviewer_fnHJ", "summary": "This paper propose Robust-PIFu, a novel approach designed to reconstruct 3D human models from a single image, especially under the occlusion setting. By harnessing the power of large-scale pre-trained latent diffusion models： a disentangling latent diffusion model and a penetrating latent diffusion model, Robust-PIFu effectively separates each human and inpaints the occluded areas of the human. Subsequently, a layered-normals pixel-aligned implicit model, coupled with an optional super-resolution mechanism, is employed to reconstruct the 3D clothed human model. Experimental results demonstrate that Robust-PIFu outperforms current state-of-the-art methods in both qualitative and quantitative evaluation.", "questions": "1. I am curious about the reconstruction performance of Disentangling Diffuser on real images when there are significant areas of missing. Specifically, how well can it reconstruct the head portion if it is occluded?\n2. In the images presented in the paper, the characters are arranged side by side. If the characters were arranged vertically, such as one person standing and another crouching in front of them, would the disentangling strategy of Disentangling Diffuser still be effective?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Robust-PIFu provides a novel solution to the problem of external and internal occlusions in 3D human reconstruction from single image by means of pretrained diffusion models.\n2. Extensive experiments prove the effectiveness of the proposed methods.", "weaknesses": "1. The writing and the presentation of the paper need improving.\n2. Robust-PIFu seems to be primarily effective in reconstructing 3D humans from laboratory datasets. Its generalizable experiment in real image reconstruction scenarios is simple, as it lacks more complex real image reconstruction results.\n3. Lack of the training details of Disentangling Diffuser.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5pdxMMhKH", "reviewer": "Reviewer_qbXS", "summary": "The authors propose a method to reconstruct 3D human(s) from a single image by leveraging pretrained latent diffusion models and implicit function representation. The models are utilized for disentangling external occlusions (overlapping people), recovering from internal occlusions (viewpoint specific missing details) and an optional normal map super-resolution module to improve fidelity. The diffusion models+implicit models are trained using Thuman2.0. The method is evaluated on MultiHuman and BUFF datasets and show superior quality and performance compared to prior works.", "questions": "- Fig. 3: consider adding a caption.\n- Fig. 4 caption has incorrect layout for the titles.\n- Disentangling Diffuser: \n  * How well does the disentangling diffuser work for overlapping humans at different depths? I presume the training data had humans at similar depths. \n  * Were any augmentations applied to add external occlusions to human images? L306 is not clear about this.\n- Penetrating Diffuser: \n  * What is the reason behind generating the normal maps in two parts? Is it mostly due to memory and compute or due to bad predictions?\n- Layered Normals PIFu model:\n  * What is the need for adding SMPL-X normal maps N_s here? Since all the required information to reconstruct the human is present in N_c. An ablation on this would be useful.\n- Refining Diffuser:\n  * Why is R run 8 separate times to obtain the super-resolved normal maps? Would it be possible to super-resolve all 8 at once or 4 at a time (similar to P's output format)?\n- Fig. 7 (Row 4) - Show results for both humans.\n- For ablations on P (L481) and the implicit model (L492), what are the inputs to the implicit model in each setting?\n- For ablation on P (Fig. 8), The hidden arms of the person in each image should have been reconstructed correctly if P outputs the correct normal maps. Does P fail when there are multiple people in the input image since its OOD?\n- Fig. 22: It would be interesting to show results on these examples. This would show that even though the GT mesh has holes, the method can reconstruct plausible surfaces.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is simple and the writing is easy to follow. To highlight specific positives:\n- Leveraging open-source diffusion models: This method highlights the ability of large open-source pretrained diffusion models to tackle 2D human inpainting (disentangling diffuser) and cross-domain conditional image generation (penetrating diffuser). The pipeline chains the outputs in a straightforward manner and obtains good quality results.\n- Superior result quality: From the evaluations, it is quite clear that prior work cannot address the problem of inter-penetrating and self-occluding human reconstruction. This method achieves much higher quality reconstructions that reflects in the qualitative and quantitative experiments.\n- Experimental results and observations: The authors have provided extensive results and ablations between the paper and the appendix which is quite appreciated.", "weaknesses": "The approach overall is quite straightforward. However, I have a few concerns regarding the details of the method. Specifically:\n- Need for SMPL-X mesh: The paper mentions that prior work ECON and Wang et al. require accurate SMPL-X meshes for good predictions which Robust-PIFu does not need (L51-53). In fact, not using the SMPL-X model causes the model to have errors as shown in Fig. 14. Please clarify this point and show results when the SMPL-X input has some error.\n- Method's robustness to diverse data: The results in the main paper are quite impressive. But, I would urge the authors to add more qualitative results on a larger diversity of data to get a more holistic view of the method's robustness (to live upto its name :) ).:\n  * Ex. real-world images under multiple settings (single/multi-humans with **varying degrees of occlusion**).\n- Predicted normal maps N_s: What is the reason behind choosing an arbitrary number 8? There are numerous set of human poses that do not get covered by just 8 maps. How does the method handle such scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7m6fujjQOQ", "reviewer": "Reviewer_8tGb", "summary": "This paper proposes a method to address the common challenges in single-image to 3D human reconstruction, specifically the performance degradation caused by self-occlusion and inter-person occlusion. The authors introduce the Disentangling Diffuser, a diffusion model used for inpainting occluded regions to resolve inter-person occlusion, and the Penetrating Diffuser, which generates multi-layer normal maps to handle self-occlusion. Using these outputs, the paper proposes a PIFu-based structure for 3D human shape reconstruction.", "questions": "1. Why are there no references and comparisons to methods published after 2023?\n2. Is there a specific rationale for defining the Penetrating Diffuser views as front, back, right, and left? Could additional views be used?\n3. How does the inference runtime or computational complexity compare to other methods?\n4. When using the Disentangling Diffuser and Penetrating Diffuser for inference, how many denoising steps were performed, which scheduler was used, and was classifier-free guidance (CFG) applied?\n5. Is there an additional study on the reliability and performance of images generated by the diffusion model (particularly normal maps), given that diffusion model results can vary with different seed values?\n6. How does the viewpoint of a single-view image affect performance, and to what extent does it impact reconstruction quality?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This work takes an approach to resolving self-occlusion and inter-person occlusion without significantly altering the foundational PIFu method. \n\nBy using the divide-and-conquer strategy with the Disentangling Diffuser and Penetrating Diffuser diffusion models.", "weaknesses": "1. The method novelty is limited. The proposed Disentangling Diffuser is identical to the Stable Diffusion [1] inpainting model in both its structure and functionality, making it difficult to consider this as a unique contribution of the paper. Similarly, the Penetrating Diffuser is analogous to ICON [2], which utilizes the SMPL model for normal prediction. It is also challenging to find novelty in the Pixel-aligned Implicit Model.\n2. The paper’s structure is difficult to follow, and the notations lack clarity. In particular, section 3 lacks a formal formulation of the inputs and outputs for each model, and there is excessive repetition regarding input and output dimensions that could be streamlined. Significant revisions are needed for paper quality.\n3. The figure 3,4 and 5 are excessively large and occupy too much space.\n4. Figure 4, figure 8, 10 placement is inconvenient. It reduces readability.\n5. The paper does not reference any methods from 2024, and all comparison methods are from 2023 or earlier. Additional experiments with newer methods are needed.\n6. The experimentation and result analysis are insufficient. In particular, a quantitative evaluation of the inter-occlusion claimed in this paper is necessary.\n7. There is no explanation or reference for the evaluation metrics. A reader who doesn't know the abbreviation for 'evaluation metric' would find it difficult to understand.\n8. The requirement to manually define occlusion regions (mask) limits practical applicability.\n9. There is an insufficient ablation study regarding layer depth of outer and inner normal map, particularly concerning variations in limb length and pose.\n10. There is a lack of detail on which diffusion and PIFu models were used and the training and optimization procedure, making reproduction challenging.\n\n[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models.\n\n[2] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J Black. Icon: implicit clothed humans obtained from normals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)