Generalizable Human Gaussians from Single-View Image
Authors: Jinnan Chen
Chen Li
Jianfeng Zhang
Lingting Zhu
Buzhen Huang
Hanlin Chen
Gim Hee Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Human Gaussians
3D Human Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We design a novel approach to learn generalizable 3D human Gaussians from a single image, achieving both high-quality rendering and accurate 3D reconstruction.

Abstract: In this work, we tackle the task of learning 3D human Gaussians from a single image, focusing on recovering detailed appearance and geometry including unobserved regions. We introduce a single-view generalizable Human Gaussian Model (HGM), which employs a novel generate-then-refine pipeline with the guidance from human body prior and diffusion prior. Our approach uses a ControlNet to refine rendered back-view images from coarse predicted human Gaussians, then uses the refined image along with the input image to reconstruct refined human Gaussians. To mitigate the potential generation of unrealistic human poses and shapes, we incorporate human priors from the SMPL-X model as a dual branch, propagating image features from the SMPL-X volume to the image Gaussians using sparse convolution and attention mechanisms. Given that the initial SMPL-X estimation might be inaccurate, we gradually refine it with our HGM model. We validate our approach on several publicly available datasets. Our method surpasses previous methods in both novel view synthesis and surface reconstruction. Our approach also exhibits strong generalization for cross-dataset evaluation and in-the-wild images.

Published as a conference paper at ICLR 2025
GENERALIZABLE HUMAN GAUSSIANS FROM SINGLE- VIEW IMAGE
Jinnan Chen', Chen Li' Jiinfeng Zhang', Lingting Zhu?, Buzhen Huang', Hanlin Chen', Gim Hee Leel National University of Singapore, The University of Hong Kong jinnan. cēu nus. edu, gimhee. Tochee.leeee Lee@nus. edu.
ABSTRACT
In this work, we tackle the task of learning 3D human Gaussians from single image, focusing on recovering detailed appearance and geometry including unob- served regions. We introduce a single-view generalizable Human Gaussian Model (HGM), which employs a novel generate-then-refin pipeline with the guidance from human body prior and diffusion prior. Our approach uses a ControlNet to re- fine rendered back-view images from coarse predicted human Gaussians, then uses the refined image along with the input image to reconstruct refined human Gaus- sians. To mitigate the potential generation of unrealistic human poses and shapes, we incorporate human priors from the SMPL--X model as a dual branch, propagat- ing image features from the SMPL-X volume to the image Gaussians using sparse convolution and attention mechanisms. Given that the initial SMPL-X estimation might be inaccurate, we gradually refine with our HGM model. We validate our approach on several publicly available datasets. Our method surpasses previous methods in both novel view synthesis and surface reconstruction. Our approach also exhibits strong generalization for cross-dataset evaluation and in-the-wild images. We open-source our code at: https:: //ittbuu com poomijinnas iinann chen/ HGM
INTRODUCTION
Automatic 3D human reconstruction from single image is crucial in augmented and virtual reality (AR/VR), game industry, filmmaking, etc. Previous works rely on strong 3D supervision such as the
signed distance value occupancy (Saito al. 2019 2020 Zhang et al. 2023c Xiu al. 2022
2023 Zhang et al. 2024 Ho et al. 2024) and focus on surface reconstruction, neglecting novel view synthesis quality, resulting in smoothed and blurred textures. With the development of neural radiance fields (Mildenhall et al. 2020). novel view rendering quality has been greatly improved for human appearance modeling (Hu et al. 2023 Kwon al., 202TT: Gao et al., 2022). However, due to the ill-posed nature of single view reconstruction, the back and side views are always blurry and lack details without additional prior. Furthermore, these methods needs large amounts of query points sampled for volume rendering, which hinders practical real-time application in the industrics. Some other methods optimize underlying appearance and geometry from scratch by introducing score distillation sampling during the optimization Tang al. (024b)); [Co al. (2024). Although effective, these methods still suffer from slow optimization and over-saturation problems. Recent 3D Gaussians generation methods (Tang et al. 20224a] Yinghao al.) 2024) combine multi-view
diffusion models (Liu et al.) 2023; Wang & Shi 2023 Shi et al.) 2023 Li et al.) 2024; Xu et al.
2023) with generalizable multi-view Gaussians prediction models to generate 3D Gaussians with high quality and efficiency. We aim to extend this on human reconstruction. However, directly employ such methods to human reconstruction with complex texture and poses gives unsatisfying results due to: 1) Inconsistency across multiple views: The multi-view images generated from diffusion model lacks consistency in appearance and pose across different viewpoints. This inconsistency stems from the inherent complexity of human body structure and movement, which leads to low-quality reconstruction results. 2) Quality loss in front view reconstruction: multi-view diffusion process involves down-sampling and changing the original input image. This step results in significant quality degradation when reconstructing the front view image, compromising the fidelity to the original input. 3) Estimating SMPI--X parameters from single view input is ill- ill-posed, directly applying initially

---

Published as a conference paper at ICLR 2025
Figure 1: Our method reconstructs detailed and geometrically consistent human Gaussian models from single view images, including loosing clothes, challenging pose and in-the-wild images.
estimated SMPL-X can lead to bending legs and wrong elevation issues in previous method Xiu et al. (2022 2023 Zhang et al. (2024); [H et al. (2024).
To address the above-mentioned problems, we introduce a novel Human Gaussians Model (HGM), which supports fast and high quality rendering from single view input, and generalize well to loosing clothes, challenging poses and in-the-wild images as shown in Fig. [ We do not use multi-view diffusion models due to the multiview inconsistency and resolution degradation problem. Instead, propose a coarse-to- to-fine framework, where the diffusion model is adapted to refine back-view images rendered from our predicted coarse human Gaussians. this way, we can keep the resolution and content of the original input image for high-fidelity reconstruction. In order to model the complex structure of a human, we inject the human prior into Gaussian prediction process. Specifically, model consists of two branches: 1) The first branch is UNet to directly predict Gaussians from the input image, as inspired by image splatter (Szymanowicz et al. 2024). 2) The second branch uses learnable tokens attached to SMPL-X vertices for structural feature extraction with attention layers, and then combined with UNet features with SparseConv Graham et al. (2018) and a transformer for Gaussian enhancement. Recognizing the inaccurate estimate of SMPL-X from the pre-trained model, during inference, we iteratively refine the initial SMPL-X parameters with our HGM pre-trained with ground truth SMPL-X. Given the loss of details of the back view by directly predicting the Gaussians from single view, we further apply ControlNet to refine the back view with the control signal from the back-view image rendered from the coarse stage. We then input the original front view and refined back view images to our HGM model to get the final refined Gaussians. Meshes can be extracted from densely rendered depth map and TSDF fusion. Our model can be trained with only posed multiview images without 3D supervision and generalizes well to untrained datasets and in-the-wild images.
In summary, our contributions are:
We introduce a generate- snnerateethen--eee -heeeeeinh----eeeeee pipeline for single view human Gaussian reconstruction that leverages diffusion priors for back view refinement, avoiding the multi-view inconsistencies commonly observed in multiview diffusion models.
Our proposed dual-branch reconstruction pipeline incorporates human priors by attaching Icarnable tokens to the SMPL-X vertices for structural feature extraction. We then fuse these features from the SMPL-X branch with the U-Net branch using Sparse Convolution and transformer.
To address potential inaccuracies in initial SMPL-X estimations, we employ our Human Gaussian Model (HGM) to iteratively refine the estimated SMPL-X parameters, resulting better alignment.
Through extensive experimentation, we demonstrate the efficacy of our method in both novel view synthesis and 3D reconstruction tasks. Our approach consistently achieves state-of- state-of-the-at the-art performance on various metries and benchmarks.
RELATED WORKS
Single-view Human Reconstruction. PIFu (Saito et al. 2019), PIFuHD (Saito et al. 2020). PaMIR (Zheng al. 2021), and GTA /Zhan et al. 2023c are capable of inferring full textures from single
2

---

Published as a conference paper at ICLR 2025
image. Techniques such as PHORHUM (Alldieck et al., 2022) and S3F (Corona al. 2023) go further by separating albedo and global illumination. However, these methods lack information from other views prior knowledge, such diffusion models, often resulting in unsatisfactory textures. TeCH (Huang et al. 2024) utilizes diffusion-based models to visualize unseen areas, producing realistic results. However, it requires time-intensive optimization per subject and is dependent on accurate SMPL-X. The emergence of Neural Radiance Fields (NeRF) has led to methods (Hu et al.,
2023; Huang et al. 2023 Gao et al. 2022, Kwon et al. 202T using videos or multi-view images to optimize NeRF for the capture of human forms. Recent advances such as SHERF (Hu et al. 2023) and ELICIT (Huang et al. 2023) aim to generate human NeRFs from single images. Although NeRF- based approaches are effective in creating high-quality images from various perspectives, they often struggle with detailed 3D mesh generation from single images and require extensive optimization time. More recently, SiTH (Ho et al. (024) proposes to combine back-view hallucination model with an SDF-based mesh reconstruction model. Similarly, SIFU (Zhang et al., 2024) employs a mext-to-image diffusion- these methods require 3D annotations such as the SDF of the meshes and texture maps as strong supervision and still fail to generate renderings with high fidelity due the limited 3D training data and representation. In addition, these methods suffer from SMPL estimation errors, leading to bending legs and wrong elevation of the reconstructed 3D humans. Compared these methods, our approach can be trained solely on multi-view images and achieves much better novel view synthesis quality. Human Gaussians. 3D Gaussians (Kerbl et al., 2023) and differentiable splatting (Szymanowicz et al., (024) have gained broad popularity due to their efficiency in reconstructing high-fidelity 3D scenes from posed images using only a moderate number of 3D) Gaussians. This representation has been quickly adopted for various applications, including imag or text-conditioned 3D generation and avatar reconstruction. Among these methods, Gauhuamn and HUGS (Hu & Liu 2024; Kocabas et al. 2024) are the first to propose optimizing human Guassians from monocular human videos. However, they are not applicable to single static human images. GPS-GausSI Gaussian (Zheng ct al. (024) propose generalizable multi-view huaman Gaussian model with high quality rendering; however, it needs dense views 16 or 8, which cannot be directly applied to single-viss view human images. Our human model achieves strong generalization in generating human Gaussians from single-view images, complementing concurrent work such as Pan et al. (2024).
prior to generating consistent textures for invisible views. However,
Generalizable Gaussians with Multi-view Diffusion. The Large Reconstruction Model (LRM) (Hong et al. 2024) scales up both the model and the dataset to predict a neural radiance field (NeRF) from single-view images. Although LRM is primarily reconstruction model, can be combined with Diffusion Models (DMs) to achieve text-to-3D and image-to-3D generation as demonstrated
by extensions such as Zero ( al. 2023). Image Dream Wang & Shi. 2023) Instant3D (Li
let al. 2024) and DMV3D (Xu et al. 2023). Our method also builds on strong reconstruction model and uses pre-trained 2D DMs to provide input images missing information in a feedforward manner. Some concurrent works, such as LGM (Tang al. (024a). AGG (Xu et al., 2024). and Splatter Image (Szymanowicz et al. 2024), also utilize 3D Gaussians in a forwaforward model. LGM (Tang ct al. 2024ab combines novel view generation diffussion models with generalizable Gaussians in feedforward manner, while GRM YYighao et al. 2024) replaces the U-Net architecture with pure- transformer and scales up large resolution. However, these methods face two main challenges when using pre-trained diffusion models. Firstly, the generated input view image becomes blurry compared to the original input, which affects the subsequent generalizable Gaussian model. Secondly, diffusion models can introduce multiview inconsistency, especially for human images with different poses, making direct adaptation unfeasible. We solve these problems by using ControlNet as the refinement tools without damaging the input image quality or introducing multi-view inconsistency.
3 OUR METHOD
3.1 PRELIMINARIES
3D Gaussian Splatting (3DGS). Introduced by (Kerbl et al. 2023), 3D Gaussian splatting represents
3D assets or scenes using collection of 3D Gaussians. Each Gaussian is characterized by its center I E IR3, scaling factor s E R3, rotation r E IR3, opacity a E R, and color features c E RR. View- dependent effects can be modeled with spherical harmonics. 3D scenes can be explicitly represented by a set of Gaussians G = [G.]. where G, = {Iz, Ti 0; ) represents the attributes for the i-th Gaussian. Compared with NeRF (Mildenhall et al. 2020), 3DGS performs fast rendering by
3

---

Published as a conference paper at ICLR 2025
SMPL-X
Iterative Refine
Es timation
Text Prompt
Refined SMPL-X
Render
Grefine
cuarse
Coarse HGM
R Render
Control Net
Fusion HGM
Mesh
Leamable
Tokens
Image Features
F,
Smol
output Gaussians
ttention
SPConv
fs
Residual
Trmnx
Image Gaussians
fu
UNet
Figure 2: Our framework and HGM model. (Top) Our framework consists of three steps: 1) Coarse Gaussians prediction with iterative SMPL-X refinement. 2) Back view refinement with ControlNet. 3) Two view reconstruction get the refined Freeine- (Bottom) Our HGM model consists of two branches: Image Gaussians prediction by UNet and adding additional structural features extracted from SMPL-X branch. J Sempl sampled by the Gaussian centers from the SMPL-X volume goo and fused with fa to the fusion transformer Timiz to obtain the Gaussian output.
first projecting Gaussians onto image plane as 2D Gaussians and performing alpha-blending for each pixel in front-to-back depth order. Building this, Image Splatter (Szymanowicz al. 2024 proposes predicting Gaussians from a single image through image-to-image translation. Specifically, each pixel is converted a Gaussian with corresponding attributes, supervised by multi-view images. Our model builds on this representation by directly predicting XYZ coordinates from the image instead of the depth.
3.2 OVERVIEW
Fig. shows an overview of our framework. Given single input human image I, our aim is to predict the corresponding human Gaussians, which can be further rendered for novel view synthesis and mesh extraction. As shown in the upper part Fig. [ our proposed method consists of three
parts: I) Coarse Gaussians prediction with SMPL-X refinement (ef. Sec. 3.3) 2) Back-view
refinement with ControNet (ef. Sec. 3.4). 3) Two-view reconstruction (ef. Sec. 3.5
3.3
COARSE GAUSSIANS PREDICTION WITH SMPL-X REHNNMMENT
3.3.1
OUR HGM MODEL
The lower part of Fig. 2 shows our proposed Human Gaussian Model (HGM). The direct prediction of Gaussians from the image pixels with UNet (Szymanowiez et al. 2024 Tang et al. (024a)) lacks human shape and pose prior, thus leading to unsatisfactory results. We therefore introduce a dual branch that utilizes SMPL-X to enforce human shape and pose prior to the Gaussian prediction process. Specifically, for UNet branch, the collection of the RGB value and ray embedding for each pixel concatenated into a 9-channel feature map as the input F; {c;, o, x d,, d; == 1, 2,2,... ..., N). Our HGM model predicts Gaussians from the U-Net
G UNet(Fi),
(1)
4

---

Published as a conference paper at ICLR 2025
SMPL-X update
Render
HGM
SMPL-X Estimation
Text Prompt
HGM
Multiview
Render
ControlNet
Mask&&Normal Matching
Figure 3: Left: Our SMPL-X refinement pipeline. Right: Our back-view refinement ControlNet.
which we refer to Image Gaussians. For the SMPL-X branch, we attach learnable tokens to each of the SMPL-X vertices and extract the patch features of the image, denoted Fy. We use artes-attention between these learnable tokens and the patch features to obtain Fs. This approach takes advantage of the fact that SMPL-X vertices are defined semantically similar areas across different identities. Consequently, the learned tokens memorize the mapping from the training dataset to unseen identities during inference, effectively providing structural human priors. This mechanism enables model to capture and utilize semantic consistent features across diverse identities, enhancing its ability to generalize to new subjects. We apply SparseConvNet (Graham et al,, 2018) (SPConv vi to propagate the SMPL-X features to the predefined whole bounding box, and denote this feature as the SMPL-X volume feature: Svol = V (Fs). where F's are the SMPL-X features. The volume feature reconstructed from the SMPL-X vertex feature provides geometric
of the target human body. The centers of the Image Gaussians from G., are then used to sample the propagated SMPL-X volume features, denoted fa gool (Cu), where C. are the centers of G SMPL-X fa features are then concatenated with the features of UNet fu for each Gaussian. This concatenated feature is fed into a transformer Trmiz obtain the coarse Gaussians, i.e.:
Geooree Tim (ifar fol).
Specifically, we predict the xyz coordinates residuals for the Image Gaussians and all the other updated Gaussian features. Trymi is a transformer that contains multiple self-attention blocks among Gaussians to ensure that each Gaussian is aware the other.
3.3.2 SMPL-X REFINEMENT
Given initial estimated SMPL-X is not accurate, we leverage our pre-trained HGM to iteratively refine the SMPL-X parameters based the mask and optionally normal matching as shown in Fig. left part. Specifically, we rendered the side-view masks and normals(optionallyy) We minimize the mask and normal difference between the Gaussian and SMPL-X renderings and back-propagate the loss to SMPL-X parameters. Then we interatively input the updated SMPL-X to our HGM model, so the Gaussian is also updated to give more accurate masks. For normal matching, we use the pre-trained normal estimator from Xiu et al. (2022) that also needs SMPL-X as input and iteratively update the SMPL--X parameters. Specifically, we compute:
LSMPL-X Afront + Xssie Loide + >) Cnommal
and back propagate it to the SMPL-X parameters. Lnormal, front and Lside are losses between the rendered masks and 2D detected keypoints of SMPL-X model and the coarse Gaussians by HGM. Enorma! is the loss between the rendered SMPLX normal and the pre-trained normal estimator (Xiu et [ (022). Note that both the normal and side mask supervisions are updated the input of the HGM and the normal estimator also contains the updated SMPL-X in each iteration. We provide
more analysis SMPL.-X refinement in the appendix.
5

---

Published as a conference paper at ICLR 2025
Input
Coarse
Refined
SITH
Input
Coarse
Refined
SiTH
Input
Coarse
Refined
SITH
Figure 4: Our back-view refinement can generate more realistic back-view images, compared with back-view hallucination of Ho et al. (2024).
3.4 BACK-VIEW REFINEMENT
Back-view hallucination poses a significant challenge in single-view human reconstruction. As shown in Fig. directly using diffusion models to generate back-view image can result in incorrect perspective projection with the front-view image as well as unrealistic texture by (Ho et al. 2024). The reason is that their diffusion conditioned on the back-view mask, and thus can only be applied for orthogonal projection where back- view mask can be directly flipped with the front-view image. However, the back-view mask is not available during our inference stage since our prediction is based perspective projection. To address this issue, we adopt generate- senerate-then-refine strategy that leverages the diffusion prior to produce perspective-fitte and realistic back-view image that is suitable for the subsequent two-view fusion stage. We train ControlNet (Zhang al. 2023b) to give realistic details based on our coarse results as shown in the right of Fig. 3 Specifically, we generate coarse back-view rendering by our HGM for the training dataset and only train the ControlNet and keep the base Stable Diffusion model as fixed. The ControlNet loss is given as:
LCN -Epntt.s [Ike co co(2, L, C,, x)12]
where y the text prompt. We it as 'Best quality' and the negative prompt as 'blur, bad anatomy, bad hands, cropped, worst quality" during inference. C₁ is the coarse back-view image from our HGM rendering I which is the ControlNet condition. We carefully design the reversing process by adding small amount of noise to the VAE encoded latent of 73 to keep the original content as much as possible. The sampling process takes around 2 seconds. In Fig. 4 we show our generated and refined back-view results, comparing them with the back-view hallucination diffusion network in SiTH (Zhang et al.) 2024). Our results maintain high resolution and generate details such as the hair of the first woman and the wrinkles in the clothes. In comparison, SiTH (Zhang al.) 2024) produces artifacts and unrealistic hallucination results. Moreover, the results are also not fitted perspectively to the input image.
3.5 TWO-VIEW RECONSTRUCTION
We combine the refined perspective-fite back-view image I3 with the front-view image I and input them into the fusion HGM model to get:
Grefine HGM(L,
Specifically, our fusion HGM model retains the design of the coarse HGM model with the additional refined back-view image as the input. The coarse HGM and fusion HGM models are trained separately with ground-trute one view and two views as input, well as ground-truth SMPL-X. The objective function for HGM training includes Le color loss, Lrgs. (GG-based LPIPS perceptual loss, Lipips (Zhang et al. 2018). and C2 background mask loss Log with ground truth masks. Each of these losses has corresponding weights that are treated as hyperparameters:
LHOM Argbborb + Atpips Cipips + Abg Ligg:
(6)
where Argh >ppips >10= LHGM is applied for coarse HGM and fusion HGM.

---

Published as a conference paper at ICLR 2025
Input
ECON
SIFU
SITH
Ours
Input
ECON
SIFU
SITH
Ours
GT
Figure 5: Levraging our HGM model, SMPL-X parameters are iteratively refined to mitigate the issue of blended legs commonly seen other approaches.
3.6 IMPLEMENTATION
Our model is trained on NVIDIA RTX A6000 with batch size of 4 20 hours. Our input image size is 512x5 12 and the number of Gaussians for each view is 256x256, with total of 65,536 Gaussians per view. For SMPL-X estimation, we use PIXIE(Feng al. 202T). Network structures and more implementation details in the appendix.
4 EXPERIMENTS
We conduct experiments the publicly available 3D human datasets THuman2.0 (Yu et al. 2021) CustomHumans//H et al. 2023) and HuMMan (Cai et al. 2022) Our method is compared with state-of-the-art (SOTA) methods in both novel view synthesis and 3D mesh reconstruction. We train our HGM on 500 human scans from THuman2.0 dataset following Zhang et al. (2024). We render the images with resolution of 5 52 and using weak perspective camera on 12 fixed cameras evenly distributed with the azimuths from 0 to 360 degree. During evaluation, all the methods are tested without the ground truth SMPL--X We follow the train and test list from SIFU (Zhang et al. 2024) and SHERF (Hu et al. 2023) to evaluate our method on T'uuma2.00 and HuMMan dataset. For CustomHumann dataset we use 45 scans for cross-dataset evaluation containing loosing clothes and challenging poses. For novel view synthesis, We use PSNR, SSIM, LPIPS as evaluation metrics. For 3D reconstruction, we use commonly used Chamfer Distance (CD), Soint-oo-Surfaee Distance (P2S), and Normal Consistency the evaluation metrics.
4.1
NOVEL VIEW SYNTHESIS
Table I: Novel view synthesis comparison with SOTA methods.
THuman2.0
CustomHumans
Method
PSNR 1
SSIM t
LPIPS + PSNR t
SSIM
LPIPS
GTA Zhang al. (203cc)
19.09
0.882
0.113
19.59
0.887
0.125
SiTH
Ho et al. (2024)
17.12
0.843
0.155
18.09
0.856
0.144
LGM
Tang al. (2024a)
18.34
0.856
0.134
19.87
0.877
0.132
SV3D
Voleti et al. (2024
19.11
0.892
0.117
20.86
0.902
0.112
SIFU
Zhang (2024)
22.10
0.924
0.0794
20.83
0.898
0.117
Ours
23.54
0.938
0.0524
23.84
0.944
0.0514
For novel view synthesis, we compare method with mesh SOTA human reconstruction methods
Zhang et al. (2023c), ECON Xiu et al. (2023). SIFU Zhang et al. (2024) and SiTH (Ho et al.) 2024 as well multiview diffusion reconstruction method LGM(fintuuned with the same training
GTA
data Tang et al. 2024a) and video diffusion method SV3D Voleti et al. (2024) on THuman2.00 Yu
al. 2222 and CustomHuman Ho al. (2023). We also compare method with of HumanNeRF methods: SHERF (Hu et al. (003), MPS-NeRF (Gao et al. 2022), and NHP (Kwon
al. 2021 the HuMMan dataset (Cai et al. 2022

---

Published as a conference paper at ICLR 2025
method significantly surpasses ofate-of-the--e single-view view human reconstruction methods in all evaluation metrics for the three datasets. As shown in Fig. [ LGM Tang al. (2024a) generates incorrect blue color and inconsistent content. SiTH Ho et al. (2024) fails to model loose clothes due to the high dependency of the SMPL.-X model. Side views are blurry and unrealistic in SIFU's results. SV3D Voleti et al. (2024) generates strange colors and wrong human pose. Compared with these methods, ours generates more realistic and consistent rendering especially for the unseen regions such clothes wrinkles and hair that well-fitted to the front views and more robust to initial SMPL-X estimation errors thanks to our iterative refinement. We provide rendering videos in 360 degree comparison with other methods in the Appendix. We also provide a visual comparison with the SOTA NeRF-based method SHERF Hu et al. (2023) on the HuMMan dataset in the appendix.
As shown in the Tab. and Tab.
Input
SITH
Input
LGM
SITH
SV3D
SIFU
Ours
Input
SIFU
Figure 6: Novel view synthesis comparison with other approaches on THuman2.0 and CustomHumans dataset. The details highlighted in the red boxes.
4.2 3D RECONSTRUCTION
For mesh reconstruction we extract the 3D mesh by densely rendering the depth map with Gaussian render and using TSDFusion to extract the surface followed by fast optimizaiton based the normal map obtained in sectior3.3.2 We compare our results with SOTA human surface reconstruction methods GTA (Zhang al. 2023c ECON (Xiu et al. 2023). SIFU (Zhang al. 2024 and SiTH
8

---

Published as a conference paper at ICLR 2025
Table 2: Novel view synthesis comparison with SOTA HumanNeRF methods HuMMan.
Method
PSNR 1
SSIM
LPIPS +
NHP (Kwon et al.
2021)
18.99
0.845
0.182
MPS-NERF (Gao et al.
2022
17.44
0.824
0.193
SHERF (Hu et al. Ours
2023
20.83
0.891
0.125
23,86
0.952
0.0591
Input
ECON
SIFU
SITH
Ours
GT
Input
ECON
SIFU
SITH
Ours
GT
Figure 7: 3D reconstruction visualization compared with SOTA methods. Details are highlighted in the red boxes.
Ho al. (2024). Note that method do not the 3D ground truth for supervision, but also achieve best performance compared with all those fully supervised methods. Thanks to our interative SMPL-X refinement. Our method alleviates commonly occurring problems of bent legs and incorrect postures found in previous methods, as shown in Fig Our HGM model reconstructs more accurate geometry with the prior learned from our dual branch Gaussian reconstruction model well our innovative SMPL-X refinement. We provide qualitative 3D reconstruction comparison in Fig 7 As shown in the figure, ECON and SIFU suffer from bending legs and wrong arms problems. SiTH generates an over-smoothed surface and missing parts. Our method can reconstruct more accurate poses while preserving geometric details.
Table 3: 3D reconstruction comparison with SOTA methods. Note that only our method trained
without 3D supervision.
THuman2.0
CustomHumans
Method
Chamfer
P2S
Normal 1 Chamfer
P2S |
NC 1
ECON Xiu et al. (2023)
2.342
2.431
0.765
2.107
2.355
0.771
GTA
Zhang ct al
2023c
2.201
2.314
0.773
1.987
2.115
0.769
SiTH
Ho al. (2024)
2.519
2.442
0.786
2.223
2.584
0.785
SIFU
Zhang et al. (2024
2.063
2.205
0.792
1.864
1.976
0.778
Ours
2.134
2.118
0.823
1.729
1.835
0.834
4.3
ABLATION STUDIES
We conduct ablation studies to evaluate the effectiveness of our SMPL-X dual branch Gaussian prediction model, coarse-to-fine refinement strategy, back-view refine ControlNet, and our SMPL-X refinement. We show the quantitative ablation results in Table The performance decreases when any component is removed. SMPL-X dual branch plays an important role in adding human priors through structural features to Image Gaussians predicted by UNet. We visualize the rendered images using model without SMPL-X dual branch as the guidance and those produced by our full model, as shown in Fig. Without SMPL-X dual branch guidance, the side-view images exhibit significant artifacts,

---

Published as a conference paper at ICLR 2025
such misaligned arms and unnatural shapes of clothes and heads, highlighted in the red boxes. This demonstrates the effectiveness of our SMPL-X dual branch Gaussian prediction design. predicted Gaussians lack human shape and pose priors without SMPL-X guidance, resulting unnatural shapes and poses. Since the initial SMPI--X prediction is not accurate, we ablate the effectiveness of our iterative SMPL--X refinement SMPL-X refinement in the appendix. Two-view refinement doubles the number of Gaussians to improve the reconstruction quality. Gaussians tend concentrate more the front view without the two-view refinement strategy, leading to poorer rendering of the back part. Additionally, the diffusion-ber refinement is crucial for improving the novel view synthesis quality, especially for the back-view images as shown in Fig. The best performance is achieved with all three components.
the 3D prior learned in our HGM model. We also visualize the
Table 4: Ablation study for each component.
NVS
3D reconstruction
Components SMPL-X dual branch
PSNR(1)
SSIM(1))
LPIPS(()
Chamfer(
P2S([)
NC(1)
21.85
0.908
0769
2.421
2.543
0.776
w/o SMPL-X refine
22,86
0.921
0.656
2.245
2.346
0.798
w/o two-view refine
22.95
0.924
0.641
2.301
2.343
0.781
w/o Diffusion refine
23.11
0.921
0.637
2.145
2.176
0.812
Full model
23.54
0.938
0.524
2.134
2.118
0.823
Figure 8: Ablation studies in terms of SMPL-X dual branch guidance. For each pair of images, left one is the results of model w/o SMPL-X guidance. Details are highlighted in the red boxes.
4.4 DISCUSSIONS
Our method improves upon previous approaches like ECON[Xiu et al. (2023) and SIFU Zhang et al. (2024) by leveraging our pre-trained HGM model to incorporate 3D priors, specifically side view masks, for enhanced SMPL-X refinement. The refined SMPL--X also benefits our human Gaussians reconstruction by providing structural information encoded in learnable tokens. Our 3D Gaussians- based method offers significant advantages in rendering speed, achieving 300 FPS compared to NeRF-based methods like SHERF Hu al. (2023), which manages only FPS. Furthermore, the mesh extracted from our 3D Gaussians with normal refinement attains high 3D reconstruction quality, In summary, our method excels both high-quality rendering and accurate 3D reconstruction, offering comprehensive solution.
CONCLUSION
In this paper, we introduce a novel generalizable single-view human Gaussian reconstruction frame- work. By incorporating human priors through SMPL-X dual branch Gaussian prediction and diffusion priors using refinement ControlNet, our method effectively handles invisible parts and varying poses. By incorporating
pre-trained HGM, inaccurate SMPL-X is iteratively refined, which benefits the Gaussian reconstruction quality. Combining generalize well to unseen subjects for high-quality and view-consistent reconstruction. We validate the proposed method on several benchmarks and demonstrate that achieves state-of-the-art performance in both novel view synthesis and 3D reconstruction.
these techniques, our method
This research project is supported by the National Research Foundation (NRF) Singapore, under its NRF-IInvsttgattrrship Programme (Award ID. NRF-NRF109-0009
Acknowledgement.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorealistic monocular 3d reconstruc- tion of humans wearing clothing. In CVPR, 2022.
Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang. Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong. Mingyuan Zhang, Chen Change Loy, Lei Yang, and Ziwei Liu. HuMMan: Multi-modal 4d human dataset for versatile sensing and modeling. In ECCV, 2022.
Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan- Yee K. Wong. Dreamavatar: Text-and- shape guided 3d human avatar generation via diffusion models. CVPR, 2024.
Enric Corona, Mihai Zanfir, Thiemo Alldieck, Eduard Gabriel Bazavan, Andrei Zanfir, and Cristian Sminchisescu. Structured 3d features for reconstructing relightable and animatable avatars. In CVPR, 2023.
Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collaborative regression of expressive bodies using moderation. In 3DV, 2021.
Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, and Xin Tong. Mps-nerf: Generalizable 3d human rendering from multiview images. In PAMI, 2022.
Benjamin Graham, Martin Engelcke, and Laurens der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
Hsuan-I Ho, Lixin Xue, Jie Song, and Hilliges Otmar. Learning locally editable virtual humans, In CVPR, 2023.
Hsuan-I Ho, Jie Song, and Otmar Hilliges. Sith: Single-view textured human reconstruction with image-conditioned diffusion. In CVPR, 2024.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d, 2024.
Shoukang Hu and Ziwei Liu, Gauhuman: Articulated gaussian splatting from monocular human videos. In CVPR, 2024.
Shoukang Hu, Fangzhou Hong, Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu. Sherf: Generalizable human nerf from a single image. In ICCV, 2023.
Yangyi Huang. Hongwei Yi, Weiyang Liu, Haofan Wang, Boxi Wu, Wenxiao Wang, Binbin Lin, Debing Zhang, and Deng Cai. One-shot implicit animatable avatars with model-based priors. In ICCV, 2023.
Yangyi Huang, Hongwei Yi. Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In 3DV, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. TOG, 2023.
Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. HUGS: Human gaussian splatting. In CVPR, 2024.
Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. NIPS, 2021.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In ICLR, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image to 3d object. In ICCV, 2023.
11

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Panwang Pan, Zhuo Su, Chenguo Lin, Zhen Fan, Yongjie Zhang, Zeming Li, Tingting Shen, Yadong Mu, and Yebin Liu. Humansplat: Generalizable single-image human gaussian splatting with structure priors. arXiv preprini arXiv:2406. /2459, 2024.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for rishutesolution 3d human digitization. CVPR, 2020.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 165 2023.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In CVPR, 2024.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024a.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024b.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitrii Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from a single image using latent video diffusion. In ECCV, 2024.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation, 2023.
Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: Implicit Clothed humans Obtained from Normals. In CVPR, 2022.
Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. CVPR, 2023.
Dejia Xu, Ye Yuan, Morteza Mardani, Sifci Liu, Jiaming Song, Zhangyang Wang, and Arash Vahdat. Agg: Amortized generative 3d gaussians for single image to 3d. arXiv preprint 2401.04099, 2024.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
Xu Yinghao. Shi Zifan, Yifan Wang, Chen Hansheng, Yang Ceyuan, Peng Sida, Shen Yujun, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024.
Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In CVPR, 2021.
Hongwen Zhang. Yating Tian, Yuxiang Zhang, Mengcheng Li. Liang An, Zhenan Sun, and Yebin Liu. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2025
Zechuan Zhang. Li Sun, Zongxin Yang, Ling Chen, and Yi Yang. Global-correlated 3d-decoupling transformer for clothed avatar reconstruction. In NIPS, 2023c.
Zechuan Zhang, Zongxin Yang, and Yi Yang. Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction. In CVPR, 2024.
Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In CVPR, 2024.
Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditoneed implicit representation for image-based human reconstruction. In PAMI, 2021.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
We introduce the following content in the appendix: SMPL-X optimization and mesh optimization details, SMPL-X evaluation, back-view details and evaluation, additional comparison, experimental environment, network structures, limitations, and more visualizations.
SMPL--X optimization and mesh optimization details. During optimization, we render SMPL-X side views and compute the side-view mask loss and normal loss for a total of 45 iterations. SMPL-X parameters are updated at each iteration. The updated SMPL-X parameters are fed into HGM to update GS every 15 iterations (3 times in total), reducing the overall optimization time. The initial SMPL-X are not input to HGM for GS prediction until after the first 15 iterations because poorly-aligned SMPL--X can lead to degraded GS. For the front view, we utilize the original image mask instead of the one rendered from GS to stabilize the process. We simultaneously render 12 views (one front-view and all the other views are considered as side views) to compute the mask loss. The loss weights are set as follows: Arontt 10, Aesde 1, and A, 0.5. For the normal loss, only utilize the front and back views with pre-trained normal estimator from ICON. Throughout the optimization process, HGM remains fixed while only SMPL-X parameters are updated. The elegance of our method lies in its iterative nature: GS refines SMPL-X and better-aligned SMPL-X estimates feed back into the HGM model generate improved 3D Gaussians, which in turn enhance the reconstruction. We show the visualization of our side-view mask rendered from iteratively reconstructed Gaussisnas by HGM, initial SMPL-X. refined SMPL-X, and our imial-extracted meshes in Fig- IT As shown in the figure. the side view masks effectively help refine the initial SMPL-X error for accurate reconstruction. For mesh refinement, we minimize the L1 loss between the predicted normal map and the rendered normal map. We also add Laplacian loss for the preservation of the local structure.
Additional comparison with TeCH. We compare our method with TeCH on the Customhumans dataset quantitatively in Tab. [ TeCH needs 4-5 hours for each sample, so we use 10 samples from the CustomHumans dataset for comparison. TeCH Huang et al. (2024) has several obvious limitations compared with ours: 1) The geometry refinement from SDS not stable and the surface broken as shown in the left part of Fig[evven though capturing more high-frequency geometric details. 2) Slow optimization: It needs 4-5 hours optimization, while ours use only around 90s. 3) The caption guidance can sometimes be incorrect. For example, as shown in the left part of Fig 9 the wrong caption of the gender resulted in wrong face reconstruction. 4) SMPL-X error leading to bending legs and wrong geometry. which is the same issue in SIFU, SiTH, ECON and GTA as shown in SIFU, SiTH, ECON and GTA.
Table 5: Additional evaluation with TeCH.
PSNR(1)
SSIM(1) LPIPS( CD(L)
P2S(J)
NC(1)
Ours TcCH
24.56
0.949
0.051
1.715
1.844
0.833
23.87
0.927
0.079
2.232
2.432
0.778
SMPL-X refinement evaluation. We evaluate using SMPL-X initializations from PyMAF-X Zhang et al. (223a)) and PIXIE PIXIE/Feng et al. (202T). We compute the MPJPE (mm) using the first 22 body joints defined in SMPL-X on both THuman2.0 Yu al. (202T) and CustomHumans (o et al. (2023) datasets. In the Tab 6 'Initial' means the direct estimation from SMPL-X predictors. "w/o videsviews' represents optimization without side-views mask loss. 'ours' refers to our optimization with all losses, including the side-views mask loss. We can see from the table that our method successfully refines the initial SMPL-X estimates using side-view priors from our HGM, which significantly reduces the error compared with without using side-view masks. Although PyMAF-X provides better initial SMPL-X estimates than PIXIE, both methods achieve comparable MPJPE scores after optimization, as the side-view mask loss guides them toward similar convergence points. This also shows that our method is robust to diverse SMPL-X initial estimators and can effectively improve the initial SMPL-X estimation.
Backview refinement details and evaluation. We use the original ControlNet architecture and initialize the ControlNet with the ControlNet-tile model. ControlNet-tile is originally trained as an image super-resolution model. We finetune the ControlNet part with our constructed data pair at
14

---

Published as a conference paper at ICLR 2025
Table 6: SMPI-X refinement evaluation in terms of MPJPE.
PIXTE
PyMAF-X
Dataset
Initial()) 75.79
w/o side-views())
Ours(L)
Initial())
w/o side-views () Ours(
CusaomHumans
65.33
39.11
65.20
58.12
39.78
Thuman2.0
80.11
72.36
44.30
71.18
65.65
44.84
Input
TeCH
Input
TeCH
Ours
Ours
Figure Visual comparison with TeCH on loose cloth and challenging pose cases.
learning rate of 1e-5, with the base SD1.5 keep fixed. Data pair construction involves first training our HGM using single-view imput without full convergence. Subsequently, we perform inference, downsample and render the back-view. The resulting back-view renderings, intentionally designed to have lower quality, serve as conditioning inputs for our ControlNet training. To validate the effectiveness of proposed back-view refinement strategy, we performance quantitative evaluation with SiTH Ho et al. (2024) and Huang et al. (2024) for back-view quality on the CustomHumann dataset. We use SSIM. LPIPS and KID as evaluation metrics between the ground truth and generated back view images. SiTH generates view using pure hallucination. which always generate unrealistic image shown in Fig. 4 and Tab. 7 TeCH use SDS loss to optimize the back-view. However, the back view always fits to the wrong SMPL-X pose and imprecise text description, which leads to lower generation quality.
Table 7: Backview evaluation.
SSIM(T) LPIPS(() KID(x 10 ()
Ours
0.949
0.079
9.26
SiTH
0.855
0.123
29.8
TeCH
0.876
0.118
20.3
Experimental environment. experimental environment is PyTorch 2.2.1 and CUDA 12.2.
We conduct all the experiments on NVIDIA RTX A6000 GPU, The
Network structures. Our UNet model consists of down blocks, middle block and 5 up blocks. with an input image size of 512x512 and an output Gaussian fcature map size of 256x256. We use 2 131, 072 output Gaussians. Each block contains
input views, resulting in total of 256 x 256 several residual layers and an optional down-sample or up-sample layer. For the last 3 down blocks, the middle block, and the first up blocks, we insert cross-view self-attention layers after the residual layers. The final feature maps are processed by x 1 convolution layer to produce 14-channel pixel-wise Gaussian features. We adopt SiLU activation and group normalization for the UNet. Our Trmiz share the same structure, consisting of multiple self-attention blocks. Specifically, they each have 5 up blocks and down blocks. The sample channels are [64, 128, 256, 512, 1024]. and the up-sample channels [1024, 512, 256, 128, 64]. The input dimensions for Tr Trair is 256.
15

---

Published as a conference paper at ICLR 2025
Input
SHERF
Ours
Figure 10: Novel view synthesis comparison SHERF on HuMMan dataset.
respectively. The output dimension for both is 14, which matches the dimension of the Gaussian features. For the attention blocks, use a memory-efficien attention implementation.
Initial
Refined
Side mask
Final mesh
Initial
Refined
Figure 11: SMPL-X refinement visualization.
Currently, method is hard to generate high-quality hands and faces, which could potentially be solved by using the SMPL-X model and regional diffusion guidance such as SDS loss for further refinement. Also, the mesh extraction process from reconstructed gaussians is not straight forward, incurring additional optimization with estimated normal map as supervision.
Limitations.
Additional results. We give visual comparison with the SOTA NeRF-based method: SHERF Hu et al. (2023) shown in 10, While SHERF predicts blurry results and loses fidelity, our method preserves high-frequency details and generates realistic back views such as wrinkles and hair that fit well to the front views.
16


Reviews:
[{"review_id": "haluRKFddd", "reviewer": "Reviewer_ckon", "summary": "This work learns 3D human Gaussians from a single image, aiming to recover detailed appearance and geometry, including unobserved regions. \nThe paper proposes a single-view generalizable framework that utilizes a generate-then-refine strategy guided by SMPL and diffusion priors. \nTo enhance realism and accuracy, the method refines the SMPL model and employs SMPL priors to improve initial coarse human Gaussian estimations using sparse convolution and attention mechanisms.", "questions": "The inference time is not reported; could you provide the specific timings for each step?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem addressed is quite challenging, and the designed network framework is reasonable, although the process is somewhat lengthy.\n- The overall writing of the paper is clear and coherent.", "weaknesses": "- The results are quite unsatisfactory, as there is a noticeable discrepancy between the synthesized face and the original image. While this issue is mentioned in the limitations section, Figure 9 suggests that the images represent two different individuals.\n- The comparison with other methods is insufficient, including approaches like TeCH[1] and GTA[2].\n- The qualitative results appear somewhat blurry, with some input images also seeming unclear, particularly in the video results.\n- Minor issues:  change \"in 2\" to \"in Eq. 2\" in line 248; clearly indicate the correspondence between F_I and F_S in the figures; missing references such as [3].\n\n[1] TeCH: Text-guided Reconstruction of Lifelike Clothed Humans\n\n[2] Global-correlated 3ddecoupling transformer for clothed avatar reconstruction\n\n[3] HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors\n\n----\nI may change my opinion depending on the authors' rebuttal and whether they can address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcnnsRooll", "reviewer": "Reviewer_Kuxz", "summary": "This paper proposes a novel approach to reconstruct human models from a single view. It first uses a coarse HGM model to predict GS representation. Besides feature from image space like the splatter image paper, this HGM model additionally aligns features based on the fitted SMPL-X template via a dual path. Then, the SMPL-X parameters and GS prediction are iteratively refined to improve mutual alignment. Finally, a ControlNet model generates a back view, which, combined with the front view, is processed by a two-view HGM for the final reconstruction.", "questions": "The patch feature $F'_I$ on line 236 is ambiguous. it is unclear if this is derived from DINO or a trained model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Feature Combination: The integration of pose-aware and pixel-space features improves the quality of the coarse GS prediction.\n2. Enhanced Quality: This approach surpasses existing single-view human reconstruction methods in output quality.", "weaknesses": "1. Incremental Novelty: There are several papers on 3D reconstruction topic that have adopted diffusion model to synthesize more views as the first step. This paper improves similar idea by using stronger controlNet signal, i.e. the rendering of course GS. This idea, while effective, is not very inspiring to me.\n\n2. SMPL-X Refinement Concerns: Although justified in ablation studies, the proposed iterative SMPL-X refinement is a EM-like approach, while the previous works always refine SMPL parameters regarding the original input image. The proposed iterative refinement may depend heavily on initial GS and SMPL-X accuracy, making it sensitive and prone to degenerate solutions. Also, the wording in Section 3.3.2 is a bit redundant and would benefit from refinement for clarity and conciseness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcMnuxo1Z2", "reviewer": "Reviewer_e633", "summary": "This paper presents a generalizable single-view 3D human reconstruction system, which uses 3D Gaussians as 3D representations. Although the 3D Gaussian representation is promising due to its rendering speed, naively adapting 3D Gaussians in 3D human reconstruction often produces errored reconstructions. To resolve this issue, this paper introduces generate-then-refine pipeline that 1) reconstructs 3D Gaussians, 2) refines SMPL-X & back-view, and 3) reruns the 3D Gaussian reconstruction. The experiments demonstrate that it achieves state-of-the-art in novel-view synthesis and 3D reconstruction compared to existing methods.", "questions": "1) According to the paper, it looks like back-view refinement is quite effective. If back-view refinement improves the final Gaussians, can the improved Gaussian be a good source for back-view refinement? In other words, I wonder if performance can continue to improve if processes 2) and 3) in Figure 2 are performed cyclically. This experiment would be a good way to demonstrate how much the paper's approach improves reconstruction performance.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "In recent years, 3D Gaussian splatting (3DGS) has shown impressive results in 3D vision fields. Despite the promising results of 3DGS, the exploration of adapting 3D Gaussians for single-view 3D human reconstruction is still lacking. This paper points out the challenges of adapting 3D Gaussians for 3D human reconstruction. I think solving the proposed generate-then-refine pipeline could be one of the good research directions in 3D human reconstruction. The proposed approach sounds plausible, and can be plugged into most reconstruction algorithms. The paper is written in an easy-to-understand manner.", "weaknesses": "I have several concern points below.\n\n1) About SMPL-X refinement strategy\n\nIt is unclear why the proposed SMPL-X refinement pipeline actually helps. As described in Section 3.3.2, the SMPL-X is optimized by two-type losses: normal loss and rendering loss. The optimization targets of two losses can be inaccurate: normal GT is from normal estimator, rendering T is from coarse Gaussians. Furthermore, as the coarse Gaussians are processed based on the initial SMPL-X (Figure 2), it seems that coarse Gaussians will not deviate much from the initial SMPL-X. It is questionable whether the proposed SMPL-X improvements always guarantee improved reconstruction quality. For example, if using a distinguished SMPL-X estimator, the refinement pipeline can lead to performance degradation.\n\n2) Validation of SMPL-X refinement\n\nAlthough SMPL-X refinement is claimed to be an important contribution, there is no detailed experiment on it. Although Table 4 indirectly demonstrates this, more explicit experiments on SMPL-X improvements (measuring MPJPE for before refine. / after refine.) are required.  Additionally, to strengthen the effect of SMPL-X refinement, experiments on diverse SMPL-X initial estimator [1,2,3] are also needed.\n\n\n3) Description about ControlNet in back-view refinement\n\nThe paper does not specified how ControlNet in back-view refinement is trained. Back-view refinement is also a major contribution of this paper, so it is important to clarify the training procedure for the reproducibility. I would recommend describing ControlNet's architecture, learning strategy, and training dataset. Specifically, I curious about how to augment coarse image (ControlNet condition) for the training. Furthermore, I also recommend that conducting a quantitative comparison with SiTH's back-view estimation (Figure 4) to justify the proposed refinement module.\n\n4) Writing\n\na. In Section 3.3.2, it is unclear whether SMPL-X only optimizes parameters or whether SMPL-X and HGM are jointly optimized.\n\nb. In L123, it is an overclaim that using multi-view images is main strength over existing methods. This algorithm setup using 12 rendered multi-view images (L350) is not significantly different from that using a 3D scan. Fundamentally, this algorithm also used 3D scans (L349), and did not show results when trained with 12 multi-view images for the existing methods.\n\n\n\n[1] Moon et al., Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation, CVPRW 2022.\n\n[2] Lin et al., One-stage 3d whole-body mesh recovery with component aware transformer, CVPR 2023.\n\n[3] Zhang et al., PyMAF-X: Towards well-aligned full-body model regression from monocular images., IEEE 2023.\n\n\nIn conclusion, the proposed approach looks plausible and well-designed, but I am concerned that the proposed generate-then-refine strategy is actually effective. I think it could be a good academic paper if it includes a more clear explanation about SMPL-X refinement (1, 2) and back-view refinement (3), which are the core of the generate-then-refine strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)