HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors
Authors: Panwang Pan
Zhuo Su
Chenguo Lin
Zhen Fan
Yongjie zhang
Zeming Li
Tingting Shen
Yadong MU
Yebin Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Single Image Human Reconstruction
Gaussian Splatting
Human Prior
Diffusion
Latent Reconstruction Transformer
Primary area: machine_vision

Abstract: Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images or time-consuming per-instance optimization significantly hinder their applications in broader scenarios. To tackle these issues, we present **HumanSplat**, which predicts the 3D Gaussian Splatting properties of any human from a single input image in a generalizable manner.
Specifically, HumanSplat comprises a 2D multi-view diffusion model and a latent reconstruction Transformer with human structure priors that adeptly integrate geometric priors and semantic features within a unified framework. A hierarchical loss that incorporates human semantic information is devised to achieve high-fidelity texture modeling and impose stronger constraints on the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis. Project page: https://humansplat.github.io.

HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors
Panwang Pan"1, Zhuo Sutt1, Chenguo Lin +1.2, Zhen Fan', Yongjie Zhang', Zeming Li!
Tingting Shen?, Yadong Mu?, Yebin Liu
Equal contribution
i Project lead $ Corresponding author
ByteDance, Pekingg University, XXiamen University, *Tsinghua University
Abstract
Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images coneuconn per-instance opti- mization significantly hinder their applications broader scenarios. To tackle these issues, we present HumanSplat, which predicts the 3D Gaussian Splatting proper- ties of any human from single input image in generalizable manner. Specifically, HumanSplat comprises 2D multi-view diffusion model and latent reconstruction Transformer with human structure priors that adeptly integrate geometric priors and semantic features within unified framework. A hierarchical loss that incorporates human semantic information is devised to achieve high-fidelity texture modeling and impose stronger constraints the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-ar methods achieving photorealistic novel-view synthesis. Project page: https: htt////hmanppllee gi github.... io.
Introduction
Realistic 3D human reconstruction is fundamental task in computer vision with widespread applica- tions in numerous fields, including social media, gaming, e-commerce, telepresence, etc. Previous works for single-image human reconstruction can be broadly categorized into explicit and implicit approaches. Explicit methods [1, 2, 3, 4], such those based parametric body models like SMPL [5, 6], estimate human meshes by directly optimizing parameters and clothing offset to the observed image. However, these methods often struggle with complex clothing styles and require lengthy optimization. Implicit methods [7, 8, 9, 10, 11] represent humans using continuous functions, such occupancy [7], SDF [12], and NeRF [13]. While they excel modeling flexible topology, these methods are limited in scalability and efficiency due to the high computational cost associated with training and inference. Recent advances in 3D Gaussian Splatting (3DGS) [14] have provided balance between efficiency and rendering quality for reconstructing detailed 3D human models, which however rely on multi-view image [15, 16, 17] or monocular video [18, 20] input. Recent popular human reconstruction studies [21, 22, 23, 24] focus on the score distillation sampling (SDS) tech- nique [25] lift 2D diffusion priors 3D, but time-consuming optimization (e.g.. 2 hours) is required for each instance. Some generalizable and large-rcconsrruction- mowelol-ooo----oooeeoo -ased works [10, 26, 27, 28] can directly generalize the regression of 3D representations but either disregard human priors or require multi-view inputs, limiting the stability and feasibility in downstream applications.
In this paper, we propose novel generalizable approach HumanSplat for single-image human reconstruction by introducing a generalizable Gaussian Splatting framework with a fine-tuned 2D) multi-view diffusion model and well-conceived 3D human structure priors. Different from existing human 3DGS methods, our approach directly infers Gaussian properties from single input image,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

23
TeCH
PSNR
(5.6 Min)
Tecli (4.5 Hlurs)
LGM
20
10'
10'
10'
Input Image
LGM (9.5 Sec)
Ours (9.3 Sec)
Reconstruction time(seconds) logurithmic scale
(a)
(b)
Figure I: Our method achieves state-of-the-art rendering quality while maintaining the fastest runtime. (a) Qualitative results: IGM [29] and GTA [30] are generalizable but lower quality, TeCH [21] exhibits issues with multi-face rendering and is time-consuming. In contrast, our method achieves higher fidelity in much shorter time. (b) Performance and runtime comparison: metrics are evaluated on the challenging Twindom dataset.
climinating the requirement for per- per-instance optimization densely captured images. This empowers our method to generalize effectively diverse scenarios while delivering high-quality reconstructions.
The key insight behind our method is to reconstruct Gaussian properties from the diffusion latent space generalizable end-to-end architecture, integrating a 2D generative diffusion model as appearance prior and human parametric model structure prior. Specifically, facing this under-constraint reconstruction problem from single-view input with extensive invisible regions, we first leverage 2D multi-view diffusion model (denoted novel-view synthesizer) to hallucinate the unseen parts of clothed humans. Then a generalizable latent reconstruction Transformer proposed to enable interaction among the generated diffusion latent embeddings and the geometric information of the structured human model, enhancing the quality of 3DGS reconstruction. During interactions. to mitigate the limitations of inaccurate human priors like the SMPL model, devise projection strategy to strike a balance between robustness and ficxibility, projecting 3D tokens into the 2D space and conducting searches within adjacent windows using projection-aware attention. Another specific challenge of human reconstruction to capture fine detail visually sensitive areas, such as the face and hand. To address this problem, we exploit the semantic cues from the structure priors and propose semantics-guided objectives to further promote the fine details reconstruction quality, By synergistically amalgamating these crucial components into unified framework, method achieves state-of-the-ar performance in striking the right balance between quality and efficiency.
In summary, the main contributions of our work are as follows:
We propose a novel generalizable human Gaussian Splatting network for high-fidelity human reconstruction from single image. To the best of knowledge, it is the first leverage latent Gaussian reconstruction with 2D generative diffusion model in an end-to-end framework for efficient and accurate single-image human reconstruction.
We integrate structure and appearance cues within a universal Transformer framework by leveraging human geometry priors from the SMPL model and human appearance priors from the 2D generative diffusion model. Geometric priors stabilize the generation of high-quality human geometry, while appearance prior helps hallucinate unseen parts of clothed humans.
We enhance the fidelity of reconstructed human models by introducing semantic cues, hierarchical supervision, and tailored loss functions. Extensive experiments demonstrate that our method achieves state-of-the-ar performance, surpassing existing methods.
Related Work
Single-Image Human Reconstruction. Single-image human reconstruction methods fall into explicit and implicit approaches. Explicit methods use parametric body models [5, 6] to estimate

---

minimally clothed human meshes [1, 2, 31, 32, 33, 34, 35, 36, 37], adding clothing via 3D offsets [3, 4, 38, 39, 40, 41] or garment templates [42, 43, 44]. However, these methods are constrained by topology, particularly with loose clothing. Implicit methods [7, 8, 9, 10, 11, 22, 27, 45, 46], utilizing representations like occupancy, signed distance fields (SDF) and Neural Radiance Fields (NeRF), offer flexibility with topology, allowing for accurate depiction of 3D clothed humans. Recent methods [30, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] combine implicit representation with explicit human priors for better robustness. Among these, GTA [30] uses Transformers with fixed embeddings for translating image features into 3D tri-plane features. TeCH [62] employs diffusion- befuusion-bbsed models for visualizing unseen areas but needs extensive optimization and accurate SMPL-X models. Another approach. HumanSGD [22], uses diffusion models for texture inpainting
but struggles with mesh inaccuracies from [8]. NeRF-based methods like SHERF [10] generate human NeRF from single images using hierarchical features, while ELICIT [11] leverages CLIP [63] for contextual understanding. These NeRF-based methods produce high-quality images but often struggle with detailed 3D mesh generation and require extensive optimization time.
Human Gaussian Splatting. While implicit representations like SDF or NeRF struggle with balance optimization efficiency and rendering quality, the high efficiency of 3DGS [14] has advanced 3D human creation. Recent 3DGS methods have targeted multi-view videos, monocular video sequences, and sparse-view input. Multi-view video methods like D3GA [15], HuGS [16], and 3DGS-Avatar [17] exploit dense spatial and temporal information for detailed models. Additionally, consistency,
HiFi4G [64] combine 3DGS with a dual- -raph mechamism to preserve spatial-tempor ASH [65] utilizes mesh UV parameterization for real-time rendering, and Animatable Gaussians [66] improve garment dynamics via pose projection mechanism and 2D CNNs. The monocular video only provides sufficient observation as humans move, the monocular human Gaussian Splatting methods [18, 19, 20, 67, 68, 69, 70, 71, 72, 73] often resort to using LBS for mapping to a canonical space and introducing additional regularization terms. For sparse-view images, GPS-Gaussian [28] achieves high rendering speeds without per-subject optimization by using a feed-forward way with efficient 2D CNNs encoding from diverse 3D human scan data. In comparison, our work stands out as the first one-shot generalizable human GS method, using only single image input and achieving high-qu -ighquulitty reconstruction without any optimization and fine-tuning.
Generalizable Large Reconstrucion Model. Large reconstruction model (LRM) [74, 75] presents that with sufficient parameters and training datasets, deterministic feed- forward Transformer [76] is capable of learning a triplane feature for volumetric rendering from a single-view image. Targeting hybrid 3D representation
general 3D object, TrislaneGaussian [77] combines 3DGS with triplane for fast rendering, where point clouds are first extracted from an image and then projected on triplane features decode 3DGS attributes. Most recent methods further leverage 2D diffusion models that can generate multi-view images simultaneously [78, 79, 80, 81] to provide plausible and abundant inputs for the following reconstruction, so they focus on the sparse multi-view reconstruction task. Instant3D [81] follows the technique of LRM and produces triplane features from 4 posed images by Transformer encoder. CRM [82], MeshLRM [83] and InstantMesh [84] replaces triplane NeRF by FlexiCubes [85] to extract meshes directly. LGM [29]. GRM [86] and GS-LRM [87] predict 3DGS attributes from each input pixel [88, 89] and combine the Gaussians from multiple viewpoints as the final 3D output. For human task, CharacterGen [90] follows Instant3D [81] to adopt two- stage approach: first, generating a 2D multi-view canonical images, then using multi-view SDF reconstruction for cartoon digital characters, without direct interaction between stages. A more compact and elegant method HumanLRM [27] replaces training data of LRM [74] from general objects with human data to predict triplane NeRF, showcasing the capacities of the large model. While it relies heavily on the dataset, often leading to issues like missing limbs due to a lack of prior knowledge. In contrast, our 3DGS-based method leverages the latent diffusion and reconstruction model in an end-to-end framework and introduces human priors by addressing their inaccuracy with specific projection strategies, achieving better generalization and stability with even less data.
3 Method
3.1 Preliminary
SMPL [5] is a widely-used parametric human body model that created by skinning and blend shapes, and learned from thousands of 3D body scans. The SMPL model [5] utilizes shape parameters
3

---

Input Image (frompt)
Hurnan Stradar Koozzsyy Scrastei Prie
TT-MII Hanma Geradti Tastiir
Condition
Ciroand Trat
Mlickr
1D0S with seruatic Midt-sto Rendeiing
(4) Novel View Synthrsiser
Recunstruction Trunsformer
Semantlcs Gendetlis-Gulded Objectives
Spatial Attenionn Tempural Atiertion
④ Concaesation
Figure 2: Overview of HumanSplat. (a) Multi-view latent features are first generated by a fine-tuned multi-view diffusion model (Novel View Synthesizer in Sec. 3.3). (b) Then, the Latent Reconstruction Transformer (Sec. 3.4) interacts global latent features (Sec. 3.4.1) and human geometric prior
(Sec. 3.4.2). (c) Finally, the semantic-guided objectives (Sec. 3.5) are proposed to reconstruct the final human 3DGS.
G E R.10 and pose parameters E R24x3 to parameterize the deformation of the human body mesh M 0) 3 0 R6890x3
3D Gaussians Splatting [1 [14] a recently proposed 3D representation that formulates 3D content by
set of Np colored Gaussians G {g:) - Each Gaussian g: =exp (-) (4;) T ' (4)) has opacity o; E R and color E R attributes, where μ4; E R33 is its location and Σ; E R3x3 implies its scaling 8 E R$ and orientation quaternion qi E R 3D space. These attributes constitute G IR., NR x11 can represent 3D object, and by projecting to the image plane with opacity-based color composition in the depth order, it can be differentiably rendered into 2D images in real-time.
3.2 Overview
Given single input image of human body Io E "xxx33 our task is reconstruct 3D representation, i.e., 3DGS in this work, from which novel-view images can rendered. As illustrated in Fig. 2, our method leverages 2D diffusion model and novel latent reconstruction Transformer that adeptly integrates 2D appearance priors, human geometric priors, and semantic cues within a universal framework. First, we SMPL. estimator [91, 92] predict human structure priors M, and CLIP [63] to generate the image embedding of the input image c. A latent temporal-spatin diffusion model [93], referred
the novel-view synthesizer (Sec. 3.3), is fine-tuned and produces multi-view latent fcatures { F, E Rxxwxx] i=1' where N is the number of views, and h, w and are the height, width and channel of the latent features. Then, we utilize a novel latent reconstruction Transformer (Sec. 3.4) that adeptly integrates human geometric prior and latent features within universal framework, which predicts Gaussian attributes G == {(1 qi S1; Ci2 ;i 1,... Np) and rendered into new images, where Np represents the number of Gaussian points. Furthermore, to texture modeling and better constrain the estimated multiple views, we designed a hierarchical loss that incorporates human semantic priors (See 3.5). Note that during training, the network trained using 3D scan data to ensure accurate supervision from various viewpoints, where the registered SMPI. is fitted via the multi-view version of SMPLify [92]. During inference, the model employs PIXIE [91 to predict human structure priors M and synthesize novel views from only single image input based the trained model, without any fine-tuning and optimization.
N
achieve
3.3 Video Diffusion Model as Novel-view Synthesizer
We take advantage the pre-trained video diffusion model SV3D [93] as appearance prior. For the input image L.. leverage CLIP image encoder [63] to obtain the image embedding and pre-trained VAE [94] E to acquire latent feature F, as conditions. Subsequently. we progressively denoises gaussian noises into temporal-continuous N multi-view latent features {F JN by spatial-
temporal UNet Da [95] with the objective:
Ee-p(e) [ (e,, IN Fo. NF
(I)

---

where { is the noise-cone corseecor multi-view latent features, { relative elevation and azimuth angles to Ip. p(c) the noise probability distribution and X(e) E R+ is the loss weighting term related to the noise level Since SV3D is merely pre-trained on the dataset of objects [96], we further fine-tune using human datasets to improve its effectiveness in the human reconstruction task. More technical details are available in Appendix A.
a, J, is the corresponding
3.4 Latent Reconstruction Transformer
Latent reconstruction Transformer contains two parts: latent embedding interaction (Sec. 3.4.1) and geometry-aware interaction (Sec. 3.4.2), which seamlessly incorporate latent features from novel- view synthesizer and human structure priors into the reconstruction process. The two components together form cohesive framework for accurately recovering intricate human details.
3.4.1 Latent Embedding Interaction
We utilize pretrained VAE [94] encoder & to encode the input image Io into latent represen- tation F. E {To) Roxxxe and combine it with the generated (F, ) (Sec. 3.3). Following previous works [97, 98], latent representations are concatenated with their corresponding Plücker embeddings [99] along the channel dimension, resulting in a dense pose-conditioned feature map. These latents are then divided into non-overlapping patches [100] and mapped to d-dimensional latent tokens by a linear layer. An intra-altention module is followed to thoroughly extract spatial correlations of latent tokens, as shown in Fig. 3, by standard Transformer [76] with Nima blocks of multi-head self-attention and feed-forward network (FFN):
F; |FFN((Sl(AAteenioo((E))])]
3.4.2 Geometry-aware Interaction
Human Geometric Tokenizer.
Given a human geometric model M 00000x obtain the feature representation, we project M onto the input view and compute the feature vector through bilinear interpolation on the feature grids by
(M(M) K (Ro. to):
where Rg, to are camera extrinsic parameters of the input view, K is the intrinsic parameters. After the projection operation, we obtain geometric human prior tokens by concatenating M and Fo (MM)) and mapping them to d dimension. Fo [MM) stands for querying the input latent feature Fo from projected locations. Similar to Latent Embedding Interaction (Sec. 3,4.1), an intra- attention module is utilized interact among these human prior tokens and get human geometric
features H E IR 6890xd
Human Geometry-aware Attention. Empirically, we find that incorporating human priors [5, 6, 101] (e.g., SMPL model) into human reconstruction faces a significant dilemma of the trade-off between robustness and flexibility [30, 102]. On the one hand, the SMPL model provides geometric priors that alleviate common issues (e.g., broken body parts and various artifacts). On the other hand, struggles to accurately capture and represent broad spectrum of complex clothing styles, particularly more fluid garments like dresses and skirts. This limitation highlights a fundamental shortcoming in the model's capacity accurately depict diverse apparel. As shown in Fig. HumanSplat addresses inaccuracies in human priors by projecting 3D tokens into 2D space and conducting local searches within adjacent windows, efficiently using priors while minimizing redundancy. Specifically, we introduce projection- -wware aware attention within the inter-attention module, utilizing window ([Kwin X kwin) and employing masked multi-head cross attennint Here, the latent features
(F.) serve queries, and the human prior tokens H serve keys and values. The process can be formulated as follows:
(F (F
(Croosssttttnnnionnnaaas ((F,) H)) N
where feature interactions occur when II, K, R., t.)) is within the window of F,, is noteworthy that the complexity of the cross-attention operation is reduced from O(L Ln) to OlL K2 compared to the vanilla multi-head cross-attention setting, where Lp and Ly represent the lengths of the latent and human geometric tokens respectively.
5

---

Fb
Geometry-aware Interaction
PRORARD
[R]
Nimal-atee Syrihecttur
Inter-attrationn Module
F
X.N
Inpot
Neighhor gaussiass
Latent Embedding Interaction
Human Goometry- Interaction
Figure 3: Illustration of latent reconstruction Transformer. It first divides Fu and F; into overlapping patches, which are then processed through an intra-attention module (Sec. 3.4.1). Within the iter-attention module (Sec. 3.4.2), we introduce the projection- W W(Kwin x Kwin), and the attributes of 3D Gaussians are decoded with a Conv x layer.
attention with window
3.5
Semantics-guided Objectives
From each output token F,, we decode the attributes of pixel-aligned Gaussians G in the corre- sponding patch by convolutional layer with a kernel. These attributes are used render novel-view images through 3D Gaussian Splatting rasterization. An ideal training objective should ensure that the rendered outputs closely match the supervised images, striving for overall consistency between the reconstructed renderings and the ground truth. However, human attention tends to be particularly focused on facial regions, where intricate details play a pivotal role in perception. Therefore, enhance the fidelity of reconstructed human models, we optimize our objective functions to preferentially focus on the facial regions.
Hierarchical Loss. Traditional approaches often overlook the semantic richness inherent in human anatomy, leading to reconstructions that lack crucial details and accuracy. To address this, we propose a novel framework that harnesses semantic cues, hierarchical supervision, and tailored loss functions guide the training process. This ensures that not only are the overall structures of the human body accurately represented, but also that critical areas such as the face are reconstructed with exceptional detail and fidelity. Specifically, by incorporating a human prior model equipped with semantic information, which adheres to the widely-used definition of 24 human body parts, such as "right hand", "left hand", "head", etc. The ground-truth body part segmentation obtained from SMPI. meshes with predefined semantic vertices [103]. Therefore, we can utilize different attention weights for different body parts and render different levels real images from different viewpoints to provide hierarchical supervision. This approach significantly facilitates the precise localization of essential body parts, including the head, hands and arms. Please refer to Appendix A.2 for more detailed procedural insights. The overall loss is defined weighted sum of part-specific losses:
CH
ArAy les) + Ap i.))),
(5)
where i E {1,..., and E {l,..., m) denote different resolution levels and human parts. 1 and I, are the predicted and ground-truth data for part j. X; and A, are weighting factors that reflect the relative importance of each resolution level and each body part, respectively. LMSE measures the MSE loss and Lp measures the perceptual loss [104].
Reconstruction Loss. Reconstruction loss LRee based on the Nrender rendered multi-view images is defined as
Nealer
LRec
LMSE (I, I.) + Am AMSE + Ap, 1.),
(6)

---

Table I: Quantitative comparison texture against other methods. indicates that this method is
fine-tuned on our training dataset. i indicates that this method requires per-instance optimization.
Category
THuman2.0 [105]
Twindom [106]
Method
Time
Diflusion
Human Prior
PSNRT
SSIM
LPIPS)
PSNRT
SSIMT
LPIPS,
PlFu [107]
X
18.093
0.911
0.137
30.0s
LGM" [29]
X
20.013
0.893
0.116
19.840
0.851
0.292
9.5s
GTA [30]
X
/
18,050
17.669
0.741
0,418
43s
SIFU [102]
22.025
0.921
0.084
19.714
0.832
0.312
44s
SIFUT [102]
/
22,102
0.923
0.079
6min
Magic123 [108]
14.501
0.874
0.145
HumanSGD [22]
V
17.365
0.895
0.130
7min
TeCH' [21]
25.211
0.936
0.083
21.192
0.884
0.188
4.5h
HumanSplat (Our)
24.033
0.918
0.055
23.346
0,913
0.125
9.3s
where I; and I are the ground-trus images and render images via Gaussian Splatting, M, and M; are original and rendered foreground mask. Am and Ap are hyperparameters for balancing three kinds of losses. The final training objective is combination of the two loss terms: L = Lu + LRee.
4 Experiments
4.1 Implementation Details
Training Details. HumanSplat is trained on 500 THuman2.0 [105), 1500 2K2K [109], and 1500 Twindom [106] high-fidelity human scans. We evenly position 36 cameras across each of three hierarchical cycles capture the full body, half body, and face, with rendering resolution set to 512 x 512. We conduct 200 epochs of 256-res training with a learning rate of 1e-5 and a batch size of 32 over days on A100 (40G VRAM) GPUs, while 512-res finetuning costs 2 additional days. We train our model with Adam W [110] optimizer, whose 31, 32 are set to 9 and 0.95 respectively. weight decay of O. 05 is used on all parameters except those of the LayerNorm layers. We use cosine learning rate decay scheduler with a 2000-step linear warm-up up and the peak learning rate is set to 4e-4. For parts related to the head, hands, and arms, X are set to 2, while the rest human part are set to 1. The parameters A₁ Ap and Am are set to 1. The model is trained for 80K iterations on 256-res and then fine-tuned on 512-res for another 20K iterations. To enable efficient training and inference, we adopt Flash-Attention-y [111] in the xFormers [1]2] library, gradient checkpointing [113], and FP16 muixedpprcisiio [114]. Fine-tuning the novel view synthesizer takes about 18 hours. Please refer to Appendix A.I 1 for more details about novel view synthesizer fine-tuning.
The video diffusion model takes about 9s generate multi-view latent features while the subsequent latent reconstruction for 3DGS takes only about 0. 3s. Additionally, it can render novel views at rate exceeding 150 FPS on a NVIDIA A100 GPU.
Inference Time.
4.2 Comparison
Quantitative Comparison. We conduct quantitative comparison on 21 THuman2.0 [105] and 51 Twindom [106] scans, rendering textured meshes from multiple angles and using PSNR, SSIM [115], and VGG-LPIPS [104] as evaluation metrics. To ensure fairness, we fine-tune LGM [29] our datasets and fully optimize each TeCH [21] instance for comparison. As shown Tab. 1, HumanSplat consistently outperforms previous generalizable methods across all datasets. Specifically, HumanSplat surpasses SIFU [102] by +1.92 (8.72% in PSNR and reduces LPIPS from 0.079 to 0.055. Notably, HumanSplat also excels the more challenging Twindom dataset, outperforming TeCH [21] by +2.15 (10.1%%) 16%) in PSNR and reducing LPIPS from 08888 to O. 125. Although the values can further decrease with model training, control the convergence to the current extent in the dataset for better generalization under more general in-the-win wild scenarios. Besides, we also report the reconstruction times for 512 512 input images NVIDIA A100 GPU. HumanSplat achieves remarkably fast reconstruction time, approximately 9.3 seconds on a single NVIDIA A100 GPU, making it much more practical than TeCH's 4.5-hour reconstruction time.
Qualitative Comparison. As illustrated Fig. and Fig. 5, by incorporating semantics-guicee objectives, Humansplat achieves more detailed and higher fidelity results against GTA [30] and
7

---

In-the-wild Images
Ground Truth
Ours
ToCH
GTA
LGM
Figure 4: Qualitative comparison of ours against TeCH [21], GTA [30] and LGM [29] on Thu-
man2.0 [105], Twindom [106] and in-the-will wild images. Our method achieves the highest quality. Note that TcCH achieves clearer results but fails to preserve the face identity.
LGM [29]. Although TeCH [21] also generates high figh-quality images, its SDS optimization often results in overly saturated multi-face outcomes [116, 117]. Moreover, while TeCH delivers the highest PSNR on the Thuman2.0 dataset, the LPIPS metric in Tab. confirms that our method provides more realistic results. As shown in Fig. 6, Humansplat outperforms HumanSGD in- the-wild images from Adobe Stock, effectively predicting 3D Gaussian Splatting properties without per-instance optimization. Notably, the results on in tn-lhe-wild images further demonstrate strong generalization capability and our model provides superior textures on both the front and invisible regions, highlighting the effectiveness of incorporating appearance and geometric priors.
4.3 Ablation Study
Latent Reconstruction Transformer. HumanSplat leverages the Stable Diffusion latent space for feature extraction. Experimental results demonstrate that utilizing the compact latent space for reconstructions leads to better rendering quality compared to using the pixel space, detailed in

---

Input Imaage
L.GM*
Input Image
Input Imugg
Figure 5: Qualitative results showcasing reconstructions of humans challenging poses, diverse identities, and varying camera viewpoints from mnll images.
Input Imsage
HumanSGD
Ours
Inpat Image
HumanSGD
Ours
Figure 6: Qualitative Comparison of ours against HumanSGD [22] on in the-wildd images.
Tab. 2 (a). We also find that patching in the time dimension for latent decoding by temporal VAE [95] adversely affects the quality, so we employ the vanilla VAE [94] decoder in this work.
Human Geometric Prior. Tab. 2 (b) illustrates the significance of human geometric prior in our approach. By utilizing SMPL as the geometric prior for human representation, we assess the robust- ness of HumanSplat with respect to different SMPL parameters. We report both the estimated [2] and Ground Truth SMPL parameters [92] and compare the model that does not incorporate a human structure prior. These experiments provide insight into the benefit of incorporating human structure prior, as significant performance degradation observed without it. Furthermore, thanks to treat human priors keys and values, HumanSplat exhibits robustness in testing against challenging poses with imperfect estimation, with only minor declines in metrics. The effectiveness of the human geometric prior is also demonstrated qualitatively in Fig. 7 (a).
Window Size. As analyzed in Sec. 3.4.2, human priors are crucial for the robustness of the human reconstruction model. As evidenced in Tab. 2 (c), when the window size Kwin is set to 1 (dubbed "Projection Only"), employing human priors significantly enhances performance, improving PSNR metric from 22.635 to 23.333. We further explore setting window size to Kwin 2 in a latent space with resolution of x 64, which further improves the model's PSNR metric from 23.333 to 24.294. This improvement is attributed to Humansplat with Kwin 2 is capable of handling loose clothing and tolerating imprecise SMPL parameters. In Tab. 3, we present results for the 2K2K dataset across various window sizes. Further refinement with a window size of Kwin 3 continues to enhance the results, whereas using Kwin = does not yield additional improvements.
Semantics-guided Training Objectives. Thanks to the incorporation of semantic- -aaantec-guiedd hierarchical losses, HumanSplat can generate higher-quality textures for both the face and hand, while also promoting the convergence of the training process, depicted in Fig. 7 (a) and (b). Furthermore, Fig. 7 (c) demonstrates that HumanSplat further produces 3D coherent segmentation results, which have potential applications in editing and specific generation of individual parts of 3D humans.

---

Inpuc Image
w/0 Inior
W/O Ex
Set
Labeling
(e)
(6)
Figure 7: (a) Qualitative evaluation of human prior and hierarchical loss Cy. PSNR/SSIM values shown at the top of each image, presenting the effectiveness of the strategies in the full pipeline. (b) LPIPS scores during training on the 2K2K validation set. (c) HumanSplat can provide 3D coherent segmentation results for potential applications.
Table 2: Ablation study of several proposed designs on 2K2K [109] evaluation dataset.
(a) Reconstruction Space.
(b) Geometric Prior Initialization. (c) Different projection methods.
Space
PSNR SSIM LPIPS]
Method
PSNR SSIM LPIPS
Methed
PSNR1 SSIM LPIPS]
Pixel Space
0.066
GT 24.633
Prior
Temporal VAE [95]
24.133
0.902
0.047
w/o SMPL 22.635
0.893
0.182
Projection Only
23.333
0.916
0.057
Ours [94]
24.293 0.915 0.040
Ours [2]
24.293
0.915
0.040
Ours (Kuin 2) 24.293
0.915
0.040
Table 3: Evaluation of different window sizes Kwin the 2K2K [109] evaluation dataset.
Method
PSNR
SSIM
LPIPS +
w/o Human Prior
22.635 (-1.658)
0.893 (-0.022)
0.182 (+0.142)
K Wiin
23.333 (-0.960)
0.916 (-0.001)
0.057 (+0.017)
Kwin
24.383 (+0.089)
0.931 (+0.016)
0.041 (+0,001)
Kwin
24.013 (-0.280)
0.922 (+0.007)
0.048 (+0,008)
Kwin 2 (Ours)
24.293
0.915
0.040
5 Conclusion
We present HumanSplat, pioneering generalizable human reconstruction network that derives 3D Gaussian Splatting properties from single image. This model integrates generative diffusion and latent reconstruction Transformer models with human structure priors, enhanced by a tailored semantic-aware hierarchical loss. These innovations achieve high-fidelity reconstruction results in a feed-forward manner without any optimization or fine-tuning, especially in the important focal areas such the face and hands. Extensive experiments demonstrate that HumanSplat surpasses existing state-of-the methods in both quality and efficiency, including robust performance in handling challenging poses and loose clothing. This capability opens up broad spectrum of potential applications.
Limitations and Future Works. Our method has some limitations that can be addressed in future works: (a). While effective with most attire, the model may struggle with intricate garments and accessories. Future enhancements should incorporate 2D data and innovative training strategies, which is promising direction for improving the handling of diverse and unusual clothing styles. (b). Although already efficient, increasing the computational speed by constraining Gaussian's 's number combining compact 3DGS with texture representation could further facilitate real-time applications, particularly on mobile devices. (c). Animating the reconstructed human models currently requires post-processing. Future work could introduce canonical space to reconstruct animatable avatars directly, streamlining the process and improving efficiency.
10

---

References
[1] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[2] Yao Feng, Vasileios Choutas. Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collabo- rative regression of expressive bodies using moderation. International Conference on 3D Vision (3DV), 2021.
[3] Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from a single image. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[4] Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Detailed human avatars from monocular video. In International Conference on 3D Vision (3DV), 2018.
[5] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions graphics (TOG), 2015.
[6] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[7] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Hao Li, and Angjoo Kanazawa. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[8] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo, PIFuHD: Muvti-Level Pixel- Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[9] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchiseseu. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings the Computer Vision and Pattern Recognition (CVPR), 2022.
the IEEE/CVF Conference on
[10] Shoukang Hu, Fangzhou Hong. Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[11] Yangyi Huang, Hongwei Yi, Weiyang Liu, Haofan Wang, Boxi Wu, Wenxiao Wang, Binbin Lin, Debing Zhang, and Deng Cai. One-shot implicit animatable avatars with model-based priors. In IEEE Conference on Computer Vision (ICCV), 2023.
[12] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2019.
[13] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023.
[15] Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.0858/ 2023.
[16] Arthur Moreau, Jifei Song, Helisa Dhamo, Richard Shaw, Yiren Zhou, and Eduardo Pérez- Pellitero. Human gaussian splatting: Real-time rendering of animatable avatars. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
11

---

[17] Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[18] Mengtian Li, Shengxiang Yao, Zhifeng Xie, Keyu Chen, and Yu-Gang Jiang. Gaussianbody: Clothed human reconstruction via 3d gaussian splatting. arXiv preprint arXlv:2401.09720, 2024.
[19] Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[20] Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[21] Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text- guided Reconstruction of Lifelike Clothed Humans. In International Conference on 3D Vision (3DV), 2024.
[22] Badour AlBahar, Shunsuke Saito, Hung- Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang. Single-image 3d human digitization with muide- diffusion. SIGGRAPH Asia 2023 Conference Papers, 2023.
[23] Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[24] Xihe Yang, Xingyu Chen, Daiheng Gao, Shaohui Wang, Xiaoguang Han, and Baoyuan Wang. Have-fun: Human avatar reconstruction from few-shot unconstrained images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[25] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2023.
[26] Jiteng Mu, Shen Sang, Nuno Vasconcelos, and Xiaolong Wang. Actorsnerf: Animatable few- shot human rendering with generalizable nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[27] Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy. and Jimei Yang. Template- frree vinw 3d human digitalization with ceffusiionguidded Irm. arXiv preprint arXiv:2401.121777 2024.
[28] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[29] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. European Conference on Computer Vision (ECCV), 2024.
[30] Zechuan Zhang, Li Sun, Zongxin Yang. Ling Chen, and Yi Yang. Global-correlate 3d- decoupling transformer for clothed avatar reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[31] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
12

---

[32] Muhammed Kocabas, Chun-Hao P. Huang, Otnar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[33] Hongwen Zhang. Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images. IEEE Transactions on Patlern Analysis and Machine Intelligence (PAMI), 2023.
[34] Jiefeng Li, Siyuan Bian, Qi Liu, Jiasheng Tang, Fan Wang, and Cewu Lu. NIKI: Neural inverse kinematics with invertible neural networks for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[35] Xin Chen, Zhuo Su, Lingbo Yang, Pei Cheng, Lan Xu, Bin Fu, and Gang Yu. Learning variational motion prior for video-based motion capture. arXiv preprint arXiv:2210.15134, 2022.
[36] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. VIBE: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogmition (CVPR), 2020.
[37] Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Danilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[38] Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Video based reconstruction of 3D people models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), 2018.
[39] Thiemo Alldieck, Marcus A. Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from single RGB camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[40] Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hierarchical mesh deformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[41] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-Degree textures of people in clothing from a single image. In International Conference on 3D Vision (3DV), 2019.
[42] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi- Garment Net: Learning to dress 3D people from images. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[43] Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. BCNet: Learning body and cloth shape from a single image. European Conference on Computer Vision (ECCV), 2020.
[44] Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Mare Pollefeys, and Michael J Black. Learning disentangled avatars with hybrid 3d representations. arXiv preprint vxXiv:2099006441 2023.
[45] Sang-Hun Han, Min-Gyu Park, Ju Hong Yoon, Ju-Mi Kang. Young-Jae Park, and Hae-Gon Jeon. High-fidelity 3d human digitization from single 2k resolution images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[46] Muxin Zhang, Qiao Feng, Zhuo Su, Chao Wen, Zhou Xue, and Kun Li. Joint2human: High- quality 3d human generation via compact spherical embedding of 3d joints. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[47] Tong He, John P. Collomosse, Hailin Jin, and Stefano Soatto. Geo-PIFu: Geometry and pixel aligned implicit functions for single-view human reconstruction. In Advances Neural Information Processing Systems (NeurIPS), 2020.
13

---

[48] Enric Corona, Mihai Zanfir, Thiemo Alldieck, Eduard Gabriel Bazavan, Andrei Zanfir, and Cristian Sminchisescu. Structured 3d features for reconstructing relightable and animatable avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[49] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: humans Obtained from Normals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Implicit Clothed
[50] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). 2021.
[51] Zeng Huang. Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. ARCH: Animatable Reconstruction of Clothed Humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[52] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. ARCH++ Animation- Ready Clothed Human Reconstruction Revisited. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), 2021.
[53] Yukang Cao. Guanying Chen, Kai Han, Wenqi Yang, and Kwan-Yee K. Wong. JIFF: Jointly- aligned Implicit Face Function for High Quality Single View Clothed Human Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2022.
[54] Tingting Liao, Xiaomei Zhang, Yuliang Xiu, Hongwei Yi, Xudong Liu, Guo-Jun Qi, Yong Zhang, Xuan Wang, Xiangyu Zhu, and Zhen Lei. High-Fidelity Clothed Avatar Reconstruction from a Single Image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), 2023.
[55] Xueting Yang, Yihao Luo, Yuliang Xiu, Wei Wang, Hao Xu, and Zhaoxin Fan. D-if: Uncertainty-aware human digitization via implicit distribution field. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[56] Yukang Cao, Kai Han, and Kwan Yee K. Wong. Sesdf: Self-evolved signed distance field for implicit 3d clothed human reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[57] Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd camera. In European Conference Computer Vision (ECCV), 2020.
[58] Zhuo Su, Lan Xu, Dawei Zhong, Zhong Li. Fan Deng, Shuxue Quan, and Lu Fang. Robustfu- sion: Robust volumetric performance reconstruction under camannobbjec interactions from monocular rgbd stream. IEEE Transactions Pattern Analysis and Machine Intelligence (PAMI), 2022.
[59] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[60] Yuheng Jiang, Kaixin Yao, Zhuo Su, Zhehao Shen, Haimin Luo, and Lan Xu. Instant-nvr: Instant neural volumetric rendering for human-object interactions from monocular rgbd stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[61] Xiaozheng Zheng, Chao Wen, Zhuo Su, Zeran Xu, Zhaohu Li, Yang Zhao, and Zhou Xue. Ohta: One-shot hand avatar via data-driven implicit priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[62] Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. International Conference on 3D Vision (3DV), 2024.
14

---

[63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. et al. Learning transferable visual models from natural language supervision. In International conference on Machine Learning (ICML), 2021.
[64] Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong. Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[65] Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, and Mare Habermann. Ash: Animatable gaussian splats for efficient and photoreal human rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[66] Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning pose- dependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[67] Xiaozheng Zheng, Chao Wen, Zhaohu Li, Weiyi Zhang. Zhuo Su, Xu Chang, Yang Zhao, Zheng Lv, Xiaoyuan Zhang, Yongjie Zhang, Guidong Wang, and Lan Xu. Headgap: Few-shot 3d head avatar via generalizable gaussian priors, 2024.
[68] Mingwei Li, Jiachen Tao, Zongxin Yang, and Yi Yang. Human101: Training 100+ fps human gaussians in 100s from I view. arXiv preprint arXiv:23/2.15258, 2023.
[69] Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Chen Zhao, Haocheng Feng, Errui Ding, and Jingdong Wang. Gva: Reconstructing vivid 3d gaussian avatars from monocular videos. arXiv preprint arXiv:2402.16607 2024.
[70] Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. SplattingAvatar: Realistic Real-Time Human Avatars with Mesh- Embedded Gaussian Splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[71] Jing Wen, Xiaoming Zhao, Zhongzheng Ren, Alexander G Schwing, and Shenlong Wang. Gomavatar: Efficient animatable human modeling from monocular video using gaussians- on-mesh. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[72] Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilddis. Gart: Gaussian articulated template models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[73] David Svitov, Pietro Morerio, Lourdes Agapito, and Alessio Del Bue. Haha: Highly articulated gaussian human avatars with textured mesh prior. arXiv preprint :xXx:24404.0105. 2024.
[74] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In International Conference Learning Representations (ICLR), 2024.
[75] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151, 2024.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[77] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable ingle-view 3d reconstruction with transformers. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
15

---

[78] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. International Conference on Learning Representations (ICIR). 2024.
[79] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d genera- tion. arXiv preprint arXiv:2312.02201, 2023.
[80] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 2ero123++; a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.151100 2023.
[81] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gener- ation and large reconstruction model. In International Conference on Learning Representations (ICLR), 2024.
[82] Zhengyi Wang. Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. European Conference Computer Vision (ECCV), 2024.
[83] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
[84] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large recon- struction models. arXiv preprini 2rXiv:2404.07191, 2024.
[85] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Gruphics (TOG), 2023.
[86] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403. 14621, 2024.
[87] Kai Zhang. Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference Computer Vision (ECCV), 2024.
[88] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[89] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[90] Hao-Yau Yang Peng, Jia-Peng Zhang, Meng-Hao Guo, Yan-Pei Cao, and Shi-Min Hu. Charactergen: Efficient 3d character generation from single images with multi-view pose canonicalization. ACM Transactions on Graphics (TOG), 2024.
[91] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collabo- rative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV), 2021.
[92] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[93] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. European Conference on Computer Vision (ECCV), 2024.
16

---

[94] Robin Rombach, Andreas Blatimann. Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[95] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231.155127 2023.
[96] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[97] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. International Conference on Learning Representations (ICLR), 2024.
[98] Eric Ming Chen, Sidhanth Holalkere, Ruyu Yan, Kai Zhang, and Abe Davis. Ray conditioning: Trading consstccyy for photo-realism in multi-view image generation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[99] Julius Plücker. Xvii. new geometry of space. Philosophical Transactions of the Royal Society of London, 1865.
[100] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
[101] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons- Moll. Learning to reconstruct people in clothing from single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[102] Zechuan Zhang, Zongxin Yang, and Yi Yang. Sifu: Side conditioned implicit function for real-world usable clothed human reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[103] Smpl segmentation. https://messcapade wwteeeiwiiiiiieeeeeeeeeee wiki/assets bww. smpl/smpl_ www.tetseggeeettiionnnno json.
[104] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[105] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[106] Twindom. https: twindom. com.
[107] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[108] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia- Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[109] Sang-Hun Han, Min-Gyu Park, Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park, and Hae-Gon Jeon. High-fidelity 3d human digitization from single 2k resolution images. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
17

---

[110] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[111] Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. arXiv preprini arXiv:2307.08655 2023.
[112] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transformer modelling library. https: htte:://ggttttt.teeeee hwtpc///cccccbooeeeeerrrretettea 2022.
[113] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.061 74, 2016.
[114) Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR), 2018.
[115] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing (TIP), 2004.
[116] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2023.
[117] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich, Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2023.
[118] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[119] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[120] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[121] Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong. Liefeng Bo, et al. Videomv: Consistent multi-view generation based on large video generative model. arXiv preprint arXiv:2403. 10.003//2/10 /20/0, 2024.
[122] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406609956. 2024.
[123] Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
18

---

A Implementation Details
A.1 Additional Novel View Synthesizer Details
Conditions of Diffusion Model. SVD and SV3D [93, 95] replace CLIP text embeddings [63] with the image embedding of the conditioning. Compared to text prompts, image prompts provide information more accurately and directly, and also save time that would otherwise be spent on text inversion or image caption [21, 102]. SV3D takes image and the relative camera orbit elevation and azimuth angles as inputs to generate several corresponding novel-view images, which are continuous in 31) video with the camera moving. In our work, we finetuned SV3D on human datasets with relative camera elevation and azimuth angles to generate 4 orthogonal views, which we found to be sufficient for reconstruction [102].
space and can be regarded
Transformer Block. The CLIP embeddings obtained from the conditioning image is fed into the cross-attention layers of each Transformer block, acting as keys and values, while the feature that layer serves as the queries. Moreover, the camera trajectory and the diffusion noise timestep are incorporated into the residual blocks.
Fine-tuning Details. We involves the widely used EDM [118, 119] framework, incorporating a simplified diffusion loss for finetuning. We fine-tune the novel-view synthesizer at 512-res uses 1000-step warmup with the peak learning rate 1e-4 in the cosine learning rate decay schedule. We use per-GPU batch size of 8 objects during 256-res training, and per-GPU batch size of 4 during 512-res finetuning stage. For each instance, we use input views and 4 novel supervision views at each iteration of 256-res training and 512-res finetuning.
Triangular CFG Scaling. Classifier-free Guidance (CFG) [120] a widely used technique to trade off controllability with diversity. However, this scaling causes the last few frames in our generated orbits be over-sharpened. To tackle this issue, we use a triangle wave CFG [93, 121] scaling during inference: linearly increase CFG from at the front view to 2.5 at the back view, then linearly decrease back to
3D Consistency Evaluation. We also conduct an evaluation of the 3D consistency of our novel-view synthesizer using the advanced local correspondence matching algorithm, MASt3R [122]. This approach involves one-to-one image matching between input views and their generated novel views, with the average number of matching correspondences serving as our metric. Our synthesizer exhibits a notable improvement in 3D consistency for human -viw generation after fine-tuning with human datasets. Specifically, the original SV3D achieves an average of 723.25 matching points, while our novel-view synthesizer increases this number to 930.13. In comparison, the ground truth demonstrates matching point number of 106.33. This evaluation highlights that fine fine-tuning on human datasets significantly enhances multi-view consistency metrics.
A.2 Additional Latent Reconstruction Transformer Details
Network Architecture. The network architecture of the proposed latent reconstruction Transformer is shown in Fig. 8. The framework is composed of latent encoder with intra- and inter-attention and a Gaussian parameter prediction module. takes initial multi-view latent embeddings as input and performs projection-awart attention to aggregate human geometric information. From each output token, we decode the attributes of pixel-aligned Gaussians in the corresponding patch with a Conv 1 x layer.
Hierarchical Loss. Our approach is efficient and can be utilized online without pre-computation. We calculate the visible face triangles given the mesh and camera parameters. Then each visible triangular face assigned the corresponding semantic label [103]. Additionally, it provides labels with superior 3D consistency and avoids issues associated with training instability. First, the hierarchical loss specifically focused on the head and hands, excluding hair and clothing, which can be further supervised using reconstruction loss LRe.. Second, instead of using segmentation to reconstruct and then concatenate different components, we design our loss function based on part segmentation. Thank to this design, HumanSplat is tolerant of inaccuracies in segmentation.
19

---

Latent Multi-view Features
3DGS Attributes
Gaussian Attr. μ:: position
Patchify
1x1 Conv
q: rotation
base scale a: opaciry
Linear Embedding
Unpatchify
C: cocolrr
Linear Embedding
Intra-Attention
X iintr
Deconv Block
Inter-Attention
XN inter
Figure 8: Detailed network architecture of latent reconstruction Transformer.
B
More Results
For practical applications demanding high input view reconstruction quality, which could potentially be directly derived from the input image. We improve the weights of the reconstruction loss for input views (referred to as reweighting loss), demonstrating that the current model with reweighting loss exhibits sufficient capacity render input views with higher fidelity, illustrated in Fig.9. We also conduct an ablation experiment for the reweighting loss function on the 2K2K Dataset [109]. There slight declines in quantitative metrics, specifically, HumanSplat with reweighting loss achieves a numerical change of -0.141, 141, -0.015, and +0.006 in PNSR, SSIM, and LPIPS. This suggests that reweighting loss is not "a bag of freebies" and HumanSplat without reweighting loss strikes balance between the quality of novel views and the input views.
As for novel view and novel pose rendering, more results shown in Fig. 10 and Fig. 11. For single input image, we use Open-AnimateAnyom [123] for novel pose synthesis and HumanSplat for novel view synthesis. This functionality enables dynamic timestamps and real- real-time rendering of novel views, thereby enhancing immersive virtual exploration. Please also kindly refer to our supplementary video for more results.
C Broader Impacts
While our model demonstrates the capability to reconstruct photo-realistic 3D avatars, it also in- troduces potential risks such as privacy violations. To mitigate these concerns, it is imperative to establish robust ethical guidelines and legal frameworks. This necessitates collaborative effort among researchers, developers, and policymakers to promote the responsible use of this technology and safeguard against its potential misuse.
20

---

w/jo rww.ightinggoo
reneighing Ioes
wyjo rowdiahiing
with rowdighene Iom
Osex
Ourx
SIPU
sedd----llddd Imsage
(hi Imrgee In Twinders Elatose
Figure 9: Ablation Study on Reweighting Loss. Comparison of the original HumanSplat, trained with reweighting loss, and SIFU featuring texture.
(a)
(b)
Figure 10: Qualitative 3D Gaussian Splatting results of diversified evaluation dataset. (a) Input Image. (b) Novel view Rendering Results.
21

---

Novel Pose Synthesis
Novel View Syntheei
Figure 11: Qualitative 4D Gaussian Splatting results on In-the-wild images, including novel view and pose rendering images. (Please zoom in for detailed view)
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
Code will be available at: hit.aaannspatt..ttu.............
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.co.. //nip..c wttpccccpppccccceeececccccccccct guides/ /CodeSubmic /utoi////iiddeesoeeeutttiiiiii ssissssillllo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Sec. 4.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eeuri....ee cc
dnne?
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects..
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: See Appendix C.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "oqt9OFVtXk", "reviewer": "Reviewer_2EjT", "summary": "- The paper, HumanSplat, focuses on photorealistic novel-view synthesis of humans from a single image.\n- The key idea is to use a multi-view synthesizer based on SV3D to hallucinate the other views + latent reconstruction transformer to predict the 3DGS.\n- Importantly, HumanSplat does not use optimization and is much faster than existing methods.\n- Baselines:  Magic, HumanSGD, TeCH\n- Evaluations are done on Thuman2.0 and Twindom datasets. Metrics: PSNR, SSIM, LPIPS\n- The proposed methods consistently outperforms the existing methods.", "questions": "- Understanding the generalizability of novel-view synthesizer. Is it limited to full-body images? Any quantitative insights into 3D consistency.\n- Qualitative comparisons to HumanSGD on stock images.", "limitations": "Yes. The authors provide a discussion about utilizing 2D data and handling diverse clothing styles.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, organized and easy to follow.\n- The key idea of multi-view diffusion plus transformer to predict parameters of gaussians in the context of human avatar creation is novel.\n- An important technical contribution is the archictecture of the latent reconstruction transformer with geometry-aware interaction.\n- The experiments are done on multiple datasets along with informative ablative studies.\n- Table. 1 highlights the speed-up gains compared to existing methods. HumanSplat is almost as fast as LRM but utilizes an explict human prior and achieves much better performance.", "weaknesses": "- Measuring 3D consistency of SV3D: The novel views generated by fine-tuned SV3D is important for predicting the gaussian parameters. An ablative study on the effectiveness of the novel synthesizer would be really helpful to understand the in-the-wild generalization of the proposed method. 3D consistency could be measure using stereo-based matching or local correspondences.\n\n- Qualitative results: Although, Table. 1 showcases better performance than HumanSGD on THuman2.0 dataset, notably in all metrics. The qualitative comparison between HumanSGD and HumanSplat is missing. HumanSGD results demonstrate much better modelling of human skin tone and overall 3D geometry. I understand the code is yet to be released, however, would it be possible to compare on similar stock images?\n\n- Minor: consider renaming \"project-aware\" -> \"projection-aware\". The writing at times is a little too non-technical, eg. L206-207, consider rephrasing such instances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and thorough feedback. It is inspiring to hear that you find the paper well-written, organized, and easy to follow, that the method in the context of human avatar creation is novel, and that our experiments are comprehensive. Below are our clarifications for your concerns.\n\n**Q1: Measuring the 3D consistency of novel views generated by fine-tuned SV3D and providing any quantitative insights into 3D consistency?**\n\n**A1**: Your insight regarding the novel-view synthesizer's 3D consistency metrics is enlightening.  As suggested, we utilize the cutting-edge local correspondence matching algorithm, MASt3R [a], to measure the **3D consistency of the novel-view synthesizer**.  Specifically, we performed one-to-one image matching between input views and their generated novel views, using the average number of matching correspondences as a metric. \n\nQuantitatively compared to the original SV3D [93], our synthesizer demonstrates a significant enhancement in 3D consistency for human novel-view generation after fine-tuning using human datasets.\n\n|                 | Matching Points Number |\n| -------------   |:-------------:|\n|   SV3D [93]   |   723.25          |    \n|  Novel-view Synthesizer (Ours) |  930.13           |       \n|  Ground Truth   |   1106.33         |     \n\nWe will incorporate it into the ablation study that fine-tuning on human datasets improves the multi-view consistency metrics.\n\n**Q2: The generalizability of the novel-view synthesizer (including in-the-wild scenarios)? Is it limited to full-body images?**\n\n**A2**: The novel-view synthesizer, inheriting the generalization and multi-view consistency of SV3D [93] and fine-tuned on human datasets, exhibits robust multi-view generation capabilities and has enhanced generalization abilities within the human domain for different regions and in-the-wild scenarios, including 'head only,' 'upper body,' and 'full body.' Our SV3D model outputs intermediate feature maps, which are not visually indicative of quality. However, the human GS generation based on in-the-wild images demonstrates the generalization ability of our synthesizer, as shown in **Figure 2 and Figure 3 of the PDF attachment**.\n\n**Q3: The qualitative comparison between HumanSGD and HumanSplat is missing. How does it compare with HumanSGD in \"in-the-wild\" images from the Adobe Stock Website?**\n\n**A3**: In response to your insightful request and suggestion, we provided a qualitative comparison against HumanSGD in-the-wild image from the Adobe Stock Website in **Figure 2 of the PDF attachment**. Notably, our results demonstrate satisfactory outcomes without per-instance optimization.\n\n**Q4: Minor issues.**\n\n**A4**: We will rename it to \"projection-aware\" and modify L206-207 to \"Therefore, to enhance the fidelity of reconstructed human models, we optimize our objective functions to preferentially focus on the facial regions.\" in our revision, eliminating non-technical parts and ensuring consistency throughout the manuscript.\n\n\n---\n**Reference**：\n\n[a]. Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. \"Grounding Image Matching in 3D with MASt3R.\" arXiv preprint arXiv:2406.09756 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UuaMhtShb1", "reviewer": "Reviewer_zEoa", "summary": "This paper proposes HumanSpat, a method that predicts 3D Gaussians from a single image of a human. The method comprises a 2D multi-view diffusion model and a latent reconstruction transformer that integrates human body prior.", "questions": "In Table 1, how was the reconstruction time computed? PIFu is very lightweight and should not take 30 seconds per human.", "limitations": "Limitations are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Well-designed generalizable model that incorporates human prior. The model design is intuitive and clear. The fusion of the human structure prior makes a lot of sense. The end-to-end design addresses major limitations in existing works, such as lacking of human prior makes the prediction missing limbs (as in Human lrm) and two stage approaches where the errors in the human pose prediction affects the clothed reconstruction.\n\n+ Reconstruction time. Powered by 3DGS, the method has minimal reconstruction time and can render novel views at real-time speed.", "weaknesses": "- Missing ablation on human prior. The paper should include ablation experiments where there is no human structure prior, as well as experiments where the human structure prior is estimated instead of GT. These experiments can provide insight on the benefit of the specific human structure prior fusion design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and valuable feedback. It is truly inspiring to know that you appreciate the intuitive and clear design of our model, which offers competitive reconstruction times and can render novel views at real-time speeds. \n\n**Q1: Missing ablation on human prior? The paper should include ablation experiments where there is no human structure prior, as well as experiments where the human structure prior is estimated instead of GT. These experiments can provide insight on the benefit of the specific human structure prior fusion design.** \n\n**A1**: Thank you for your suggestions. Below are our clarifications for your concerns.\n1. **During training**, we employ SMPL parameters fitted from scans (Ground Truth) to ensure multi-view semantic consistency when computing the hierarchical loss, which helps maintain training stability.\n\n2. **During testing**, we utilize the off-the-shelf PIXIE [89] to predict SMPL parameters. We report both the estimated and  Ground Truth SMPL parameters and compare the model that does not incorporate a human structure prior, as shown in **Table 2 (b) and (c)  of this paper**. We further demonstrate the results of ablation experiments as follows:\n\n|               | PSNR $\\uparrow$| SSIM $\\uparrow$  | LPIPS $\\downarrow$ |\n| ------------- |:-------------:|:-------------:|:-------------:|\n| GT SMPL parameters| 24.633  | 0.935  | 0.025 |\n| Predicted SMPL parameters|  24.374 (-0.259) | 0.928 (-0.007) |  0.036 (+0.011)|\n| w/o SMPL parameters |  22.635 (-1.998) | 0.893 (-0.0420) | 0.182 (+0.157) |\n\nThese experiments provide insight into the benefit of incorporating a human structure prior, as significant performance degradation is observed without it. Furthermore, thanks to our model's fusion design ($k_{win}$), HumanSplat exhibits robustness in testing against challenging poses with imperfect estimation (additional in-the-wild results can be found in **Figure 3 of the supplementary material PDF**), with only minor declines in metrics using predicted SMPL parameters. \n \n**Q2: How was the reconstruction time computed? PIFu is very lightweight and should not take 30 seconds per human.**\n\n**A2**: \nThe reconstruction time in Table 1 was calculated from a single image (512$\\times$512) to the corresponding 3D representation. Although PiFu is a lightweight network, its inference involves densely sampling the 3D grid and evaluating each point to determine its inclusion in the surface, followed by marching cubes to extract the surface mesh. This runtime and reconstruction resolution are dependent on the number of sampling points. The official implementation of this process takes approximately 30 seconds on two NVIDIA GV100 GPUs, as detailed in the follow-up work [a] by the same authors of PiFu.\n\n---\n**Reference**:\n\n[a]. Li, Ruilong, et al. \"Monocular real-time volumetric performance capture.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16. Springer International Publishing, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o9kuQrCEdb", "reviewer": "Reviewer_6Egy", "summary": "This paper proposes a generalizable human rendering framework from single images. The proposed method relies on different priors and achieves state-of-the-art results on multiple datasets.", "questions": "Please see the weakness section.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The topic of human modeling/rendering from partial observation is important and interesting.\n- Thanks to 3DGS, the proposed method achieves a good balance of rendering quality and speed.", "weaknesses": "- Qualitative visual results on the rendered images are not satisfactory. It is acceptable that renderings of unseen views of the human can be relatively poorer. It is surprising to see renderings of exactly the input views are also in low quality (Figure 4). This is usually a sign of poor model capacity, which may be attributed to the proposed objective functions or the model itself.\n- It is not clear how the 'parts' of human are defined. Are they integer based index as in iuv? If so, how to obtain the ground truth body parts labels? Are they provided by the dataset or they have to be pre-computed by some off-the-shelf models. What if there are prediction errors in the 'ground truth' semantic labels? Is the proposed method robust to this kind of errors? I also have doubts on its effectiveness in terms of the overall rendering quality since there is no ablation study provided to show how it improves quantitative/qualitative results.\n- As presented in Table 2 (b), it seems that the proposed method is very sensitive to geometric prior (i.e. SMPL estimations). Since the model relies on PIXIE for SMPL prediction, what if there are errors in the predictions (most likely for challenging poses or complex clothes/background), how will the proposed method perform under those cases? This on the contrary shows advantages of geometric prior free methods such as LRM and HumanLRM. If there is a benchmark on human rendering with challenging poses, I am afraid those methods will perform better than the proposed one.\n- I wonder why there are not qualitative comparisons with SIFU but only quantitative metrics?\n- I'd suggest to change Figure 1 (b) to a different style. The dashes in the graph are misleading and confuse readers as they are in different scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments! Below are our clarifications for your concerns.\n\n**Q1:  Why are the input views in low quality (e.g., Fig. 4)? Is this a sign of poor model capacity or inappropriate objective functions?**\n\n**A1**:\n1. (a) HumanSplat reconstructs 3DGS from a single image in a **generalizable** manner, a process that is inherently ill-posed. To enhance its capability to handle diverse scenarios, the quality of the input views is sacrificed for overall balance. For additional details, please refer to **Fig 3 of the attached PDF**. (b) **Similar artifact patterns in the input views** have been reported in Fig. 4 of GTA [30] and Fig. 5 of SIFU [101]. \n\n2. To further confirm the **model's capacity**, we adjusted the weights of the reconstruction loss for the input views ($\\times$4), illustrating that the current model with reweighting loss has sufficient capacity to render input views with higher fidelity, as demonstrated in **Fig. 1 of the attached PDF**.\n\n 3. **Ablation on Reweighting Loss Function  on 2K2K  Dataset [104].** While qualitative improvements are noticeable by reweighting loss, there are slight declines in quantitative metrics, showing that reweighting loss is not \"a bag of freebies\".  We originally chose to balance the quality of the novel view and the input view, and as your suggestion, we will include this analysis in the revision.\n\n|     | PSNR  $\\uparrow$ | SSIM  $\\uparrow$  | LPIPS  $\\downarrow$ |\n| ------------- |:-------------:|:-------------:|:-------------:|\n| w/o reweighting loss (Ours)  |     24.374       |   0.928    |  0.036   |\n| with reweighting loss        |     24.233       |   0.913    |  0.042    |\n\n**Q2: How are human body parts defined? What if there are prediction errors in the 'ground truth' semantic labels? Is the proposed method robust to such errors?**\n\n**A2**:\n1. **Difinition.** We employ the widely-used definition of 24 human body parts, such as \"rightHand\", \"leftHand\", \"head\" etc. The GT body part segmentation results are obtained from SMPL meshes with predefined semantic vertices [a]. Specifically, we calculate the visible face triangles given the mesh and camera parameters. Each visible triangular face is then assigned the corresponding semantic label. our approach is efficient and can be utilized online without pre-computation. Additionally, it provides labels with superior 3D consistency, and avoid issues associated with training instability.\n\n2.  **Robustness?**  There are two primary sources of \"Hierarchical Loss Error\": SMPL misalignment and the influence of hair and clothing. (1) Robustness to SMPL accuracy? During training, we leverage ground truth SMPL parameters to ensure training stability. (2) Robustness to hair and clothing?  To validate this, we compare our model supervised with a Human Parsing model, and the corresponding metrics are as follows:\n\n| |PSNR  $\\uparrow$ | SSIM  $\\uparrow$  | LPIPS  $\\downarrow$  |\n| ------------- |:-------------:|:-------------:|:-------------:|\n| with Semantic SMPL (Ours)  |     24.374       |   0.928    |  0.036   |\n| with Human Parsing      |   23.912        |   0.911    |     0.050        |\n\nThe ablation study demonstrates the superiority of our method, primarily because the hierarchical loss is specifically focused on the head and hands, excluding hair and clothing. In contrast, the 2D Human Parsing model does not provide supervision that is consistent in 3D.\n\n**Q3: Is the proposed method very sensitive to geometric priors? How does it handle challenging poses, and what are the advantages over methods that do not use geometric priors?**\n\n**A3**:\n1. SMPL provides geometric priors that alleviate broken body parts and artifacts (please refer to **Fig. 3 of the attached PDF**). Prior-based methods reduce the demand for large datasets, enhance generalizability, and prevent overfitting. For instance, HumanLRM requires 9.4K scans compared to our method which requires only 3.5K human scans.\n\n2. We have implemented two key strategies to handle misalignment from imperfect SMPL: (1) Using human priors as keys and values in the Latent Reconstruction Transformer, and (2) Setting a window size $k_{win}$ for fault tolerance. Our generalization ability for challenging poses is shown in **Fig. 3 of the attached PDF**. In the following table, we simulate erroneous poses with random SMPL initialization and demonstrate that our window strategy helps reduce the influence of erroneous human priors.\n\n  \n|   | PSNR $\\uparrow$| SSIM $\\uparrow$  | LPIPS $\\downarrow$ |\n| ------------- |:-------------:|:-------------:|:-------------:|\n|  Baseline     |      24.374         |     0.928          |    0.036           |\n|  $k_{win}$=2 + Random SMPL |   19.894      |     0.876    |    0.366   |\n|  $k_{win}$=3 + Random SMPL |    20.609   |    0.883 |    0.327      |\n|  $k_{win}$=4 + Random SMPL |     21.295   |    0.889  |    0.242          |\n|  $k_{win}$=64 (w/o human prior) |    22.635         |    0.893          |    0.182   |\n\n\n3. For extreme cases, we acknowledge these as limitations. We will add discussion to the limitation part and consider integrating a better SMPL parameter optimization process.\n\n**Q4: Why there are no qualitative comparisons with SIFU?**\n\n**A4**: \n1. Up to now, the official repository of SIFU [101] only generates coarse geometry and texture. This is inconsistent with their paper because texture refinement significantly impacts the qualitative outcomes. This is unfair to report the qualitative results (**Fig.1 of attached PDF**). \n\n2. if they update their code during the revision, we promise to complete qualitative results.\n\n**Q5: Change Figure 1 (b) to a different style.**\n\n**A5**:  Thanks for your suggestion, and we will make it more distinct.\n\nWe promise to incorporate the addressed issues into the final version. \n\n---\n**Reference**：\n\n[a]. https://meshcapade.wiki/assets/SMPL_body_segmentation/smpl/smpl_vert_segmentation.json", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZccGuFInwq", "reviewer": "Reviewer_eC4L", "summary": "This paper presents a 3D human reconstruction system that takes a single RGB image and outputs 3D Gaussians, which can render the reconstructed 3D humans to any viewpoint. In contrast to existing 3DGS works, which requires per-instance optimization and cannot be generalized to unseen identities, the proposed system can be used for any unseen identities in zero-shot manner. To achieve this, the authors utilizes 2D mulit-view diffusion model and latent reconstruction transformer with human structure priors. A hierarchical loss, which incorporates human semantic information, is introduced. Experimental results demonstrate the powerful capability of the proposed work.", "questions": "Please see the weaknesses section.", "limitations": "Please see the weaknesses section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Utilizing diffusion-based generative models to render unseen novel viewpoints is a reasonable choice considering the aiming task is a significantly ill-posed one. The latent reconstruction Transformer effectively incorporates the human structure prior with the generative model’s output. The hierarchical loss could be useful to enforce a consistency between 3D human geometry and appearance.", "weaknesses": "1. Unclear writings\nOverall, technical details are not enough to fully understand the manuscript. \n1-1. For example, Sec. 3.3 describes the video diffusion model. I can’t fully get how the ‘video’ diffusion model can be used for the ‘novel-view’ synthesizer. I understand that a video of an object can model different viewpoints when a camera is moving, but there should be a clear justification and reason why the authors chose this generative model for the novel-view synthesizer. In addition, it seems the video diffusion model does not take the target camera pose. Then which novel view is modeled with the video diffusion model given no camera pose? \n1-2. Also, for the hierarchical loss, how can the authors access the target human image of certain body parts? Do the authors already have GT body part segmentation? If the authors simply use rendered body part segmentation from GT SMPL meshes, I don’t think that is a good choice as SMPL meshes only model naked bodies, while there could be hair and loose clothes in the image.\n\n2. Lack of comparison to SOTA methods\nPlease compare with more SOTA methods, such as ICON and ECON.\n\n3. Lack of in-the-wild results\nAs the proposed work aims to generalizable 3D human reconstruction, there should be more diverse generalized results. Only three results in Fig. 4 show the results on in-the-wild inputs, which have quite simple poses. Please report more results from input images with diverse **human poses**, identities, and camera pose.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your recognition of the insight behind our method and its model designs. Below are our clarifications for your concerns.\n\n**Q1:  How the ‘video’ diffusion model can be used for the ‘novel-view’ synthesizer?  Why the authors chose this generative model for the novel-view synthesizer? Which novel view is modeled with the video diffusion model given no camera pose?**\n\n**A1**: \n1. SV3D [91] is a cutting-edge video diffusion model for **object** novel-view synthesis, and it's finetuned from SVD [93], an image-conditioned video generative model.\nSV3D takes an image and the **relative camera orbit elevation and azimuth angles** as inputs to generate several corresponding novel-view images, which are continuous in 3D space and can be regarded as a video with the camera moving.\n2. Thanks to the seamless continuity of images in the generated video, SV3D presents state-of-the-art 3D consistency in the object novel-view synthesis task, so we adopt it as the foundational model for our novel-view synthesizer.\n3. In our work, we finetuned SV3D on human datasets **with relative camera elevation and azimuth angles to generate 4 orthogonal views**, which we found to be sufficient for reconstruction (refer to SIFU [101]).\n4. We sincerely appreciate the reviewer's keen observation and commit to addressing  **omission of target camera pose input in Equation 1** in the revised manuscript.\n\n**Q2: For the hierarchical loss, how can the authors access the target human image of certain body parts? Do the authors already have GT body part segmentation? While there could be hair and loose clothes in the image, how do the authors deal with it?**\n\n**A2**:\n1. **Definition.** We employ the widely-used definition of 24 human body parts, such as \"rightHand\", \"leftHand\", \"head\" etc [a]. The GT body part segmentation results are obtained from GT SMPL meshes fitted from scans with predefined semantic vertices and faces. Specifically, we calculate the visible face triangles given the mesh and camera parameters. Each visible triangular face is then assigned the corresponding segmentation label, repeated across all available scans.\n\n2. **Explanation.**\n\n- **Robustness to hair and clothing**? \nFirst, **The hierarchical loss is specifically focused on the head and hands**, excluding hair and clothing which can be  further supervised using reconstruction loss $L_{Rec}$. \nSecond, instead of using segmentation to reconstruct and then concatenate different components, we design our loss function based on part segmentation. This approach is tolerant of inaccuracies in segmentation.  \n- **Comparison.** We also compare our model supervised with a Human Parsing model while still showcasing the current Hierarchical Loss's superiority, as shown in the following table. Besides, compared to 2D image-based human parsing methods, our approach can be utilized online without pre-computation. Additionally, it provides labels with superior 3D consistency, and the supervision signals it offers help to avoid issues associated with training instability.\n\n| |PSNR $\\uparrow$| SSIM $\\uparrow$  | LPIPS $\\downarrow$ |\n| ------------- |:-------------:|:-------------:|:-------------:|\n| with Semantic SMPL (Ours)  |     24.374       |   0.928    |  0.036   |\n| with Human Parsing      |   23.912        |   0.911    |     0.050        |\n\n\n**Q3: Lack of comparison to SOTA methods Please compare with more SOTA methods, such as ICON and ECON.**\n\n**A3**: \nWe appreciate the reviewer's suggestion and provide the following detailed explanation:\n1. Focus on Appearance Results: While ICON (CVPR-2022) and ECON (CVPR-2023) primarily focus on geometry, our work emphasizes appearance results, evaluated through image rendering metrics. These metrics are crucial for assessing final reconstruction fidelity, which geometry-focused methods alone do not capture. Therefore, our comparisons target methods that prioritize both appearance and geometry.\n2. Comparison to Recent SOTA Methods: We have compared our results with the latest SOTA methods: GTA (NeurIPS 2023), SIFU (CVPR-2024), and TeCH (3DV 2024). These methods have been shown to outperform ICON and ECON in their respective papers. Notably, TeCH is a follow-up work by the same authors of ICON and ECON.\n\nAs suggested, we will highlight the rationale for selecting the latest SOTA methods for comparison and cite ECON in the main paper during the revision. Additionally, we will consider including a comparison of our appearance reconstruction results with ICON and ECON's geometry reconstruction results in the supplementary materials.\n\n\n**Q4: Lack of in-the-wild results? As the proposed work aims to generalizable 3D human reconstruction, there should be more diverse generalized results.**\n\n**A4**: We thank the reviewer for the insightful suggestion. In response, we provide more in-the-wild results with diverse human poses, identities, and camera poses in **Figure 2 and Figure 3 of the PDF attachment**  in our \"global response\" in the rebuttal.\n\n\n---\n**Reference**：\n\n[a].  https://meshcapade.wiki/assets/SMPL_body_segmentation/smpl/smpl_vert_segmentation.json", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to all the reviewers for their valuable, constructive, and thoughtful feedback. It is truly inspiring to hear that the majority of reviewers recognize that:
- The proposed method is meaningful (6Egy), efficient (2EjT, 6Egy), and addresses major limitations in existing work (zEoa).
- The quantitative and qualitative experiments are extensive (2EjT).
- The model design is novel and make sense (2EjT, zEoa, eC4L).
- The paper is well-written, well-organized, and easy to follow (2EjT).

We have responded thoroughly to all reviewers **in the corresponding rebuttal text input box** and have included additional qualitative results in the **supplementary material PDF** to address concerns raised by the reviewers.

**Answer for unsatisfactory results in input views**: 
In Figure 1, we adjusted the weights of the reconstruction loss $L_{Rec}$ between the input and novel views, resulting in improved quality of the input views. Simultaneously, we showcased the qualitative results of SIFU without  texture refinement.




**Answer for in-the-wild results**: 
In Figure 2, we provided additional results for in-the-wild images sourced from **Adobe Stock**. In Figure 3, we demonstrate the generalizability of our model with in-the-wild images in **challenging scenarios**, and highlight its superiority over Geometric Prior Free methods (e.g., LGM) in alleviating broken body parts and artifacts.

Thank you very much for your time and valuable input! If there are any further concerns that we have not yet addressed, we would be eager to hear from you.

Meta-review: - This paper presents a model for single-view 3D human reconstruction with Gaussian splats. It uses a fine-tuned diffusion-based novel view synthesizer to generate multiple views of the input and proposes a prior-aware latent reconstruction model for Gaussian splat prediction. The reviewers (**2EjT,**  **zEoa, eC4L**) ****acknowledge the technical contribution of using masked cross-attention to incorporate the SMPL mesh prior as tokens.
- Concerns are mainly raised around the influence of introduced SMPL prior (**zEoa, 6Egy,  eC4L**) and missed visual comparisons (**2EjT, 6Egy)**
- In the rebuttal, the authors did a good job of addressing the questions by providing ablation for SMPL pose priors and additional visual results for HumanSGD and in-the-wild results.
- The AC agrees with the reviewers and recommends its acceptance, and also requests the authors to incorporate the reasoning and results in the responses to the final version.
Decision: Accept (poster)